We called it material too. I don't buy the explanation that fabric is more specific. Fabric is just structured substance.
Another old word for it was "stuff"
Maybe the lesson is that whatever it is called, that word soon gets applied to a much broader selection of, em, material?