Back in the day when everyone I knew called fabric 'material' the word was NOT used to refer to lumber, stone, concrete, etc. I have only heard 'material' used for building supplies in the last 20 years. It could be a regional thing. But you never see a 'materials store.' It is a building supply store, a lumber yard, stone yard, etc. That said, the definition in the dictionary cover anything of substance.
Isn't there a blog called 'Material Girls'? Or am I confused with something else?