I think of fabric as tools, or pantry/kitchen supplies.
Some people don't like "extra" in their house--their cupboards are basically bare. These are people who were actually "afraid" during Y2K. Others, who work from a "full pantry" idea, weren't fearful--they already have enough on hand, probably, in a pinch, for a month of food. (Maybe not ideal menus, but food nonetheless).
It is my guess that these same people who get uneasy at the thought of having extra fabric around, probably don't buy extra food--they don't work out of the "pantry method" of food shopping--they prefer more day to day or week to week shopping.
I think it is more of personality thing.
I honestly don't think it is hoarding if you are constantly using it.
I love having fabric around--but too much can be too much to organize. When you spend more time organizing, cleaning up around, etc. than using/sewing with your fabric, then you've probably got too much.