I only buy for projects but I make sure I absolutely have enough so my stash is building very slowly but my money can earn interest in the bank where as if I buy fabric just to be buying fabric my money sits and no interest. Also, when I come across a pattern I want to make I can choose at that time and make sure I have plenty for my project instead of going to a stash and finding the "perfect" fabric only to realize I don't have enough and stores don't carry it any more.
Your money, build your stash as you want and can afford not by what others have.