I agree with you. When we were younger, my Mom sewed all of our clothes, curtains, furniture covers, dance review costumes, etc. and never threw away the scraps or leftover yardage. She always saved it for "maybe one day I will need it". Unfortunately, she never used it and when we did hit her stash, some of the fabric had dry rotted! Even though I am a sewer, I have never held onto to fabric scraps. My name "scrapbeagle" originated when I became a scrapbooker, which is a whole other stash happy hobby.
I am new to quilting, and although it is very hard not to buy when you see the fabric, I am trying not to build too big of a stash. There is nothing wrong with having or not having a stash. We will each find our own path in how we accomplish our quilting. I am fortunate to have fabric and local quilting stores close to my house, and some do not have that advantage.