I have a decent sized stash, most of it gifted to me. I love to look at the fabric; I have 2 walls of shelves so I can see every piece. I also have multiple small pieces, and shapes which I have leftover after cutting out for my quilts. I personally think they are too small to use - some are about 1 1/2 inch square, some are longer strips. I keep thinking I will make a crumb quilt, or a crazy quilt, but that hasn't happened yet; I keep getting distracted by another pattern that I want to make. I probably should just give up and throw them away, but I just can't bring myself to do that yet.
My whole point is that you should build a stash, only if you are comfortable with scrappy-type quilts. If you are only doing specific quilts, then buy for that one quilt, with a smidge over ( in case of mistakes), but it is very easy to become overwhelmed at how fast you stash can grow. If you like scrappy quilts and have the storage to put it, then go for it. If this is your choice, you can find good deals at estate sales, thrift shops. or no-longer-worn clothes from your closet. Family and friends will often be on the lookout for fabric, as well.
only you can know whether to build a stash, or to only buy fabric for a specific quilt. whichever you choose, happy quilting.