I do both - sometimes buy the fabric just because I like it so much and sometimes with a pattern idea in hand. I sort my fabric in groups - what fabrics go together to make a quilt. I have groups of two to 6 fabrics ready to use in a quilt. All I have to do is find the pattern I like for each group and then sew.
I also have a group of fabrics that are orphan fabrics, meaning they have no companion fabrics right now. I keep a small photo album in my purse filled with fabric swatches and I take it to the quilt shop with me to find companion fabrics at some point in time.
Right now I have groups of fabrics for 8 quilts. From them I will probably have enough scraps for at least 2 more quilts. This method works great for me.