I immediately thought "both" but as I've reflected on this question I realize that I'm probably more driven by the pattern. There are SO MANY lovely quilts that I'd like to make, so many patterns to try out that when I find the right fabric for a pattern, I make it. I've recently been on a buying kits kick which is new to me and now have a few of those to sew together which I guess still falls into the pattern category.

Great question. I enjoyed exploring my thought process.

For me it depends. On the fabric I come across or have, or a pattern that looks intriguing. So, I would say, both, either or.

I would say both ways too. When three fabric stores went out of business here in my area, I bought as much as I could afford. Buying as many pieces as possible in the line that was available. I also got many neutrals that are easy to use with many patterns. The reductions were so good, I just didn't pass that up.
On the other hand, if I fall for a pattern I have to make, I kit it up and that gives me a grab and make easy project. I store it all together, so I don't use it accidently in something else. Like many of the gals I also have bought backings and battings to finish up tops I have finished and would like to see it completed.
I need to buy neutral threads for quilting. I have used much of them up and they are a must to have on hand. So. I need to find some on sale soon.

It can go either way. Often it's fabric first, because I fall in love with something and get inspired.

I'm in the "Both" camp and a couple of times it's been the backing fabric that's inspired all of the other decisions.

It's always fabric first,, then scramble to find a pattern.

Both. Lately it's been fabric and find a pattern. As my name says, I like the fabric.

Some of my best quilts have been pattern first.

I'm in the both camp too, but usually I fall in love with the fabric first, then find a pattern.