Both! Sometimes the pattern inspires me, other times the fabric.
I'm currently working on a Labyrinth Way quilt which has long been on my bucket list. First it was the 3D effect of the design. I kept looking for good occasions to make it and it didn't come up. When I went looking for fabric, I couldn't find all 4 at one time if the fabric was on sale. If I found the perfect fabric, it wasn't on sale and I was too cheap to buy. Years pass...
A month or so ago I found a wild print on sale that just appealed to me and and said it was time to make the project. I've gotten a pretty good start on it and should be able to show pictures soon. I very rarely make a quilt more than once, other than standards for donation projects, but I always learn so much in making one I'm generally tempted to play with another.
Other times I look at what fabrics I do have since I do have a large stash, and think about what can I do with what I have. So recently I decided I had too much green and then saw an idea for a project that would at least make it easier to close the tote!