I have to answer both+ Sometimes I see a pattern and know just the fabric to use. Other times, I have to find the right fabric (preferrably from my ever-growing stash). Sometimes, I have a fabric in mind and find a pattern. When I have a specific person in mind, the picture gets a bit more complicated and I may play around with different versions until I pick one.
On occasion, I started with one pattern and while looking through the stash, run across a fabric that dictates a different pattern.