I do the same thing. I try to start withthe pattern then find the fabric. I've also been trying to go through my stash before I go shopping for fabric for a new project.
The other thing to look at is the pattern of the fabric and what it might lend itself to. Is it large patterns that you don't want to cut into little pieces? Blenders that would complemement other fabric? Once in a great while the fabric will yell at me what it wants to do! :0)