It works all ways for me - sometimes fabric, no pattern - sometimes pattern, no fabric. It will just sit and stew until I pull it all together. I do try to be careful not to let the cart get too far in front of the horse, as I find having too many unplanned projects, or UFO's, a little stressful. If I had your dilemma, I would probably haunt pinterest looking for inspiration.