I TOTALLY GET IT! I've done this too! I tried taking swatches to the store with me, but that can get real unhandy. And I hate cutting into any fabric until I'm sure I want to use it. Now, I go to the paint store first. B4 I buy any fabric, I choose a bunch of their paint color cards that I like together. Then I take that to the LQS and stick to what's on the card (s) as close as I can. I don't get as many fabrics that way, but I also don't spend quite so much $ on gas instead of fabric.
And I end up with colors that really look good together.