"Can someone tell me how to overcome this!? "
Yes, have someone else drive you around town and make sure they never give into the desperate pleading that will occur each time you come within four or five miles of a fabric store.
If you can't find someone willing to be your personal chauffeur/voice of reason, then I'm afraid you're out of luck.