How about setting aside a weekly budget for a fabric allowance? Make a list for exactly what you have to have, not your wants. Keep the list in your purse. For me, I have to stay totally out of craft stores, or I will definetly find something to buy. I am on a strict budget now, and I will only buy something if I have the cash saved for it. That is how I control my spending.
Saying you will just stop altogether is a bit extreme. This is a reasonable compromise. If it is not on your need list, don't buy it!