Like most everyone else, I think you are right in feeling the way you do, and would contact the owner of the store. I prefer to think that the mistake was unintentional, but his response was way out of line. Maybe a weakness in math is a sore point for him so he is defensive?
A few questions: What does having bought a couple of sale items have to do with it? I'm confused by the inclusion of that as a rationale. It's on sale, but you get charged for more yardage than you buy?
I've never known that cutting 1/8ths of a yard was an issue in some places? Or heard of charging extra for small cuts? Or lots of cuts? Is that a norm in some areas?
With the economy such as it is, customer service can make or break a busines, and a happy customer can provide much word of mouth advertising. An UNhappy customer is more likely to do the same!