I think I'd say, "Gee, I'm sorry you feel that way. I guess I'll have to use that fabric for the gift I was gonna make you in something else. I wouldn't want to insult you. I didn't know it upset you so."
On the other hand, if you have borrowed money from them, pay it back before you buy anything else and say nothing more. We are always beholden to those from whom we borrow.