It is a nice wallhanging and it is patriotic. You obviously put alot of time, effort and thought into that gift. He is just being nasty. There is probably no "health" reason because he has acted like that with others and there is no excuse. He just sounds like a bully.
It all depends on the family dynamics. Me, being me--and I AM NOT suggesting this for you, I would have told him where to stick it, I would have taken it back and I would never give him anything or any part of me again, not even a card on Father's Day. My best friend would be like you and let him pass. wvdek has a great suggestion.