I would tell them that it's your fabric and your hobby and to 'Mind Their Own Business!
The next time that they bring it up tell them that you are not going to reply to their comments ever again as it is not a topic that is open for discussion. If they do bring it up again simply ignore them and hopefully they will get the message and find out that you are serious.