Was he saying it in a joking manner? Or do you think he resents your hobby and was poking fun at it in a mean way? I know from personal experience that my husband often times says things that to him are harmless, but they catch me the wrong way. (I tend to take things personally more often than most people though)
I hope y'all can talk it out and make things right. I know all too well about not having anyone to talk to about things. (((hugs)))