Yes, quilting is my hobby. "Hobby" is not a derogatory or demeaning word. Now if they said, "your silly hobby," then I'd be a little offended. :) A hobby is simply something that gives you pleasure in your leisure hours, something for which you don't get paid. If you make your quilts to sell, then I would say that quilting is your profession, not your hobby.
But if the word doesn't work for you, then tell people that quilting is your avocation. Avocation is a synonym for hobby, but it has a bit more gravitas. :)