I am 27 and have a toddler, and I just started quilting this year. I did make an unfortunate "quilt" for my mom 2 years ago for Christmas, but I didn't even know enough to actually quilt it so it's a mess. She loves it though and is getting a real replacement soon!
My husband is actually really excited that I can make stuff and likes to show it off, but most of our friends and family are the same as your husband and think I'm nuts.