I think you have a clone of my husband! I think it is the companionship that they want more than anything else. Sure, griping about the cost is also a possibility.
Stick to your guns. You've told him quilting makes you happy, so now ask him what makes him happy and join him in it occasionally, if possible.
When my husband told me he resents the time quilting takes me away from him, I proposed that every day we'd sit down together for 15 minutes and talk about all the good qualities that made us choose each other as mates, or what our individual plans and wishes were. Real Quality Time....That lasted about a week....(*grin*)
Since then, he watches TV with his head set on, and I do my quilting, reading, writing, computing,etc. A person with no hobbies gets lonely and finds it hard to relate to the sense of accomplishment a hobby brings. But, being able to negotiate and be considerate of one another has really quelled all those complaints. Hang in there.....
Marveen