Personally, I wouldn't like it. My quilting time is just that - my time. We do a lot together as a family, and I love socializing and being with people, but I definitely need some alone time every single day and most of the time I get that in my quilt room.
But, if you and your hubby would find it enjoyable to share that time together, then I say go for it!