That's exactly what I'm doing now. I recently developed bone cancer in my right femur, and my sewing room had been upstairs. The stairs were becoming increasingly more difficult for me to manage, so my DH suggested moving it all downstairs into the living room. We, like you, rarely use it -- we always entertain in the family room. It has worked out very well for me. And I love having all my sewing 'stuff' so close.
Janet