Go for it Girl!
Our home was the same - we barely EVER used the living room as such and before my hubby built my quilting studio (when we did our major reno) I changed my rooms around and used the living room as my quilting room.
If we did have guests over - after they got over the initial ????? over the switch, they agreed it made much more sense.
It's your home, so you get to call the shots. If you and your family members are/can be happy with the switch - that's all that really counts.