Go with your plan and enjoy your big, ideal room!
Perhaps the 2nd bedroom while used for storage in the closet,
could be used as the sitting room when you do have visitors.
Realistically though, how often do you have company?
And is the company beyond your immediate family?
To me, if people are more worried about your set-up in your home than coming for the visit ... then they really are not there to visit with you!!!! Worse than Quilt Police, IMHO!
Another thought ... with a fresh start in your new smaller place, all on one level,
perhaps you will find that you will be able to keep things a bit tidier and you will be happier with how it looks.
Before you move, take the time to do some planning and set it up the way you want to.
Happy planning ... and moving to your new Home Sweet Home!