I always take my fabrics to natural sunlight when picking the colors. Generally that means outside for the final decision. Therefore, the color of my room doesn't matter. It's off-white since we rent. Our apartment is partially underground, so it's like a dungeon in the living and dining rooms as there's no sun on that side of the building. Lots of trees close to the building.
I'd make the room a color you feel good in. If that's sunny yellow go for it.