Too me, the key is "sleep" or "appearance". I'd put the prettier quilt on during the day and remove it to sleep. Our guest room usually has a quilt on the bed over the comforter. Most of our guests take off the comforter as it's too hot.
I too like the lightweight of a down comforter... if the room is cool enough.