Yes you will lose the insulating power of the down. Maybe just have the quilt come off the bed at night and then fluff up the down blanket and lay only that on the bed. There's nothing like the featherweight warmth of a down comforter. Have you thought of making a duvet quilt top for the down comforter?