I don't think kneading your bread on a cold counter top would hurt it. I store yeast dough in the refrigerator for days and it is fine. The only thing it might do to it is your dough might rise a little bit slower. I make caramel rolls all the time and store the rolls in the refrigerator before baking them.
Bread sounds good. It is cold here, I might have to make some today.