speak to him, he sounds like a very reasonable and kind man from your description.
i don't see why you should hold these feelings in and express them to people on a board when you could be talking to your DH.
i'm sure if you explained it to him just like you did here he would agree with you and your relationship will be better for it.