If this is very unlike him to say something this hurtful then maybe something medically is wrong with him. If this is normal for him it sounds like you should just say Happy Fathers Day and leave it at that. If he says where is he's present then say 'here I am'.