Your health department sounds like it likes to overstep it's bounds. Put up a No Trespassing sign and make them produce a warrant to come on your property. I've never heard of a health department caring about the exterior paint on a house before, most of them have all they can handle just making sure restaurants aren't poisoning people.
Rodney