I think your concerns are totally valid. I also think you need to find a new doctor. Is this guy an allergist? If not, I'd be for finding one.
While doctors aren't there to tell us what we want to hear, but instead to diagose and treat, they still need to listen to their patients. What good does it do to write scripts for a patient that won't take the medication anyway?