I don't allow our dogs in many spots of the house, and certainly not where I sew or quilt. But I do at times get a customer quilt that will have dog or cat hair on it (or smoke! Yuck--it immediately goes into a plastic trash bag with Irish spring soap for at least a week). I take them to the deck and shake REALLY hard and sometimes will drape them over the deck railing to air out. But I'm not allergic either.
I'd say, limit your charity work to those that don't have pets in the home. and don't feel guilty about enforcing that!