I also would never donate a quilt to charity that I would not be willing to say I made. I make a few tops (sometimes quilts) each year for donation through the Heartstrings project and I have sometimes been told by other local quilters that my quilts were too good for donation.
I think getting the pastor to speak up is a good idea. If that doesn't work I would be looking for another avenue to donate my time and work to.