I happen to agree with you that simply because a quilt is going for charity purposes does not mean any fabric can be used that doesn't match, that any form of sewing is acceptable. I would have no problem with telling people that.
You don't have to single the anything goes crowd out, but stress the fact that you put labels on these quilts and you want to be proud to put the labels on the quilts. If the offenders' noses are out of joint, too bad. You really don't want their quilts anyway.