Oh, my! I just reread your post and now see that he even returned the quilt with the request to make it bigger. Hmmm. I guess I'd keep it but I sure wouldn't make it bigger or make another for them. There are some people who don't "deserve" the efforts that go into making a quilt gift for them. Nowadays my quilts go to my family so then I'm not disappointed/offended how others respond. If I make a charity quilt, I won't know if it ends up in the dog's box. We have these expectations that a gifted quilt will be treasured and in reality they are simply another blanket to some people.
Since he returned the quilt with the request for it to be bigger--if he inquires later about the progress, you can respond with a comment similar to "after consideration, it's a big job that I don't think I can do". Might ask him if he'd like the quilt back as is if you don't want to keep it now. But No is a good word and you should have no qualms about saying it.