I wonder about the stretch? Do you have a serger? You might consider using a serger for the seams, that way the seams will have a little give when the quilt is pulled. I'd hate to see the seams pop here and there in a few years. All quilts are pulled on, from pulling it up while you're in bed, to tucking it in.
I'd love to see a photo of your sweatshirt quilt when you're finished. This sounds like a wonderful idea.
Christine-
http://quiltdasher.blogspot.com/