I don't think I would use the sweats in the quilt unless you stabilize it. What about using all the other fabric in the quilt body and then doing an applique with the sweats. Sort of like a raggy look. Cut out shape and then stitch in maybe 1/4" and say good to go. Can be very cute.