I'm so glad you've decided to wait just a bit!
Back to your original question, when you're ready to work with the sweaters, it'd be better to have the quilt top completely done with similar weight fabric. I would use the coordinating shirts, as part of the quilt backings, so that they'd be alongside fabrics similar to themselves. Not knowing the weight of the sweaters, it's hard to suggest like fabrics for piecing with them. If you have a quilting guild or LQS near you, it might help to get the advice of someone knowledgeable, who can see/feel them, then recommend appropriate fabric.
Welcome to the world of quilting!!!