I am currently working on memory quilts as well. We lost my nephew in a car accident earlier this year. My sister gave me his clothes.. I think she gave me every stitch of his clothing except underwear, and what he had on the day of the accident and what he was buried in. (I have 7 quilts to finish - 55 x 65 not big quilts) I have jeans, puffy jacket lining, flannel, tshirts, dress shirts all mixed together. I love the mixed materials.. its a neat effect. I did buy some filler cotton fabric to work in as well (Jacob loved horses and hunting and tractors - so this fabric helped incorporate those things into the quilts)
I did use fusible interfacing on everything - except the jeans. Now I am not completed, so I don't know how these will last. But so far so good. The interfacing has made it much easier to sew and handle.
I have 2 that are based on the jacobs ladder pattern - I modified them a bit just to be able to put the tshirt logos on. One for my son has no pattern.. just pulled together however it worked.. the other are just 4" blocks.
Good luck. I am sure they will be wonderful quilts