You could mix the fabrics by stabilizing the t-shirts with nylon tricot fusible. The polyester fabrics would not require stabilizing. Poly frays more easily than cotton, so you probably just want to be sure to handle it carefully when piecing. If you encounter a piece that frays a *lot*, then you might want to stabilize it or cut the piece with pinking shears.
One thing to be aware of is that cottons fade over time whereas polyesters do not. This can make a quilt look "odd" many years down the line. This kind of quilt should be kept out of sunlight as much as possible, because light fades fabric.
Still, I would use it as you describe for a memory quilt.