I think a quilt made with both the sweater fabrics and the shirts would LOOK very interesting, but might be most impractical to use. For one thing, if many of them were wool, it would have to be dry-cleaned. However, I think the idea is worth pursuing.
I would definitely advise using a larger seam allowance than the typical 1/4" on those knit sweaters. And anything very bulky probably would not work well at all. After you gain a little experience, perhaps you could buy just a few cheap sweaters and shirts at a local thrift store and experiment BEFORE you cut up the precious ones that were your father's. You could get very creative using the button plackets, collars and cuffs right in the quilt blocks. I would love to see this when you finish it.