You say some of the blouses are blends - do you mean poly-cotton blends? If they're a woven poly-cotton, they shouldn't be that hard to work with, though you do have to make sure your iron isn't turned too high, as it might melt them.
I don't have any experience with sweater knits or other knits. I'm about to do a memory quilt incorporating knits, and I think I will put interfacing on the back to make them "act" like a woven, per advice given on this board. I do plan to make a test swatch with tee-shirt material and run it through the washer and drier several times, to make sure it wears well.
Do make sure not to use any worn/weakened sections of the clothing, as those sections will wear out first, limited the useful/beautiful lifespan of the whole quilt. I made that mistake on a quilt made from my husband's shirts.