I've been using worn out washrags, doubled, for the batting in my pot-holders. It is very heavy & puffs up nicely. Since they are doubled I feel confident that they will be great for pot-holders. I have also made the mistake of piecing a tumbler block backing out of washrags, (Don't do that it's not very pretty, way too stretchy,) however it is very heavy. Having this experience has given me a new respect for terry cloth. If you're sewing the towels together I'd use a half inch seam allowance, & something like muslin to stabilize it. I do think that terrycloth in yardage would be awesome as batting or as backing.
I finished the tumbler at the retreat that i went to this last weekend. Here's a link to the picts of the retreat. My terry tumbler is about halfway down.
http://www.quiltingboard.com/t-162672-1.htm