I make rag quilts out of fleece, it doesn't rag but it still has an interesting look to it when the seams are cut, and I only use 1 ply of fleece, 2 plies is really heavy.
I didn't put a center piece in the squares in the last flannel rag quilt I did. It was a really big quilt and I thought 3 layers would be way too heavy so only did 2 layers.