I'd use the flannel backing on all of the squares if it's going to use to cover up with. Seems to me it would make it a softer 'feel' and a more even warmth.
Have you thought about using blocks with the pockets still attached? I was cutting the best parts out of some jeans that DH has outgrown and I was thinking that a pocket here and there would add interest.