I've used fleece for two baby quilt backs. I was disappointed because once they were heavily quilted they lost their "softness". Also polar fleece is elastic and stretches, which made it difficult to work with. I also made the mistake of using it for batting, which really screwed up the quilt, again because of the elasticity.
Flannel might be a better choice, but wash it first, as it shrinks at least 10 percent. :shock: