I have only tried fleece once on a baby quilt. It was not as bad an experience as I thought it would be. Turned out fine but it wasn't a large quilt.
I did read on some site that when using fleece or minky, etc. for backing, one way is stretchier than the other. Have the stretch going left to right, not up and down.