I've used fleece on MANY baby quilts. I use adhesive to baste it to the top, and I don't use a backing. I stitch in the ditch from the center out to 'set' the 4 quarters first. Then I work from the centers out in each of the quarters. I haven't had a problem with stretching.