I used fleece for several different quilts I made for gifts with extraloft batting. It worked fairly well, but I hand tied the quilts - I did not try to machine sew or hand quilt, just tied the quilte every 6 inches - at the junctions of the blocks. Also, need a bigger binding than usual. Normally I use 2 1/2 inches, but for these quilts I made 3 1/2 inches.
One was for my husband and he loves the quilt. Carries it from room to room and it keeps him quite warm (It gets cold in New England and we keep the house at a balmy 66).
So yes you can use fleece, just need to modify how you plan on finishing