You can use anything you want for quilt filling, but if you want to use a filling that won't shrink, make sure all your cottons were prewashed/preshrunk or you quilt all over very closely so they cannot shrink.
When I first read your post I thought you meant the white fleece batting. But if you mean polar fleece the only thing I would be concerned about is the color of the polar fleece showing through unless you can get white.