I'm inclined to leave it smooth side showing, for exactly the reason that willferg mentions. And the lint would be unreal once it starts to pill up.
This is the fleece in question by the way. I think my grandma had this exact color in a sweatshirt
:
[ATTACH=CONFIG]427662[/ATTACH]
Certainly not my first choice of backings, but I just quilt them
I don't pick the matches.