use what you want. the quilt police don't really exist. I consider it a personal challenge if someone tells me "you can't" do x or y. I think what is meant is that the raggy part will not be as colorful if it is white on the back. I've never thought about this or worried about it and I make a lot of raggy quilts--have used all flannel, all non-flannel, and a combination of the two.
Laurie