You can rag either side. I have done denim I-spy quilts and ragged the denim side one one and the I-spy side others. You could rag both sides depending on how you sew the seams.
i-spy side ragged
[ATTACH=CONFIG]16026[/ATTACH]
denim side ragged
[ATTACH=CONFIG]16027[/ATTACH]