I do some baby quilts with two layers of unwashed flannel, no batting. Like you, I have some cute panels that make it quick and easy. Besides, I like quilting a lot more than piecing <grin>! The result is more like a receiving blanket than a quilt, but it is very appropriate for milder climates like mine. I did one quilt with flannel plus warm and natural, but I though it was far too heavy; the flannel provides significantly more weight than a regular cotton quilt.
When I washed the the completed two layer quilt, it did shrink. However the quilting kept it together, and since both top and backing were similar weight, it shrunk pretty smoothly, if that makes sense. No wonkiness. I do think you have to pre-shrink flannel when using it with other types of fabrics, and I pre-wash it if using it in piecing.