I think it would be fine to use as batting as long as the panel colors don't show through. If the panel is very dark and your fabrics are very light, shadowing through could be a problem; otherwise it won't matter.
Flannel has a habit of shrinking a *lot*. It would be a good idea to wash and dry the flannel twice before using as batting, just to make sure it doesn't shrink so much it distorts the finished quilt when washed.