I think that it will be just fine. They are called summer quilts when there is no batting in them. I would just caution you to wash the flannel more than once because it does not shrink all the way in the first washing. I usually wash two or three times just to be sure. It would be bad if you got your quilt all together and washed again and it turned out wonky.
Debbie