Flannel will fray no matter what you do to it. It's the nature of the beast.
Wash it twice in hot water to shrink it as much as possible.
There is a product called fray check, am not sure how it would feel on a whole quilt. I have used it in small spots when doing my hand applique and the motif was fraying, but that is just a dot here or there.
With flannel the fraying can happen anywhere, anytime.
You could do a double row of stitching, stitch on the edge and then come in 1/8" and do another row of very small zig-zag. I have repaired a flannel quilt doing that. It's alot of work to do a whole quilt though.
Good luck, there is no good answer.