Getting the hang of hand quilting takes time and practice -- just like any other part of quilting! So, just relax, take your time and enjoy the ride. First, practice getting your stitches even (not small). Then work on getting them smaller.
Yes, the flannel backing is probably making it harder for you, but it will be cozy.
And, finally, there's no hand quilting police here. The baby won't know the difference!