One possibility is that you stretched the backing fabric too tight when you held it down with masking tape. It should simply be smoothed flat -- not stretched tight. When it is stretched too much, once you let go of the masking tape the fabric springs back to its original size, and this can cause wrinkling -- especially since lengthwise and crosswise grains of fabric have different stretching characteristics.
You did not mention how you basted the quilt. For hand quilting, it is usually best to thread baste.
Personally, I would also experiment with spray basting. Many hand quilters spray baste their quilts, and spray basting has the advantage of keeping all the layers continuously flat against one another (whereas with pin basting only the pinned places are perfectly flat against each other, and with thread basting there is an opportunity for layers to shift). Some hand quilters find that spray basting adds drag to the quilting needle, but others have no problem with it.
When you hand quilted with a frame before, do you mean you used a full-sized quilting frame (the kind that does not require basting)? Or do you mean a hoop (floor hoop, lap hoop, or hand held hoop)? Hoops require basting, plus you need to take the hoop off and re-position the quilt periodically. Full-sized quilting frames simply have you roll the quilt as you complete an area.
If it was a hoop, again, stretching the backing fabric while basting can cause problems with the backing. Also, if the basting is not close enough together, the layers can shift as you loosen and re-position the hoop. If it was a hoop, did you keep the quilt very loose within the hoop? (That is actually correct; a common beginner mistake is to hoop a quilt drum-tight.)
Edit: I would add that I tried hand quilting without a hoop or frame. My stitching was not nearly as nice as when I used a hoop.