Do you pivot at the point with your needle DOWN IN THE FABRIC?
To pivot, even with the needle down, the fabric can easily be "over manipulated" underneath the needle....I'm talking about down thru the layers.
After you pivot, with the needle still down, jiggle the fabric slightly to see if it wants to readjust itself on the bottom layers.
If you'll do this several times, you'll soon learn what you are doing that causes the lower layers to shift more than they should. The point is, all the layers need to be lined up with each other after the turn/pivot, to prevent that wobble at the point.
Good luck!