What I do to help with the 'swinging' - and even the entire seam - since I use the edge of my presser foot as the guide for my seam with my needle set to the 1/4" setting - I run the index finger of my left hand ever so gently along the left edge of the presser foot while holding the fabric. That helps me keep things lined up and the seam straight.
Someone else mentioned a stiletto - which is also good for this purpose - but I find that having my hand on the fabric and finger against the presser foot gives me a bit more to hang onto and keep things lined up better.
Sometimes you do need that stiletto helper as well. I generally just use my seam ripper for those times (a chop stick or skewer works as well). I also like to use one to keep the little bits of the seam allowance properly flat/folded when going under the presser foot.