When you hold down the ruler, do you hold it in the middle? I used to hold mine in the middle and it would shift on me just a tad. Now I hold down the lower third, cut up to where I still have good control, then walk my fingers up the ruler, hold it down and cut some more. Sometimes I only need 2 changes, sometimes 3.
I also stopped layering too many layers of fabric. Cutting fewer layers gives me better results.
Sometimes - depending on the pattern, I may cut a piece oversize and trim it down precisely.