Have someone watch you cut and maybe they can diagnose where the problem is.
This is a very good suggestion. I am also left handed and had the same problem. I had someone watch me cut and discovered I was slanting my blade and the ruler was moving oh so slightly. I got the grippers for the ruler and have to pay attention to the blade so I don't slant it. And lots of practice, so it isn't a problem anymore.