I was tailing off like you. I found I was taking my eye off where the edge of the fabric was as I was coming to the end of the seam. I started making sure I kept the fabric straight all the way to the needle and the problem was solved most of the time.
Then I noticed that when I set my needle in the left position and ran my fabric along the inside edge of my presser foot, my fabric wasn't on both feed dogs all the way. I didn't realize how much slippage this caused. I have Brother machines, and the feed dogs are wider than my old Singer. Now, I am trying something new. I have changed to a foot I can use with the center stitch on my machine and adjusting my needle so that I get a 1/4" seam by letting the material ride along the outside edge of my foot, thus making sure the fabric is connecting with both feed dogs. I'm having to work a bit harder to keep the seam straight but am getting a more consistent pull.