I'm wondering if, when you draw the line, the tiny width between the ruler and the tip of the pen is big enough to make a difference. Even if it's only one thread's difference, I think it would multiply by 4 by the time you use that line to measure from and stitch on both sides. So, over by two threads on one half, under by two threads on the other. And when you place two pieces together, the measurement is off.
You could check this possibility by putting your ruler on a lined piece of paper, then use your pen to draw the line. If it's over by any width at all, it is distorting your measurements.