This delema haunted me when I first started quilting and still does on a bad quilting day. My problem was in my cutting. I was holding my ruler too tight on the edge of the fabric and then when I let a tiny bit of the edge show, my blocks were a better size. Another thing a friend taught me was to square up as you go. First it might be a HST, next into a 4 patch and so on. Square the block at each step and the end results are very good, often with no squaring up needed. Good pressing also helps, set the seams before pressing to one side and then press again. I don't believe in this scant 1/4" seam and I believe using the above suggestions will solve the problem.
I know you will find your own little thing that is throwing your size off.