I would double check each seam and see if you are not veering off right or left at any point along the seam. That will distort your block size.
I have been quilting for years and sometimes my seams or points will not match no matter how I re sew them. Unless your swap is for a certain pattern then make log cabin blocks. Easy blocks of all to make.