Just looking at this block shows me that not many units are very square. Measure the centre piece, and the left side seems larger [top to bottom], and the outer units continue the problem. Can you mark the back of each piece and units. with a dot in each corner, or lines, in the correct finished size as in the illustration above. this might help
Good luck