The simple answer is that YES, they need to be close to the same size or they won't fit together correctly.
I would measure all of the blocks and sort them into 3 batches: exactly right, smaller and larger. Then I would check the seams of the ones that are off. Are the seams too small or too large?
If the difference is just a little, then I would steam press the smaller blocks into shape to block them (as has already been suggested). For the larger blocks, I would take them apart and take a slightly larger seam. You already realized that cutting them would distort the corner blocks.
Good luck and don't give up.