You need to fix that now because each difference, no matter how small, will make a big difference later and you will not be satisfied. Take the blocks apart and square them up so they are all the same, even if you have to cut them down. Either throw out the smallest block or make them all that size. You will be much happier with the final results. :D
LaughingRain