You're right, the one on the top left is the worst. When you square them up to 5 inches, is the diagonal seam exactly on the diagonal line of your square up ruler? That has to be exactly on the diagonal line when you square it up. Even if you cut them to 5 inch squares, that diagonal HAS to be on the exact diagonal line of the ruler.
I have been there!! Can you tell? That is your critical part that makes them all meet in the middle like they are supposed to. You can not adjust it as you sew the squares together because of that diagonal seam.