Once it's quilted, it won't even be seen. You're border fabric isn't directional, and the eye doesn't zoom right to that spot. I actually I had to follow the pieces around to see what where you thought the problem was.
The only way to avoid that is that the first piece (let's say the "little" one) has to have a partial seam that will be completed later. The piece would be the same size as the others (or oversized to be trimmed later). You would trim the upper edge to align with the center, like you did, and work around the center adding border in a clock-wise direction. When you came to the last border piece (the one on the bottom), that would be attached to the center, AND then it would be attached to the hanging piece of border piece #1.
I hope that makes sense.