It's bothering you now ... so ... you probably want to face the issue and deal with it now. Otherwise, you know it will bug you forever!!!!
Here's a way with out mitres that might work out for you. Cut the sides lengthwise on the fabric. Then the ends crosswise on the fabric. (Or sides crosswise and ends lengthwise). Your fabric is directional, so you will have the trick of making sure you have the design going the right way.
The next challenge you may have ..... which to me, may be more bothersome than the corners looking askew. That would be if you have to join the fabric along the sides/ends as well. Another good lesson in joining pieces and keeping the pattern working.
Honestly, it is all worth the effort! Yes, takes a little time, but the more you do this sort of a thing, the more confident you will become and not see it as such a challenge for future projects.
Just be sure to give yourself some pats on the back for tacking it and how much better it looks.
Good Luck!