To provide an option, you could trim a small amount from where the seam is (usually 1/4" wide, but be careful.) You don't want that seam to get too thin. This will give more room for the binding.
I agree that leaving alone is the best solution, if it visually works for you.