I would leave it, too, especially since your quilt is scrappy. It would bother me if there was only one beige fabric throughout, but with scrappy, anything goes.
Also, I forget who says this, but you pay for both sides of the fabric, so feel free to use them both.