I treat borders and binding like frames for artwork. I think it will be fine and wont detract at all.
Just imagine if you were frame the quilt in a picture frame, and go from there. I personally like adding borders in a quiet complimentary color, so that it really does look like a picture frame, and match my binding to the border.
If you look at artwork, some use a very ornate frame (pieced border), some use a sleek frame (plain border), and some go without a large frame (no border) but the whole purpose is to accent the artwork. So my recommendation is to look at your quilt, think about how you would actually frame it if you had to. Then go with the borders/binding that follows your vision.