On someone else's quilt that I was giving away and won't see too often, it wouldn't bother me and I'd probably leave it. If it bothered me that much and I was going to see it daily, I'd take it out. That's me though.
Can the untrained eye really notice a difference when viewing the whole quilt?