If this fabric is giving you problems now - think what it might be doing five years from now after some use and a few washings.
I agree that it would be a big undertaking to pick it apart at this stage and replace the problem fabric - but do you want to deal with it for the life of the quilt?