If it really bothers you, bite the bullet and replace that particular piece. I have taken pieces out and replaced them on occasion. You can always hand stitch them back in using a ladder stitch. Kind of a pain in the patooty but worth it in the end. I have found adding patch material can add extra bulk, and besides I'd always know there was a boo-boo there. I feel your pain. Good luck and I hope whatever method you choose works well for you. :)