I don't get to picky over threads if the material is frayed a little. I only remove long threads and other loose threads that cling to the fabric. If there isn't a lot of contrast in the material, or very light material, it shouldn't matter to much. Don't over work or worry yourself to distraction over threads showing through. enjoy!
peace