It could also be the way you are 'unpicking' it.
When I am ripping something out, on the front side of my fabric, I just clip a stitch every 5 or 6 stitches with my seam ripper, then I go to the back side, grab the back side thread end at the beginning of the line of stitching and just pull it away from the fabric. The entire back length of thread comes out in one piece, and my entire piece is now apart.
There may be a few tiny cut pieces of thread clinging between the fabric pieces, but they just brush off.