Beware of ballpoints - mentioned by another poster earlier. I had been using them without an issue for HST and then grabbed one from my local bank and discovered to my horror that the bit of ink at the cut line of the HST bled right onto the front of the HST fabric when it was washed. Lavender and cream tone-on-tone. Awful, and couldn't get it out.
I know what you mean about the fabric pulling, because that happened to me with the .008 permanent micro-marker I used - it scraped on the fabric because it was too fine a point on the pen.
I use colored pencils in a near-match to the front of the fabric now, or Frixion. they ride smoother.
FYI I put samples of Frixion-erased fabric in the freezer and in the fridge, and the lines never came back. It would have been red on solid yellow and there wasn't a trace of it.