I recently used an orange chalk marker and had a heck of a time getting it all to come out. Several washings later - with a bit of 'scrubbing' (with an old toothbrush & some dawn dish soap) and the stubborn marks finally were almost gone. No one else would notice them but of course I did since I was looking for them.
I've not had issues with any colors of those markers in the past. I think this time I pressed just a bit too hard to get the marks to stay put. I think that was probably the main issue.