I had the same type of problem with the sewline pencil for marking quilts. The white and the yellow lead were fine, then I used the green. What a disaster! I tried everything to get it out and never did. Of course it was on a very light quilt so it bugs me no end!!
What type of marker were you using?
OH, and I had tested it before using it to mark the whole quilt, but it came off then, go figure...