Just a quick reminder before you wash it - look at it very closely after washing to make sure the marks came out. If you put it in the dryer and they aren't all out, the dryer will probably heat set it in.
I use a product called AWESOME to get most any stain out, but haven't tried it on pencil. You can buy it at Dollar Tree, Dollar General, and I think I even saw it at Wal-Mart. I use it on everything - clothes, general cleaning, furnace air filter, grease hood on stove, in my water for carpet cleaning, etc etc etc.....it really is awesome. lol. I think that is what I would try first if I had your problem right now.