Use those caramel colored gummy erasors. I have seen them at quilt shops. Probably available at office supply stores too. The soft, gummy texture graps the fibers, etc out of the cuts in the mats. For general cleaning the mat off, i use the plastic scrubby with the knob handle.
OLFA recommends soaking them in the tub too. Directions on the Olfa website. Not sure about other brands tho.....i love Olfa . What Jan said!
sandy