Have not had trouble with Frixion pens. Only used them for marking appliqué lines.
I used an orange Crayola washable marker on a cream quilt one time, to mark quilting lines. 5 years, YES, 5 years later, I decided to wash it before I went to all the trouble of quilting. The marker washed out and I had to remark it to quilt. That was 4 years ago and I do not have any residual marker on the quilt.