I think is really depends on what color of chalk was used. White seems to always come out. The color additives in other chalks can stain just like any paint. Always test a scrap of fabric before using. Not every brand of chalk marker is as good as another. When dressmaking, I prefer to use a watercolor pencil to mark things but that is on the inside of the project).
Did you by any chance iron or use the dryer on the chalk marks? You might have accidently set the color onto your project.