Some of the colored chalks say on the packaging that ironing over the marks can make them difficult or impossible to remove. Check your instructions on that.
So far the only colored chalk I had trouble removing was the blue pounce powder. I had a devil of a time getting it out after I marked a stencil pattern with it and quilted over it. Luckily after much scrubbing with a brush most of it was out and what remained wasn't obvious or offensive since it was on a blue sky fabric. That was the last time I used that blue pounce powder, though.