I never use colored chalk for the same reasons. But the purple (if quilting right away) or the blue (washes away with a spritz of water) work well in the white areas.
If you think the colored chalk is leaving a stain because of an oil in it, you might dust some corn starch on it and use a Q-tip to rub it in and get out by shaking it. I've also used a little oil release on a Q-tip and dab it on the stain when I've had some machine oil on the quilt.