you are not catching enough of the fabric to get past the fraying-
widen your satin stitch- or fold under the edges (either by adding interfacing (right sides together sew all the way around- slit the interfacing- turn right side out- place applique- then satin stitch around the finished edge)
or if you could try some fray check along the edges- and widen your satin stitch so it grabs more of the appliques