That is why you have to secure the inside edge with a smaller zz stitch or another tight stitch to hold it. I would suggest that you go around the inside edge at least twice. Then on top of that with placing the center of the flower in will add another security.
And also you can use this technique for making straight fringe too.