Part of your particular problem is that you are trying to fray a curved line; consequently some of your fraying is on the straight of the goods, but some is on the bias. Fraying works best on the straight of the goods.
Have never tried it on the bias, but I would suspect you'll have to help it along with your scissors. Good luck! It's a cute quilt! :thumbup: