It could be your stitch (which one are you using?)/your tension needs adjusting, your needle is dull or you need a stabilizer on the back. That's all I've got.
Of course, there's always the sewing machine gremilins!
If you do find the solution, you can go back and trim the frayed edges and use fray check to keep them from fraying again.
Good luck! :thumbup: