I don't trim them off, just try to make sure I can iron to the dark side. If I have one fabric that frays really badly, I will sew the quarter inch seam, then sew another at 1/8 inch. Sometimes, I have been known to sew a 1/8 inch seam (stay stitch) on only the badly fraying piece. That prevents a double seam that might not be as flat as I want it. It's double sewing, but it solves the problem of further fraying.