Years ago I was taught to fold the material and pin the edges with rust-proof pins. If it is a large amount of material, you will have to refold and repin when drying it. I personally haven't had any trouble with fraying using this method. It takes a little more time, but doesn't get tangled.

I just washed 2 huge loads of fabric(big sale/rain/high humidity). some pcs were scraps. some flannel was less than8" sq.. had hardy any fraying, much less than any load, I've previously washed! I think it was because the washer was filled to the max!--LESS room to move around. I was pleasantly, surprised! I also used my gentle/ slow agitation speed& shorter cycle!