The selvages have a looser weave, and therefore aren't as strong. Over time and many washings, the seams with selvages (from what I'm told) won't hold up as well as the rest.
I think if you fold your fabric and then turn it so that you fold the fabric again so that the selvages are together, you can cut them off in pretty short order.