If the fabric seems to be askew when I'm folding it over, I abandon going by the selvages. I can usually see the grain on the under piece. Once that is straight, regardless of selvage position, I then put the grain of the top piece straight visually, then trim as usual. I don't like tearing edges (not that I have the strength), as it ruins the fabric quite a ways in.