I think it depends on the fabric and how densely woven those selvedges are. On some fabric the selvedge is barely noticeable but on others it's really dense and sort of ripply - those dense ones I wouldn't base my cut off of because I don't think they're straight enough, I feel like they're bunching up the fabric almost like a strip of weak elastic would.
But about 99% of the time, I base my cut on a folded edge. The only times I don't is when I'm doing LOF borders, and those I rip, like NJ Quilter does.