I remove the selvedges first. I find that the selvedges are more tightly woven and can cause puckers.

Long long ago I was told to remove the selvage because it will shrink at a different rate as the rest of the material. There are some fabrics that do shrink differently along the selvage, cotton comes to mind. That's my 2 cents worth.