I think there are different kinds of contamination. There are all kinds of chemical contamination on new fabric and clothing. These are probably worse than the clothing you might buy at the second hand level. When I am lucky to pick up fabric at a yard sale or 2nd hand shop it goes right into the washer. Lots of people like to use old shirts, ties, sheets, etc. in quilts. I would always wash these before using them.
I guess I wash everything before I use it, even when I buy new clothing.