I was in the camp of Not washing fabric before I got a terrible allergy to a fabric I was working with that I did not pre wash, and when ironing it, it off gassed so much that my face got chemically burnt. Not to mention what we inhaled when the house stunk.
I am now in the always wash before I work with it camp.