I generally disregard any advice that includes 'you should' or 'you must' - unless it is accompanied by an explanation of why I should or I must.
I wash fabric for two reasons - the sizing in fabric irritates me, so it has to be washed out. The other reason is that I buy the fabric from stores, which are usually clean and neat, but who knows about the store next door or warehouse it came from? I wash fabric to ensure that no pests survive to propagate in my home.