I would wash (actually - just flush the afflicted areas) if there was something really gross on it that had to be removed such as blood, vomit, urine, food and beverage stains
The handling grime ( in my world that includes starch/sizing and glue basting pruducts ) can wait until the item is finished to be washed out.
I am in tbe group that washes all washable components before using them in an item. This includes trims like rickrack, bias tape,, and zippers.. uneven shrinkage annoys me.