I wash, starch, and dry all fabric before it goes into my stash. I worked in a fabric store years ago, and you never know what customers have on their hands while they are looking at the fabric, or which bolts of materials the kids blew snot on while playing.
To me, ironing fabric and folding is relaxing and I enjoy a little break from the creative times of quilting. Plus, when I'm piecing, the fabric is already starched which helps stop a lot of the dirt from sticking. It also stiffens the seams while piecing.