I wash everything, including scraps that are given to me, but I never buy pre-cut fabrics such as jelly rolls or layer cakes. I would not wash those if I had them. If you mix fabrics that are washed and unwashed, you probably won't notice any problems unless some of the unwashed fabrics bleed (and you would have that problem even if there were no washed fabrics in the group) or unless you are using flannel or homespun, which shrink a lot more during washing.