I always wash every piece of fabric, with a couple of exceptions.
1. One-block-wonder quilts. If you pre-wash the fabric, the pattern might not line up over all the repeats, and that makes the kaleidoscope effect impossible. I always wash after the blocks are together (before the border, which is pre-washed fabric). I might not end up with the exact dimensions I was hoping for, but the pattern kaleidoscope effect remains intact.
2. Wall hangings. I figure they're not meant to be washed anyway, and since they'll generally be hanging out all the time, the more vibrant the color is to start, the longer the colors will last. Plus, working with un-washed fabric (especially the smaller pieces I generally use for wall hangings) is much easier.
For everything else, from potholders to quilts, I always pre-wash. I don't need my 29 different fabrics shrinking 20 different ways when it's time to wash the quilt. And I certainly don't need the red dye from fabric 1 running onto white fabric 2. I do not pre-shrink my batting, though. I find it's pretty consistent, and Warm and Natural only shrinks about 3%. I like that it gives the quilt a pillowy look when it shrinks (slightly).