I wouldn't wash the panels. If they are straight now, they could go wonky with washing. Fabric shrinkage is not an issue if the quilt is moderately quilted (not tied, and not quilted in lines 12" apart). Once quilted, the batting controls shrinkage of the fabric; mixing washed and unwashed fabric in this kind of quilt is fine.
I would wash only fabrics that I suspect could bleed profusely. Moderate bleeding is not a problem if you wash a quilt in a large machine (that uses a lot of water; not a domestic front-loader) in Synthrapol.