I do not wash charm packs. For the very reason you found. If they are in the same pack they are from the same line of fabric, usually, and will shrink the same once the top is put together.
HOWEVER, if you like to prewash put small pieces in a mesh bag or pillowcase and wash them that way. I've also put them in the sink in warm water and washed them by hand. Others like to wash on delicate in their washing machines. Hope this helps.