I definitely wash black, red and blue fabrics since I know they are the most likely to bleed. However after making a quilt where I did not prewash ALL the fabrics in the quilt I had some of the unwashed fabrics shrink and I had to ditch the quilt. Now I wash all the fabrics in a quilt the way it will be washed when assembled, then I dry the fabrics on high to make sure any shrinking lingering there happens.
If my fabrics have high contrast, I may also do the test where you wet small pieces of the high contrast and put a paper towel down and then stack the pieces and put a weight on them for a couple of hours. Some fabrics may sluf a little color at the seams and some light colors will grab any color near by and the wet and weight will let me know if that is an issue. I hate when there is no real color release tut the white or cream grabs the color at the seams.