An idea that just popped into my head is to add two narrow strips on adjacent sides of the smaller blocks (sort of log cabin-ish or attic window) to get those blocks the same size as your larger blocks.
I'm visualizing a light-medium color. You would need to play around with placement to see if you would want it on the inside corners or the outside corners of your finished large block. My initial thought was inside, but you would have to try it both ways to be sure which looked more pleasing.
Just set the blocks on prospective fabric the way you have them on the cutting board to audition.
This would also allow you to use different fabrics in different block sets as you mentioned wanting to do.