My quilting instructor always tells us to trim the blocks to match the smallest one. Sometimes I do, sometimes I just put a border on. The other day in our BOM class somehow my block came out way bigger than it should have. (might have been a cutting problem) Anyway, instead of cutting it down, I just added a little more to it and said, "Theres my center medallion." Personally for what you want to do I'd trim them all to the smaller size. Starch stretching can sometimes really mess you up when it's washed--especially if the fabrics weren't prewashed.