Years ago when I started making quilts for our bi-annual (is that right? every 2 years) family reunions I came up with this great idea that I would keep one block from each quilt and wind up with a sampler quilt. Great idea - however, I have no clue as to where those extra blocks went - probably someplace very special. So, I believe that you have come up with a great idea and no it will not be ugly - it will be a very special quilt with special memories.