Gotta say - been there, done that! I was making a Triple Irish Chain and mis-cut for a couple of the blocks. I didn't have enough fabric, so I pieced it together. I think I have a couple of blocks in that quilt where I had to piece one particular fabric to get what I needed. Only I know it is there, and I look at is as a "where's Waldo" type thing.
I say go for it!!! Use the fabric you love, it will be wonderful in the end.