I say go for it, then you'll be able too see the beautiful fabric in your finished quilt every day!
I do have fabric that I love so much that I only want to use it for a project I'm going to keep. Then whatever piece is left over I hang onto and only want to use if it's something really special.