Maybe you should add some chino to soften it up. I don't think it looks scrappy at all. Your theme is shirts -- all kinds -- and the design is fabulous. I made a tote with my husband's old shirts and love working with shirting fabric. In fact, I made a special trip to NYC garment district just to stock up on shirting. There was an amazing variety. Keep going -- finish it -- I think you may love it in the end!