Yes, most likely the design will be affected with that many fewer fabrics. Are you able to post a picture of the pattern, or a link? If so, we could give more specific input about how you might proceed. There are definitely patterns out there that use fewer fabrics, it's just a question of finding one. If I recall, some of the BQ patterns work with just a few fabrics.