If you really like the (not yellow) fabrics and really want a quilt that showcases them, then yes, I would pull it apart and remove the yellow. (And I would choose one or more of the colors in the print fabric if you were going to return to the same design as you have.)
I say this coming from having experienced similar dissatisfaction; if you KNOW you're not going to be happy with the results, start over.
-- Jillaine