When I started putting in gardens at our home, I loaded the beds with yellow and white flowers. It was a bit bright and hot looking. I traded some of the yellow flowers for pinks and purples, added a few blues, put in more evergreen shrubs, and now it is the way we love it. I think now the yellow flowers accent rather than overwhelm.
I feel the same about yellow in quilts. I love yellow. But in quilts as in gardens, a little goes a long way.