As already said ... get what you like the most.
Some people's skin do look better with gold or silver .... so it may be worth a consideration.
Or perhaps will muddy the thoughts!
Another thought ... how about mixing the metals.
There are some lovely setting that have white and yellow gold .... or one of my most favourite is to see tri-golds together.
It really goes back to my first comment ........... get what you like!
(and phooey on everyone else!
)