How do you get "yes" as an adverb? an adverb is like quickly, slowly, - modifies a verb. I would say it is an interjection, can also be a noun. But not an adverb.
I would rather see YAY than Yea - I have even seen Yeah, as in Yeah, I won a prize. to me that is Yeah, like slang for Yes.
All this reminds me of a funny story. A friend was in a museum where there was a big Egyptian display. A lot of hieroglyphics. He was in the same room with a couple who were looking at a large hieroglyphic writing and wondering what all the different symbols meant. The woman said, "There sure are a lot of ravens. I wonder what the raven means?"
He walked behind them and said quietly, "It means 'the'."