I am not "drawn to" brown or orange, either, but am planning a fall wallhanging and table runner in which I will need these colors. I will probably do a Hallowneen wallhanging, too, someday so will need the orange for that. I don't "do" gray, either, at least not yet. I may someday need some to set off something else, but don't forsee it.
I don't have white, either, but might get some if I see the need. I, too, do mostly tone-on-tones because they are so pretty! I might use a narrow solid border or binding from time to time.