We had almost that same situation.
When we bought the house, the wood trim ( not window and door frames) on the house was painted that "fence red" (with tri colored 1970s brick!?!?!), as was the fence, and the 10x16 shed (with dark brown trim, yet). I'm not a fan of the color, and I think as it fades it gets worse. Also, the darker colors absorb the heat and fade the paint faster, and I think they age the wood faster.
Fast forward 5 years. Yes, I lived with it for 5 years, despite hating the color, because DH doesn't mind it. I painted the wood trim on the house a nice neutral beige, then we put in a garage. I got to pick the siding color. Bisque. A beige of sorts. The company that put our garage package together said it was the most popular color they sold. I mentioned to DH at that point that we might need to paint the shed at that point too, because the red would just not go anymore. He still didn't want to do it. Fast forward another 2 years.
We did all the window trim on the house and redid the garage trim - nice clean white. DH starts eyeballing the shed. Finally comes to me and says.... "it doesn't go, does it?" I wasted no time getting the stain! I wasn't waiting for him to change his mind again.
Two coats of primer, done in a day, and the shed was white. 2 coats of stain (I think it said 4 hours apart on the can) the next day and the shed was Beige with White trim. It's a weekend job.
Now I just have to get that fence done. Coincidentally, it needs to be replaced this year... I'll by the stain and have it in the garage before he can decide he wants it the same color.
It took me almost 8 years, but I'll get the yard looking the way it should.
I vote the same color as the house. It unifies the yard.