Yes, I think that if you add some other neutrals s/a black, tan, etc. along with the red and gold, it will make it a little more manly.
I'm having the same problem with some shopping bags that I'm making for my son. If they're to frilly, he won't use them...at least not in public...LOL. I went with some neutral, necktie prints in black, grey, camel, etc. I plan to put bright red handles on them to jazz them up.
~ Cindy