I think you need to keep the dark brown, dark green and dark blue for contrast. They draw the eye.
Can you try making each color group shade from lightest to darkest (so the darkest color of each group will be next to the lightest of the next group)?
I love the effect you're going for - and can't wait to see the end result!