I think you're on the right track for color families, just not quite the right go-together fabrics yet. Blue and rust are classic complementary colors, they just have to be shades and hues that coordinate. To me it's the brown in the center that does not go with the other colors (except the blue). Try auditioning a different shade of brown, a bit lighter perhaps, and less reddish. I hope you include a blue, as it will add life to your color palette.