I've had exactly the same experience with blue fabrics. For me it's an issue with the neutrals – some blues pull toward being matched with creams, some more towards white. It's the difference (to me anyway) between the subdued and the bright. Yes, you could probably throw them together, but you are right, the feeling isn't there...