It's not ugly.
Try some neutral colors to bring out the blues and the yellows. Go to a fabric store and lay out the squares on bolts of cloth until something pleases you.
Try putting the patches also in another formation. I know for myself I don't like the non uniform look. For me I'd probably be putting the yellow and glowy ones in middle, sash it and then surround with blue? Or do something in a pattern. Rearrange till it suits you
Or you could try instead of neutral or complementing colors, try contrasting ones.
Blue and yellow are contrasting, if blue isn't working, try yellow or even heck try purple. You could try batiks as well.