I just showed it to my DH. He doesn't see it as a swasticka (SP??) and his mind is very logical. He's an engineer, so I know designs are very natural for him. Anyway, I'm also a Jew and would normally be quite sensitive to something like that. Again, I don't see it either.
That being said, I think the image is there for you, and you will be happier if you rearrange the block. Especially since you keep lamenting over it!
Your other squares are quite pretty by the way!