I didn't see a swastika until I read your post, but I showed it to my husband without any forwarning and he immediately said it was a swastika.
I still think the block is very pretty. Maybe you could add some decorative trim or embellished stitching (crazy quilt style) to detract from the spinning.
Also, I think it's unlikely to keep that look up if you make several more and put them together. I would think it'd look more like stairsteps.