Once you see the swastika pattern you'll never look at the quilt in a good way so it would be better to change it now.
Take your block apart so you have four rail fence blocks. Add a solid square. Rail fence, square block, rail fence square block..... It will become a whole different pattern. Or take apart and rearrange the four rail fence blocks.