To make sure I understand correctly, they want you to pay to ship back something that was a mistake on their part -- you ordered in good faith. If you had ordered blue fabric and they sent yellow, would they expect you to pay to ship it back?
It might benefit you to call again and ask to speak to someone else. I'm sure it can be resolved without cost to you.