For the most part...insecurity. I worry that what I send will either not be good enough, liked, or be enough for the person I send to.
I did the Secret Santa swap this year, and I still worry about what I sent to Jerrie, and often wonder if she is able to use what I sent! (Now that I see more of Jerrie's posts, I know that the fat quarters I sent could have been MUCH different for her tastes!)
It boils down to the idea that I usually refrain from swaps because I don't want to disappoint anyone... :)