I agree. I think some of our frustration is more with not knowing that our packages arrived safe and sound than with not getting something in return.
I think it is a real safe-guard in a swap like this to actually get delivery confirmation and a receipt so that if a package doesn't arrive, there is proof that you sent it. I would be very upset if someone thought I was a "deadbeat" if a package did not arrive, even though I sent it.
I also take pictures of what I am sending out, just in case.