Since there are so many unpleasant experiences listed, I thought I'd add that I have a close friend with whom I cheerfully exchange books several times a year, and we both return them faithfully. I have known her since 1970, longer than I've known my husband.
It's apparently just too easy for some people to ignore property rights. I wouldn't even consider borrowing something that I wouldn't be able to replace if it gets damaged or lost.
We all need to be braver about the business of not wanting to offend people who have clearly not been too worried about offending us. It should not be that difficult to say, "Sorry, no." You don't owe anyone a loan or an explanation. In fact, if it's a previous offender, you owe it to yourself to mention that they didn't return or damaged the last thing they borrowed. Maybe they'll suddenly remember where it is.