It only takes one word ............... NO!
You shouldn't feel you have to give any reasons for your decisions.
Aquot, that I came across a few years ago, on this matter ...
"Saying no, is saying yes to yourself!"
I only wish I knew who to give credit to these empowering words.