I have a standard reply that fits many scenarios: "thank you for asking me to ....fill in the blank( useful,because it catches them off guard). However,due to a conflict in my schedule,I'm unable to do it at this time. Please ask me again. That's all I say. No need for justification,guilt ,hogwash.