Is it said more jokingly than seriously? If so, then try to take it as a compliment. But if they are getting mad at you about it, then... Wow - you seem to run into some very forward and grabby / greedy people. :roll:
Try to ignore them and go on with your caring and good works. You are storing up your treasures in heaven!