I like, & often use the same phrase listed already - "don't let the person live rent free in your brain" - you're the only one concerned, or they wouldn't have made the thoughtless comment - they don't deserve your angst.
I, too, have had to quit working - for health reasons that made it impossible to continue caring for laboring women & babies - but my time is just as valuable (& so is yours) as anyone working for a paycheck.
I used to hear, "You have more time than I; my kids are little. Your kids are in high school". Heck, I was busier with teens than I ever was nursing a baby & changing diapers! lol
And now that I'm retired, I'm still busy every day, even if the thing I want to do today is nap or sew or listen to music - its MY time to use as I choose. I help where I can, but have learned to say "No" in order not to overcommit to too many worthy causes.
You are always important to God; do what you are able & look around at our great world to find something to smile about, & forget the rest.
I'll say a prayer for you.
Jan