It wouldn't be wrong for you to tell him, as he goes out the door, that NEXT year, he is to figure out a different way to celebrate with the kids.....that it's now time for you to have separate families. I had to do that with my ex. It's YOUR home --- he can't intrude unless you let him..............and if the kids can't understand, then they'd better learn. I find it amazing tha you allowed this for so long. My ex was abusive, too. I allowed it for 2 yrs, then told him he wasn't welcome in my home anymore. The kids survived.....and I'm still very close to them -- they actually understood!
and 2nd thought ---- why should you have tha anxiety!!! This is your holiday, too! And the kids know by now that you can have a civil divorce -- but civil does not mean you have to allow him in your house.