Never. I am 52 and have always been excited and enjoyed watching my children's next steps. They have both been to college and lived on their own we are on the after post college transition transition. Which means that DD wanted to go back to school and in order to facilitate her paying for it she is back home. DS had a horrid breakup with his girlfriend and needed a transition housing situation. The world is a more difficult place to grow up into than it was when I was 20.
I am not sure that there is any such thing as an empty nest anymore and even if there is being a mom is a lifetime position.