Go with your plans! As stated the child will not remember! If anyone has hurt feelings it will be the parents , and they should have said something much earlier. Someday the parents will be in the same situation.
I sometimes have this issue with family "planning" things and giving limited notice. As none of my family lives within 600 miles . It poses a significant problem to travel on "limited"notice. I had to give up the guilt or learn to live on someone elses schedule. Guess which one I chose! I'm much happier living my life.