My thought is that your plans have been made long before the invitation was received. Therefore, your husband should graciously represent both of you at his grandson's party, supportively explaining that you had a trip planned a year ahead that could not be cancelled. And, you should make plans to do something just from you to the grandson as a token gesture of good-will.
Just what I would do...take what works for you & leave the rest.