My sticking point would be missing my son's games. My son plays soccer and my husband and I really enjoy ourselves at the games.
If that's not such a big deal to you, then I say go. I did the same thing you did a few years ago - there were meetings and retreats that I'd skip because I didn't know the other members of the group very well, and there was always a scheduling conflict of some kind. Then I decided I needed a break from my life and actually went on one of the retreats. I had SO MUCH FUN and made SO MANY FRIENDS!!! Don't ignore the impact of spending a few hours with some like-minded girlfriends. There is nothing else that rejuvenates me so well.