In my opinion you and your DH had a wonderful vacation, you were together and you both spent time doing something you you love - what's wrong with that?
My girlfriend and I meet up with some online quilting pals at an annual weekend retreat held in a quilt shop. Her DH doesn't understand why she would do that and her reply was that it's the only time she can sew when she wants to and for as often and as long as she wants to - no phones to answer, housework to do, dinners to cook, etc., etc. and all the while hanging with some great friends. We also can hit the Lancaster fabric stores if we want to take time away from quilting - what more could we want?
We're both disappointed that we can't make the retreat this year so I'm having a Jersey Girls Mini Retreat this weekend at my house for her and my sister.