I had a bil that sounds just like yours. The 4 of us traveled a lot for many years together. We would go weekly to the Lancaster, Pa. area (1 hour away). We would drive one week and he the next. The particular Friday, he was filling up the car with gas on 30 there at the shops and I asked him if we could stop at a particular store. He stuck his head in the window and said, "You are travelling with the wrong person." After that, my husband drove or we did not go with them.
Some years went by, we visited but did not go with them anywhere. She started to be a real problem, which turned out to be all timers (we use this) and we helped him put her in a home, etc. Then the police called us one night and here he was in his pjs out in the middle of a 4 land highway trying to find her. We went over and my husband spent the night with him, I came home. In a few weeks his son put him into the same home with her. They had to separate them because she got violent. She passed in two years and in a year he passed. We often think maybe there was something going on in the home that we will never find out. His son has sold the home with everything in it. We used to do crafts, craft shows, and go from Tennessee to Branson, to Maine anytime we wanted. Later on you may find something out that you will be surprised at. Maybe jealousy of some kind. I think that is what happened to us.