Yes, I would definitely worry! I have never personally owned a pitbull, and I never will. A couple of years ago not far from where I live an 11-year-old boy was killed by his family's pitbull. The boy and his brother were horsing around, chasing each other through the house, and yelling....just having fun. The 11-yr-old sort of fell onto the couch as they tussled a bit and the dog leaped on him there. The dog couldn't be pulled off and mauled the boy so terribly that he died. This was a dog which supposedly was the sweetest dog in the world, having grown up with this family and never caused a bit of trouble. Now their child is dead!
I know it's probably wrong of me to condemn an entire breed, but that's how I feel about pitbulls. I've had many large dogs throughout my life and have never feared any of them, but would not ever trust a pitbull.
As far as your grandchildren, yes I would worry. That dog was used to living with adults, and to now put him in a home with children who run and scream and play would not be very wise in my opinion. And, yes, it's just my opinion, but you asked.