You do what you gotta do. Tomorrow is a brand new day. You do need a retreat space. Especially with new activities in the house. Raising a family is stressful, doing it in someone else's space makes it even more stressful. I vote the camper would be your sewing room and 'retreat'.
People are more important to me than fabric. Family, friends and loved ones take priority over my hobbies.
The best for you and your family.