I think I would sit my husband down and explain to him that you are not happy that your space was invaded. Remind his that he wouldn't like it if the grandchildren invaded his woodworking area and played with his tools or scattered his supplies all around. In fact, I'm sure he would be rather upset.
I'm very fortunate that my husband grew up with a mother that sewed all the time except on Sundays. She didn't drive and she wasn't able to work outside the home. So she spent her extra time sewing for her family. Her sewing machine was in the dining room and Rich says he has fond memories of sitting at the dining table talking to her while she sewed. Anyway, he understands that the smallest snippet of fabric might be needed and if it is on the floor he carefully picks it up and lays it on my sewing machine table.
Rich even goes with me to the fabric stores, quilt shows, craft fairs, etc. If he gets a little restless he will go outside and find a comfy seat and sit and wait for me no matter how long I take to shop or look.
Shirley