I don't think you have anything to be alarmed about. When people "follow" you, it simply means that they are looking at the things you've pinned on your boards. They do not have access to any personal info, unless you upload your own pictures of your house with your address clearly visible, or pictures of your credit cards, etc.
Pinterest is basically a visual bookmarking site. You know how you bookmark something you want to come back to? Well, this is the same thing, except the bookmarks are all pictures. My pinterest board is full of recipes; pictures of quilts (VERY inspirational); funny pictures of silly animals (makes my youngest son giggle); beautiful kitchens (so if I ever get the chance to renovate I have some wonderful ideas); lovely places around the world that I hope to see someday, and so forth. The only thing that is on my Pinterest board is my name, and you do not have to use your real name if you choose not to.
You can change your settings to not receive notifications of when people follow you, etc.