You could very well be allergic to the sizing in the fabrics (yarns, too). This allergy can cause red splotches to appear on your skin. I had a cousin with this allergy. She said she was covered in red splotches and so she stayed home and sewed and knitted. The more she stayed home and did that the worse the red splotches got.
I'm trying to remember what she said she was allergic to. I'm thinking she said the sizing was Quaternium 15, but she said she was also allergic to cinnamon and nutmeg (from the bark on the tree those come from). Now you can get cinnamon and nutmeg extract. I don't know if she would have been allergic to those. The quaternium 15 is in a ton of consumer products like shampoos and detergents. I think she had to check for synonyms of quaternium 15, as well. My cousin had tests from a dermatologist.
Bottom line, yes, you probably are allergic to something from/in the fabric. My cousin is no longer living, so I can't ask what she had to do to remove the sizing from the fabric. She had a washing machine, so she wouldn't have been limited to handwashing. I would wear gloves while washing. For long term, you might want to invest in an apartment sized washer and dryer. Even the scented dryer sheets that the previous person used in the laundromat could set off your allergies.