Anyone who has experience making rag quilts knows sometimes a slight tug of the material will pull off pieces of fabric. Very easy to stick in the mouth and swallow. No I don't raise my children in bubbles, but I also don't want them dead. My great uncle died at 2 1/2 years old from a choking hazzard. So, I wouldn't intentionally put danger in my child's path, either.
If you feel it's safe enough, that's your decision.