The same thing happened to me. I ran it through the dryer a couple times and more came off. Also, I had a sweater "de-piller" (don't know if that is what it's actually called) that helped to remove the pieces that didn't want to come off in the dryer.
I love your quilt and fabrics! It looks wonderful!