I only have this problem with jelly rolls that have the pinked edges. On the other hand I seem to get more of this little clutter from any black fabric. I suspect it is the color that is so much easier to see.
I agree, a roller or one of the old fashioned clothes brushes will do the trick.