Some come off more than others. It's amazing how much some of the glitter fabrics can shed and keep shedding for years. Others have practically zero shed and keep all the shine. If you really don't like it and a soft brushing/prewash isn't enough, then I'd figure it would damage the fabric too much to take it off -- but what does the back side of the fabric look like? You get to use whatever side works better for your project.
I've always liked metallic fabrics but in the last few years I've become aware that not everyone feels that way -- mostly because people have started handing it to me because they heard I would take it.
Glitters are my least favorite, but I recently worked with a pink piece that was just lovely. It cut nice, sewed nice, didn't shed, nice hand -- but definitely sparkly! I also had the chance to work with one of the very shiny plastic looking more solidly coated type and it was lovely to work with. Didn't have any problems with a hot iron or anything like that, my pressed seams were crisp and it had a nicer than expected hand fee;.