I usually wind up taking them out - can't stand anything scratchy. Just pick the stitches from the label side and you won't hurt the garment.
And some labels that aren't so bad, when you wear another garment OVER it, all of a sudden start scratching.