You can try heating it with a hair dryer, then rolling it off with a finger or piece of fabric. Heat softens adhesives used in labels. This is how I usually remove labels as it allows all of the adhesive to come off with the paper. In this case, though, it may not remove all of the adhesive. If so, I would probably move on to using Goo Gone, allowing it so soften the adhesive, then rubbing with a piece of scrap fabric.