Next time, try using a Pentel Gel Roller for Fabric - it says permanent - and I believe that it will be.
I ordered mine on-line -
I have not been pleased with the Pigma pens, either. By the way, they were designed for use on paper - even though the fabric stores are selling them.