When working with velvet you have to be carful not to crush the pile. Try a little glue stick in just the seam allowance and see if that holds it long enough to stitch the seam. Try it on scraps first to see if it helps. I would suggest little strips of fusible but ironing on velvet or velveteen will sometimes cause marks.