I've also used velvet fairly extensively in clothing. A lot will depend on whether it's velvet or velveteen. Velvet is rayon or silk with rayon backing. Velveteen is cotton and a bit more forgiving. I second the needleboard but have also found that a plush towel is a fair substitute if you don't want to invest in a needle board. Be very careful with the pressing. As SueZQ said, velvet marks easily. Velveteen not so much.
You can used flannel instead of backing to make it a bit lighter. If you want to use backing, I'd look for the lightest weight batting you can find. There are some very thin ones out there.
I'm not sure about trying to hand quilt. I think the big stitch as mentioned earlier in this thread would be the only way you could do that.