I have not done it, and I think you are extremely brave to try it!
My suggestion is to cut a strip and handle it for awhile. That will tell you a lot about how much it will fray. I'm thinking it will fray and shed fluff a lot. Not sure that stabilizing it with a fusible will help, as the fluff will be coming off the top -- not the fused part of the velvet. Plus fusing will stiffen an already thick fabric, making it harder to turn it over the edge.
Have you considered that the purpose of a binding is to take a lot of abuse for the quilt? Velvet does not take abuse well.
Velvet is slippery, and will show marks from the presser foot. I think you could sew it to the front by machine, by I would definitely recommend finishing it by hand.
Will be very interested to hear how this goes.