I have ready your interesting idea and The Lord of the Rings sound very different, however, guess your waiting on the however, I personally would not use velvet for a baby quilt. It will not stand up to the frequent washings a baby quilt will need. Also velvet has a pile much like fur and needs to be cut all the one way or it will look odd. Not easy when you are working with small pieces of cut fabrics.
I am meaning this to be constructive and hopefully you will find some of the many beautiful and innovative fabrics out there that will give you just as nice effect as velvet and stand up better in wear.
Good luck
Elle