I haven't made one yet but have "researched" them. From what I gather, you can use just about any fabric. What I would watch out for is the purpose of your quilt and if the materials are washable. There are washable wools and rayon velvets. I am in the process of making a quilt now that didn't start out to be a crazy quilt, but now has elements of appliqué', crazy quilting, stupendous stitching and anything else you could possibly think of!
Oh this all sounded so familiar to me I just ran down to my quilting room and found MY book: Crazy Shortcut Quilts by Marguerita McManus and Sarah Raffuse. Duh, me!
It says, "If you want to use silk, lawns, laces, sheers and other delicate fabrics, we suggest you learn how to best stabilize that fabric and do some practice sewing... and "Heavy-weigth fabrics like wools, velvets, corduroy, etc. may stretch more than cottons and may also need a lightweight stabilizer to make them manageable during the cutting and piecing stage..."
It also mentions that you should think about the use of your quilt; will it be washed a lot and if so, maybe cottons would be better is the jest of it.
No matter what material you use, I think it will be fabulous and fun. Go for it!