You can use any fabric, however, here are some things to watch out for:
- excess fraying - (a lot of the dressy fabrics do) use a larger seam allowance or use a serger to seam and overcast the edges. also consider quilting it by stitching down the seam allowances to prevent further fraying.
- deal with washability issues ahead of time by prewashing all fabrics. this way you get the shrinking done and you can see how the fabric reacts to water and drying (some dressy fabrics look horrible after water hits them!)
Otherwise...go for it.