I have made a couple of full size quilts and a wall hanging using only silk fabric. I start by ironing on a very thin interfacing that is made for silk. Then I can cut out my pieces and sew them together with no trouble. Even the miniature quilt ( the wall hanging ) which had pieces as small as 1/4" finished squares came out fine.
I wouldn't attempt it without the interfacing though, as it would slide and ravel on you as you worked with it.