Just think of crazy quilts: They use all sorts of fabric, but because they can behave unpredictably, they are generally sewn on a foundation like a muslin. No, you do not need to stick to cotton, but if you do, then you know what to expect (how the fabric will behave). You can certainly mix fibers but I would recommend that you prewash everything to take out the shrinkage factor. Prewashing will also show if any fabrics will not hold their shape, burn during pressing, or otherwise misbehave.
Just have fun and see what works for you.
Now if you wanted to join a swap, then cotton is the rule, and there is a swap police.