I agree with pennyswings. I know I persue quilting because of the love I have of working with fabrics. I have two hobbies. The scrapbook side of my craft room is filled with hundreds of dollars of free product that was given to me by a family member who works for a manufacturer. The embellishments sit, unopened, for the most part because scrapbooking doesn't give me the same satisfaction that quilting does.
My quilting side of my craft room is much more modest. There, things have been gathered a little at a time, over the period of many years. But working with the fabric brings me peace. I can spend hours with my sewing machine, and find myself cursing the night when I have to stop. If quilting brings you joy, you'll find a way to stretch the budget to make the hobby work for you. Good luck!