It just happens. See, there are these two fat quarters that you bought, and then you were a bad fabric mommy and you left them alone in the dark.. and the next day you heard the call. There were twins. Then those twins needed more fabric and they had triplets. :D
Trust me, it grows. If you decide that you might want to do a scrap quilt, and you keep your scraps, it grows exponentially.
If I like it - I buy a yard and a half.
If I love it - I buy 2/3 yards (depending on price)
If I can't live without it - I buy as much as I can afford.
If you shop sales, shop remnant bins, shop thrift stores and Goodwill, or online, or EBay, or swaps. If you swap with people you'll have to get fabric to match. :D