If you have significantly more cotton fibers incoming than outgoing, then I guess it does start to become something sort of like hoarding. If you just have a lot of volume flowing through your hands, then you're just productive and busy.
I like to give my fabric collection a reality check every now and then. I force myself to commit to use it or lose it. If I realistically don't want to use it, then it's time to give it to someone who will, sell it, or donate it. Interestingly, some of my favorite quilts have been made when I faced a use it or lose it moment -- somehow that removed the mindblock I had which was keeping me from using it up, waiting to find just the right pattern or the perfect recipient, and just get going and do it.
My guideline for myself is that all my fabric needs to fit within a specific space-- I have a dresser with drawers designated for color ranges, a collection of shoe boxes on top for scrappy bits, and one tote for larger yardages. Anything more coming in translates to something has to move out-- ideally in the form of a quilt.