I feel your pain. We are planning to move next year, so I am having to go through a lot of stuff including fabric. I once gave away all of my whites and blacks, figuring I would never make a white and black quilt, and lived to regret it! So I am careful now.
One thing I have done is purchase clear plastic bins that I like (they have handles that lock the lids in place) so I can actually see what I have. When going through bins, I find there are fabrics I actually don't like anymore; those can go. I have two entire bins of calico, which I no longer really care for (am more into batiks and modern fabrics these days), so I will go through them and get rid of at least half of the calico.
I am getting rid of fabric I don't much care for anymore. I am getting rid of bins of scraps because, over the years, as much as I admire the scrap quilts of other people, I have found that I just don't want to have to deal with all those scraps. I enjoy making quilt tops that are fairly simple and at least partially controlled. I'm sure other quilters will enjoy making something with the scraps; I just feel that I don't have the desire (or the time, or the energy.....).
I have a bin of fat quarters. Discovered the hard way that I don't often use fat quarters in my quilts. I am planning to pull just the pieces that I really like, in the colors I like to use, and will get rid of all those "weirdo" prints and colors I am unlikely to ever want to use.
I think it's fine to keep everything else, especially if it's organized into bins so I can find things. It will be easy for the executor of my estate to donate the bins to worthy causes, since he doesn't quilt! No emotional attachment there. The charities can wait until I don't want it anymore.
My 2 cents worth.....
There were some interesting posts in this thread when I asked about downsizing:
http://www.quiltingboard.com/general...d-t248285.html
Edit: I should add that I'm approaching 70 and aware that I will never be able to use up all the yardage I have. Still, I enjoy having it. Like you, I enjoy buying more too. I've just learned to limit that buying to about 5% of what I used to buy to "build up my stash". I'm much more selective now about what I purchase!