I find I tend to feel very 'claustrophobic' when there is lots of clutter around. I don't have that issue with fabric 'stash' per se as I've only ever bought for projects at the time I start them. I just don't have room to have a 'stash'. Plus, early in my quilting 'career' I read here and other sites about many people either feeling overwhelmed by their stash or many fabrics just no longer appealing to them. Since I tend to change my mind frequently about preferred colors; furniture arrangements; etc., concluded having no stash was the answer for me.
As zozee says, 'things' are not human and, thus, have no feelings. Purge guiltlessly. Don't bother so much with the reorganization until you think you are done purging. Just put things away as you can. Just because you put the seam ripper in 'that' drawer today does not mean it needs to stay there after tomorrow.
Good luck.