Goodness, no, you're not the only one. I organize mine, then I think of something I need. I go back, pull it out and if I don't use it all it's not put back where it came from so often. I didn't use to be this way, but I guess as I get older, I either get lazier or something.
So don't feel you are the only one.