You need to start thinking like a vitner. Fabrics, like fine wine, need to age awhile to get better. So at least once a week you need to stroke them, admire them and talk to them lovingly telling them how wonderful they are going to look once you decide how they are going to play together.
You're not a collector - you're an artist who needs time to imagine your next piece of art. Give up the guilt and feel free to dream.