Like you, I keep all my flannels together. I keep what few garment fabrics I have separate also. For my regular quilt fabric, I don't pay too much attention to color. I separate by value. Since any fabric can be either a light, medium or dark depending on what fabric is adjacent to it, this helps me select what works best.