Hot water and laundry detergent does wonders!! The towels are far nicer when I use hot, than cold or lukewarm! (sorry, no direct experience with a "gi")
Once you get it figured out, then perhaps you'll have to "teach" DH to unpack it right after each workout.
On the other hand ... if it is not bothering your DH, should you be letting it bother you?