I have several neutrals and will hold each in turn, next to the group of fabrics. Whichever I notice the least, is the winner.
My 'neutrals' are light grey, dark grey, off white and putty (a sort of light tan). I also have a sort of greyish blue and a sort of greyish green.