The only remotely realistic reason to avoid white would be because whites (and blacks) are hard to match up. If you put a dozen whites next to each other, there will be subtle differences. That is one of the reasons you need to be careful with home dyeing - plain white and optical whites take dyes differently.
Think of all those Baltimore Album quilts done on white backgrounds. By any chance, did the people who told you this work on commission? Plain fabrics are generally cheaper than high-priced blenders and such.