Unless your white fabric is a light weight cotton, I don't think you have a problem. You've already noted that your white is soft white, so I think your neutral wool batt would not create a problem. Adding an extra layer as you have suggested may create worse problems, if the fabric should bunch up in the layers.
Here's my experience ... I used it with my embroidered snowman quilt ... the blocks are a pure white with variegated blue embroidery. To me, there is no shadowing or colour distortion and the white still reads as totally white.
If you want to check things ... you could always do a trial sandwich with the same fabrics as in your quilt.