The more layers, the more likely the fabric can shift slightly as you cut. I usually only cut 2 (but sometimes up to 4) layers at a time as I am unable to hold the ruler firmly enough to cut more without shifting.
It depends on what you can manage without getting that slight movement which can end up making a big difference to your block sizes or crooked cuts.