It partly depends on the size of quilts/blocks you work on the most.
When I do smaller than 8" blocks I tend to use my 8x8 or 9x9 rulers a lot and my 4x12 (I think)
When I am doing larger projects I use my Big Momma which is 8x12 or my 8x24 and my 18x18 the most.
I have plenty of rulers for special purposes but in the end it usually comes down to a few workhorses.