This is the million dollar question.
If I have a pattern, I buy an extra quarter yard more for just in case mistakes, and left overs go into a scrappy quilt.
If I am buying because I like it:
if the fabric is a neutral and on sale, I go for 3-5 yards depending on the sale
if it is something I like and I like the price 2 yards
If it is something I fell in love with and not on sale, I go with as little as a half a yard