I use both, but only because they're the same brand and the lines match up. But I use the mat lines for inches, I can't get the hang of half and quarter inches on the mat for some reason, but I then line up my ruler lines with the mat lines. The only time I need to be careful is when I need to make 1/8 inch cuts because that ruler is a different brand and doesn't match up right. Lucky for me, I hardly ever make 1/8 inch cuts!
But personally, I think it's consistency that's the best. If you use the same method throughout the project, you won't have a problem