Well yeah, it matters, and it can be different for each ruler depending on brand and age (years of cutting shaves the edges).
Measure with a fine line ruler or a piece of graph paper so you know what's accurate for the ruler you are using...and then use only that ruler for that particular project. Most of mine are accurate to the inside of the line, not the outside and there's almost a mm difference between the two.