I gave up the expectation of a perfectly finished quilt long ago BUT I do still strive for it. I do my best at every step of the process and, as a rule, I do trim my blocks (or partial blocks depending on the design). I find it increases the odds that I'm kinda/sorta close to my desired block/quilt measurement when finished. Regardless, after piecing the blocks together, sandwiching the layers, quilting and binding - I often do not end up with a perfectly square quilt. And I am perfectly fine with that, too.
To answer your actual question - I'm good with up to 1/8" fudge factor but any more then the block comes apart and I try to figure out what can be done to better achieve my desired finished size. Might I add that it's usually a BH Mystery quilt with all those tiny pieces that I'm wrestling with to finish well. Love her mystery quilts for that reason - I've learned so much from her instruction and having to repeat blocks 100's of times.