I have small pieces of fine sandpaper stuck to the back of my ruler with rubber cement. You can peel it off if it doesn’t work for you.
I use Scotch removable tape (blue label) a Lot to hold wiggly things in place. It comes up more easily than painter tape, plus you can see down through it. It removes little to no lint from the fabric. Maybe you can tape the end of your fabric to the mat. You can reuse it a few times, but not nearly so many as painter tape.
I use a weight on the ruler, as QuiltE suggested, sometimes more than one.
There is an omnigrid ruler, very long, that has a lip on the end to hook over the end of your mat or the edge of the table. You could put the weight on it.
One thing that has Really Helped me in trimming up blocks is to put the block on a slightly larger scrap of batting. This cuts down on the wiggling and rock&roll from the block seams. There are special rulers for that issue (Bloc-loc?) but they are pricey, and most of us have batting scraps on hand.
My challenges are visual, but these things have really helped. Hoping some of this might help you.
hugs,
charlotte