I've never worried about being on the straight of grain either. I just make sure that my fabric is somewhat slack when sewing and I press rather than iron, all to avoid stretching. It works. I do sometimes tear an edge of my fabric if there's no way I can get the selvage edges to meet otherwise and if I can spare the inches.
Quilting is about scraps and scraps aren't perfect.
I learned to piece first by hand, and yes it's so wonderful how precise that can be. Our teacher for my first (and only) quilting class was not as worried about being on grain (just try to be as close as possible), and our quilts were great. I remember her focus on NOT getting too concerned if things weren't perfect, that they will all be beautiful in the end. I take that with me today. And looking at some of the quilts that I hated then because of imperfections, but feel a bit sentimental about now, I definitely think she was right.