I have more difficulty going backwards than side to side or forward.
I have noticed that it is because I can't see where I am going, causing me to move the quilt with a jerkier and more inconsistent motion...
Maybe try again, and see if it feels less comfortable for you too?
I have moved around my machine to get around this. Leave the quilt in the machine, needle down. Move to the side of the machine and continue so you are now sewing to the side instead of going back wards, if that makes sense?