No, it's not just you. If I'm doing straight line quilting using a walking foot it always takes longer than FMQ because of having to reposition or at times turn the quilt at the end of each line plus just having to sew slower so the walking foot can do its job properly. Most of the time if I want to do straight line quilting I FMQ with rulers so I can avoid dealing with those issues.
Susan Cleveland has a neat trick of just sewing in reverse instead of turning the quilt for nonFMQ straight line quilting. But it requires a machine that can be set to keep sewing in reverse rather than one where you have to hold a lever or button down with one hand while reverse stitching, otherwise it's hard to manage anything other than a very small quilt.
Rob