Hi, I've also had to move the needle over to get a scant seam although most of the time I use the 3.5. But I haven't had to move it as much as you mention.
When you say it clears the little hole, do you have the APC engaged? Because when I do, I can't change the position of the needle.
Maybe take it to your dealer and have them check it out. There could be a minor adjustment needed.
BTW, are you a member of the Janome 7700 group on Yahoo? If not, it's a great group and there's a lot of help with the machine.