Are you sewing around outside curves? If so, you must stop with the needle down in the background fabric in the straight part of the stitch before adjusting the fabric. If you adjust when the little zig is in the patch it will open up the buttonhole stitch.