Not knowing what machine/needle plate you have, it is definitely possible (and will happen) that if you are using a zig-zag plate rather than a straight stitch (single hole) one, and you raise the foot with the needle still down, you can cause needle deflection when moving the fabric. Think about it - the needle is held by the needle bar at the top, but there is only the needle plate at the bottom to keep it aligned vertically. If that needle plate is a zig-zag one, you have the width of the slot in the needle plate that the needle can be pulled out of vertical if the fabric is pulled.