If I understand you correctly, the nut is working itself loose from the screw.
If that's the case wrap some plumber's tape around the threads of the screw and replace the nut. That should help tighten it up so the nut stays put until you want it to move. Good luck!