I use YLI brand .003 nylon thread, which is finer than fine hair. Have never had a problem with it. Coarser nylon will act more like fishing line.
You need to wind the bobbin slowly when using nylon thread, to avoid stretching the thread as it goes onto the bobbin. You also want to use a metal bobbin, if at all possible, to avoid having the bobbin distort, and you want to underfill the bobbin (3/4th fulls is good).
Did you reduce the upper tension? Nylon stretches, so you need to reduce the upper tension to place less pressure on it as you sew.