Not all domestic sewing machines will take a size 18 needle. What is the brand and model of your machine? Which threads have you tried? What foot are you using?
I would stick with a size 16 needle (topstitch is best, but a regular sharp needle will work too; not ballpoint).
My guess is that the tension settings may be wrong for FMQ. Although this Youtube video has advice for longarmers, it works well for domestic machines also:
http://www.youtube.com/watch?v=Q1mRhcquZTM
Basically it involves reducing bobbin thread as much as possible, then adjusting top tension until you get a balanced stitch. If you start out with bobbin tension too high, top tension will also need to be very high to get a balanced stitch -- and this tight tension results in fraying of the thread.