Is your monofilament thread nylon? The way I have secured it is with the tiny stitches. In terms of stitch size setting on my machine, that doesn't matter as much as the thickness of what you are working with. I recommend taking a sample of the layers you are working with and experimenting. When I do applique with monofilament thread, all it takes for me is about 5 tiny stitches at the 2mm to 3mm setting on my machine. When quilting through batting, though, the setting needs to be longer in order to get the thicker stuff to feed far enough to make a tiny stitch (instead of just stitching in place).
There are "invisible" threads made out of polyester rather than nylon. I haven't worked much with those, so not sure if they handle exactly the same.