Harriet Hargrave has quilted many quilts using nylon monofilament thread. YLI brand is a good one. For nylon, you need a thread that is as fine as hair; the nylon threads sold in stores is usually way too thick.
Nylon may yellow somewhat over the years and may also become brittle; however, in general it holds up pretty well. I have seen no evidence that it cuts into the fabric. You do not want to use nylon monofilament in baby quilts just in case a thread gets pulled out; it is so fine and strong, it can wrap around a baby's toe or finger and cut off circulation.
The experience I have had with polyester invisible thread is that it is not as fine or transparent as nylon. I tried it for machine applique but do not like it as much as nylon for that purpose. Quilting with it may be a different story. The advantages of polyester over nylon are that it won't yellow and won't become brittle over time.