I truly love quilter's cotton, but have had many tell me that they like flannel on the back because it feels "softer".
Of course, all of us know that what you start with isn't always what you end up with. It depends how densely you quilt, what type of thread you use to quilt (or if you hand tie) & what you use for batting. My personal bias is that there is nothing like the feel of quilter's cotton with a higher loft cotton batt (Warm & Plush, for me) that is quilted 1-2" apart. That warm, wrinkly fabric draped around me or my loved ones is so cozy & comforting.
I would really encourage you to see if you can't find some completed quilts with different backings at your LQS or guild to actually feel as a completed product before deciding what you think is the "softest" choice for material. There are many different choices -- felted wool, flannel, minky, traditional cotton ... or even some non-traditional options like fleece, cotton chenille, dupioni silk, or satin. If you ask 10 different people, you may well get 6 different answers; the only opinion that matters is yours.