If you're after soft and cuddly, do not use Warm and Natural, it quilts up rather stiffly.
Wool is fantastic, very breathable and has a gorgeous drape! I love Quilters Dream Wool, it's very soft.
Pair it with a nice flannel back and it just doesn't get any softer. I would avoid fleece simply because it's poly and won't breathe.
You can get flannel in solid colors, which would look out of place with a more elegant top.