I use wool blanket off cuts.
Tried to practise hand quilting on one of my pot holders yesterday and it was a nightmare as it was so thick. After that I tried to practise hand quilting with just some off cuts of thin batting. Easy! So I guess I would recommend wool blanket off cuts only if you're machine quilting.
The funny thing is that as I've given away every single pot holder I've made so far I can't report back on how they are to use. I might try and fix that by making one for my own use today.