I've never used just cotton batting in my pot holders. I've always used my OLD ironing board covers, because of all the heat. I always make my top and bottom different so I know what side to place the ironing board cover face down on.
Good luck, I'm sure how ever you do it you will be happy.