I believe the directions from InsulBrite say to use one layer of InsulBrite and two layers of cotton batting. I've made several that way and they do a good job of diffusing the heat.
You can buy the silver stuff that goes on the ironing board on the internet and that's what I used to use before InsulBrite came along. I would use a layer of the silver stuff toward each outside of the pot holder (two layers total) with two layers of batting in between. That way, no matter which way you held the pot holder, you were always protected.
Either way works well.