Usually, I grab my potholders with the 'top' side (which is usually the more decorative side) next to my hand. Could be I am a little OCD, but that is what I do.
I have just recently finished making casserole carriers for both my DD and DDIL for Christmas, and this bit of information was quite helpful as I sandwiched the fabrics.
I suppose this subject is pretty much like the Ann Landers agrument of which way to place your toilet paper -- rolling off the top or rolling from the back and under the roll.