i haven't come up with any system yet that's better than trial and error. there are some brands that i'm consistently pleased with; others i've learned to hate; and some that are great most of the time, with the occassional unpleasant surprise.
and it seems no matter how little or how much i've paid for something, the truth is only revealed or confirmed once it's been washed. (you have noooooooooooo idea how much that ticks me off. that's precious time "wasted" that i'd rather spend actually using the stuff. mumble ... grumble ... frizzlefrazzle :hunf: )
if i can see through them when i hold them up to the light, i pass. if they are already frayed and frazzled while still on the bolt, i sneer snottily and move on. if they feel slippery, as opposed to silky, i know they'll be a pain to work with. they had better be inexpensive (no more than $3.50 per yard) or they go back on the rack.
my favorites are fabrics that have a medium weight, a nice "classy" drape, and a bit of "grab". by that, i mean that it will naturally "stick" to another layer of itself, or to other fabrics. if it does that before it's washed and with the added bonus of a silky feel and a certain indescribable lustre, i'm in love. those will get the highest rating on The Internationally Famous PatriceJ Snooty Scale.
i also like the "workhorse" fabs. they might not be fancy or shopworthy, but i can buy lots of them when i need to; they come out of the dryer looking and feeling like they did before the first wash. they're great for kids quilts because they'll hold up in the wash, but nobody will need to feel badly if they get abused or damaged to too much love.
you guys know me. i could go on and on and on until your eyes roll back into your heads with boredom. it's that complex.
i'm slowly building a list of brands that perform consistently well for me and of those that don't.