Yep the rising cost of fabric is due to a very very large number of factors, the drought (which is affecting many parts of the world, IIRC, not just the US) being just one factor.
I personally don't much care about organic fabric vs. regular fabric, although I am perfectly willing to spend the extra money on organic foods, especially certain foods. For fabric, the look and feel of the fabric is more important to me, especially since someone already mentioned- I'm not washing the stuff in organic detergent!