Only about 15% of wholesale fabric cost is determined by the price of raw cotton. A lot of the rest of the cost is in petroleum and labor. Labor costs in China, where most of the weaving is done, are up substantially, and likewise in So. Korea and Japan, where much of the fine fabric is printed. Labor costs are also up substantially in Indonesia, where many of the batiks are made. Likewise there are huge increases in the cost of dyes, most of which are made from petroleum products. Finally, there are a lot of transportation costs connected with getting the fabric from the cotton in the field to the finished product on your sewing machine.
Think of all the steps involved and the transport involved, and you'll see why petroleum costs are the largest factor in driving higher fabric prices.