you should just be walking up and down 7th avenue. there are shops of decorating and quilting and garment fabrics for blocks. and the cheapest of the cheap. give yourself some time though. you'll get lost in all those treasures!
they are all wholesale houses but they sell to 'off-the-street' traffic, too.
i spent my last incarnation as a ny girl and that was where i did all my fabric shopping from age 13 to age 25, when i moved to nj. i want to go home!