I haven't ever done a row robin - just a couple round robins. One didn't go so good. It would be good to know someone's track record in swapping, meeting obligations, etc.
If the colors were not too specific, it could work. Patriotic, Autumn, etc.
Another possibility would to send out color samples to the four who would be adding a row at a price of a stamp.