For the last 20 years, my stone fruit guy (peaches, plums, apricots) at the farmer's market also says to never refrigerate them.
My tomatoes are rarely fully ripe when I buy them, usually they sit on a plate in my cupboard until used. Keeping them enclosed helps them ripen better than just being on the counter. You can also put them in a paper bag to help keep the ripening gasses in.