The Shelburne Museum, south of Burlington, has a fantastic quilt display and dozens of wonderful historic buildings (including Adirondack "cottages"), in addition to demonstrations of things like blacksmithing. You can even stay in the wonderful estate house, built by a Vanderbilt heiress over 100 years ago. There is a wonderful store outside the Museum complex that sells a wide variety of the old "penny candies."
If you go as far north as Burlington, the Univ. of Vermont campus has some magnificent historic buildings (1824 Lafayette Bldg., actually dedicated by Lafayette on his return trip to the US) and the Student Center, which was designed by Richardson over 100 years ago. It's also only a few miles from the site of the VT Quilt Show.
In Bath, Maine, there is a wonderful museum for shipbuilding; better yet, just off the highway, is Halcyon Yarn. Its catalog is great, but the contents of the store have to be seen to be believed. If you also knit, crochet, or weave, it would take you as long to go through the shop as it would for your husband to tour the shipbuilding museum, and they are only about 6 blocks apart!