Lishui City Museum is the best indoor stop in town, especially if you are using the city as a base for the surrounding counties. Its exhibitions cover local history, natural life, and the crafts most associated with the area: Longquan celadon, sword-making, and Qingtian stone carving.
The museum sits near the riverfront park, so it pairs well with an easy walk along the water. English interpretation can be limited, but the objects are strong enough to make the visit worthwhile.