The Gero Onsen Museum is a small stop, but it is useful if you want to understand what you are soaking in. Exhibits cover the geology of hot springs, the qualities of Gero's water, bathing customs, and the history of the town's onsen culture.
It pairs well with a public bath or foot bath nearby. Visit before bathing if you like context, after bathing if you like warm indoor explanations.