The big constraint on Åndalsnes is the Trollstigen pass. It opens in late May or early June once the upper switchbacks have been cleared of snow, and it closes again in October — sometimes earlier in a bad year. If the road is closed you lose the Golden Route to Geiranger and one of the best reasons to be here.
June to early September is the safe window. Long daylight, the trains run their summer service, the gondola is operating, the boat tours run on the fjord, and the ridges are walkable. July brings the most tourists and the highest hotel prices; June and early September are quieter and the light is arguably better. Outside the summer season the town shrinks back to its everyday life — quiet, with most of the visitor infrastructure shut, but the Rauma Line train still runs and winter ski touring on the surrounding peaks is excellent if you know what you are doing.
Bring layers and rain gear in any season. The Romsdalsfjord generates its own weather and a sunny morning at sea level can be a cloud-bound afternoon on the ridge.