Summer (June–August) is the obvious choice, when the town is at its liveliest and the warm, shallow bay is genuinely swimmable. July and August are festival months: the White Lady Days festival (late August full moon), the Old Music Festival, the Tchaikovsky Festival, and the Blues Festival all happen within a few weeks of each other. Accommodation fills up quickly then — book ahead.
May and September offer pleasant weather, smaller crowds, and better prices. The wooden architecture looks especially good in the golden light of autumn. Winter is quiet and grey; the town does not disappear entirely, but many tourist facilities close or reduce hours. The bay sometimes freezes, which is atmospheric in its own way.