May, June, and September are the best months to visit. The weather is warm enough for kayaking and walking, the castle gardens are at their best, and the crowds are manageable. September in particular has a quiet, almost melancholy quality — the tourists thin out but the weather often holds, and the light on the castle and river in the late afternoon is exceptional.
July and August are peak season. The town is genuinely busy — narrow lanes clogged, restaurant queues at lunch, the castle courtyards full. It is still worth coming, and the summer festival calendar adds life (the Baroque Arts Festival in late July brings period music and theatre to the castle), but adjust your expectations and book everything in advance. Arriving early in the morning and staying overnight makes a significant difference.
Winter is quiet and atmospheric in a different way. The crowds are gone entirely, snow on the castle and rooftops is picturesque, and prices drop. The castle is on reduced hours (some tours suspended) and most kayak operators are closed, but the town itself is worth visiting if you come for the architecture and atmosphere rather than the activities. Christmas brings a modest market.
Avoid Czech public holiday weekends — Easter, the May Day holidays, and national day weekends — when domestic visitors fill the town to capacity.