Masters week (the first full week of April) is when Augusta becomes impossible to visit spontaneously. Every hotel room within an hour's drive books out months in advance, prices multiply by four or five times, and the city is consumed by golf. If you're coming for the Masters, plan and book the moment tickets become available. If you're not, avoid that week entirely.
Spring (March, and then late April through May) is otherwise the best time to visit. Temperatures are warm but not punishing, the azaleas are blooming — Augusta is a genuinely beautiful city in flower — and the downtown is at its most pleasant. Autumn (October and November) runs it close: the heat has broken, the crowds are gone, and the light on the Savannah River is exceptional.
Summer is very hot and humid. June through August regularly sees temperatures above 35°C with oppressive humidity. Outdoor sightseeing, including the Riverwalk and the Canal towpath, is uncomfortable in the middle of the day. If you visit in summer, plan outdoor activities for early morning or evening.
Winter is mild by Northern standards — temperatures rarely drop below freezing — but the city is quiet and some attractions have reduced hours.