Elba's tourist season is unmistakable: June through early September is high season, with August solidly booked weeks in advance and prices peaking. Portoferraio in mid-August is a small town overrun by Italian families on their ferragosto holidays — fun if you like the atmosphere, frustrating if you wanted a quiet beach.
The shoulder seasons are the better times to visit. Mid-April to mid-June has reliable warm weather, the sea warming up by late May, the wildflowers blooming, and the cliff hiking trails (especially around Enfola and Capo Bianco) at their best. September into early October is similarly pleasant — the sea is still warm from summer, the crowds thin out from the first day of school, and the prices drop sharply.
Winter (November to March) is quiet. Many hotels and restaurants close; the ferry runs less often; rain comes in occasional bands. The forts are still walkable, the old town has the air of an out-of-season Italian harbour town, and you can walk the Ghiaie beach alone. The Napoleonic anniversaries in early May (his arrival) and late February (his departure) bring small festivals and re-enactments worth timing a visit around.