Late spring through October is the season. Crabs are running, the marinas are full, the bar decks are open, and the bay is warm enough to swim by Memorial Day. May and June are arguably the best — soft-shells are in, the weather is reliably warm but not yet humid, and the worst of the summer crowds haven't arrived.
July and August are hot and humid, the way the Chesapeake always is, with afternoon thunderstorms and mosquitoes after dusk. September and October are perfect: fewer boats, cooler nights, oysters back on the menu.
Winter is bleak in the way only a flat tidewater town can be. Most restaurants and shops close from December to March. The wildlife refuge is at its best then for migratory waterfowl, if you can stand the cold.