The first week of October is fair week, and it transforms the town. Hundreds of thousands of people converge for the Fryeburg Fair, which means lodging within 30 miles books up weeks in advance and the roads get genuinely congested. If that is your reason for coming, plan well ahead. If it is not, consider going before or after.
Summer -- late June through August -- is the river season. Water levels are good for paddling, the days are long, and the valley is at its most lush. Fryeburg itself stays quiet relative to the Maine coast; there is no beach tourism here to inflate prices or crowd the roads. September is underrated: the river is still paddleable, the fair crowds have not arrived, and the foliage begins in the hills above town.
Winter brings reliable cold and enough snow for Shawnee Peak to operate most of the season. The town is quiet, lodging is cheap, and the White Mountain resorts across the border in New Hampshire provide alternatives if you want bigger terrain. Spring is mud season -- manageable but not the best time to visit.