The Aylmer and District Museum occupies the old town hall on Talbot Street West and does a solid job of telling the story of the surrounding agricultural district — the tobacco and dairy farms, the early settlers, the Indigenous peoples who lived in the region before European contact. The displays are locally focused and genuinely informative rather than generically "heritage."
The building itself has been carefully preserved and is arguably the most architecturally distinguished structure in town. If you are passing through Aylmer, a half-hour here is worthwhile.