White's Castle stands at the bridge over the River Barrow, giving Athy a medieval marker right in the centre. It was built in the early 15th century and later rebuilt, damaged, and reused as a prison.
The castle is more of a landmark than a long visit, but it makes sense of the town's river crossing. See it before walking the towpath or heading into the Shackleton Museum nearby.