The ruined walls at the very top of the village belong not to one castle but to three — the medieval fortresses of the Lapopie, Gourdon, and Cardaillac families, who shared the cliff-top among them in something like an early condominium arrangement. Each held one corner; together they made the village more or less impregnable from above.
The castles were dismantled on Louis XI's orders in the 15th century after the families had picked the wrong side one too many times. What remains is a scramble of low walls and one stub of square tower, but the climb up is worth it for the view: directly down to the river, across to the cliffs opposite, and out over the rooftops of the village. The walk takes ten minutes from the church and is free at any hour.