The Bishop's Palace of Castres is the most important architectural set piece in the town: a 17th-century building designed by Jules Hardouin-Mansart, the architect of Versailles, and built between 1666 and 1670. It served as the bishop's residence until the diocese was abolished after the French Revolution; today it houses the town hall and the Musée Goya.
The formal garden behind the palace was laid out by André Le Nôtre, again the same designer as the gardens of Versailles, and is one of the few documented Le Nôtre gardens to survive outside the Île-de-France. It is small but immaculate — clipped box parterres, low yew hedges, a central walk on the axis of the palace — and the symmetry and proportion are textbook of the master at his most disciplined. Admission to the garden is free during daylight hours.