St Xavier's, founded in 1923 by Jesuits of the Madurai Mission, is the oldest college in Tirunelveli district and one of the better arts and science colleges in southern Tamil Nadu. The main quadrangle preserves a handful of substantial early 20th-century buildings in a restrained Madras–Gothic style around a wide green lawn, with the college chapel on the north side.
Visitors can wander the grounds during the day (there's usually no formal check). It is not a tourist sight as such, but for anyone interested in the legacy of Jesuit education in India — or anyone wanting a quiet shaded walk in the middle of Palayamkottai — it's worth a 20-minute stop.