Nielsen Park is a harbour reserve in Vaucluse, with shaded picnic lawns, bushland, and a small protected swimming beach. It has been a public reserve since 1911 and still feels more local than the big surf beaches.
The best reason to come is the Hermitage Foreshore Walk, which runs along the western edge of Vaucluse with repeated views across Sydney Harbour. It is an easy way to pair a swim with a scenic walk.