The Drostdy Museum is Swellendam's main cultural anchor, set in and around the old magistrate's residence and outbuildings. Whitewashed Cape Dutch architecture, period rooms, and local-history displays give the town's age a physical centre.
It is the best place to start because it explains why Swellendam feels older and more settled than many road towns on the N2. The surrounding streets also make a good short walk for gables, gardens, and mountain views.