The town centre is small and entirely walkable; you can reach the harbour, the High Street, the Seabird Centre, and the Law on foot in under 15 minutes from the railway station.
For the surrounding villages — Dirleton, Gullane, Yellowcraig, Tantallon — local buses (East Coast Buses 124 in particular) run along the coast road every half-hour or so. Cycling along the John Muir Way is a good option in summer; the route is signposted and runs from the town toward Tantallon and Dunbar in one direction, Aberlady and beyond in the other.
For more remote sights (the inland villages of East Lothian, Lammermuir Hills) a car is easier than the bus.