What the locals call Banff Castle is in fact two things stacked on top of each other: the surviving earthworks of a 12th-century royal castle, and a Georgian mansion built within them in 1750 by John Adam for Lord Deskford. The original medieval fortress was visited by Edward I of England in 1296 and demolished to make way for the present house, but the ditch and rampart are still clearly visible around the perimeter.
The mansion itself is now in community use and not generally open to the public, but the grounds — high on a bluff above the harbour — are accessible and offer one of the better views over Banff Bay and across to Macduff.