The ruins of a small Norman castle sit on top of Castle Hill, the rocky headland that separates North Beach from South Beach. There is not much castle left — a section of curtain wall, the lower part of the gatehouse, and a watchtower — but the view from the top is the best in town. You can see along the coast in both directions, across to Caldey Island, and back over the rooftops of the walled town.
A statue of Prince Albert stands on the seaward side of the hill, a Victorian addition that locals either love or roll their eyes at. It is a short walk up from the harbour or the seafront; the best time to go is for sunrise or sunset.