Marisco Castle stands close to the landing side, where the island first feels inhabited. It is compact rather than romantic, but its position makes sense immediately: whoever held this corner controlled the practical way on and off Lundy.
The castle is part of the island's long pattern of private owners, raiders, royal claims, and odd experiments in self-rule. See it early, then walk north with that history in your head.