The Ann Bryant Art Gallery is a calmer East London stop, set in a Victorian house in Southernwood. Its collection and exhibitions focus on South African art, with enough local character to make it more than a bad-weather fallback.
Pair it with the East London Museum if you want a town-based morning before going to the beach. The building itself also helps, adding a little old-city texture to a place better known for surf and port industry.