The Lillesand Town and Maritime Museum is the best place to put the pretty harbour into context. Its collections cover shipping, local trade, domestic life, and the coastal culture that shaped the town.
It is a modest museum, which suits Lillesand. Visit early in the day, then walk the streets and harbour with a better sense of why these small white towns mattered along the south coast.