Hunger by Knut Hamsun (1890) — the first great Norwegian modernist novel, following a starving writer through the streets of Christiania (Oslo), but Hamsun spent his formative years in Bergen and the book's atmosphere of provincial pride colliding with artistic ambition is pure western Norway. Hamsun won the Nobel Prize in 1920, though his later politics were catastrophic.
Peer Gynt by Henrik Ibsen (1867) — Ibsen wrote his dramatic poem while abroad but the world it conjures is the Norway he knew from his years in Bergen as a theatre director. The trolls, the Button Moulder, and Peer's journey around the world and back to his fjord home is the defining work of Norwegian romantic nationalism.
Sofies verden (Sophie's World) by Jostein Gaarder (1991) — a Bergen schoolgirl begins receiving mysterious letters introducing her to the history of philosophy. Gaarder is a Bergen native and the city's relationship with learning and sceptical inquiry runs through the book. A good introduction to Western philosophy wrapped in a mystery story, and better than it sounds.
The History of Norwegian Literature by Harald Beyer — if you want a proper grounding in the literary culture that produced Ibsen, Bjørnson, and Hamsun, this is the standard English-language reference. Bergen was at the centre of the 19th-century literary revival, and the context helps.