Hiroshima by John Hersey (1946) — Not set in Kobe, but the nuclear age it describes ended the war that reshaped this port city. More relevant is the way Hersey's documentary approach illuminated civilian experience of disaster. Kobe was firebombed in 1945; the 1995 earthquake killed more than 6,000 people. Reading Hersey first makes both events feel less abstract.
The Japan I Never Knew by Cathy N. Davidson (1993) — A candid account of a year in Japan by an American professor, including time in the Kansai region. Davidson is honest about cultural friction and the difficulty of understanding Japan from outside, which is refreshing.
Silence by Shusaku Endo (1966) — Set partly in the period of Portuguese missionary activity in Japan, the Namban era that produced the art in the Kobe City Museum. Endo's novel about a Jesuit priest in 17th-century Japan is one of the finest Japanese novels and gives direct context to what you see in Kobe's collections.
The Great Earthquake and Fire of 1906 — Kobe's 1995 earthquake has been written about extensively in Japanese but relatively little in English. Kevin Rafferty's City on the Edge: The Transformation of Kobe (1995) is out of print but worth finding; it covers both the city's commercial history and the earthquake's impact.