Yokohama — Westerner's Paradise by Hamish Todd — a readable history of the foreign settlement era and the city's transformation from fishing village to international port. Good background for exploring the Yamate district.
The Makioka Sisters by Junichiro Tanizaki — not set in Yokohama itself, but this epic novel of pre-war Japanese bourgeois life captures the Kansai merchant culture that was closely linked to the port trade of the era. A window into the world the port city was part of.
The Open City: Yokohama's Foreign Settlement — an illustrated academic study of the Kannai district's architecture and the foreign concession that shaped modern Yokohama. Useful for anyone interested in the Meiji-era Western buildings that survive in the Yamate area.