Moby-Dick by Herman Melville (1851) is the unavoidable starting point. The opening chapters set in New Bedford are among the finest in American literature, and understanding them before arriving changes what you see when you walk the streets of the historic district. The Seamen's Bethel, Johnny Cake Hill, and the character of the city are all there in the text.
In the Heart of the Sea by Nathaniel Philbrick (2000) tells the story of the whaleship Essex, sunk by a sperm whale in 1820 — the real-world event that inspired Moby-Dick. Philbrick is a superb narrative historian and the book is essential reading for understanding what whaling actually involved, and what the men who did it experienced.
The City That God Forgot by Tracy Kidder is an older work of journalism set in New Bedford that captures the city's post-whaling decline and the lives of its working-class communities. Harder to find but worth tracking down for anyone interested in the city beyond its 19th-century monument status.