Master and Commander by Patrick O'Brian. The first of the great Aubrey–Maturin series, much of it set in or sailing out of Portsmouth in the Napoleonic Wars. Reading it before you visit HMS Victory makes the lower deck feel a great deal more real.
A Study in Scarlet by Arthur Conan Doyle. Doyle wrote his first Sherlock Holmes novel in lodgings in Southsea in 1886, while waiting in vain for medical patients. The Holmes-related collection at Portsmouth Museum is the world's largest.
The Mary Rose: Tudor Ship's Bone by Margaret Rule. The chief archaeologist's account of the recovery of Henry VIII's flagship. Written with academic precision but utterly gripping — it makes a visit to the Mary Rose Museum twice as good.
Pompey by John Sadden. A short, well-illustrated social history of the city — the docks, the dockyard workers, the trams, the Blitz, Portsmouth FC. Particularly good on the period from the 19th century onwards.
Nicholas Nickleby by Charles Dickens. Most of the second half is set in Portsmouth — Vincent Crummles's travelling theatre company perform here. Dickens, born in the city in 1812, drew on memories of his own childhood visits.