Dracula by Bram Stoker (1897) — the inevitable starting point. The chapters set in Whitby capture the harbour, the East Cliff, and the storm-wrecked Demeter beneath the abbey with a tourist's eye for the actual place. Stoker stayed in Royal Crescent and lifted local names and folklore straight out of the town directory.
Sylvia's Lovers by Elizabeth Gaskell (1863) — set in a thinly-disguised Whitby ("Monkshaven") during the press-gang years of the Napoleonic Wars. A long, slow, atmospheric novel of whaling town life, deeply researched, and a good antidote to the Dracula image.
The Voyages of Captain James Cook — any decent annotated edition. The colliers Cook learned his trade on were built in Whitby; reading the Pacific journals back in Whitby's old town, with the harbour right outside, makes the local museum more vivid.
Possession by A. S. Byatt (1990) — partly set on the Whitby coast, with key scenes around the cliffs and the abbey. Useful background to the strange Victorian seriousness about jet, mourning, and the sea that still hangs over the town.