Saratoga Longshot by Stephen Dobyns (1976) — the first in Dobyns's long-running series about Charlie Bradshaw, a private detective in Saratoga Springs. The novels are rooted in the race track milieu and the particular social atmosphere of the city during the August meet. Good, low-key entertainment that will make you feel like you know the place before you arrive.
The Great Bets by John Eisenberg — covers the history of American thoroughbred racing, with several chapters devoted to Saratoga and the famous upsets the track has witnessed. Better on history than atmosphere, but authoritative.
Yaddo: Making American Culture by Micki McGee (2008) — a history of the Yaddo artists' colony and its place in 20th-century American literary and artistic life. Packed with names and anecdotes. A good read if you want to understand why the retreat produced so many important American writers.