For a feel of the city before you arrive, Michael Ondaatje's In the Skin of a Lion is the great Toronto novel — the building of the Bloor Street Viaduct and the waterworks told through the immigrant labourers who did the work. Hugh Garner's Cabbagetown is the classic chronicle of that neighbourhood during the Depression, when it was a working-class slum rather than a restored showpiece.
For non-fiction, Robert Fulford's Accidental City traces how Toronto grew into the place it is, and Shawn Micallef's Stroll: Psychogeographic Walking Tours of Toronto is the best companion for exploring on foot, neighbourhood by neighbourhood.