Portrait of the Artist as a Young Dog by Dylan Thomas. The closest thing Thomas wrote to a Swansea memoir — ten linked short stories of childhood and adolescence in the city. Read this and the city centre, the Uplands and the bay start to make sense in a different way.
Under Milk Wood by Dylan Thomas. Set in a fictional Welsh coastal village rather than Swansea itself, but inseparable from the cadences of South Wales speech you will hear all around you.
Eye of the Beholder by Stevie Davies. A novel of grief and second chances set partly in Swansea and the Gower, by a writer who lives in the city and teaches at the university.
The Story of Wales by Jon Gower. Not a Swansea book as such, but the best single-volume narrative history of Wales, by a Welsh-language poet and BBC broadcaster — useful background for any trip to the country.