The Civilization of Angkor by Charles Higham is the most readable single-volume account of the Khmer civilisation and its temples. Higham is an archaeologist who has spent his career in mainland Southeast Asia and writes with authority and clarity — good preparation before arriving, and good context for what you see once you are there.
River of Time by Jon Swain is a memoir by a British journalist covering Cambodia and Vietnam during the wars of the 1970s, including the fall of Phnom Penh. Swain's account of Cambodia in its darkest period makes the subsequent recovery and the country's endurance as a destination both comprehensible and moving.
First They Killed My Father by Loung Ung is a memoir of survival during the Khmer Rouge years, told from the perspective of a child who lost most of her family. It is not about Siem Reap specifically but it is the essential text for understanding the catastrophe that Cambodia lived through and why the country is the way it is today.
The Gate by François Bizot is a deeply personal account of a French ethnologist who was captured by the Khmer Rouge in 1971 and survived through an unlikely relationship with his captor, the infamous Duch. Bizot had spent years studying Cambodian Buddhism and the book illuminates both the civilisation and its near-destruction.