Lose Your Mother by Saidiya Hartman. A searing, personal account by the African-American academic of a year spent travelling along the slave routes of Ghana, including extended time in Cape Coast. Hartman is honest about the discomforts of African-American 'roots' tourism and about her own complicated relationship with the slave fortresses. The single best book to read alongside a visit to the castle.
The Door of No Return: The History of Cape Coast Castle and the Atlantic Slave Trade by William St Clair. A detailed but readable history of Cape Coast Castle specifically, drawing on the surviving British East India and Royal African Company records. Particularly strong on the day-to-day administration of the slave trade and the lives of the European traders.
Homegoing by Yaa Gyasi. A novel that opens in Cape Coast Castle in the 18th century and follows two half-sisters and their descendants — one line shipped into American slavery, the other married to a British slaver — across seven generations. A vivid, accessible introduction to the long shadow of the Door of No Return.
Bury the Chains by Adam Hochschild. A narrative history of the British abolitionist movement, with substantial sections on the Cape Coast Castle era. Less specific to Ghana than the others, but the best overview of the British end of the trade.