Kimiidera is a hillside Buddhist temple south of the centre, reached by a steep approach and a long flight of steps. The climb is part of the visit: at the top you get temple halls, a large Kannon image, and views back across Wakayama Bay.
It is especially good in cherry blossom season, but it works year-round as a quiet counterpoint to the castle. Pair it with Wakanoura if you are spending the better part of a day on the south side of the city.