Jagannath Temple stands near the palace area and is worth a look for its architecture as much as its religious role. The octagonal base and layered forms make it more distinctive than many small city temples.
It is an easy add-on after Ujjayanta Palace. Go respectfully, and expect an active place of worship rather than a museum piece.