Haibara House is another preserved samurai residence in the old castle town. It served guards and retainers rather than high lords, so the appeal is in the scale and detail: tatami rooms, timber framing, and the practical layout of a working household.
See it with Orii House if you have time after the castle and Raikyu-ji. The pair gives Takahashi a fuller shape than a one-sight castle stop.