Usk Castle was founded in the early Norman period and still occupies the best piece of ground in town. The remains are fragmentary rather than grand, but the towers, walls, gardens, and view over the roofs make it the natural first stop.
Check opening days before you climb up, as the castle is privately owned and visitor access is not daily year-round. When it is open, it gives Usk the historical weight that many small market towns lack.