Offa's Dyke Path passes through Hay-on-Wye, making the town a useful stop on one of Britain's classic border walks. You do not need to be walking the whole route to enjoy it; short sections north or south give a quick taste of the Wye Valley and the rising country toward the Black Mountains.
For a spare afternoon, ask locally about current path conditions and choose a simple out-and-back. The weather can change quickly once you leave the shelter of town.