The Reykjadalur valley walk is one of the best free outdoor experiences in Iceland. A marked trail starts from the car park at the edge of Hveragerði and climbs gradually through an open valley full of geothermal vents and steaming ground, reaching the hot river in about 3 km (roughly one hour each way at a moderate pace).
The river is geothermally heated to a comfortable bathing temperature in the middle section — changing boards are provided (they are basic wooden platforms, not facilities). The hike itself passes through striking scenery: the valley sides are layered orange and yellow from mineral deposits, and on clear days the view back toward the coast and across the lowlands is excellent. Go on a weekday to avoid crowds, and in winter for the additional drama of steam rising from the valley against snow. Pack a towel, a swimsuit, and decent waterproof footwear — the trail can be muddy.