Uplistsikhe — the name means "lord's fortress" in Georgian — is one of the most remarkable ancient sites in the South Caucasus. Cut into a sandstone cliff above the Mtkvari (Kura) River about 15 km east of Gori, it was inhabited from around the first millennium BC through to the medieval period and contains hundreds of caves and carved chambers representing the accumulated work of many centuries.
The site is substantial and surprisingly coherent despite its age. You can walk through carved streets, enter large ceremonial halls with elaborate ceiling decorations, see the remains of pagan temples that were later adapted as Christian churches, and explore the network of tunnels that allowed defenders to move through the cliff. The main ceremonial hall — sometimes called the Queen Tamar's Hall — has a remarkable carved ceiling and an antechamber that gives directly onto the cliff face above the river.
The small on-site museum has good contextual information. The site is open daily and tickets are inexpensive. Getting there from Gori: shared taxis (marshrutka) run from the Gori bus station, or hire a taxi — about 30 minutes each way. The walk down from the main car park to the cave entrance is easy; the walk through the site involves some scrambling on uneven surfaces. Uplistsikhe makes an excellent complement to the Stalin Museum and fortress — all three together fill a full day from Tbilisi.