The 14th-century fortress at the southwestern end of the walled town was built by Florence in 1353, after San Gimignano submitted to Florentine rule, partly as a symbol of control and partly as a genuine military installation. Most of the structure was demolished by the Florentines themselves in the 16th century, when it became strategically irrelevant. What remains — a single tower and the outer walls — has been converted into a public garden and vineyard, and it's now one of the most pleasant spots in town.
The views from the remaining tower are excellent: you get the full panorama of the surrounding countryside, and a good angle back on the town's tower cluster. Admission is free. It's a ten-minute walk from the main piazza, which means it's quieter than the central sights, and the garden is a good place to escape the midday heat.