Fort Saint-Elme stands on a rocky spur above the town, a 16th-century star fort with commanding views over the coast and the mountains behind. It was built by the Spanish Crown during their control of Roussillon as a complement to the Château Royal below — together the two fortifications controlled access to the port from both sea and land. Vauban's modifications in the 1670s after French annexation were minimal, suggesting the original design was considered adequate.
The fort is privately owned and opens for guided visits in summer. The walk up from the town takes about twenty minutes along a winding path; the views from the walls, which look north along the Côte Vermeille and south towards Spain, are among the best on this section of coast. On clear days the Spanish coast is visible.