The coastline around Jelsa is studded with rocky coves — pebble and stone rather than sand, as is standard on Hvar — that offer clear, calm swimming with good visibility to the bottom. The nearest and most popular are a short walk or five-minute drive from the harbour: Mina beach to the east, and a handful of smaller inlets to the west that you can reach on foot along the coastal path.
For anything more secluded, rent a small motorboat from the harbour (no licence needed for small engines, rates by the day or half-day) and explore the coastline under your own steam. The coves on the south side of Hvar, facing open sea, are wilder and harder to reach but emptier in summer. Water temperature reaches a very comfortable 25°C in August.