Skutulsfjörður is the westernmost branch of Ísafjarðardjúp, the long fjord system that defines this part of the Westfjords. The town of Isafjordur sits at its head on a low spit of sand, hemmed in by mountains rising 600–700 metres on either side. The fjord stays ice-free year round, which is the historic reason for the town's existence.
The road around the fjord on the southern side gives the best views back to the town — particularly at sunset in summer, when the cliffs across the water glow pink for hours. In winter, the surrounding peaks accumulate enough snow to support a small ski area at Tungudalur, 5 km from the town centre.