Downtown Newburyport is built for walking — the whole historic core, from High Street down to the boardwalk, fits inside a 20-minute stroll. The Clipper City Rail Trail is a paved 2.5-mile path from downtown out toward March's Hill, useful for bike commuters and pleasant for an evening walk.
You'll want a car for Plum Island, the Parker River refuge, and any day trip. The MBTA Newburyport commuter rail line ends at the city station off Route 1, with an hour-and-ten-minute run into Boston North Station — handy for visitors arriving by air.