The Museum of Viljandi is compact, local, and useful before or after the castle hills. Its displays cover the town and county from archaeology through the middle of the 20th century, with enough context to make the streets outside read differently.
The building helps: the museum is housed in a former pharmacy in the old centre, so it fits naturally into a walk through town rather than feeling like a separate outing.