What is called the "castle" is really the complex around Saint Wenceslas Cathedral on a low rise above the Morava river — the original 11th-century ducal seat of Moravia, fortified, rebuilt and repurposed many times. The Romanesque palace, the cathedral, and the surrounding cloister are now best visited through the Archdiocesan Museum, which uses the historic spaces to display religious art from across Moravia.
The museum (closed Mondays) won the European Museum of the Year award shortly after its opening in 2007. Highlights include the Romanesque chapel, the wooden ceiling of the Bishop's Hall, and a serious collection of Gothic panel painting. Even if you don't go in, the cathedral square and the small park around it are worth a stroll.