The Hayashibara Museum of Art sits near Okayama Castle and holds works connected to the Ikeda family, the former lords of the domain. Expect swords, lacquer, Noh costumes, calligraphy, and ceramics rather than a vast encyclopaedic collection.
It is a good short stop after the castle, especially if you like Japanese decorative arts. Check exhibition schedules, as smaller museums in Japan often rotate displays from storage.