The castle is open year-round and summer is the most comfortable time to explore the ruins. Yorkshire weather being Yorkshire weather, rain is possible at any time of year — pack accordingly.
The market is at its best on Wednesdays and Saturdays, when the stalls are up and the town centre is at its busiest. If your main purpose is the castle, the day of the week matters less.
Pontefract Racecourse is one of England's oldest flat-racing venues, with racing running from May through October. Race days bring crowds to the town; if you are not there for the racing, it is worth checking the fixture list before you visit so you know what to expect.