Go in the cooler months if you can. Summer heat in Kharga is punishing, often over 40°C, and the distances between sites make that more than an inconvenience.
Winter days are much easier for temples, forts, and desert drives, though nights can be cool.