Holi and Janmashtami are the most intense times to visit, with large crowds, colour, processions, and heavy pressure on transport and rooms. They are memorable, but only if you are ready for the crush.
For calmer temple visits, come outside the biggest festival weeks and start early in the morning. Summer heat is harsh across the Yamuna plain; winter and the cooler shoulder months are easier for walking between temples.