A morning at the Azrou Rock and a wander through the medina is enough to see the town itself. If it is Tuesday, the souk is unmissable; on other days, the Ensemble Artisanal gives a good sample of local weaving.
The real reason most travellers come is the surrounding country: the cedar forests, the Barbary macaques, and the high lakes of Ifrane National Park. Allow at least a full day for that, with a picnic and a hired taxi.