Abstract

Distilled diffusion models suffer from a critical limitation: reduced samplediversity compared to their base counterparts. In this work, we uncover thatdespite this diversity loss, distilled models retain the fundamental conceptrepresentations of base models. We demonstrate control distillation - wherecontrol mechanisms like Concept Sliders and LoRAs trained on base models can beseamlessly transferred to distilled models and vice-versa, effectivelydistilling control without any retraining. This preservation ofrepresentational structure prompted our investigation into the mechanisms ofdiversity collapse during distillation. To understand how distillation affectsdiversity, we introduce Diffusion Target (DT) Visualization, an analysis anddebugging tool that reveals how models predict final outputs at intermediatesteps. Through DT-Visualization, we identify generation artifacts,inconsistencies, and demonstrate that initial diffusion timestepsdisproportionately determine output diversity, while later steps primarilyrefine details. Based on these insights, we introduce diversity distillation -a hybrid inference approach that strategically employs the base model for onlythe first critical timestep before transitioning to the efficient distilledmodel. Our experiments demonstrate that this simple modification not onlyrestores the diversity capabilities from base to distilled models butsurprisingly exceeds it, while maintaining nearly the computational efficiencyof distilled inference, all without requiring additional training or modelmodifications. Our code and data are available athttps://distillation.baulab.info