On-Policy Self-Distillation without Any Supervision

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for posttraining large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine “self”-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model’s own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a selfconsistency threshold. It then conditions the model’s distribution on the pseudosolution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO.

Introduction. Post-training has emerged as a key driver of advances in the reasoning capabilities of large language models (LLMs), with progress largely propelled by supervised fine-tuning (SFT) [Ye et al., 2025, Wen et al., 2025, Li et al., 2026], knowledge distillation from stronger teachers [Abdin et al., 2025, Lu and Lab, 2025], and reinforcement learning with verifiable rewards (RLVR) [MiniMax et al., 2025, Guo et al., 2025, Team et al., 2025]. Among the many recipes, on-policy distillation bridges SFT and RL by training on the model’s own generations, reducing the train-inference mismatch [Gu et al., 2024, Agarwal et al., 2024] and catastrophic forgetting [Shenfeld et al., 2026b] of teacher-forced SFT, while retaining dense tokenlevel supervision instead of sparse scalar rewards. Subsequent work has continued the effort of OPD along its objective, supervision, and systems dimensions. DistiLLM [Ko et al., 2024] introduces skewed KL, while DistiLLM-2 applies asymmetric objectives to teacher- and student-generated responses [Ko et al., 2025].

Discussion / Conclusion. We showed that the ground-truth solution in on-policy self-distillation can be replaced by the model’s own majority-vote consensus: the agreeing rollout serves as the teacher’s reference, and distillation is applied only to self-inconsistent rollouts. With the OPSD recipe otherwise untouched, this label-free variant outperforms its supervised counterpart on the five-benchmark average, and a grid over two model sizes and both reasoning modes shows the advantage reproduces at both scales in non-thinking mode and a tie in thinking mode, where the base model is already strong enough that little headroom remains. The result suggests that, in the regime of on-policy self-distillation—a base model competent enough to vote well and fallible enough to have headroom—the binding constraint is not access to gold solutions but the machinery for surfacing and correcting the model’s own inconsistencies, a machinery that requires no supervision.