S²D-OPD：按 JSD 筛选状态的选择性蒸馏监督

View PDF HTML (experimental)

Abstract:Direct On-Policy Distillation (Direct-OPD) transfers reinforcement-learning-induced policy improvements from a small model to a larger student by using the token-level log-ratio between post-RL and pre-RL checkpoints as dense supervision on the student's own rollouts. This transfer rewards the policy shift at every state, yet the log-ratio measures only relative change: it can stay fixed even as the probability mass that both checkpoints assign to the student's candidate tokens vanishes. Through an exact construction, we show that the Direct-OPD reward and its update can remain unchanged while the Jensen-Shannon divergence (JSD) and both KL directions between the checkpoints vanish with this mass, and we note that a small JSD bounds how much the teacher's behavior changed. Motivated by this analysis, we propose Selective Supervision for Direct-OPD (S$^2$D-OPD), which ranks student-sampled states by their teacher-reference JSD and masks Direct-OPD supervision at low-divergence states, retaining only the top 10% of states per response. Across two teacher pairs and four student models ranging from 1.7B to 8B parameters, S$^2$D-OPD improves held-out accuracy over dense Direct-OPD on AIME and HMMT benchmarks in seven of eight settings and matches it in the eighth, without extra forward passes.

Comments:	19 pages. Yibo Zhao and Zixuan Yang are equal contributors and may list their names in either order on their CVs
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2609.29142 [cs.LG]
	(or arXiv:2609.29142v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2609.29142 arXiv-issued DOI via DataCite

Submission history

From: YiBo Zhao [view email]
[v1] Thu, 24 Sep 2026 07:19:10 UTC (2,568 KB)
[v2] Sat, 26 Sep 2026 07:54:04 UTC (2,324 KB)