HuggingFace Daily Papers（社区热门论文）· HuggingFace Daily Papers（社区热门论文）·· 7 天前AI 评分43
RIDE：在表征空间外推 RL 诱导方向，让学生模型逼近或超越教师模型
AI 导读
RIDE（RL-Induced Direction Extrapolation）将强化学习相对基座检查点的表征偏移作为方向，在每一层和每个 token 位置计算教师与其 RL 前检查点的残差，并把学生隐状态回归到沿该残差外推的目标上。
正文
Abstract:On-policy distillation (OPD) trains a student to match the teacher's next-token distributions on the student's own trajectories and has yielded substantial empirical gains. Generalized variants allow the student to surpass the teacher by extrapolating an implicit reward in output space. The language-model head, however, attenuates this change anisotropically: much of the change encoded in the teacher's hidden states reaches the logits at a small fraction of its weight, and the sampled-token log-probability ratios on which output-space extrapolation relies inject noise that the extrapolation amplifies, making training unstable. We observe that reinforcement learning (RL) shifts a model's internal representations relative to its base checkpoint, and that the direction of this shift can be measured at every layer. Motivated by this observation, we propose RIDE (RL-Induced Direction Extrapolation), which extrapolates the RL-induced change directly in representation space: at every layer and token position, RIDE computes the residual between the teacher and its pre-RL checkpoint and regresses the student's hidden states toward targets displaced beyond the teacher along this residual. Conditioned on a sampled trajectory, this regression is equivalent to maximizing a linear directional reward defined by the residual under a quadratic penalty centered at the teacher, which makes explicit how the objective moves the student along the RL-induced direction while limiting its deviation from the teacher. Across four base/RL-teacher pairs spanning different scales, architectures, and pre-training lineages, RIDE approaches or exceeds the RL-trained teacher on every pair and is the only method whose mean does so, and it consistently outperforms output-space extrapolation, which degrades the student whenever the teacher is close to its base. Project page: this https URL.
| Comments: | 19 pages |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.36484 [cs.LG] |
| (or arXiv:2609.36484v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.36484 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hao Li [view email]
[v1]
Tue, 29 Sep 2026 01:44:39 UTC (641 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org