HuggingFace Daily Papers（社区热门论文）· HuggingFace Daily Papers（社区热门论文）·· 7 天前AI 评分42
Where the Model Changes Its Mind：用后见分歧定位实现高效可验证奖励强化学习
AI 导读
研究者提出 Hindsight-Divergence Localization（HDL），利用后见诱导的 token 对数似然变化来选取分支点，只从关键位置生成续写并仅用新生成的后缀更新策略。在数学、代码和智能体任务上，相比同等组规模与训练步数的 GRPO，HDL 生成 token 最多减少 2.5 倍、rollout 墙钟时间提速 1.8 倍，智能体任务性能最高提升 12.5 分。
正文
Abstract:Group-relative methods for reinforcement learning with verifiable rewards (RLVR) learn from differences in rollout outcomes. Independently sampling complete trajectories is costly and does not explicitly explore the decision space at critical positions. Feedback on a completed trajectory can reveal which earlier choices the policy reconsiders, suggesting where to sample alternative continuations. We introduce Hindsight-Divergence Localization (HDL), which uses hindsight-induced changes in token log-likelihoods to select branch points. HDL generates a small number of complete root trajectories and fills each training group with continuations from the selected positions under the original task context. Each continuation reuses its root prefix and contributes policy updates only through its newly generated suffix, reducing generation cost while focusing additional exploration and learning on decisions after branching. Experiments with three models across math, code, and agent tasks show gains in both rollout efficiency and task performance. Compared with GRPO at matched group sizes and training steps, HDL yields up to a 2.5$\times$ reduction in generated tokens and a 1.8$\times$ speedup in rollout wall-clock time. Despite this reduced generation budget, HDL improves performance across all three domains, with gains of up to 12.5 points on agent tasks.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.36864 [cs.LG] |
| (or arXiv:2609.36864v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.36864 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Fanchao Chen [view email]
[v1]
Tue, 29 Sep 2026 07:05:29 UTC (413 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org