HuggingFace Daily Papers（社区热门论文）· HuggingFace Daily Papers（社区热门论文）·· 5 天前AI 评分43
为 Looped Transformers 解码：LoopCD 免训练对比解码实现近乎免费的提升
AI 导读
研究者提出免训练对比解码框架 LoopCD，通过对比最终预测与较早循环轮的中间表示来引导 token 选择，分为 logit 空间的 LoopCD-Logits 和零输出开销的 LoopCD-Hidden 两种形式。
正文
Abstract:Looped Transformers achieve parameter efficiency by repeatedly executing a shared block across recurrent loops. Each loop yields an intermediate representation decodable for the same next token, yet standard decoding discards earlier states. Because earlier loops embody less computation, recurrence inherently supplies aligned weak-and-strong prediction pairs without auxiliary models or external training. We introduce LoopCD, a training-free contrastive decoding framework that guides token selection by contrasting the final prediction with an earlier recurrent pass, operating either in logit space with one extra output pass (LoopCD-Logits) or in hidden-state space with zero output overhead (LoopCD-Hidden). Across four looped Transformer families, LoopCD delivers substantial, consistent gains at full recurrent depth: LoopCD-Logits raises Ouro-2.6B-Thinking's AIME 2024 pass@1 from 61.88% to 73.33%, while LoopCD-Hidden lifts Huginn's HumanEval pass@1 from 22.56% to 31.71%. Crucially, these performance gains enable halving the number of recurrent loops while still matching or exceeding full-depth unguided baselines, reducing forward FLOPs by 22.5% to 48.2%. By transforming intermediate recurrent states into effective guidance signals, LoopCD achieves superior decoding quality while substantially reducing inference compute.
| Comments: | 32 pages, 19 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.02185 [cs.LG] |
| (or arXiv:2610.02185v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02185 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ruixiang Zhang [view email]
[v1]
Thu, 1 Oct 2026 17:58:38 UTC (3,635 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org