HuggingFace Daily Papers（社区热门论文）· HuggingFace Daily Papers（社区热门论文）·· 8 天前AI 评分40
TaH2：用自适应循环 Transformer 改进测试时扩展
AI 导读
针对循环 Transformer 在测试时扩展上的不足，研究者提出 TaH2，通过前瞻深度监督联合后训练主干网络与迭代决策器，让额外迭代只作用于真正受益的 token。在 AIME 基准上，TaH2 将准确率-算力斜率提升 53%（2.74 对 1.79），同等测试算力下峰值准确率超出非循环基线约 3.4 分；迭代深度从 2 增至 8 时，其增益从 +2.8 分扩大到 +3.9 分。代码已开源。
正文
Abstract:Looped transformers have demonstrated promising parameter efficiency by reusing layers for latent computation. Prior studies compare looped and non-looped models at matched parameters or per-token FLOPs. However, to the best of our knowledge, whether looping improves test-time scaling as outputs grow longer remains underexplored. Through post-training looped transformers, we study the accuracy-compute slope, measured as the accuracy gain per doubling of test-time decoding FLOPs. We find that existing looped transformers often yield steeper slopes than their non-looped baseline, yet underperform it at matched compute. While fixed-depth looping spends extra iterations on every token, our analysis shows that many tokens do not benefit from extra iterations. We therefore propose TaH2, which enables the model to focus extra iterations on the tokens that benefit from looping. It jointly post-trains the backbone and an iteration decider through lookahead depth supervision, which uses online labels indicating whether further iteration improves the prediction. TaH2 improves both the efficiency and attainable accuracy of test-time scaling. On challenging AIME benchmarks, TaH2 improves the accuracy-compute slope by 53% (2.74 vs. 1.79) over the non-looped baseline, exceeding the baseline's peak accuracy by about 3.4 points at matched test-time compute. As the maximum iteration depth increases, existing looped models largely plateau, while TaH2's gain over the non-looped baseline continues to grow from +2.8 points at depth 2 to +3.9 points at depth 8. Our code is available at this https URL.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| MSC classes: | 68T50, 68T07, 68T05 |
| ACM classes: | I.2.7; I.2.6; I.2.8 |
| Cite as: | arXiv:2609.35748 [cs.CL] |
| (or arXiv:2609.35748v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.35748 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yichen You [view email]
[v1]
Mon, 28 Sep 2026 17:56:55 UTC (847 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org