HuggingFace Daily Papers（社区热门论文）· HuggingFace Daily Papers（社区热门论文）·· 10 天前AI 评分42
DepthBench：衡量残差连接如何提升 Transformer 计算深度
AI 导读
研究者推出 DepthBench，一个在固定模型规模与预训练配方下系统改变宽深比（d_model/n_layer）的受控基准，用于衡量架构深度能否转化为有效计算深度。
正文
Abstract:Depth is a natural way to increase the computational capacity in Transformers, yet the contribution of deeper layers can diminish as depth grows larger. Recent approaches enhance normalization (\text{e.g.}, LayerNorm Scaling) or residual connections (\text{e.g.}, mHC, AttnRes) to enable better information flow and depth utilization. However, it remains unclear whether they truly translate increased architectural depth into effective computational depth, and whether their reported gains stem from better access to information across depth, or unaccounted-for confounding factors. In this paper, we introduce \textbf{DepthBench}, a controlled benchmark for studying computational depth across various architectures. We systematically vary the width--depth aspect ratio ($d_{\text{model}}/n_{\text{layer}}$) from shallow--wide to deep--narrow shapes, while keeping the model size and pre-training recipe fixed. Across 10 representative architectures, we find that the benefit of allocating more capacity to depth is strongly architecture-dependent. Standard Pre-LN and most of its norm- and scaling-based variants provide little benefit and can even degrade performance as models become deeper and narrower, whereas HC and Full AttnRes improve consistently even at extreme deep shapes. These gains extend beyond pre-training loss and consistently translate into improved domain-specific performance and effective computation. Controlled layer-level analyses further show that the gains of HC and Full AttnRes are associated with more effective utilization of additional layers, revealing distinct mechanisms of computational depth across architectures. Overall, our results identify residual connection design as a key determinant of whether depth can serve as a meaningful scaling axis by enabling additional architectural depth to translate into effective computation.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.32534 [cs.CV] |
| (or arXiv:2609.32534v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2609.32534 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Keyu Wang [view email]
[v1]
Sat, 26 Sep 2026 12:15:42 UTC (12,815 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org