HuggingFace Daily Papers（社区热门论文）· HuggingFace Daily Papers（社区热门论文）·· 8 天前AI 评分34
FRAC：用分数阶状态空间转移实现长序列建模的选择性 SSM 架构
AI 导读
研究者提出选择性 SSM 架构 FRAC，用分数阶动力学将传统 SSM 的指数遗忘替换为幂律长记忆，并通过有限状态、对数间隔的指数模式求和近似重尾目标核，使其成为可并行训练与预填充、同时保留有界状态自回归解码的高效循环模块。在包括 1.3B 参数语言建模在内的实验中，FRAC 的长上下文表现持续优于当前最优 SSM 基线，短上下文性能也保持竞争力。
正文
Abstract:State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance. Most modern SSMs rely on ODE-based dynamics that lead to exponential forgetting, limiting their ability to retain information over broad temporal ranges. We introduce FRAC, a selective SSM architecture derived from fractional dynamics that replaces this exponential decay with power-law long memory. To make fractional dynamics practical, FRAC approximates the heavy-tailed target kernel with a finite-state, log-spaced sum of exponential modes. This construction turns fractional memory into an efficient recurrent module with parallel training and prefill, while retaining bounded-state autoregressive decoding. Extensive experiments, including 1.3B-parameter language modeling, demonstrate that FRAC consistently improves long-context performance over state-of-the-art SSM baselines while staying competitive on short-context. These results show that fractional dynamics provide a practical and effective prior for long-context SSMs.
| Comments: | NeurIPS 2026 (Oral) |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.36314 [cs.LG] |
| (or arXiv:2609.36314v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.36314 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ivan Kobyzev [view email]
[v1]
Mon, 28 Sep 2026 21:48:14 UTC (117 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org