MiniMax Sparse Attention（MSA）块状稀疏注意力
MiniMax 提出块状稀疏注意力 MSA，基于 GQA 构建。轻量级 Index Branch 为每个 GQA 组独立选择 Top‑k KV 块，Main Branch 仅对选中块执行精确块稀疏注意力。在 109B 参数多模态模型上，MSA 与 GQA 性能持平，1M 上下文下每 token 注意力计算减少 28.4 倍。配合协同设计的 GPU 内核，H800 上实现 14.2 倍 prefill 和 7.6 倍 decoding 端到端加速。推理内核与基于 MSA 的多模态模型已公开发布。
MiniMax这个稀疏注意力把长上下文推理计算砍掉28倍，而且直接开源了高效CUDA kernel和模型，做agent和代码仓库级推理的团队可以赶紧试试。
Authors:Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Jinkai Hu, Jiayao Li, Rui Gao, Zekun Li, Songquan Zhu, Jingkai Zhou, Pengyu Zhao
Abstract:Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale. We introduce MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA). A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group, enabling group-specific sparse retrieval while maintaining efficient block-level execution; the Main Branch then performs exact block-sparse attention over only the selected blocks. Designed around a principle of simplicity and scalability, MSA is deliberately streamlined, making it straightforward to deploy efficiently across a broad range of GPUs. To translate sparsity into practical speedups, we co-design MSA with a GPU execution path that uses exp-free Top-k selection and KV-outer sparse attention to improve tensor-core utilization under block-granular access. On a 109B-parameter model with native multimodal training, MSA performs on par with GQA while reducing per-token attention compute by 28.4x at 1M context. Paired with our co-designed kernel, MSA achieves 14.2x prefill and 7.6x decoding wall-clock speedups on H800. Our inference kernel is available at: this https URL. A production-grade natively multimodal model powered by MSA has been publicly released at: this https URL.
| Comments: | 30 pages, 14 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.13392 [cs.AI] |
| (or arXiv:2606.13392v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.13392 arXiv-issued DOI via DataCite |
Submission history
From: Xunhao Lai [view email]
[v1]
Thu, 11 Jun 2026 14:23:41 UTC (3,976 KB)
[v2]
Fri, 12 Jun 2026 09:42:25 UTC (3,976 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org