NatureBench：AI编码智能体能否匹配Nature系列论文已发表SOTA？
NatureBench是一个跨学科基准测试，包含90个从Nature系列同行评审论文中提取的任务，用于评估AI编码智能体能否超越复现、实现发现。基准基于NatureGym自动化管线，为每个任务提供标准化容器化环境，解决环境碎片化问题。在严格禁用网络搜索的协议下评估10种前沿智能体配置，最强模型仅在17.8%任务上超过已发表SOTA（g>0.1准则）。分析表明，智能体成功主要依赖方法论翻译，失败主因为方法选择错误和计算预算不足。已发布基准、NatureGym管线及公共排行榜。
这个基准把AI agent丢进Nature论文的复现池里游了一圈，发现最强的配置也只能在17.8%的任务上超越SOTA，而且靠的是方法翻译而非发明——对做科研agent的团队来说，既是冷水也是路线图。
Authors:Yuru Wang, Lejun Cheng, Yuxin Zuo, Sihang Zeng, Bingxiang He, Che Jiang, Junlin Yang, Yuchong Wang, Kaikai Zhao, Weifeng Huang, Kai Tian, Zhenzhao Yuan, Jincheng Zhong, Weizhi Wang, Ning Ding, Bowen Zhou, Kaiyan Zhang
Abstract:We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. Evaluating ten frontier agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 17.8% of tasks under the g>0.1 criterion. Analysis of method pathways reveals that agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding. We release the benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction. Code: this https URL
| Comments: | Add results of GLM-5.2 and MinMax-M3 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.24530 [cs.CL] |
| (or arXiv:2606.24530v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.24530 arXiv-issued DOI via DataCite |
Submission history
From: Kaiyan Zhang [view email]
[v1]
Tue, 23 Jun 2026 12:58:23 UTC (5,509 KB)
[v2]
Mon, 6 Jul 2026 16:56:53 UTC (5,516 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org