IndustryBench：探究大语言模型的工业知识边界
研究团队发布IndustryBench，这是一个基于中国国家标准（GB/T）和工业产品记录构建的2049项中文工业采购问答基准，并提供了多语言对齐版本。构建中，基于外部搜索的验证环节拒绝了70.3%的大语言模型生成问题，凸显了仅靠模型过滤的不可靠性。对多语言模型的评估发现：最佳系统得分（0-3分制）仅为2.083分，提升空间巨大；“标准与术语”是普遍能力短板；扩展推理会因引入无依据的安全关键细节而降低多数模型的安全调整分数；安全违规检查会显著改变模型排名。研究表明，工业领域的大语言模型评估需基于源文本、具备安全意识，而非依赖简单的聚合准确率。
工业采购场景下，LLM的准确率远不够用，而且推理模型越想越多反而越不安全，这个基准把幻觉和安全风险摆上了台面。
Authors:Songlin Bai, Xintong Wang, Linlin Yu, Bin Chen, Zhiang Xu, Yuyang Sheng, Changtong Zan, Xiaofeng Zhu, Yizhe Zhang, Jiru Li, Mingze Guo, Ling Zou, Yalong Li, Chengfu Huo, Liang Ding
Abstract:In industrial procurement, an LLM answer is useful only if it survives a standards check: recommended material must match operating condition, every parameter must respect a regulated threshold, and no procedure may contradict a safety clause. Partial correctness can mask safety-critical contradictions that aggregate LLM benchmarks rarely capture. We introduce IndustryBench, a 2,049-item benchmark for industrial procurement QA in Chinese, grounded in Chinese national standards (GB/T) and structured industrial product records, organized by seven capability dimensions, ten industry categories, and panel-derived difficulty tiers, with item-aligned English, Russian, and Vietnamese renderings. Our construction pipeline rejects 70.3% of LLM-generated candidates at a search-based external-verification stage, calibrating how unreliable industrial QA remains after LLM-only filtering. Our evaluation decouples raw correctness, scored by a Qwen3-Max judge validated at $\kappa_w = 0.798$ against a domain expert, from a separate safety-violation (SV) check against source texts. Across 17 models in Chinese and an 8-model intersection over four languages, we find: (i) the best system reaches only 2.083 on the 0--3 rubric, leaving substantial headroom; (ii) Standards & Terminology is the most persistent capability weakness and survives item-aligned translation; (iii) extended reasoning lowers safety-adjusted scores for 12 of 13 models, primarily by introducing unsupported safety-critical details into longer final answers; and (iv) safety-violation rates reshuffle the leaderboard -- GPT-5.4 climbs from rank 6 to rank 3 after SV adjustment, while Kimi-k2.5-1T-A32B drops seven positions. Industrial LLM evaluation therefore requires source-grounded, safety-aware diagnosis rather than aggregate accuracy. We release IndustryBench with all prompts, scoring scripts, and dataset documentation.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.10267 [cs.AI] |
| (or arXiv:2605.10267v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.10267 arXiv-issued DOI via DataCite |
Submission history
From: Songlin Bai [view email]
[v1]
Mon, 11 May 2026 09:30:48 UTC (8,978 KB)
[v2]
Tue, 12 May 2026 12:43:13 UTC (8,978 KB)
[v3]
Wed, 13 May 2026 06:35:15 UTC (8,978 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org