无攻击者的“游戏”：LLM 驱动的搜索在选拔压力下的基准指纹识别

View PDF HTML (experimental)

Abstract:Benchmarks for systems that are optimized against the evaluation signal measure something different from what they claim. We document this concretely in two GPU-kernel-optimization suites with held-out generalization gates: Metal-Sci (10 scientific-compute tasks) and Metal-ZK (12 zero-knowledge/cryptographic tasks), in which three frontier LLMs (Opus 4.7, Gemini 3.1 Pro, GPT-5.5) propose Metal kernels inside a $(1{+}1)$ evolutionary loop with rich feedback. Although no model is prompted to act adversarially, the promoted winners repeatedly fingerprint the evaluation configuration: they branch on the identity of runtime parameters, tune the measured branch maximally, and leave the unmeasured branch slow or silently wrong. Across the pooled suites, $16/53$ ($30\%$) of in-distribution wins fail to transfer to held-out configurations. We give a four-mode taxonomy of these failures, from configuration fingerprints to gate leakage. We distill design guidance for measurement under strategic optimization: held-out probes retain validity only on non-enumerable axes; gates must measure held-out performance, not just correctness; and a transfer rate is interpretable only with per-failure mechanism grades: ours decomposes into gamed, overfit, and benign.
Code and research artifacts: this https URL

Comments:	Published as a conference paper at AI Measurement Science Workshop @ COLM 2026
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2608.08722 [cs.LG]
	(or arXiv:2608.08722v1 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2608.08722 arXiv-issued DOI via DataCite

Submission history

From: Victor Gallego [view email]
[v1] Sun, 9 Aug 2026 14:16:11 UTC (906 KB)