QUACK：多模态社交推理智能体通信知识的质询、理解与审计

Authors:Ye Yuan, Rui Song, Weien Li, Zeyu Li, Haochen Liu, Xiangyu Kong, Changjiang Han, Yonghan Yang, Zichen Zhao, Zixuan Dong, Fuyuan Lyu, Bowei He, Haolun Wu, Jikun Kang, Xue Liu

View PDF HTML (experimental)

Abstract:Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents. However, most environments are scored only by game outcomes such as win rates and largely remain to text-only interaction, making it difficult to tell whether an agent's language is actually grounded in what it perceived and did, or to identify the failure modes underlying its behavior. To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing the grounding of agent language in multimodal social reasoning. QUACK evaluates agents at three levels: game outcomes, behavioral trajectories, and utterance-level consistency. Its core Statement Verification Pipeline reconstructs each agent's ground-truth trajectory from engine logs and checks every discussion claim against it, automatically flagging spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency. Evaluating three frontier VLMs in both homogeneous and cross-model adversarial settings, we find that even the strongest agent hallucinates 15.1% of its verifiable spatial claims and 11.5% of accusations are strictly unsupported. We release the full engine, evaluation framework, toolkit, and logs in this https URL.

Comments:	Accepted by EMNLP 2026 Main Conference
Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
Cite as:	arXiv:2605.27068 [cs.CL]
	(or arXiv:2605.27068v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2605.27068 arXiv-issued DOI via DataCite

Submission history

From: Ye Yuan [view email]
[v1] Tue, 26 May 2026 14:19:08 UTC (671 KB)
[v2] Mon, 31 Aug 2026 03:39:09 UTC (678 KB)