EVA-Bench：端到端语音智能体评估新框架
EVA-Bench是一个端到端语音智能体评估框架，解决了模拟真实对话与测量全范围语音故障两大挑战。它通过动态多轮机器对话和自动验证进行仿真，并提出了衡量任务完成度、音频保真度的EVA-A指标，以及评估对话体验的EVA-X指标。框架包含三个领域的213个场景及鲁棒性测试集，采用区分峰值与可靠能力的测量方法。在12个系统的测试中发现，无系统能在两项核心指标上同时超过0.5，峰值与可靠性能差距显著，且口音与噪声扰动暴露出明显的鲁棒性缺陷。该框架已开源。
EVA-Bench 把语音代理评估从「能对话就行」推进到「对话质量+鲁棒性」的全维度打分，还开源了 213 个企业场景，做语音助手的团队该认真看看。
Authors:Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz, Oluwanifemi Bamgbose, Fanny Riols, Hoang H. Nguyen, Raghav Mehndiratta, Lindsay Devon Brin, Joseph Marinier, Hari Subramani, Anil Madamala, Sridhar Krishna Nemala, Srinivas Sunkara
Abstract:Voice agents are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses realistic conversation simulation and comprehensive voice-specific evaluation. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates dynamic bot-to-bot audio conversations with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy) and EVA-X (Experience). EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and multi-trial measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; (2) peak and reliable performance diverge substantially (median pass@k--pass^k gap of 0.44 on EVA-A); and (3) accent and noise perturbations expose substantial robustness gaps, with effects varying across architectures, systems, and metrics (mean $\Delta$ up to 0.314). We release EVA-Bench under an open-source license.
| Comments: | Accepted to EMNLP 2026 (Findings) |
| Subjects: | Sound (cs.SD); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.13841 [cs.SD] |
| (or arXiv:2605.13841v3 [cs.SD] for this version) | |
| https://doi.org/10.48550/arXiv.2605.13841 arXiv-issued DOI via DataCite |
Submission history
From: Hoang Nguyen [view email]
[v1]
Wed, 13 May 2026 17:58:52 UTC (970 KB)
[v2]
Wed, 27 May 2026 20:23:25 UTC (971 KB)
[v3]
Wed, 9 Sep 2026 17:57:23 UTC (716 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org