Vera：大规模LLM智能体安全测试框架
Vera是一个端到端自动化安全测试框架，通过三阶段自增强流水线对LLM智能体进行规模化的安全检验：文献驱动探索持续发现新兴风险；组合生成跨维度构造可执行安全用例；自适应执行在隔离沙箱中运行异构agent并基于环境状态与工具调用证据验证结果。在OpenClaw、Hermes、Codex、Claude Code四个生产级agent框架上测试，多通道攻击下平均攻击成功率达93.9%。同步发布Vera-Bench，包含1600个可执行安全用例，覆盖124个风险类别。代码已公开。
对 OpenClaw、Hermes、Codex、Claude Code 等主流 Agent 框架的自动化安全测试，攻击成功率高达 93.9%，并开源了 1600 个可执行的安全用例，做 Agent 产品的团队不应错过。
Authors:Yunhao Feng, Ruixiao Lin, Ming Wen, Qinqin He, Yanming Guo, Yifan Ding, Yutao Wu, Jialuo Chen, Zhuoer Xu, Xiaohu Du, Jianan Ma, Zixing Chen, Xingjun Ma, Yunhao Chen, Xinhao Deng
Abstract:LLM agents increasingly perform autonomous actions through external tools, leading to complex and evolving safety risks. However, existing safety testing targets expert-designed safety violations, and the corresponding outcomes are evaluated by hard-coded rules, making them costly to extend as agents evolve. To this end, we present Vera, an end-to-end automated safety testing framework that instantiates software engineering testing principles for non-deterministic agents through a three-stage, self-reinforcing pipeline. First, a literature-driven exploration continuously discovers and structures emerging risks into taxonomies of safety risks, attack methods, and tool execution environments. Second, combinatorial composition across taxonomy dimensions produces executable safety cases, each specifying a concrete safety goal, a programmatically constructed initial state, and a deterministic verification predicate grounded in observable artifacts. Third, adaptive execution runs heterogeneous agents in isolated sandboxes where a control agent steers multi-turn interaction based on runtime observations, while evidence-grounded verifiers judge outcomes from environment state and tool-call evidence rather than model self-report. We evaluate Vera on four production agent frameworks (OpenClaw, Hermes, Codex, Claude Code), revealing substantial safety weaknesses, with average attack success rates reaching 93.9\% under multi-channel attacks; we also release Vera-Bench, comprising 1600 executable safety cases spanning 124 risk categories across three execution settings. These results indicate that modular, executable testing infrastructure is essential for rigorous and maintainable safety evaluation of rapidly evolving agentic systems at scale. The code is publicly available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.01793 [cs.AI] |
| (or arXiv:2607.01793v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.01793 arXiv-issued DOI via DataCite |
Submission history
From: Yunhao Feng [view email]
[v1]
Thu, 2 Jul 2026 07:08:26 UTC (539 KB)
[v2]
Sat, 4 Jul 2026 02:16:40 UTC (539 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org