WorldMemArena：通过行动-世界交互循环评估多模态智能体记忆
针对现有基准无法精确诊断多模态智能体记忆在动态环境中的具体失败阶段，研究提出了“行动-世界交互循环”记忆模型，并构建了WorldMemArena基准。该基准包含400个多会话多模态任务，涵盖“终身进化”和“智能体执行”两类场景，支持对记忆写入、维护、检索和使用的阶段级评估。研究首次对长上下文、RAG等手工设计系统与基于框架的记忆智能体进行直接比较，发现记忆写入与存储质量的提升不直接带来性能改善，且多模态记忆在利用视觉证据及跨领域稳定性上仍存在挑战。
首个能定位多模态 Agent 记忆“写、维、取、用”哪一步出问题的基准，头对头比较长上下文、RAG 和自管理记忆，结论是写得好未必用得上，做 Agent 的值得认真看。
Authors:Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu, Yepeng Liu, Lin Long, Yichen Guo, Nuo Chen, Zhaotian Weng, Elena Kochkina, Simerjot Kaur, Charese Smiley, Xiaomo Liu, James Zou, Sheng Liu, Yuheng Bu, Songyou Peng, Xin Eric Wang
Abstract:Multimodal large language models are increasingly deployed as long-horizon agents, where memory must do more than recall: it must track an evolving world, revise what has gone stale, and surface the right evidence at decision time. Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions, leaving us unable to localize failures to writing, maintenance, retrieval, or use. The rise of agent harnesses that author their own memory sharpens this gap, since we have no principled way to compare hand-designed pipelines with self-managing alternatives. To close these gaps, we formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle, and instantiate it in WorldMemArena: 400 multi-session multimodal tasks spanning Lifelong Evolution (evolving personal and task states) and Agentic Execution (memory from real observations, actions, and feedback), annotated with gold memory points, updates, distractors, and evidence chains for stage-level diagnosis. This enables the first head-to-head comparison of long-context, manually designed (RAG and external memory systems), and harness-based memory agents. Results show that: (1) better memory writing and storage do not guarantee better performance; (2) multimodal memory still struggles to fully use visual evidence; (3) systems are unstable across domains and degrade on realistic agentic trajectories; and (4) harness memory is more flexible but remains costly and less reliable.
| Comments: | 25 pages, 8 figures |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.29341 [cs.CV] |
| (or arXiv:2605.29341v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.29341 arXiv-issued DOI via DataCite |
Submission history
From: Chengzhi Liu [view email]
[v1]
Thu, 28 May 2026 04:27:20 UTC (9,232 KB)
[v2]
Mon, 1 Jun 2026 07:01:55 UTC (9,232 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org