世界模型与语言模型：论具体推理与抽象推理的互补性

AI 导读

本研究探讨了世界模型与多模态大语言模型在预测未来状态时的互补性。世界模型可生成具体的视觉未来轨迹，但可能视觉合理却任务错误；多模态大语言模型则擅长抽象推理。为此，研究提出了“受控的具体推理”框架，并构建了VRQABench和OpenWorldQA两个基准。同时，提出了Privileged-Future On-Policy Self-Distillation（PF-OPSD）方法，该方法在训练时利用真实未来视频作为特权上下文评估推理轨迹，但部署时无需真实未来。实验结果显示，PF-OPSD在两个基准上分别比基线高出10.6%和10.9%，并提升了对噪声或冲突轨迹的鲁棒性。

推荐理由

世界模型靠视觉预测，语言模型靠抽象推理，这篇把两者真正拧在一起了。用未来视频做自我蒸馏提升 10%，还给全开源，做 agent 决策的可以认真看看‘什么时候不信自己的眼睛’是怎么训出来的。

正文

View PDF HTML (experimental)

Abstract:World models and multimodal large language models (MLLMs) provide complementary capabilities for predicting future outcomes from static visual observations. World models can generate concrete visual rollouts of possible futures, while MLLMs can reason abstractly over questions, goals, and rules. However, generated rollouts are stochastic and may be visually plausible but task-incorrect, making it necessary to determine when visual simulation is useful, whether a rollout is credible, and how it should influence the final answer. We formulate this problem as controlled concrete reasoning, where a model learns to invoke, verify, and integrate visual future simulation alongside abstract reasoning. To study this setting, we construct two human-verified benchmarks, VRQABench for controllable spatial lookahead and OpenWorldQA for open-domain physical prediction, and propose Privileged-Future On-Policy Self-Distillation (PF-OPSD). During training, PF-OPSD uses ground-truth future videos and answers only as teacher-side privileged context to evaluate on-policy concrete-reasoning trajectories, while the deployable student never observes true futures at test time. Experimental results show that PF-OPSD outperforms baseline by 10.6% and 10.9% on VRQABench and OpenWorldQA, respectively, while increasing robustness to noisy or conflicting rollouts. Our code and dataset are available at this https URL.

Comments:	EMNLP 2026
Subjects:	Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
Cite as:	arXiv:2606.03603 [cs.CV]
	(or arXiv:2606.03603v2 [cs.CV] for this version)
	https://doi.org/10.48550/arXiv.2606.03603 arXiv-issued DOI via DataCite

Submission history

From: Yucheng Zhou [view email]
[v1] Tue, 2 Jun 2026 13:07:49 UTC (559 KB)
[v2] Mon, 31 Aug 2026 10:56:25 UTC (701 KB)