JoyAI-VL-Interaction：实时视觉-语言交互智能
JoyAI-VL-Interaction是一个8B参数的视觉优先交互模型，能像人一样持续观察实时画面，自主决定每秒钟保持沉默、回应或将复杂问题委托给后台模型。模型擅长视觉触发响应和时间感知，并涌现出引导用户切换屏幕、根据幻灯片即兴讲解等未专门训练的能力。配合完整可部署系统，支持实时视频流输入、可插拔ASR/TTS模块、记忆、可视化UI及可连接任意API或智能体的后台大脑。在六个真实场景中，人类评估者更偏好该模型而非豆包和Gemini的应用内视频通话助手。这是首个开源且附带训练配方、数据和完整部署系统的视觉驱动交互模型。
这是首次把视觉主动交互能力做成开源模型和完整系统，模型不再等问题才回答，而是自己决定何时开口或保持沉默，对做实时助手和直播产品的团队有直接参考价值。
Authors:Dingyu Yao, Junhao Zhou, Chenxu Yang, Chuanyu Qin, Xiangyu Zeng, Yifei Li, Haowen Hou, Zheming Liang, Congcong Wang, Kaiwen Tuo, Jun Zhang, Yuhan Zhu, Yuhang Cao, Shenglong Ye, Shuai Xie, Shuhuan Gu, Haoyang Huang, Qingyi Si, Nan Duan, Jiaqi Wang
Abstract:Many moments in the real world do not wait for a user to ask. A fire starts on a security monitor, an expression flickers across a video call, or a product a viewer wants flashes by in a livestream. Yet today's large models remain mostly turn-based by design: they answer only when addressed, and even video-call apps that appear interactive still operate as question-answer systems, reacting only when polled or prompted. We argue for a different paradigm: a model that is present in the world like a person. It continuously watches what is happening now, decides on its own whether to speak or stay silent, interacts in real time, and delegates to a background model when the problem is hard. To advance interaction models and their adoption across domains, we make two fully open-sourced contributions. First, we release JoyAI-VL-Interaction, an 8B-scale, vision-first VL-interaction model. The model makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model, and it excels at vision-triggered responsiveness and time awareness. We pair it with a transferable training recipe, from which capabilities we never trained for emerge, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck. Second, we release a complete, deployable system built around that model. The system streams any ongoing video into the model, making it genuinely present in the world. All other components are pluggable, including ASR/TTS modules, memory, visualization UI, and a background brain that can connect to any API or agent. Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin. To our knowledge, this is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system.
| Comments: | v2 |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.14777 [cs.CV] |
| (or arXiv:2606.14777v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2606.14777 arXiv-issued DOI via DataCite |
Submission history
From: Chenxu Yang [view email]
[v1]
Wed, 10 Jun 2026 03:43:50 UTC (14,024 KB)
[v2]
Thu, 24 Sep 2026 10:09:36 UTC (16,703 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org