HuggingFace Daily Papers（社区热门论文）· HuggingFace Daily Papers（社区热门论文）·· 2026-06-09精选AI 评分73
Flow-DPPO: 面向流匹配模型的散度近端策略优化
AI 导读
针对流匹配模型中在线强化学习比率裁剪策略约束不当的问题，Flow-DPPO 提出用散度近端约束替代。关键洞察是流模型每步策略为高斯分布，可精确计算新旧策略间的 KL 散度。Flow-DPPO 采用非对称散度掩码，仅在梯度更新偏离信任区域且超阈值时阻止更新。实验表明，Flow-DPPO 获得更高奖励，KL 近端效率更优，缓解了灾难性遗忘，促进多目标均衡，并能在比率裁剪失效时支持稳定的多轮训练。代码已开源。
推荐理由
用 KL 散度代替比值裁剪来解决流匹配 RL 训练的不稳定，理论简洁，代码已开源，做图像/视频生成优化的同学可以跑一下。
正文
Abstract:Recent work has demonstrated that online reinforcement learning (RL) can substantially improve the quality and alignment of flow matching models for image and video generation. Methods such as Flow-GRPO and CPS cast the denoising process as a Markov Decision Process and apply PPO-style ratio clipping to enforce a trust region. However, we argue that ratio clipping is structurally ill-suited for flow models: the probability ratio between new and old policies is a noisy, single-sample estimate of the true policy divergence, leading to over-constraining in some regions of the trajectory and under-constraining in others. We propose Flow-DPPO (Flow Divergence Proximal Policy Optimization), which replaces ratio clipping with a divergence proximal constraint. A key observation is that the per-step policy in flow models is Gaussian, enabling exact and cheap computation of the KL divergence between old and new policies. Flow-DPPO employs an asymmetric divergence mask that blocks gradient updates only when they simultaneously move away from the trusted region and violate the divergence threshold. Experiments show that Flow-DPPO achieves higher rewards with better KL-proximal efficiency, alleviates catastrophic forgetting, promotes balanced multi-objective optimization, and enables stable multi-epoch training where ratio clipping degrades. Code and models are available at this https URL.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.11025 [cs.LG] |
| (or arXiv:2606.11025v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.11025 arXiv-issued DOI via DataCite |
Submission history
From: Xiangxin Zhou [view email]
[v1]
Tue, 9 Jun 2026 15:59:57 UTC (5,245 KB)
[v2]
Sat, 27 Jun 2026 07:44:26 UTC (5,245 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org