GRAFT：用跨模型轨迹交换提升 RLVR 训练效率

View PDF HTML (experimental)

Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) methods such as GRPO rely on successful self-generated trajectories, but finite rollout budgets can produce all-fail groups with no reward-based policy-gradient signal. While additional rollouts improve the chance of success at higher cost, successful trajectories missing from one model's rollouts may already have been discovered by another. Indeed, we observe that heterogeneous models often succeed on complementary prompts, creating opportunities for mutual learning without a designated stronger teacher. To exploit this complementarity, we propose GRAFT (Gated Replacement of Answer-Failed groups with peer Trajectories), an off-policy-aware framework that replaces all-fail groups with informative peer groups. GRAFT transfers both successful and unsuccessful peer responses with peer-computed advantages, while controlling cross-model mismatch through sequence-level compatibility weighting and token-level importance ratio clipping. Across three heterogeneous model pairs and five mathematical reasoning benchmarks, GRAFT consistently improves both models over GRPO with the same per-model rollout budget, gaining 2.1 points on average and up to 4.5 points in model-level average performance. Stored peer trajectories preserve most of the gains, improving over GRPO by 1.8 points on average without simultaneous co-training.

Comments:	29 pages, 11 figures, 9 tables
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
ACM classes:	I.2.6; I.2.7
Cite as:	arXiv:2609.37868 [cs.LG]
	(or arXiv:2609.37868v1 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2609.37868 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Doohyuk Jang [view email]
[v1] Tue, 29 Sep 2026 15:47:25 UTC (612 KB)