PORTool: 基于奖励树和重要性感知的策略优化方法，用于多工具集成推理
研究团队提出PORTool算法，以解决多工具集成推理中仅依靠结果奖励导致的信用分配模糊问题。该方法通过重要性感知策略优化，在结果级监督下强化智能体的工具使用能力，同时实现步骤级奖励分配。PORTool生成奖励树来明确关键决策步骤，从而更精确地引导模型学习有效的工具调用序列，提升复杂任务解决的效率和可靠性。
不少 Agent 团队训练时都遇到过奖励信号太稀疏的问题，PORTool 试着把奖励细粒度化，给了个可实操的解法，做工具调用智能体的值得深读。
AuthorsFeijie Wu†**, Weiwu Zhu, Yuxiang Zhang, Soumya Chatterjee, Jiarong Zhu, Fan Mo, Rong Luo, Jing Gao†
Multi-tool-integrated reasoning enables LLM-empowered tool-use agents to solve complex tasks by interleaving natural-language reasoning with calls to external tools. However, training such agents using outcome-only rewards suffers from credit-assignment ambiguity, obscuring which intermediate steps (or tool-use decisions) lead to success or failure. In this paper, we propose PORTool, an importance-aware policy-optimization algorithm that reinforces agents’ tool-use competence from outcome-level supervision while assigning reward at the step level. Specifically, PORTool generates a rewarded rollout tree in which trajectories share prefixes before branching, enabling direct comparisons among alternative tool-use decisions within the same context. It then estimates each step’s importance by a correctness-dominant signal, i.e., whether descendants of that step can ultimately produce a correct final answer, plus an auxiliary term indicating whether the step’s tool calls execute successfully. Using these step-wise importance estimates, PORTool updates the policy to generate efficient tool-call steps, guided by both local comparisons within each branching decision and the overall quality of entire trajectories. Experiments show that PORTool improves final-answer accuracy while reducing tool-call steps compared with state-of-the-art baselines, and ablation studies confirm the robustness of the proposed step-wise importance estimates.
- † Purdue University
- ** Work done while at Apple
Related readings and updates.
Evaluating AI agents that use external tools requires realistic test scenarios that capture how practitioners compose tools and iterate across conversation turns. Constructing such scenarios by hand demands deep domain expertise, does not scale across tool ecosystems, and produces static benchmarks that cannot track evolving APIs. We observe that tool specifications—function names, natural-language descriptions, and typed parameter…
Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies…
来源：Apple Machine Learning Research（RSS） · machinelearning.apple.com