MULTI-REWARD REINFORCEMENT LEARNING

Advantage Aggregation with Priority for Multi-Reward Policy Optimization

PAPO · Primary Advantage Policy Optimization

Xiangchi Yuan1, Xingjian Diao2, Zheyuan Liu3, Chunhui Zhang2, Dachuan Shi1, Kejing Xia1, Zhenbang Du1, Shenglong Yao1, Soroush Vosoughi2, Wenke Lee1

1Georgia Institute of Technology2Dartmouth College3University of Notre Dame

↗ Read the paperExplore results ↓

Keep the task first.
Make auxiliary rewards work with it.

Comparison of additive fusion, hard gating, and PAPO fusion across primary and auxiliary advantages
The priority problem. Additive fusion can reverse the primary advantage. Hard gating discards auxiliary distinctions. PAPO bounds conflicting signals while preserving aligned supervision. Figure 1 from the paper.

Abstract

TL;DR. Auxiliary rewards should refine how a task is solved, without overturning the primary objective. PAPO makes this priority explicit when combining advantages.

Multi-reward reinforcement learning (RL) is increasingly used to optimize large language models for both task performance and auxiliary behavioral objectives, such as concise reasoning and efficient tool use. However, existing approaches typically combine objective-specific rewards or advantages symmetrically, implicitly allowing auxiliary objectives to influence the policy update as strongly as the task-defining objective. We term this advantage priority inversion: when primary and auxiliary advantages conflict, a sufficiently strong auxiliary signal can reverse the update direction preferred by the primary objective.

In light of this, we propose Primary Advantage Policy Optimization (PAPO), a priority-aware framework for multi-reward policy optimization. PAPO explicitly distinguishes primary and auxiliary objectives and performs asymmetric advantage aggregation. Auxiliary advantages are fully preserved when they agree with the primary signal, but are softly bounded under conflict so that they cannot overturn the primary optimization direction. The framework applies naturally to both sequence-level GRPO and token-level PPO; for PPO, we additionally use a multi-head critic to estimate objective-specific returns before advantage fusion. We evaluate PAPO on competition-level mathematical reasoning and agentic tool use with multiple rewards across PPO- and GRPO-based training. PAPO consistently improves primary-task performance while retaining strong auxiliary-objective performance.

01 / MOTIVATION

When efficiency overrides correctness

A shorter answer is useful when it is correct. Fewer tool calls are useful when the task still succeeds. Yet a large auxiliary advantage can overwhelm a conflicting primary advantage under a weighted sum.

PRIMARY OBJECTIVE

Solve the task

Answer correctly or complete the tool-use task.

AUXILIARY OBJECTIVE

Solve it efficiently

Use fewer reasoning tokens or fewer tool calls.

Advantage priority inversion: an update can reward a response with a negative primary advantage, simply because its auxiliary advantage is large enough.

02 / METHOD

Preserve agreement. Bound conflict.

PAPO estimates objective-specific advantages before combining them. Aligned signals add together; conflicting auxiliary signals are smoothly bounded relative to the primary advantage.

PAPO architecture: sequence-level advantage aggregation for GRPO and a multi-head critic with token-level aggregation for PPO
One fusion rule, two optimization families. GRPO combines sequence-level advantages. PPO uses objective-specific value heads and GAE before token-level fusion. Figure 2 from the paper.

Sequence-level GRPO

Independently normalize each objective within the rollout group, fuse primary and auxiliary advantages, then normalize the fused group before optimization.

Token-level PPO

Use a multi-head critic to estimate each objective’s return. Compute GAE separately and apply PAPO at each token, keeping critic targets unfused.

INTERACTIVE EXAMPLE

Try conflicting reward signals

Move the auxiliary advantage to see when additive fusion changes direction.

Additive fusion
PAPO fusion

Illustrative fusion with α = 1, before any subsequent group normalization. This is an advantage-level demonstration, not a training simulation. At zero primary advantage, the aligned branch uses the auxiliary signal.

03 / EXPERIMENTS

Better task performance.
Strong auxiliary behavior.

Qwen3-8B results on mathematical reasoning and agentic tool use. All values below are taken from Tables 1 and 2 of the paper.

62.1%

Math accuracy · GRPO-based PAPO
+5.2 pp over GRPO

93.0%

Math length compliance
GRPO-based PAPO

89.93%

Tool-use accuracy · GRPO-based PAPO
0.33 average tool calls, matching GRPO

Overall mathematical reasoning performance
MethodOptimizationAccuracy (%) ↑Length compliance (%) ↑

PAPO is highlighted within each optimization family. Auxiliary performance is not uniformly best: for tool use, PPO-based PAPO uses 0.56 calls versus PPO’s 0.37, with higher accuracy (88.25% versus 86.25%).

04 / TRAINING DYNAMICS

Learning both objectives together

The paper’s training curves show PAPO improving the auxiliary length objective while maintaining stronger primary-task accuracy across GRPO- and PPO-based optimization.

Training and evaluation accuracy and length reward over optimization steps for PAPO and baseline methods
Figure 3 from the paper. GRPO-based training appears on the top row and PPO-based training on the bottom row.

BibTeX

@misc{yuan2026advantage,
  title={Advantage Aggregation with Priority for Multi-Reward Policy Optimization},
  author={Yuan, Xiangchi and Diao, Xingjian and Liu, Zheyuan and Zhang, Chunhui and Shi, Dachuan and Xia, Kejing and Du, Zhenbang and Yao, Shenglong and Vosoughi, Soroush and Lee, Wenke},
  year={2026},
  url={https://xiangchi-yuan.github.io/data/ICLR_2027_Advantage_Integration.pdf}
}

Manuscript citation; publication details can be updated upon release.