Behavior Knowledge Merge in Reinforced Agentic Models

Xiangchi Yuan1, Dachuan Shi1, Chunhui Zhang2, Zheyuan Liu3,
Shenglong Yao1, Soroush Vosoughi2, Wenke Lee1
1Georgia Institute of Technology, 2Dartmouth College, 3University of Notre Dame
xyuan300@gatech.edu
Paper arXiv Code

Teaser Image

Figure 1: RAM merges behaviors from multiple specialized RL agents into a single generalist agent by preserving unique task vectors.

Abstract

TL;DR: We propose RAM (Reinforced Agent Merging), a new framework to merge RL-trained agents. Unlike SFT merging, RAM explicitly preserves "unique" parameter updates that encode task-specific behaviors, preventing the performance dilution common in standard averaging methods.

Reinforcement learning (RL) is central to post-training, particularly for agentic models that require specialized reasoning behaviors. In this setting, model merging offers a practical mechanism for integrating multiple RL-trained agents from different tasks into a single generalist model. However, existing merging methods are designed for supervised fine-tuning (SFT), and they are suboptimal to preserve task-specific capabilities on RL-trained agentic models.

The root is a task-vector mismatch between RL and SFT: on-policy RL induces task vectors that are highly sparse and heterogeneous, whereas SFT-style merging implicitly assumes dense and globally comparable task vectors. When standard global averaging is applied under this mismatch, RL's non-overlapping task vectors that encode critical task-specific behaviors are reduced and parameter updates are diluted.

To address this issue, we propose Reinforced Agent Merging (RAM), a distribution-aware merging framework explicitly designed for RL-trained agentic models. RAM disentangles shared and task-specific unique parameter updates, averaging shared components while selectively preserving and rescaling unique ones to counteract parameter update dilution. Experiments across multiple agent domains and model architectures demonstrate that RAM not only surpasses merging baselines, but also unlocks synergistic potential among agents to achieve performance superior to that of specialized agents in their domains.

Motivations

We evaluated RAM across multiple environments including Meta-World and Minecraft.

Quantitative Results

Table 1: Comparison of RAM against standard merging baselines (Task Arithmetic, TIES, DARE) on the Meta-World benchmark.

Impact of Sparsity

We analyze how the sparsity of the task vectors affects the merging performance. As shown below, RL agents exhibit significantly higher sparsity compared to SFT models.

Sparsity Analysis

Method

Our framework, Reinforced Agent Merging (RAM), addresses the task-vector mismatch between RL and SFT. The pipeline consists of three main steps:

Method Pipeline

1. Disentanglement: We separate shared knowledge (common across agents) from unique knowledge (task-specific).
2. Selective Preservation: Instead of global averaging, we apply a mask to preserve the magnitude of unique updates.
3. Rescaling: Parameters are rescaled to ensure the merged model retains the specialized capabilities of the original agents.

Results

Poster

BibTeX

@article{yuan2026ram,
  title={Behavior Knowledge Merge in Reinforced Agentic Models},
  author={Yuan, Xiangchi and Shi, Dachuan and Zhang, Chunhui and Liu, Zheyuan and Yao, Shenglong and Vosoughi, Soroush and Lee, Wenke},
  journal={arXiv preprint arXiv:26XX.XXXXX},
  year={2026}
}