s³

every dot is an agent · move your cursor through the logo

Simple Scaling of Swarms
via Cooperative Multi-agent RL

Kunal Jha† Gonçalo Faria† Shuyue Stella Li† Scott Geng† Faeze Brahman Noah A. Smith Yulia Tsvetkov Max Kleiman-Weiner‡ Natasha Jaques‡
UWUniversity of WashingtonAi2Allen Institute for AI
† Equal contribution  ·  ‡ Equal advising  ·  Correspondence: kjha@uw.edu
Read the paper Cite

Can a group of AI agents learn to work together without anyone telling them how? We train swarms of language models with no leader, no assigned roles, and no turn order. They share one reward and can message each other whenever they want, and over the course of training they figure out how to organize.

Agents explore on their own, tell each other what they find, and pick up where a teammate left off.

A cartoon of the idea, not one of our experiments.

01 / Motivation

Most multi-agent systems are organized by hand

When people run many language models together, they usually decide ahead of time how the models should work together. One model plays the orchestrator and hands out subtasks, others debate in rounds, or each gets a role like planner or critic.

These structures are written before the agents ever see a task, and they don't adapt to it. They also fail in familiar ways. Orchestrated systems duplicate work, send each other too many updates, and have trouble reconciling subagents that disagree.

Human groups are different. How people split up work and share information depends on the problem in front of them, and a lot of that structure came from cultural evolution rather than anyone's plan. Part of why centrally planned economies struggled is that no planner could know everything the people doing the work knew.

In AI, learned behavior has repeatedly beaten hand-designed behavior. Reasoning models got much better by learning to reason through reinforcement learning, not by following strategies people wrote down for them. We think the same should be true for how groups of agents organize. If a task is best solved with a coordinator, the agents should learn to pick one. If it's best solved by splitting up the work, they should learn to split it.

Left: multi-agent systems organized by static harnesses. Right: s3 agents decide for themselves when to work alone, share findings, or read messages.
Figure 1. Left: prior systems use a structure designed in advance, like a controller that assigns tasks or a leader that runs a shared forum. Right: in s³, each agent decides for itself how long to work alone and when to share findings or read teammates' messages. The organization comes out of training.
To our knowledge, s³ is the first method where reinforcement learning trains asynchronous language model swarms with no prescribed roles or communication structure to coordinate well enough to beat best-of-N sampling from solo agents trained with the same budget. The swarms also keep their advantage when we run them with more agents than they were trained with.
02 / Method

Copies of one model, one shared reward

A swarm is N copies of the same model. Each agent works on its own copy of the task and can't see what its teammates are thinking or what their tools return.

The only way agents can coordinate is by what they choose to tell each other. So learning to organize really means learning to communicate: what to share, who to share it with, when, and when it isn't worth saying anything.

Everyone on the team gets the same reward. The team succeeds if any agent solves the task, and successful teams pay a small cost for the fraction of their token budget they used:

\[ R \;=\; {\color{#5fd3ab}S}\left(1 - {\color{#f5b04a}\lambda} \min\!\left(1, \frac{{\color{#9fc5ff}\sum_{i=1}^{N} \lvert\tau_i\rvert}}{N L}\right)\right) \]
team success (any agent)token cost, λ = 0.25tokens used by the whole team

For each problem we run 8 independent teams and compare them, the way GRPO compares responses. Every token from every member of team \(g\) gets that team's advantage:

\[ A_g = \frac{{\color{#5fd3ab}R_g} - {\color{#f5b04a}\bar R}}{\sigma_R + \epsilon} \]
this team's rewardaverage over the 8 teams on the same problem
A few design choices
01

Two messaging tools

Agents can call message(to, text) to write to one teammate or everyone, and read_messages() to check their inbox. There's no limit on messages and no reward for sending them. Teammates get random names in a shuffled order each episode, so the policy can't learn conventions tied to a specific name.

02

Reward the team's best outcome

We use the best result on the team, not a sum or an average. A sum would make a bigger team worth more just for being bigger. An average would reward each agent for being right on its own, which pushes agents to solve the problem separately and then argue about it.

03

Compare teams, not teammates

We never try to work out how much each agent contributed. Comparing independent teams on the same problem takes care of that: whatever separates the teams that succeeded from the ones that failed gets reinforced, and that includes helping a teammate.

03 / Infrastructure

What it took to make training work

The objective is simple. Most of our effort went into the harness and the trainer. For RL to discover useful communication, an agent has to be able to act on a teammate's discovery while it's still useful, and that requirement shaped most of the choices below.

asynchrony

Agents don't take turns

Each agent generates text, calls tools, and continues as soon as its own tool result comes back. One agent can share a discovery while another is waiting on a tool, so the team isn't held to the pace of its slowest member.

notifications

Messages arrive as notifications

When a message comes in, the harness adds a notification to the recipient's history with unread counts by sender, but not the contents. An idle agent wakes up right away, and a busy one sees the notification when its current request finishes. It's up to the agent whether to read it. An agent that stops calling tools goes idle and spends no tokens.

chunked generation

Sharing the inference server

All agents in a swarm share one inference server, with a limited number of sequence slots and limited KV-cache memory. If a few agents run long generations, everyone else waits. We split generation into requests of at most 1,024 tokens that continue from the exact same prefix. This gives every agent a chance to make progress and gives notifications a place to land, without adding turns or extra prompts.

off-policy correction

Training while sampling

The inference server keeps collecting episodes while the trainer updates, and we accept rollouts up to four updates old. For exploration, inference also runs in FP8. To correct for the mismatch, we use IcePop to mask tokens with extreme importance ratios, plus score centering to remove the bias that masking introduces over time.

loss masking

Only train on what each agent wrote

Messages from teammates, tool results, and notifications are in an agent's context, but we mask them out of the loss so an agent never learns to imitate another agent's words. We keep the exact sampled tokens and their log-probabilities across interruptions, so the importance ratios are exact.

isolation

Keeping comparisons clean

Each team attempt has its own message board, and each agent has its own random seed, so the order the server answers requests never changes what an agent samples. Teams count equally in the loss no matter how long they talk. In exploration, every agent sees the doors in the same order, so any diversity in their search has to be learned.

Making the comparison fair

A swarm can look better just because it runs more agents. Best-of-N sampling is a strong baseline, and it keeps improving as N grows. So every learner we compare gets the same problems, the same number of agent trajectories per update, the same per-agent token budget, and the same optimizer. Our solo baseline is a single agent trained with that budget. At evaluation we run N independent copies of it and count a success if any of them solves the problem.

04 / Tasks

Two reasons groups help, and why both get harder with more agents

Both of our tasks are simple on purpose. Each one isolates a different reason a team can do better than an individual, and they get harder in opposite ways as the swarm grows.

▼ harder as N grows

Information aggregation

Each agent gets one private number between 1 and 99, written in English ("forty-two"). The team has to return all N numbers in sorted order. Sorting is easy once you have the numbers. The hard part is coordinating: each agent has to share its number, read everyone else's, keep track of who sent what so repeated messages don't create duplicates, and only submit once it has heard from everyone. Messages can arrive in any order.

This is a stripped-down version of a common situation: agents reading different files in a codebase, searching different sources for a report, or sensors each seeing part of a scene. The answer only exists once the pieces are combined, and adding more independent agents doesn't help.
\[ P_{\mathrm{coord}}(N) \le \min\!\left(1,\, N q^{N-1}\right) \]

If each message a recipient needs gets through with probability \(q<1\), success eventually drops off exponentially with N. An agent's inbox can also be in \(2^{N-1}\) states depending on who has reported, which is 8 with four agents and 32,768 with sixteen. Agents have to learn a general rule, like waiting until they've heard from N−1 different teammates.

▲ baseline improves as N grows

Decentralized exploration

Each agent explores its own copy of an unknown tree of rooms, starting from the same place. Every junction has 3 doors, junctions are ten moves apart, and one of the 27 rooms at the bottom is the exit. Searching the whole tree can take 780 moves, usually more than an agent can afford with its 8,192 tokens. The first agent to reach the exit solves it for the team.

This is a simple model of searching together: parallel research agents, a distributed bug hunt, or the thousands of agents recently put on a single open math problem. Anyone could get there alone, but a team that shares what it finds can split up the space and avoid repeating each other's dead ends.
\[ P_{\mathrm{search}}(N) = 1-(1-p)^N \]

Here the independent baseline keeps getting stronger as you add agents. Communication only helps if it beats a best-of-N baseline that rises with N.

Illustrations

Two cartoons of what communication buys a team. The first follows our information aggregation task. The second is a stylized search problem, not our maze, meant to show how one agent's find can speed up another's.

05 / Results: information aggregation

Swarms learn to share what they know

s³ learns to aggregate information reliably and does much better than controls trained with the same budget.

93%
success for s³ with 4 agents
35%
training on the swarm's own successful episodes (online SFT)
13%
rewarding each agent only for its own answer
Bar chart of information aggregation success
Figure 2. Held-out success with four agents, the training swarm size. Bars show mean ± SE across six seeds. All three learners get the same training budget, so the gap isn't explained by more interaction or more successful examples.

s³ agents also send fewer messages. Each one broadcasts its number about once, and every agent sends about the same amount. The controls send more messages overall, but a few agents send most of them while the rest say very little.

Delivery concentration, reciprocity, and message counts for each method
Figure 3. Communication structure in evaluation episodes with four agents. Delivery concentration measures whether a few agents do most of the sending, and reciprocity measures how much communication goes both ways.

Running bigger swarms than we trained

We train with four agents and then evaluate with up to sixteen, without retraining. With eight agents, twice the training size, s³ solves over 25× as many problems as the strongest baseline, and it stays ahead through twelve agents. At sixteen, every method struggles. What the swarm learned transfers, but it doesn't remove the growing difficulty of tracking everyone's information.

Success and token efficiency across evaluation swarm sizes
Figure 4. Held-out results across swarm sizes for information aggregation (top) and decentralized exploration (bottom). Swarms train with four agents (dashed line). Token efficiency is solve rate divided by the fraction of the population's token budget used, counting failures.
06 / How coordination emerges

What changes during training

These agents start from the base model with no demonstrations, so everything they learn about coordinating comes from RL.

Early in training, about half of swarms have at least one agent that never sends its number, and most of the rest talk in direct messages. Partway through, swarms switch to broadcasting, at first more than once. By the end, about 80% of swarms settle on one broadcast per agent, and agents learn to share their own number before checking their inbox.

Why this pattern? Early on, the few swarms that already broadcast once per agent succeed about 30 percentage points more often than swarms on the same problem that use direct messages. The shared reward keeps reinforcing that difference until it takes over.

Individual rewards create a social dilemma

When each agent is rewarded only for its own answer, sending your number only helps your teammates. Each agent is better off waiting for someone else to go first. Over training, more agents read before they send, and many stop sending at all.

This is a public goods game. Everyone would do better if everyone shared, but each agent does better individually by holding back, and when everyone holds back, nobody shares. With a shared reward the dilemma goes away, because an agent benefits whenever any teammate gets the answer right.

You can see this in the "Individual reward" setting of the first illustration above.

Training dynamics of communication structure, read-before-send rate, and exploration success by exit door
Figure 5. (a) Information aggregation swarms go from incomplete sharing, through repeated broadcasts, to one broadcast per agent. (b) With individual rewards, more and more agents read before sharing. (c) In exploration, communicating swarms succeed at about the same rate no matter which door leads to the exit.
07 / Results: decentralized exploration

Swarms learn to split up the search

In exploration any agent can find the exit alone, so the question is whether communicating helps beyond simply running more agents.

Both swarm and solo policies start with a short round of supervised fine-tuning on scripted demonstrations of valid exploration (only the swarm demonstrations include messages). RL then trains on larger, harder trees.

−19 pts
success with four agents when messaging is turned off
~½
the room overlap of swarms that can't communicate
4 ≈ 6
four s³ agents nearly match six solo-trained agents
Bar chart of decentralized exploration success at N=4
Figure 6. Held-out success with four agents. Turning off messaging while keeping the shared reward costs 19 percentage points.

Every agent sees each room's doors listed in the same order, so agents with similar habits tend to take the same first door. Swarms that can't communicate do fine when the exit is behind that first door, worse when it's behind the second, and much worse when it's behind the third. s³ swarms succeed at about the same rate wherever the exit is, because agents claim different branches and report the ones they've finished (Figure 5c).

As we add agents, communicating swarms reach high success sooner. Four s³ agents nearly match six solo-trained agents, and six nearly match eight. With six agents, s³ beats best-of-six by about 7 percentage points, with no statistically detectable loss in token efficiency. At the largest sizes independent search also gets close to perfect, so the main benefit of communication is getting there with fewer agents.

08 / Ablations

What matters

shared reward

It works for both tasks

In exploration, individual rewards do about as well as s³, because hearing about a teammate's dead ends helps you reach the exit yourself. In aggregation, sending your number only helps others, and individual rewards fall apart. A shared reward works in both cases, which matters when you don't know the structure of a problem ahead of time.

communication

The messages are doing real work

Keeping the shared reward but turning off messaging drops exploration success with four agents by 19 points and roughly doubles how much agents repeat each other's search. The gains come from coordinating, not just from the team objective.

demonstrations

Exploration needs a starting point

Without fine-tuning on demonstrations first, both swarm and solo learners stay much less successful, and the swarm has no significant edge. With demonstrations, both start much higher, and RL keeps improving the swarm from there.

09 / Next steps

Where this goes next

We're currently scaling s³ to larger models and more sophisticated tasks.

Our experiments so far use two tasks and small models (Qwen3-0.6B for aggregation and Qwen3-1.7B for exploration), so there's a lot left to test. Looking through the training traces also points to specific failure modes. Agents sometimes submit a wrong answer even when they have every number, repeat a teammate's search, or trust a report they haven't checked. Training agents to keep track of what their teammates know, and whether those reports can be trusted, could make bigger swarms more reliable without more communication.

Our main takeaway is that agents can learn to collaborate in the same training process where they learn the task. With a shared goal and infrastructure that lets them act asynchronously, useful ways of sharing information and dividing work show up during training. Because those behaviors carry over to larger swarms, training a small swarm can pay off when you deploy a bigger one. We're excited to see what this makes possible for multi-agent systems more broadly.

Read the paper
Citation

Build on this work

@article{jha2026s3,
  title   = {s$^3$: Simple Scaling of Swarms via Cooperative Multi-agent RL},
  author  = {Jha, Kunal and Faria, Gon{\c{c}}alo and Li, Shuyue Stella and Geng, Scott and
             Brahman, Faeze and Smith, Noah A. and Tsvetkov, Yulia and
             Kleiman-Weiner, Max and Jaques, Natasha},
  journal = {Preprint},
  year    = {2026},
  url     = {https://scalingswarms.com}
}