Method: Diversity Control

How is behavioral diversity measured and controlled?

How do we measure behavioral diversity?

Each agent's policy defines a probability distribution over actions given an observation, so the difference between two agents can be measured as a distance between their action distributions. System Neural Diversity (SND) summarizes team diversity as the average pairwise distance between agent policies across observations. We use the Wasserstein-2 distance, which remains well defined when two distributions do not overlap, whereas the Kullback-Leibler divergence grows without bound.

Two Gaussian distributions with shrinking variance: the Wasserstein-2 distance approaches the distance between means, while the KL divergence tends to infinity. — As the variance of two Gaussian policies shrinks, the Wasserstein-2 distance approaches the distance between their means, while the KL divergence diverges. From Bettini et al. (2024).

What does SND reveal about team structure?

Beyond a single value, the matrix of pairwise distances shows how diversity is distributed across the team. In Navigation with three agents and three goals, all pairwise distances are uniformly high, indicating three distinct policies. With two goals, two agents share a goal and converge to similar behavior, with a pairwise distance of 0.28, while the third agent specializes. This reveals an emergent sub-team structure.

Pairwise distance matrix for three goals with uniformly high off-diagonal values. — Learned behavior (top) and pairwise Wasserstein-2 distances between agents (bottom). Agent 1 is orange, agent 2 is blue, and agent 3 is green.

Pairwise distance matrix for two goals with a low distance of 0.28 between agents 1 and 2. — Learned behavior (top) and pairwise Wasserstein-2 distances between agents (bottom). Agent 1 is orange, agent 2 is blue, and agent 3 is green.

How does DiCo control diversity?

Diversity Control (DiCo) decomposes each agent's policy into a shared homogeneous component and an agent-specific heterogeneous component. During training, DiCo estimates the SND of the heterogeneous components and rescales them by the ratio of the desired SND to the estimated SND. This constrains the team's behavioral diversity to the desired target while reinforcement learning updates the policies.

DiCo pipeline: policy decomposition into shared and agent-specific components, SND computation, and rescaling toward the desired diversity. — DiCo policy decomposition, SND estimation, and rescaling. From Bettini et al. (2024).

Why does the choice of target matter?

DiCo enforces a diversity target but does not choose it. In Navigation with three agents and three goals, a target of SND 1 achieves the highest reward and the shortest episodes, as agents cover all goals. A target of 0 forces identical behavior and plateaus at approximately half the reward, while 0.5 learns more slowly. The best target differs across tasks, which motivates selecting it automatically with ESC.

SND 0

SND 0.5

SND 1

Mean reward for DiCo with fixed targets of 0, 0.5, and 1. — Navigation, three agents and three goals, with DiCo fixed at SND 0, 0.5, and 1: mean reward, episode length, and measured diversity.

Episode length for DiCo with fixed targets of 0, 0.5, and 1. — Navigation, three agents and three goals, with DiCo fixed at SND 0, 0.5, and 1: mean reward, episode length, and measured diversity.