How does mutual shaping through diverse training compare to population-level diversity effects?
This explores two different senses of 'diversity' in training — diversity that agents create by shaping each other during training (mutual/role-based specialization), versus diversity (or its loss) measured across whole populations of models or outputs — and what the corpus says about how they relate.
This reads the question as a contrast between two scales of diversity. 'Mutual shaping through diverse training' is the small-group story: agents that improve each other while learning, like generation-and-critic pairs trained on distinct role-dependent data so they don't collapse into the same behavior Can multiple agents stay diverse during training together?, or critique steps folded into the training loop that keep the solution space wide instead of letting it narrow to a single winning path Do critique models improve diversity during training itself?. 'Population-level diversity effects' is the wide-angle story: what diversity looks like when you survey many models or many outputs at once. And here the corpus delivers an uncomfortable surprise — across 70+ models and 26K open-ended prompts, different LLMs independently converge on near-identical answers, an 'Artificial Hivemind' that quietly cancels the benefit you'd expect from ensembling diverse models Do different AI models actually produce diverse outputs?.
The striking thing is that these two scales pull in opposite directions, and the reason is the same underlying force. Mutual shaping works precisely because it engineers *difference into roles* — remove the critic or the summarizer and the system overfits and stalls Can multiple agents stay diverse during training together?. Population-level convergence happens because nothing engineers that difference: overlapping training data and shared alignment procedures push independently-trained models toward the same attractor Do different AI models actually produce diverse outputs?. The default of training is collapse — RL converges on one dominant pretraining format within an epoch Does RL training collapse format diversity in pretrained models? and squeezes exploration breadth in search agents through the same entropy-collapse seen in reasoning Does reinforcement learning squeeze exploration diversity in search agents?. Mutual shaping is one of the few mechanisms that actively resists that default.
But population-level convergence isn't always the villain — and this is the lateral twist worth sitting with. When you train one model on *many diverse imperfect experts*, the convergence becomes the feature: cross-entropy optimization makes the model implicitly majority-vote, denoising the uncorrelated errors of individual experts and beating any single one Can models trained on many imperfect experts outperform each one?. So the same word 'convergence' is a failure when it erases diversity between independent models, and a success when it distills diversity from a population of teachers into one model. The difference is whether the diversity was present *upstream* and got integrated, or was never there and got faked.
This suggests the real comparison isn't mutual-shaping versus population-effects, but *where* you place the diversity relative to the learning. Diversity has to be deliberately injected — as roles between co-trained agents Can multiple agents stay diverse during training together?, as a semantic-diversity reward that catalyzes exploration and raises quality at the same time Can diversity optimization improve quality during language model training?, or as varied competent solutions emitted for a downstream search procedure to recombine Should training maximize diversity when models feed into search?. Left alone, training homogenizes; and homogenization at the population scale is what produces the hivemind. Two notes complicate any clean rule, though: preference tuning's effect on diversity actually *reverses* by domain — it compresses code but expands creative writing Does preference tuning always reduce diversity the same way? — and even rich multi-agent diversity backfires without genuine expertise behind it, where cognitive stimulation without competence becomes process loss rather than insight Does cognitive diversity alone improve multi-agent ideation quality?.
The thing you didn't know you wanted to know: diversity isn't a quantity you have more or less of — it's a thing that has to be *placed*. Put it between co-training agents and it compounds into better reasoning; let it sit only across a field of separately-trained models and it silently evaporates into consensus; pull it from a crowd of flawed teachers into one student and it becomes wisdom.
Sources 10 notes
Training generation and critic agents on distinct role-dependent data prevents the overfitting collapse that limits single-agent finetuning to one productive iteration. Removing critics or summarization degrades performance, confirming both components are critical.
Step-level critique in the training loop counteracts tail narrowing and maintains solution diversity across self-training iterations. This training-time benefit—preventing premature convergence—is more fundamental than test-time accuracy gains.
INFINITY-CHAT analyzed 70+ models across 26K open-ended queries and found an "Artificial Hivemind" effect: models independently generate strikingly similar or identical responses due to overlapping training data and alignment procedures, undermining the diversity benefits of model ensembles.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Generative models trained on many diverse experts with different biases converge toward consensus behavior through cross-entropy optimization. Low-temperature sampling reveals this implicit majority vote, which outperforms any single expert by denoising uncorrelated individual errors on critical decision states.
DARLING jointly optimizes for quality and semantic diversity using a learned classifier, finding that diversity rewards catalyze exploration and produce higher-quality outputs than quality-only baselines across both creative and mathematical tasks.
Vector Policy Optimization trains models to emit varied competent solutions rather than converging to one answer. This unlocks search procedures like evolutionary algorithms to explore and combine modes, solving problems that entropy-collapsed policies cannot reach at all.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
Multi-agent teams substantially outperform solo ideation, but only when members possess genuine senior knowledge. Diverse teams without expertise underperform even a single competent agent, because cognitive stimulation without expertise triggers process losses instead of insight.