Chapter 11. The Academic Frontier: What Science Says About Swarms

The headline result of the 2023–2026 academic scrutiny is unpleasant for enthusiasts: most of the measured "multi-agent advantages" reduce to three things — extra inference compute, answer diversity, and aggregation — and not to "collaboration" at all. A genuinely cost-matched baseline of chain-of-thought + self-consistency proved nearly unbeatable, and swarms win only in narrow but real niches: breadth-first search beyond a single context window, ensembles of heterogeneous models, and verification. Meanwhile the main cause of swarm failures is not model weakness but orchestration errors and the absence of result verification.

The Reference Point: Voting as the Toughest Baseline

The anchor result against which everything else is measured is "More Agents Is All You Need" (Li et al., TMLR, February 2024): LLM accuracy grows with the number of independently sampled agents combined by a simple majority vote — without debate, roles, or message-passing. An ensemble of ~15 Llama2-13B instances catches up with a single Llama2-70B on GSM8K, and the gain for the 13B model is +24 pp (0.35→0.59). The gain correlates with task difficulty; the experiments were pushed to ~40 agents.

The peak of the "collaboration beats the frontier model" claims was Mixture-of-Agents (Wang et al., ICLR 2025): a layered composition of open-source models scored 65.1% on AlpacaEval 2.0 against GPT-4o's 57.5% (the Together blog). An important caveat from the dossier: the result was not compute-matched, the leaderboard has since been deprecated, and later audits attribute most of the MoA effect to extra compute and aggregation.

The practical takeaway: any swarm design must first beat sampling-and-voting at an equal token budget. As shown below, that rarely succeeds.

The Rise and Fall of Multi-Agent Debate

The history of debate is a model example of how science tests a beautiful idea.

• 2023: Du et al. (ICML 2024) report that multi-round debate among LLM instances improves math, reasoning, and factuality.
• 2024: "Should we be going MAD?" (Smit et al., ICML 2024) — debate "does not reliably outperform" self-consistency and is fragile to hyperparameters.
• 2025: "Stop Overvaluing Multi-Agent Debate" (Zhang et al.) run 5 MAD methods across 9 benchmarks and 4 models: MAD "often fails to outperform simple single-agent baselines like CoT and Self-Consistency" while consuming noticeably more compute. The only durable lever is model heterogeneity (Heter-MAD).
• 2025: "Debate or Vote" (NeurIPS 2025 Spotlight) proves that the belief trajectory in a debate is a martingale: debate by itself does not change expected correctness, and most of the gains attributed to MAD come from plain voting.
• 2025: "Can LLM Agents Really Debate?" (Wu et al.) — what matters is the models' reasoning strength and the group's diversity; structural knobs (turn order, confidence visibility) barely matter, and majority pressure suppresses independent correction.