Chapter 8. Observability and Evaluating Swarm Quality
A you cannot trace is a swarm you cannot improve: per the taxonomy, 41–86.7% of open multi-agent-system runs end in failure, and evaluating by the final result alone overstates quality by 20–40%. and are not operational hygiene but the swarm's primary feedback loop. As Braintrust puts it: "the teams that win are the ones that close the loop between a production failure and a fix the fastest."
The Trace Matters More Than the Result
The 2026 consensus is to evaluate the trajectory, not just the outcome. judged by their final answer "pass 20–40% more tests than a full trajectory evaluation shows" (Confident AI, 2026). The reason lies in the nature of the failures: (arXiv:2503.13657, 1,642 annotated across 7 frameworks, NeurIPS 2025) shows that the largest failure modes — step repetition (15.7%), divergence between reasoning and actions (13.2%), being unaware of termination conditions (12.4%) — live inside the trace and are masked by a plausible final output. A separate warning: incorrect and incomplete verification together account for 17.3% of failures — a weak verifier agent is itself a source of failures, not a defense.
The HAL meta-study (Princeton, arXiv:2510.11977: 21,730 runs, 9 models × 9 , ~$40k of compute) adds two sobering facts: raising reasoning effort _lowered_ accuracy in most runs, and inspection of logs caught agents that "search HuggingFace for the benchmark instead of solving the task." HAL's conclusion: no score is interpretable without the full and logs.
The practical loop for trajectory : checks of tool selection and its arguments (not an LLM judge), plus an "every failure becomes a new regression test" cycle. The typical form in CI — about 30 golden-replay cases per PR, a run under 5 minutes, and a merge block when metrics regress (Confident AI CI/CD guide).
flowchart LR
A[Production failure] --> B[Human review of the trace]
B --> C[New eval case]
C --> D[CI regression suite]
D --> E[Merge blocked on regression]
E --> ABenchmarks in 2026: What to Trust
SWE-bench Verified is dead as a frontier signal: in February 2026 the OpenAI Frontier team stopped reporting it after an audit found defects in roughly 60% of the 138 hard tasks, and — crucially — models reproduced the gold patches verbatim from the task ID alone, a clean fingerprint of contamination (Digital Applied). The replacement — SWE-bench Pro (Scale AI, ICLR 2026): the same models scoring 81% on Verified drop to 46–58% (CodeSOTA). The second confounder is scaffolding: the same model swings ±12 points depending on the . The lesson for a : a score is the score of the "model + harness" pairing, and quoting a number without specifying the harness is meaningless.
This is where the open part of the chapter ends. The rest opens with a free registration — no payment, just an email address.