Right About Failure, Wrong About Which Collaboration Pays: A Cost-Aware Protocol Routing Review

Source Document

Chih-Hsuan Yang, Jingyan Jiang, Cheng-Hau Yang, Vikram Vasudevan, Huihuo Zheng, Venkatram Vishwanath, Rajeev Thakur, "LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks", arXiv:2608.14927 [cs.AI], submitted 2026-08-14, license CC BY 4.0, DOI 10.48550/arXiv.2608.14927. Affiliation: six authors at Argonne National Laboratory, one at Oregon State University. Funding: U.S. Department of Energy, Argonne Leadership Computing Facility (Contract No. DE-AC02-06CH11357). The full text was checked against a primary-source snapshot collected 2026-08-19T21:45:37Z (via GitHub Actions) — this session's own WebFetch access was blocked (EGRESS_BLOCKED) across three 10-minute retries, so the snapshot fallback procedure was used.

This is a preprint without peer review. All authors work at a government national laboratory or a university rather than a commercial model vendor, which lowers the incentive to favor a particular product, and funding comes through a DOE compute-facility allocation rather than corporate sponsorship. Worth noting: the solver used throughout, gpt-oss-120b, is an open-weight model released by OpenAI. The paper positions itself against a related but distinct literature — model routing and cascades (RouteLLM, FrugalGPT, CARROT and similar) that choose which model to use — while this work asks a different question: given the same model, which collaboration protocol is worth escalating to.

Study Overview

Two questions drive the paper. Can a model predict when its own direct answer is likely wrong? And separately, does that same signal tell you which collaboration protocol is worth its added cost? The authors argue these are not the same question, and treating them as one is a mistake.

The primary benchmark is 4,181 competition-level math problems (Omni-MATH 2). Every problem is run under all four protocols with the same gpt-oss-120b solver — direct solving (Baseline), iterative self-correction (Single), planner-executor-reviewer collaboration (PER), and multi-agent deliberation (Broadcast) — producing a fully matched dataset. Because every protocol's outcome is observed for every problem, the paper can compute a retrospective fixed-order oracle (the cheapest protocol that would have worked) and compare real routers against that upper bound offline. A robustness check extends the design to JEEBench (engineering), SciBench (college science) and LAB-Bench (biology), plus a second solver (Gemma-4-31B-it), for ten model-condition settings in total.

Key Results

On the 423-problem held-out test split, representative policies trade off solve rate against token cost as follows. Excess is the mean token overpayment relative to the realized oracle.

Two very different approaches land near the same ~78% solve rate at very different cost: the self-confidence gate spends 45.0K tokens, roughly two-thirds of the frozen router's 71.3K. Even so, the gap to the retrospective oracle (92.4%) stays at 14.4 points, and across six held-out router evaluations that gap to the oracle never closes below 18.5-28.9 points.

Router errors also run in opposite directions. Tier-majority under-escalates on 27.4% of problems and over-escalates on 12.5%. The gpt-oss-120b router cuts under-escalation to 18.0% but pushes over-escalation up to 33.3%, and the higher-solve Llama and Gemma frozen routers reduce under-escalation to 6-11% while over-escalating on 63-71%. A "smarter" router, in other words, is not automatically a cheaper one.

Failure-risk prediction and collaboration-value prediction are measured on different bases. Applying the same post-answer gpt-oss-120b confidence score to increasingly specific targets:

AUROC declines gently (0.88 down to the low 0.70s), but AUPRC drops sharply from the 0.90s into the 0.10s. Part of that is falling prevalence (PER-first success occurs in only 8.8% of cases), but the underlying message holds regardless: the same confidence score is useful for "this will probably fail" and much weaker for "and specifically, PER is worth it."

Credibility Assessment

Three things earn trust here. The matched design runs every problem through all four protocols, so router comparisons are not confounded by differences in solver capability. Robustness is checked across four benchmarks of quite different character (math, engineering, science, biology) and two solver families, for ten total settings. And the authors are explicit about their own limits — the bootstrap intervals reflect variation across problems, not run-to-run reproducibility, and the oracle is a diagnostic ceiling rather than a deployable policy, both stated plainly in the paper.

The caveats matter too. Every problem-protocol pair is a single deterministic run at temperature 0, so run-to-run variance is not measured. Cost is tracked only in tokens; the authors explicitly note that latency, monetary price and parallelism are out of scope. The oracle's cost ordering (Baseline

Reviewer's Judgement

First, the paper's real contribution is less the specific numbers and more the act of splitting "will this fail" from "which collaboration is worth paying for" into two distinct problems. Teams often collapse both decisions into a single confidence score; the AUROC 0.88 versus AUPRC 0.10-0.17 gap puts a number on why that habit is risky.

Second, the self-confidence gate (78.0% at 45.0K tokens) is one of the few results here that is directly actionable today. But it is strictly a binary Baseline-versus-Single decision — it does not decide whether to go further to PER or Broadcast, and it would be a mistake to read it as a full four-protocol router.

Third, an 18.5-28.9 point oracle gap surviving in held-out evaluation reads as a clear "not yet" signal against promoting a learned router to fully autonomous production decisions. Until that gap narrows, wrapping the router's escalation calls in human review or A/B verification is the safer path.

Putting It to Work

• Start with a binary gate — before automating expensive protocols like PER or Broadcast end to end, deploy a cheap binary gate (confidence-based direct-answer-vs-one-more-pass) first. Per this paper, that gets a comparable solve rate at 45.0K tokens versus 71.3K for a full router.
• Instrument failure-risk and collaboration-value separately — do not monitor "probability the answer is wrong" and "probability this specific protocol is worth its cost" under the same metric. Track protocol-specific value with prevalence-sensitive measures like AUPRC.
• Measure your own router's directional bias — conservative policies tend to under-escalate (missing solvable problems); aggressive frozen-LLM routers tend to over-escalate (paying for unnecessary collaboration). Measure both error rates on your own data before trusting a vendor default.
• Track cost beyond tokens — the paper itself flags tokens-only cost measurement as a limitation. Latency, API spend and parallelism headroom all need their own monitoring before a routing decision is deployment-ready.
• Double-check escalation to expensive protocols — route promotion decisions into PER- or Broadcast-level collaboration through human review or sampled A/B checks rather than a fully autonomous router. The 18.5-28.9 point oracle gap quantifies exactly the risk this step is meant to catch.

Conclusion

What this paper delivers honestly, in one experiment, is both a working low-cost confidence gate and proof that the same gate does not solve which collaboration protocol to escalate to. A binary gate reaching 78.0% solve at 45.0K tokens is worth piloting now; fully automating promotion to PER or Broadcast is not, given how much of the oracle gap remains unclosed. The same tension shows up when sizing execution budgets for subagent workflows — see Subagent Workflow Budget and Resume Design for the operational side.

References

• LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off — arXiv abstract (source)
• Full HTML text of the same paper — used to verify tables and AUROC/AUPRC figures (via snapshot)
• Subagent Workflow Budget and Resume Design — sunny34.com blog