100% vs 25%: Reviewing PayPal's T2MO Framework for Coding-Agent Cost Routing

Source Document

Srinivasan Manoharan, Junhua Zhao, Fangbo Tu, Haifeng Wu, Jian Wan, Maliah Rajan M, Ashwin Hegde, Mithun Sasidharan, Kalyan Chakravarthi Podamekala, "Task-to-Model Optimization for Enterprise LLM Coding Assistants: A Data-Driven Framework for Cost-Optimal Routing", arXiv:2608.08528 [cs.LG], submitted 2026-08-09, License: arXiv.org perpetual non-exclusive license.

This is a preprint without peer review (classified cs.LG, though the content reads closer to an industry system-design report grounded in a production deployment). All nine authors use PayPal email addresses (@paypal.com), and the "motivating enterprise environment" the paper describes reads, in context, as PayPal's own deployment. We found no separate funding source or conflict-of-interest disclosure within the scope we checked. The full text was verified against a snapshot GitHub Actions collected from the arXiv HTML page at 2026-08-12T22:07:13Z (docs/research-authoring/snapshots/2026-08-13/2608.08528.txt, sha256 8b7d77df…), because this run's session egress proxy was fully blocked — including the unrelated control target example.com — and stayed blocked after three retries at 10-minute intervals.

Study Overview

The authors don't number formal research questions, but the motivating problem is clear: when enterprises try to cut coding-assistant inference spend by shifting traffic to models with lower token prices, end-to-end cost — including retries, escalations, and developer wait time — can actually rise. T2MO (Task-to-Model Optimization) addresses this with a nine-stage pipeline built from production session telemetry: instrumentation → Clio-style bottom-up clustering to discover a task taxonomy (weighting nodes by cost share rather than request count) → difficulty grading by an LLM judge (calibrated against at least 300 human-labeled sessions, requiring human-human Cohen's kappa of at least 0.6 and judge-human agreement of at least 0.7) → distribution-matched benchmark construction inside the same agentic harness developers actually use (at least 30 tasks per cell) → candidate model evaluation across four dimensions — task completion, cost, latency, operational fit → deriving the cost-optimal model mix → a rolling 12-month spend forecast → staged rollout from static policy to shadow classifier, verified cascade, and intelligent router → governance. The core formula defines the expected cost of routing to a candidate model as "candidate cost + (1 − pass rate) × (incumbent cost + developer wait cost)" (Eq. 1), and proves mathematically that this rule never produces worse realized cost than naive token-cost minimization (Proposition 1). A candidate only displaces the incumbent once its pass rate clears the "routing boundary" derived in Eq. 2.

Key Results

Monthly coding-assistant inference spend in the motivating environment is roughly $3.05M, with about 98% concentrated in two frontier-class models. Clustering production traffic by cost share yields these top five task categories:

The paper benchmarks two subcategories from this taxonomy — Frontend & UI Development (7.1% of total spend) and Git Workflow & Repository Management (5.4%) — inside the actual production agentic harness, measuring pass rates for the candidate model GLM 5.2.

The same model's pass-rate curve bends in opposite directions across the two subcategories. Git Workflow holds 100% through Medium and only drops to 47% at Difficult, while Frontend & UI already halves to 50% at Medium. Because only cells that clear the routing boundary (Eq. 2) get displaced, just one Frontend cell (Easy) qualifies, versus two Git Workflow cells (Easy and Medium). The paper lists monthly savings for these displaced cells as $26K, $20K, and $26K respectively — and re-deriving the formula (e.g. $3.05M × 7.1% × 35% × 0.35 ≈ $26.5K) matches those figures exactly. But the difficulty split (35%/45%/20%) and displacement fraction (δ=0.35) feeding that calculation are values the paper itself labels "an illustrative placeholder pending the difficulty grader's actual per-cell output" — the pass rates and subcategory cost shares are measured, but the dollar savings are a worked example of the methodology, not a confirmed deployment outcome.

Credibility Assessment

Three things support trust here: pass rates were measured inside the same agentic harness developers actually use rather than via raw API calls; the routing rule itself carries a mathematical proof (Proposition 1) that it never beats out to worse cost than token-cost minimization; and the numbers are grounded in real production data (roughly $3.05M/month, category-level cost shares). The caveats are real too. There's no peer review, and all authors appear to work at the company that operates this framework, creating an apparent conflict of interest with no separate disclosure. The authors themselves acknowledge in Section 17 ("Limitations") that this is a single-enterprise case study without live A/B results or ablations. Contracted per-token prices are deliberately withheld, so readers cannot independently verify a key cost input, and as noted above, the headline savings figures are worked examples rather than measured outcomes — easy to miss on a quick read of the table.

Reviewer's Judgement

First, in this reviewer's judgement, the weakest link in the paper is how lightly the illustrative nature of the headline savings is flagged — tucked into a table caption rather than the main narrative. A single assumed coefficient, δ=0.35, drives the entire savings figure, and nowhere in the text is that value grounded in measurement. Measured figures (pass rates, cost shares) and assumed ones (difficulty split, displacement fraction) sit side by side in the same table row, which makes it easy for a fast reader to conclude "$72K/month in confirmed savings" when that isn't what was shown.

Second, the paper never states outright that all authors work at the company running this framework, but the email domain and spend scale make it effectively self-evident in context. That said, explicitly acknowledging the absence of failure analysis and A/B results in Section 17 is unusually candid for an industry paper — a caveat worth reading alongside, not instead of, the trust it earns for the authors' transparency.

Third, independent of the dollar figures, the observation that pass rates for the same model swing in opposite directions by task category and difficulty (Git Workflow 100%→47%, Frontend 100%→25%) is measured, and the conclusion that coding tasks shouldn't be routed as a single bucket holds regardless of how much weight you put on the savings numbers.

Putting It to Work

• Bake retry and escalation cost into the routing formula explicitly — comparing token prices alone overrates cheaper models with high failure rates. Compare expected cost as in Eq. 1, adding (1 − pass rate) × (incumbent cost + wait cost).
• Instrument a task × difficulty grid, not a single category level — as these two subcategories show, the same model's pass rate can bend in opposite directions by difficulty.
• Separate measured from assumed figures before reusing a vendor's or paper's savings claim — don't cite coefficients like δ or a difficulty split as confirmed until you've replaced them with your own telemetry.
• Evaluate candidate models inside your actual deployment harness — single-shot API performance can diverge from real usage patterns involving retries and tool calls.
• Ship override controls, kill switches, and retry/revert telemetry from stage one — move from static policy to shadow classifier to verified cascade in stages, but let developers pin back to a specific model at any time.

Conclusion

T2MO argues for assigning enterprise coding-assistant traffic to models based on expected completion cost at the task-category-by-difficulty level rather than token price, and backs that argument with measured pass-rate data that swings in opposite directions across subcategories for the same model. But the specific savings figures the paper presents are a worked example built on assumed coefficients, not a validated deployment outcome — worth adopting the methodology while re-verifying the numbers against your own data. For a related angle on evaluating model performance through a cost lens, see The Highest-Success-Rate Agent Was Also the Most Expensive: Designing Performance Checks Beyond Accuracy.

References

• Task-to-Model Optimization for Enterprise LLM Coding Assistants — arXiv abstract (source)
• Full HTML text of the same paper — used to verify the results tables (via snapshot)
• The Highest-Success-Rate Agent Was Also the Most Expensive: Designing Performance Checks Beyond Accuracy — sunny34.com blog