The Patch Always Applies, the Meaning Doesn't: Reviewing Kozuchi Agent

Source Document

Mehdi Bahrami, Kosaku Kimura, Satoshi Munakata, and 23 others, "Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair", arXiv:2608.15579 [cs.SE], submitted 2026-08-16, DOI 10.1145/3832783.3834531, licensed CC BY 4.0. Affiliation: Fujitsu Research (Japan, USA, Europe, China). Accepted to the Industry Showcase track of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE 2026), to be presented in Munich in October 2026.

This is a peer-reviewed industrial experience report — though the Industry Showcase track applies a lighter review bar than the research track, weighing the credibility of an industrial deployment story over academic novelty. All 26 authors work for Fujitsu, and the paper is Fujitsu's own researchers evaluating a coding agent Fujitsu built in-house — an unambiguous conflict of interest: the vendor grading its own product. To its credit, the paper maps this boundary itself, in a table distinguishing which claims rest on controlled ablations versus operational observation alone. Because this session's egress was blocked, the full text was cross-checked against a snapshot (a copy of the primary-source HTML) collected by GitHub Actions at 2026-08-18T21:42:47Z.

Study Overview

The research question: how far can a locally hosted, non-fine-tuned open-weight model go on SWE-bench-style issue resolution when paired with an auditable repair harness and cross-agent selection? The targets are Python SWE-bench Verified (500 instances) and Java Multi-SWE-bench (128 instances). The method has three layers. First, an eight-phase graph built on mini-swe-agent (reproduce → synthesize tests → localize → fix → verify → … → final report) structures long-horizon execution. Second, a small deterministic toolset — line tracing, caller discovery, guarded editing — cuts down edit-format failures. Third, the same model (Qwen3.5-27B) runs eight independent times (K=8); each run's self-generated bug-reproduction and regression tests are then cross-applied to every other run's patch to score and select the final submission. The design deliberately avoids hidden benchmark tests or a learned verifier — mirroring the fact that no hidden feedback exists at merge time in production.

Key Results

The tables below were checked against Table 3, 5, 7, and 8 of the full HTML text. The per-repository breakdown in Table 5 was independently summed and matches the 374/500 headline exactly.

The Python headline is the official cloud evaluator result. The largest repository, django/django (231 instances, 46.2% of the set), resolves at 76.6%, above the overall mean; the lowest, pylint-dev/pylint (10 instances), resolves at only 30.0% — the paper itself warns that low-count repositories should be read as failure clusters, not stable rank evidence.

The selector's contribution across the eight runs uses a different basis (cumulative vs. single-run) and is broken out separately.

The selector adds +14 instances over the order baseline and +35 to +37 over the single-run mean, reaching 92.2% of the oracle ceiling (376/408). The character of the remaining failures is clear: of 126 unresolved instances, 91.3% (115) are patches that apply cleanly but fail hidden tests — a semantic error — while zero instances fail because the patch format itself was invalid. 494 of 495 runs (99.8%) produced a syntactically valid patch.

Credibility Assessment

Grounds for trust: the same model and harness were transferred unchanged from Python to Java, and per-phase message share matched within ±5 points across languages, showing the harness effect is not language-specific. Internal arithmetic checks out — the 12-repository sum matches the headline exactly. A sensitivity sweep on the selector weights (w_B=0.3, w_R=0.7) is reported (at most 35/495 candidates and 5/500 outcomes shift across seven nearby weight pairs), supporting the claim that the weights were not tuned post hoc.

Caveats: all 26 authors work for Fujitsu, the company that built the system under evaluation. The industrial-impact claims (5→1 touch-points, ~75 to ~5 minutes per cycle) are explicitly labeled by the authors as "order-of-magnitude estimates," not measurements from a controlled developer pilot. Core mechanisms — the phase graph, handover, tool sandbox — are backed only by "operational signatures," not component-removal ablations, a boundary the paper's own evidence table (Table 1) concedes. "First among open-weight" does not mean best overall: the system is significantly behind two closed-frontier combinations (Sonar/live-SWE + Claude-Opus-4.5) by 22 instances, and its lead over the closest open-weight peer, Lingxi v1.5 + Kimi-K2 (+18 instances), is not statistically significant (p=0.054). No funding source is stated separately, but since every author is a Fujitsu researcher, internal research funding is the reasonable inference.

Reviewer's Judgment

First, I judge the most practically valuable number in this paper to be the failure breakdown, not the 74.8% headline. That 494/495 patches are format-valid while 91.3% of failures are still semantic suggests the bottleneck for adopting a coding agent has already moved from "can it produce a patch" to "is the patch correct." Further investment in format validation is likely past its point of diminishing returns.

Second, the authors report matter-of-factly that the selector reaches only 92.2% of the oracle ceiling (a 34-instance, 6.8-point "selector regret") — but I read that gap itself as the more actionable finding. 408 instances are solvable by at least one of eight runs, yet only 376 are actually picked: the distance between "generating more candidates" and "picking well" remains large. Investing in selection logic may pay off faster than scaling candidate count further.

Putting It to Work

• Invest in semantic checks, not format checks — once patch-apply reliability is already in the high 90s, the next lever is the quality of self-generated regression tests standing in for hidden tests, not linters or formatters.
• Cross-score candidates with independently generated tests — cross-applying each run's own tests to every other run's patch gives a pre-merge selection signal without any hidden feedback.
• Instrument rework loops separately — track cycles like CODE_FIX↔VERIFY_PATCH, and set an early-exit threshold where rework concentrates.
• Right-size candidate count (K) after a difficulty pass — 234 of 500 instances are solved by every one of the eight runs, so a cheap difficulty triage can cut compute by varying K per instance.
• Always pair a leaderboard "#1" with its subset — cite rank claims together with the comparison set (open-weight vs. overall, which language) to avoid overstatement.

Conclusion

Kozuchi Agent is not a new model but a case study in lifting an open-weight 27B model into the leaderboard's upper tier through an auditable harness and a low-cost selection mechanism. The real contribution is not the 74.8% headline but the diagnosis behind it: the remaining gap is semantic, not formatting. Given that every author has a stake in the outcome and several core mechanisms are reported without ablation, treat the numbers as a directional signal from Fujitsu's own operating environment, and validate the selector and phase graph independently on your own harness before adopting. For the cost trade-off between adding candidates and improving selection accuracy, see The Economics of Performance Checks.

References

• Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair — arXiv abstract (source)
• Full HTML text of the same paper — used to verify tables and arithmetic (via snapshot after a session outage)
• The Winning Agent Was Also the Most Expensive One — sunny34.com blog