Production Rose, Public Records Thinned: A Simulation Review of Agentic Coding in Open-Source Communities

Source

Mengying Zhou, Yongjie Yin, Yang Chen, "From Social Coding to Agentic Coding: Productivity and Relational Reconfiguration in Open-Source Communities", arXiv:2608.03585 [cs.AI, cs.CY], submitted 2026-08-04, License CC BY-NC-SA 4.0. Author affiliations, marked only by superscripts in the original, are not present in the copy this review cross-checked.

This is a preprint that has not been peer reviewed. Within the scope this review could verify, no funding source or conflict-of-interest disclosure appears, and the paper states only that "code and corresponding data will be released later," so the reproduction path is currently closed. The point to state first is that none of the figures below come from observed reality — they come from an LLM-based multi-agent simulation, initialized with the GitHub history of 1,084 real developers, in which LLM agents act out how they would behave if a coding agent were available.

The full text was cross-checked against a snapshot the collector script (fetch_source_snapshot.py) fetched from the arXiv HTML page at 2026-08-10T12:31:59Z (docs/research-authoring/snapshots/2026-08-10/2608.03585.txt, sha256 861ebaa2…), because this run's session egress proxy was fully blocked.

What the paper asks

Three questions. RQ1: how do community productivity and efficiency change once a coding agent (CA) is introduced, and how evenly does adoption spread? RQ2: does task execution shift from public human interaction toward private human-agent collaboration, and what short-term relational change accompanies it? RQ3: does the public record produced under agent involvement remain useful to later developers?

The method is counterfactual simulation, not observation. From a real GitHub developer dataset, the authors selected 1,084 developers with at least one commit every week from January 21 to March 18, 2018, at least 50 historical commits, and contributions to three or more non-fork repositories. The pipeline runs Initialization (backgrounds built from historical records) → Warmup (four weeks of real commits injected as few-shot ICL to restore recent behavioral patterns) → Simulation (branching the same community snapshot into No-CA and CA conditions for four weeks). Under CA, developers who used at least seven languages and logged at least 15 commits during warmup are pre-designated "seed adopters" and exposed first; subsequent diffusion follows each developer's Innovation Diffusion Theory (IDT) category. The primary experiment runs DeepSeek-V4 with three independent runs per condition; GLM-5.2 and Qwen3.7 each get one paired cross-check run.

Simulation validity was checked against real activity: developer-day MAE is 1.86 and RMSE 3.80, and mean total activity per developer is 31.0 (real) versus 32.2 (simulated), with medians 28 versus 25. This validation only confirms the simulation reproduces existing activity patterns — it says nothing about whether developers would actually behave this way under the counterfactual condition of CA availability.

Results

Productivity and adoption metrics show a clear split: task completion and speed improved sharply, while adoption itself stayed a minority behavior.

Adoption capping at 26.0% means 74.0% of developers never used a CA by the end of the simulation. Yet the CA-assisted commit share climbed to 65.0%, because the minority who adopted used it intensively and captured a disproportionate share of community output. The paper reports that per-capita task counts also diverge sharply by CA-awareness status, reading this as a "participation amplifier" effect concentrated among developers who were already active and well connected.

Task-execution pathways are also restructured. Splitting completed tasks into HHI (direct human-human), HSA (human solo), AHI (agent-mediated collaboration), and ASA (agent solo loop), No-CA contains only HHI and HSA (32.4% / 67.6%), while under CA the split becomes HHI 11.6%, HSA 31.2%, AHI 17.0%, ASA 40.3%. AHI+ASA (agent-involved) sums to 57.3%, with ASA alone the single largest mode in the table. Notably, cross-developer work itself (HHI+AHI) barely shrinks — from 32.4% under No-CA to 28.6% under CA — meaning direct collaboration didn't disappear so much as 59.9% of it moved from direct conversation to agent mediation.

At the developer-pair level, interaction breadth and short-term continuity stay largely intact after CA introduction (table above) — the paper reads this as evidence that relationships themselves have not yet weakened. The problem lies in the record those relationships leave behind. On a standardized retrieval benchmark (8,822 real commits from March 19-May 19, 2018, unseen by the agents), the real-human corpus reaches 81.1% knowledge coverage, while a size-matched CA corpus averages only 22.3% ± 2.2pp across five repeated samples — a relative drop of 72.5%. Average retrieval steps rise from 2.63 to 8.02 ± 0.10, and retrieval success falls from 82.3% to 22.3% ± 1.4pp.

A basis mismatch worth flagging — the appendix's cross-model robustness table (Table 2) reports, for a single DeepSeek-V4 run, knowledge coverage of 67.6% (real) → 13.6% (CA) and average retrieval steps of 3.54 → 8.75. Despite using the same model, these absolute figures differ from the main-text headline (81.1% → 22.3%, 2.63 → 8.02). The main text explicitly states a size-matched, five-repeat procedure; the appendix table does not state whether it size-matches or how many runs it averages, so the two tables appear to follow different procedures. Cite them separately and never blend the two. Completed-task growth also diverges sharply by model — +86.1% for GLM-5.2 and +17.3% for Qwen3.7 against +39.0% for DeepSeek-V4 — so while direction (higher productivity, capped adoption, more agent mediation, lower knowledge coverage) holds across all three models, magnitude varies by up to 5x.

Credibility

The strength lies in initialization. Backgrounds are built from 1,084 real GitHub developers (369,645 commits across the full analysis window), recent behavior is restored via few-shot ICL during warmup, and validity is quantified with MAE, RMSE, and Gini-deviation statistics. Reproducing the direction of results with GLM-5.2 and Qwen3.7 alongside DeepSeek-V4 is also a plus — all three models agree on higher productivity, capped adoption, expanding agent mediation, and falling knowledge coverage.

The caveats stem from the method's fundamental nature. First, this is not observed reality but a simulation of how LLM agents infer they would behave if a CA existed; the validity check only confirms reproduction of existing activity patterns, not behavior under the counterfactual condition itself. Second, the diffusion mechanism pre-designates seed adopters by language-count and commit-count thresholds and encodes Innovation Diffusion Theory directly into the adoption rule — the finding that gains concentrate among already-active developers may partly reflect an assumption built into the design rather than an emergent discovery. Third, code and data are not yet released, and no funding or conflict-of-interest statement was found. Fourth, the observation window is short (a 4-week simulation atop an 8-week total timeline including warmup), which the paper itself flags as limiting analysis of longer-term community evolution. Fifth, on conflicting evidence: prior work the paper cites (Ziegler et al. 2024, measuring real-world GitHub Copilot productivity) reports individual productivity gains from observed data but does not address public-knowledge loss — this paper attempts to fill that gap with simulation, not with observation.

Reviewer's take

First, the headline is "productivity up 39%," but the heavier signal, in my judgment, is that this much knowledge loss already shows up at 26.0% adoption. If public-knowledge coverage collapses from 81.1% to 22.3% with only a quarter of the community on CAs, the loss at higher adoption is territory this four-week simulation never covers.

Second, the adoption-inequality ("participation amplifier") finding is interesting but arguably circular. Seed adopters were pre-designated by researcher-set activity thresholds, and Innovation Diffusion Theory was written directly into the diffusion rule — so "already-active developers adopt first and most" reads less like a discovery and more like a reproduction of an assumption baked into the design. It's safer to carry this into practice as a hypothesis about how a particular rollout mechanism could unfold, not as an observed fact.

Third, the mismatch between Appendix Table 2's DeepSeek-V4 baseline and the main-text headline is something the paper itself does not flag. A robustness check is normally meant to confirm the conclusion holds when only the model changes under otherwise identical conditions; if the baseline itself moves across procedures, what the check is actually verifying becomes unclear. Directional agreement is valid evidence, but absolute numbers should only be quoted from the main-text table (five-repeat, size-matched).

Fourth, the coincidence that knowledge-coverage decline (81.1%→22.3%) and retrieval-success decline (82.3%→22.3%) land on nearly the same number is likely because both metrics share the same TF-IDF cosine-similarity 0.3 threshold and a substantially overlapping procedure. Read them cautiously as two expressions of one measurement, not as two independent pieces of evidence.

Applying it

• Mandate a separate public-record step — even for agent-handled work, standardize a process for leaving human-readable decision rationale, alternatives considered, and a final diff summary in the issue or PR description. What the simulation loses isn't the code — it's the narrative that leads to it.
• Re-examine newcomer onboarding paths — if a large share of newcomers rely on search alone to grasp project context, strengthen separate onboarding material (wikis, tutorials) after CA rollout to reduce dependence on the public commit log.
• Monitor adoption inequality on a cadence — track CA usage broken out by activity tier to check whether the productivity tool is concentrating only among already-active contributors.
• Pair simulated predictions with real measurement — treat this paper's absolute numbers as a directional hypothesis only; when rolling out a CA in a real organization, track issue/PR comment volume and knowledge-base reuse rate as observed metrics and compare against the simulation's predictions.
• Manage model-dependent magnitude as a range — don't adopt a single "+X% productivity" figure as a target. Within this paper alone, the range spans 17.3-86.1% depending on the model.

Conclusion

This paper's contribution isn't the already widely reported fact that coding agents raise productivity — it's dissecting where that productivity comes from. While completed tasks rose 39% and completion time fell by more than half, a substantial share of that work moved into human-agent loops that leave no public record, and as a result, the retrievability of public knowledge for newcomers dropped to roughly a quarter of its prior level. Given the methodological limits of an LLM-based simulation, the circularity in the seed-adopter design, and effect sizes that vary up to 5x by model, the higher-value move is not to port the absolute figures into an organizational target but to carry over the underlying problem framing and measurement approach: productivity metrics alone miss something when evaluating coding-agent rollout. A similar gap between raw output and delivered value is covered from another angle in Code Up 180%, Releases Up Only 30%.

References

• From Social Coding to Agentic Coding: Productivity and Relational Reconfiguration in Open-Source Communities — arXiv abstract (primary source)
• Full HTML text of the same paper — used to cross-check results tables, the appendix robustness check, and limitations (via snapshot)
• Code Up 180%, Releases Up Only 30%: QA Gate Design for IDE Agent Testing — sunny34.com blog