What the Average Success Rate Hides on Long Tasks

Alibaba Cloud's Wuying-Browser-Agent paper introduces BrowserBench, a new evaluation of 350 real-world website tasks (37.9 steps on average, across 254 sites, bilingual Chinese/English) built specifically to expose failure modes that shorter benchmarks miss. Even the top open-source model, a 27B checkpoint, only reaches 65.1% on BrowserBench — out of ten attempts, four don't finish. The strongest closed model, GPT-5.5, tops out at 67.4% on the same benchmark, missing 32.6 percentage points, or roughly one task in three.

That gap doesn't show up on shorter benchmarks like WebVoyager or Online-Mind2Web. For a deployment decision, the absolute 65.1% matters more than the "#1 open source" headline — an average alone hides exactly the long-horizon weakness you need to plan around.

Reinforcement Learning Pays Off More as Tasks Get Longer

Comparing before/after online RL (DAO-GRPO) by difficulty tier: easy tasks gained only +1.0pp, hard tasks gained +12.4pp, and tasks over 50 steps jumped from 13.3% to 26.7% (+13.4pp). The improvement concentrates where the model is weakest, not where it already performs well. The execution layer standardizes on 24 atomic actions across five categories — navigation, interaction, extraction, files, and flow control — which cuts failures caused by inconsistent action definitions rather than model capability.

From Design to Operations: A Long-Horizon Browser Agent Deployment Checklist

At the planning stage, set separate targets by task-length segment instead of one average figure: aim for 80%+ success on tasks under 25 steps and 50%+ on tasks over 50 steps, and write an explicit interaction budget cap (e.g., 100 steps) into your evaluation protocol, counting any overrun as a failure.

Failures split into two branches — budget overruns and mid-task drift. Route overruns to automatic termination plus a human confirmation request; when a stuck pattern (repeating the same screen) is detected, fall back to a safer alternate path — direct URL navigation or a fresh search — instead of blind retries. An ablation in the paper shows mixing recovery/reflection data too early in the fine-tuning curriculum actually hurts performance (35.1% vs. 38.0% with the curriculum order preserved), so treat data ordering — success trajectories first, recovery trajectories later — as an experiment variable in your own fine-tuning runs.

Your pre-launch checklist should mix short, medium, and 50+-step tasks in scenario tests and report success rate per segment, not just overall. If you score progress or success with an LLM judge, audit it separately against a small human-labeled sample and log the agreement rate — that's what separates a plausible-looking verdict from a verified one.

Standard log fields should include steps per task, failure type (budget overrun / stuck / misjudged), the judge's rationale, and latency; mask any personal data or credentials encountered on visited pages before storage, and restrict access to a minimal audit group. Reporting third-party benchmark scores (WebVoyager, Online-Mind2Web) alongside your own extends the same audit discipline — don't let a self-built benchmark carry the whole claim.

Weekly, track the top failure type per segment to prioritize retraining or fixes to action definitions where failure rates spike. Direct improvement budget toward the segments with the biggest gap — tasks over 50 steps, not the short tasks that already score well — to keep resourcing aligned with where the RL gains actually showed up.

Key Takeaways at a Glance

Long-horizon browser agents still have real limits: the top open-source model finishes only 65.1% of realistic tasks, and even the strongest closed model misses roughly one in three. Segment targets by step length instead of one average, count budget overruns as failures, sequence recovery data late in the curriculum, and audit your LLM judge against human labels — those four practices are what actually separate a deployable agent from a demo.

References

Wuying-Browser-Agent: Real-World Centric Fundamental Long-Horizon Browser Agents — arXiv

GPT-5.5 Still Misses One in Three: Reviewing Wuying-Browser-Agent — sunny34.com Research