Production traces are already an eval set
As of 2026, 89% of organizations have adopted agent observability and 62% have step- and tool-call-level detailed tracing. Yet most teams use traces only for after-the-fact debugging. When you promote risky traces (tool-call failures, latency overruns, user drop-offs) into a dataset with one click, they become regression tests that automatically flag whether the same input repeats the same failure on the next deploy. The key is elevating your real production distribution straight into a regression set, with no need to fabricate synthetic data.
Run LLM-as-judge on top of traces
Attaching offline and online LLM-as-judge to production trace datasets has become standard practice. Offline runs the judge prompt across the entire regression set before deploy and uses the pass rate as a gate; online samples live traces and scores them. Split the rubric into four axes -- correctness, tool-use appropriateness, instruction following, and safety violations -- and fix each axis to 0/1 or a 1-5 scale. Log the judge's verdicts as traces too, so you can re-audit the judge's own bias.
Bind cost to the quality gate
LangSmith has expanded from a plain observability tool into an agent engineering platform by adding Fleet and unified cost tracking. When you run the regression set, don't look at pass rate alone: log per-trace tokens, cost, and p95 latency alongside it, so a regression that doubles cost while chasing quality gets blocked on the same screen.
An A4-length guide: from planning to operations
Start planning by nailing requirements to numbers. Fix your target metrics at a regression-set pass rate of 95% or higher, zero safety violations, p95 latency of 4 seconds or under, and per-trace cost within +10% versus the prior quarter. Begin the regression set with 200-300 risky traces, and first agree in a document on the promotion criteria -- which labels and signals count as "risky."
Classify failure patterns in advance and code the recovery branches in. The main types are tool-call schema mismatches, infinite retries, context-overflow truncation, and judge-vs-ground-truth disagreement (the judge says pass but the answer is actually wrong). For recovery: on tool failure, retry with exponential backoff up to twice, then hand off to a human-confirmation queue; on latency overrun, return early with a safe, condensed response; route low-confidence verdicts to human review.
Make the stop conditions explicit. If any one of more than two retries, exceeding the cost ceiling, or detecting a safety violation triggers, halt immediately and return a fallback response. This is the last safety valve against infinite loops and cost blowups.
Operational checklists and quality control start from a standard log schema. Keep trace_id, step, tool, latency_ms, tokens, cost, and verdict as fixed fields. Before promoting to a dataset, mask PII such as emails, phone numbers, and addresses, and keep raw text under separate access controls. Pin the judge prompt version and calibrate the judge itself once a month against a golden set.
Run the continuous-improvement loop weekly. Fold low-scoring traces from online judging into the regression set, then re-run after a fix deploy to confirm the pass rate recovers. If judge-human agreement drops below 90%, redefine the rubric and refresh the golden set.
Execution summary
Promote risky traces into a regression set with one click, run offline and online LLM-as-judge on top of it, and build an operations loop that holds gates at 95% pass rate, zero safety violations, 4-second p95, and cost within +10%. With per-failure recovery branches, stop conditions, PII masking, and judge calibration in place, you can block quality regressions before they ship.