An agent changes even when left alone
Even without changing code, an agent's quality shifts over time, because the input distribution changes, the base model updates, or the persona wavers as conversations lengthen. Drift detection catches these changes early and alerts the team to possible regression. Without seeing drift, you learn of gradually declining quality only after an incident.
The key is to distinguish the kinds of drift. Different causes need different responses.
Three kinds of drift
Data drift means a change in the input distribution; model drift means the model's predictive performance degrading. In LLMs, data drift often causes model drift. Added to these is persona drift, where the agent's tone and stance waver as conversations lengthen, which can be detected even black-box.
Full guide: from planning to operations. In planning, define drift alert criteria as numbers. For example, set a threshold on an input-distribution distance metric, a baseline quality-score drop of 1% or more, and criteria for spikes in refusal and retry rates. Run scheduled automated checks to compare current outputs against historical baselines and detect performance drift regularly. Fix the baseline to metrics from a stably operating period so the comparison target does not wander.
A common failure pattern is ignoring behavioral signals. Beyond the quality score, behavioral metrics like refusal patterns, retry frequency, and response-length distribution often foreshadow drift first. To prevent this, monitor outcome and behavioral metrics together and fire an alert when either breaches the baseline. For recovery, if drift is detected right after a base-model update, revert to the prior model version or recalibrate the prompt, and hold automatic improvement until the cause is confirmed.
On the operations checklist, keep baseline and comparison history. Log which metric deviated when and by how much, and against which baseline, in standard fields. As observability fields, use input-distribution distance, quality-score trend, refusal and retry rates, and persona-consistency score. Keep masking rules so data used for drift analysis contains no personal information.
The continuous improvement loop analyzes which signal foreshadowed drift earliest weekly. Raise alert priority for signals with high early-warning power and adjust thresholds for signals that often false-alarm. Drift detection should be a system that keeps adjusting to reflect real regressions, not a fixed threshold.
Key takeaways
In short, an agent drifts even without code changes. Distinguish data, model, and persona drift, detect early with scheduled baseline comparison and behavioral-metric monitoring, and respond to post-update regressions with rollback. Observe early-warning signals and keep tuning alerts to catch gradually declining quality before an incident.