Judges are biased too
Self-improvement loops and evaluation gates mostly use an LLM as the judge. The problem is that the judge itself is systematically biased. Trusting a biased judge distorts the very direction of improvement, so the judge must be calibrated before use.
Five biases are commonly observed: position bias favoring the first-shown answer, verbosity bias rating longer answers higher, self-preference bias favoring answers from the judge's own model family, format bias favoring certain formats, and calibration drift where standards wander over time.
Fixing each bias
Each bias has a known fix. Cancel position bias by evaluating both (A,B) and (B,A) orders; counter verbosity bias with a 1-4 scale that rewards conciseness. Mitigate self-preference by rotating judges across different model families, and catch calibration drift by running baseline calibration on a schedule.
Full guide: from planning to operations
In planning, define judge reliability as numbers. For example, set a target agreement with human labels, a verdict-flip rate on order reversal of 5% or lower, and a ceiling on the score standard deviation of baseline items. Pin the judge contract (prompt and scoring rubric) so it scores the same way for the same input. Constrain scoring to a rubric narrowed to factual criteria rather than free-form to raise reproducibility.
A common failure pattern is trusting the judge without measuring bias. To prevent this, apply corrections mechanically on production traffic: shuffle order on every pairwise call, pin the judge contract, rotate judges across families, and calibrate continuously with baseline items. For recovery, if the judge scores baseline items differently than before, treat it as drift and recalibrate or swap the judge. When viewing several metrics at once, apply multiple-comparison correction to filter chance significance.
On the operations checklist, make the judge's own reliability a measured target. Recent work treats the judge as a measurement instrument, applies item response theory to its reliability, and proposes calibration-based bias correction and confidence intervals that account for the judge's sensitivity and specificity. Log judge version, contract hash, order, baseline scores, and human agreement in standard fields. As observability fields, record order-flip rate, inter-judge agreement, baseline drift, and correlation with humans.
The continuous improvement loop analyzes cases where judge and human diverged weekly. Types the judge repeatedly gets wrong get their rubric reinforced or are added as baseline items. Judge calibration should be an operational activity that keeps adjusting to absorb drift, not a one-time setup.
Key takeaways
In short, calibrate the LLM judge before use. Diagnose position, verbosity, self-preference, format, and drift biases, and apply order shuffling, a conciseness-rewarding scale, cross-family judge rotation, and baseline calibration mechanically in production. Treat the judge as a measurement instrument and observe its reliability continuously so a biased judge does not distort the direction of improvement.