The eval set sets the direction of improvement
A self-improvement loop moves in the direction that raises eval-set scores. So the eval set is effectively the objective function of improvement. If it fails to reflect the real usage distribution, the loop optimizes hard toward something useless in practice.
The core risks are overfitting and data leakage. As improvement repeats, if the agent memorizes only the eval set's traits, scores rise while real quality stalls or worsens.
Three kinds of eval set
The golden set is a small number of core cases whose answers and criteria are fixed by humans; it is an unchanging quality baseline. The fixed set is kept for regression checks, and the rotating set injects recent real cases periodically to detect overfitting. Promote a change only when it does not worsen on all three sets at once.
Full guide: from planning to operations
In planning, set each set's size and promotion conditions as numbers. For example, require zero golden-set violations, a fixed-set gain of 2% or more, and a rotating-set drop within 1% all at once to promote. The eval set must mix difficulty and type evenly; too many easy cases let scores rise while missing hard failures. So you need a rule to prioritize hard cases from failure logs into the set.
The most common failure is data leakage. If the improvement agent indirectly references eval-set answers via prompt or memory, scores rise but that is cheating, not learning. To prevent it, physically separate eval data from data used for improvement, and inject the rotating set only through a path the improvement agent cannot access. For recovery, if the fixed set improves but the rotating set drops sharply after a change, judge it overfitting and roll back immediately.
On the operations checklist, include eval-set version control. Log in standard fields which cases were added or retired when, and which eval-set version verified each change. Mask personal data in the eval set and manage sensitive cases under separate access control. As observability fields, record per-set scores, the rotating set's new-case ratio, and score divergence between sets.
The continuous improvement loop reviews cases that failed only on the rotating set weekly as golden-set candidates. Promote recurring failure types into new quality criteria to expand the golden set. An eval set should be a criteria document that keeps evolving by absorbing failures, not an asset fixed once and forgotten.
Key takeaways
In short, the eval set sets the direction of self-improvement. Anchor criteria with a golden set, separate fixed and rotating sets to detect overfitting, and physically block evaluation leakage. Observe score divergence between sets and absorb rotating-set failures into the golden set so the eval set keeps reflecting reality.