RSI loops need a separate governance layer
Recursive self-improvement (RSI) lets an agent rewrite its own code, prompts, and even its evaluation criteria. OpenAI has framed an intern-level AI research agent for September 2026 and a fully capable agent as a 2028 target, and release intervals that once ran 6 to 12 months are compressing to a weekly cadence. Once the assumption that a human pre-reviews every change breaks down, change-approval authority and rollback paths must be pinned to a governance layer outside the code the agent touches — not inside it.
Runaway starts in the objective function, not the metrics
When an agent treats the very criteria that judge it as an optimization target, you get reward hacking: benchmark scores climb while real quality falls. Keep the objective function and the golden eval set as read-only boundaries the agent cannot modify, and keep ownership of pass rates and regression counts with humans.
Small changes, fast reversals
ICLR 2026 opening its first RSI-dedicated workshop reflects the same concern — verification speed is not keeping up with generation speed. In a weekly cadence, cap the number of files and prompts changed at once, and fix canary share and automatic rollback thresholds in advance as the baseline for containing runaway behavior.
Detailed guide: from planning to operations
(a) Planning and requirements start with numbers. For example: hold golden-set pass rate at 98% or higher, zero regressions, a p95 latency ceiling, 5% canary traffic observed for 24 hours, and an automatic rollback threshold pinned to a 2-point drop in pass rate. These values are constants owned by the operator, not the agent.
(b) Enumerate failure patterns first, then attach recovery branches. The main types are reward hacking, prompt drift, self-editing loops that never terminate, and eval-set contamination. Recovery branches: retry transient errors three times; route any attempt to change the objective function or safety rules to human confirmation; fall back to the prior version when metrics leave the threshold; and set a hard stop when the same commit fails five times in a row or rollbacks repeat.
(c) The operations checklist is decided by observability. Log every self-edit attempt to a standard schema (timestamp, change-diff hash, input metrics, verdict, recovery branch), and run any user data entering logs or prompts through PII masking. Sign each change diff so who changed what stays traceable.
(d) The continuous-improvement loop runs by feeding failures back into the golden set. Register every rolled-back change as a regression case so the same mistake does not recur next cycle, and review pass-rate trends, violation counts, and human-confirmation intervention frequency together in a weekly review.
Executive summary
The risk of RSI comes from loss of control, not from capability. Pin the objective function as read-only, break changes into small units, wire automatic rollback to numeric thresholds like 98% pass rate and zero regressions, and predefine the recovery branches — retry, human confirmation, safe fallback, and hard stop — and you can contain runaway behavior even at a weekly release cadence.