The safety math for self-improving AI assumes a fixed goal — what happens when the system gets to set its own?

This explores whether self-improvement systems still have their correctness guarantees when you remove the fixed, externally-defined goal and let the system evolve its own target instead. The corpus suggests the answer flips depending on what kind of guarantee you're asking about — and that the interesting move is not preserving the static target but replacing it with a moving one that's engineered to stay honest.

Start with why anyone would remove static utility in the first place. Fixed benchmarks and reward functions saturate: once an agent gets strong enough, a stationary target becomes something to game rather than achieve, and you get reward hacking Why do fixed benchmarks fail as agents grow stronger?. So the static external utility isn't a safe anchor you'd want to keep — it's a liability that gets worse as the loop improves. The fix in that work (RQGM) is telling: it splits the search into epochs with a *stable within-epoch* criterion but *evolves* the utility across epoch boundaries. That's the key structural insight — you don't get a global guarantee, you get per-epoch guarantees, and you keep the target moving just ahead of the optimizer so it can never catch up and cheat.

But there's a deeper limit the question is circling. A whole cluster of the corpus argues that a self-improvement loop with *no* external signal at all is formally bounded no matter what — the generation-verification gap means a model can't reliably validate its own fixes, so metacognition alone hits a hard ceiling What limits autonomous capability in large language models? What actually constrains AI systems from learning misalignment?. The methods that actually work don't remove external utility so much as smuggle it back in through side doors: past model versions, third-party judges, user corrections, or tool feedback Can models reliably improve themselves without external feedback?. So 'removing static external utility' and 'removing external grounding entirely' are two very different moves — the first is often healthy, the second breaks reliability by construction.

The most concrete answer to your literal question comes from the Darwin Gödel Machine, which *deliberately* trades away formal guarantees. The original Gödel machine required a formal proof that a self-modification improves utility before applying it; DGM abandons that proof requirement and substitutes empirical benchmarking plus an evolutionary archive of variants Can AI systems improve themselves through trial and error?. It got 2.5× on SWE-bench — so yes, removing the static, provable utility does break the *formal* guarantee, and in this case that was the point. What replaces it is an empirical, statistical form of confidence, which buys open-endedness at the cost of certainty.

The surprising takeaway: the formal guarantee was never really doing the useful work. The systems that improve best keep two things the theory says you need — a *stable criterion inside each step* and an *external validator* — while throwing away the one thing that felt safest, the fixed global objective. And there's a wrinkle worth knowing: the ability to keep evolving the loop isn't uniform across models. Weak models can't even invoke their own scaffolding, strong models over-follow instructions, and the capacity to actually benefit from self-updates peaks in the middle Do stronger models always evolve harnesses better?. So whether removing static utility helps or hurts also depends on which model is running the loop.