Statement: We must pressure AI companies to immediately limit the use of recursive self improvement

Paper · Source
Frontier AI Risk & RSI

Source: Future of Life Institute · 2026-09-28

“Today, for the first time, heads of state recognise that the risks of AI cannot be managed by the companies alone. Every day, new incidents of increasing severity show that there is an urgent and vital need for independent, binding oversight of AI.

“Last week, Anthropic CEO Amodei and OpenAI CEO OpenAI proposed on-site inspectors to test the safety of their systems, and world leaders have called for the creation of an international governance institution. These are excellent first steps, but we need more – and they absolutely must not include governments building a liability shield for these companies.

“As UN Human Rights Chief Türk has urged, countries should put pressure on companies to immediately limit the use of recursive self improvement until the research to make it safe has been done. In parallel, countries should ramp up investment in hardware verification technology that can enable a future international AI deal involving both China and the U.S.

“It is now more clear than ever that private AI companies cannot be trusted to police themselves. Government intervention is urgently required to halt the ongoing race-to-replace, and help build a prohuman future with AI for everyone.”

Synthesis notes that discuss concepts related to this paper 2

Notes in this collection most semantically related to the paper, ranked by cosine similarity.

Lines of inquiry this paper opens 24

Research framings built by reading the notes related to this paper — the questions it feeds into.

How does increased AI involvement reshape human agency and accountability? How should evaluation environments be designed as security boundaries? Does extending agent autonomy erode human capacity for meaningful oversight? Can sequences of locally safe agent actions violate system-level constraints? Can AI systems achieve sustainable self-improvement without external human guidance? What organizational factors determine whether firms successfully adopt and integrate AI? How does AI adoption reshape workplace inequality and labor market outcomes? Can humans maintain meaningful oversight as AI systems become more autonomous? How can we ensure AI systems reliably follow constraints rather than circumvent them? How should organizations govern autonomy and access for agentic AI systems? Do safety evaluations reliably measure real deployed AI safety risks? Can models intentionally sandbag on capability evaluations without being caught? Can AI research automation improve discovery pace without sacrificing quality? How does AI assistance affect professional skill development and retention?