What distinguishes iterative query refinement from pure self-revision loops?
This explores the difference between loops that keep reformulating what you ask the world (refining a search query across retrieval rounds) and loops where a model just keeps rewriting its own answer with no new input — and why one tends to improve while the other tends to rot.
This explores the difference between iterative query refinement — where each pass reformulates the question and pulls in fresh external evidence — and pure self-revision, where a model loops on its own output with nothing new entering the system. The corpus suggests the dividing line isn't the act of looping at all; it's whether each iteration imports an outside signal or just re-digests the model's own prior guess.
The self-revision side has a consistent failure signature. When o1-like models revise their own reasoning, most revisions keep the wrong answer rather than fix it, and longer revision chains actually correlate with lower accuracy Does self-revision actually improve reasoning in language models?. The sharpest finding is that the *source* of the critique is what matters: external critics improve revision, while a model judging its own uncertain output tends to amplify its confidence in wrong answers instead of correcting them Does revising your own reasoning actually help or hurt?. Iterative refinement methods inherit the token-level 'overthinking' failure one level up — they accumulate noise across passes with no guarantee of improvement Do iterative refinement methods suffer from overthinking?. So a pure self-loop is structurally an echo chamber: no new information ever arrives to break a wrong commitment.
Query refinement escapes this precisely because each round injects new external evidence — retrieval is the outside signal that self-revision lacks. But the corpus is clear this only works if the architecture is built for it. Hierarchical research systems that separate query planning from answer synthesis outperform flat ones on multi-hop questions, because keeping the 'what should I ask next' component apart from the 'what's the answer' component reduces interference Do hierarchical retrieval architectures outperform flat ones on complex queries?. And the refinement loop needs room to breathe: capping reasoning per turn (not just overall) preserves the context an agent needs to actually absorb new evidence on the next retrieval round, rather than burning it all up in one turn Does limiting reasoning per turn improve multi-turn search quality?.
There's a deeper wrinkle, though — refining the query isn't automatically better than not looping at all. Fine-tuning a retrieval model on implicit queries can match query augmentation without any expansion, because the model learns to resolve ambiguity through training instead of through repeated reformulation Can fine-tuning replace query augmentation for retrieval?. And retrieval failures are often architectural rather than something more iteration fixes: fixed-interval triggering, embeddings that measure association rather than relevance, and hard mathematical limits on what a given embedding dimension can represent Where do retrieval systems fail and why?. So query refinement can hit walls that no amount of re-querying breaks — the loop has to be the right *kind* of loop.
The thing you might not have known you wanted to know: looping isn't the variable that determines whether iteration helps — *what flows in on each pass* is. Self-revision and query refinement look like the same control structure, but one closes the system off from new information and slowly amplifies its own errors, while the other only works when it stays open to an external signal and gives itself the context budget to actually use it. The Darwin Gödel Machine makes the same bet at the agent level — open-ended self-improvement works there because it validates each variant against empirical benchmarks rather than its own judgment, keeping an external reality check in the loop Can AI systems improve themselves through trial and error?.
Sources 8 notes
Evidence from QwQ, R1, and LIMO shows most revisions retain wrong answers rather than correcting them. Smaller models frequently switch correct answers to incorrect during revision, and longer chains with more revisions correlate with lower accuracy.
Revision guided by external models improves accuracy, but a model revising its own uncertain output typically amplifies confidence in wrong answers rather than correcting them. The revision source, not the revision act itself, determines the outcome.
Sequential revision methods share the same failure architecture as token-level overthinking: they accumulate noise without guaranteed improvement. Progressive Draft Refinement avoids this by compressing memory between iterations, outperforming longer reasoning traces at matched compute.
Separating query planning from answer synthesis into distinct components reduces interference and improves multi-hop query performance. This architectural principle mirrors documented benefits of separating planning from execution in agent design.
Unrestricted reasoning within single search turns consumes context needed for subsequent retrieval rounds, degrading the agent's ability to incorporate new evidence. Setting per-turn reasoning budgets, not just overall time limits, prevents this context erosion and maintains search quality across iterations.
Fine-tuned semantic search models trained on implicit queries match the performance of augmented pretrained retrievers without expanding input length. The model learns to resolve ambiguity through training rather than requiring explicit augmentation.
RAG systems fail at three structural levels: adaptive triggering (fixed intervals waste context), semantic-task mismatch (embeddings measure association, not relevance), and mathematical limits (embedding dimension constrains representable document sets). These require fundamentally different retrieval approaches, not tuning.
DGM replaces formal proofs with empirical benchmarking and maintains an evolutionary archive of agent variants, achieving 2.5× improvement on SWE-bench and 2.2× on Polyglot by discovering capabilities like better code editing and context management.