Why does single-round retrieval fail on multi-step tasks across different domains?
This explores why grabbing documents in one shot breaks down when a task needs several reasoning steps — and whether the failure is the same story across different kinds of tasks and knowledge.
This explores why a single pass at the retriever — fetch once, then answer — falls apart on tasks that need several hops of reasoning, and whether that breakdown looks the same across domains. The short version from the corpus: single-round retrieval fails because the failure is *architectural*, not a tuning problem you can fix with a better embedding. Where do retrieval systems fail and why? frames it cleanly — retrieval breaks at structural seams: fixed triggering wastes context, embeddings measure topical association rather than task relevance, and there are hard mathematical limits on how many documents a single vector can even represent. A multi-step task asks the retriever to anticipate, in one shot, evidence it can't know it needs until step three.
The most direct answer is that multi-step tasks need retrieval to *unfold* the way reasoning does. Can retrieval be extended into multi-step chains like reasoning? makes this literal: instead of one retrieval, you generate a chain of intermediate retrievals, each conditioned on what the last one surfaced — turning retrieval into something you can scale with compute, like chain-of-thought. Do hierarchical retrieval architectures outperform flat ones on complex queries? reaches the same place from the architecture side: separating query *planning* from answer *synthesis* reduces interference and lifts performance on multi-hop queries, because one component figures out what to ask while another figures out what the evidence means. A single round collapses those two jobs into one and does neither well.
The "across different domains" part is where it gets interesting, because the corpus says the *strategy* has to change with the task — there is no one retrieval move that works everywhere. Does question type determine the right retrieval strategy? splits questions into types where evidence-based ones suit plain RAG but debate, comparison, and experience questions need decomposition or aspect-specific retrieval. Can routing queries to task-matched structures improve RAG reasoning? pushes further: route each query to the *structure* it needs — a table, a graph, an algorithm, a catalogue — because forcing every query through the same chunk-retrieval pipeline ignores how differently shaped knowledge demands differently shaped access. That's why a single-round system that works in one domain quietly fails in the next.
There's also a subtler reason rooted in *when* to retrieve at all, not just how many times. When should language models retrieve external knowledge versus use internal knowledge? models each reasoning step as a decision — retrieve, or trust what the model already knows — and gets a 22% gain mostly by *not* retrieving when retrieval would only inject noise. Can simple uncertainty estimates beat complex adaptive retrieval? echoes this: a model's own calibrated uncertainty is a better trigger for retrieval than fixed heuristics. Single-round retrieval has no such dial — it fires once, blind to whether the current step even needs outside evidence.
The twist worth taking away: even unlimited context doesn't rescue single-shot retrieval. Can long-context LLMs replace retrieval-augmented generation systems? shows long-context models can match RAG on semantic lookup but still fail on structured, multi-table queries that require joins — proving the bottleneck isn't *how much* you can stuff in one pass, it's that some tasks are irreducibly sequential. And a quieter failure lurks inside multi-step systems themselves: Does limiting reasoning per turn improve multi-turn search quality? finds that letting a model reason too hard in a single turn burns the context it needs to absorb the *next* round of evidence. So the deeper lesson is that multi-step tasks don't just need more retrieval — they need retrieval and reasoning paced together, something a single round can't offer by definition.
Sources 9 notes
RAG systems fail at three structural levels: adaptive triggering (fixed intervals waste context), semantic-task mismatch (embeddings measure association, not relevance), and mathematical limits (embedding dimension constrains representable document sets). These require fundamentally different retrieval approaches, not tuning.
CoRAG extends chain-of-thought training to retrieval by using rejection sampling to generate intermediate retrieval chains. Test-time compute can scale through chain length and count, creating a compute dial—greedy decoding for speed or tree search for accuracy—just like reasoning-token scaling.
Separating query planning from answer synthesis into distinct components reduces interference and improves multi-hop query performance. This architectural principle mirrors documented benefits of separating planning from execution in agent design.
Research shows non-factoid questions split into five types, each requiring different retrieval and aggregation methods. Evidence-based questions suit standard RAG, while debate and comparison need aspect-specific retrieval, and experience/reason questions need decomposition or filtering strategies.
StructRAG demonstrates that selecting knowledge structure type based on query demands—via DPO-trained router choosing among tables, graphs, algorithms, catalogues, and chunks—improves knowledge-intensive reasoning over standard retrieval. The approach grounds this in cognitive load and cognitive fit theory from cognitive science.
DeepRAG models each reasoning step as a Markov Decision Process where the model learns when to retrieve versus rely on parametric knowledge. The 21.99% improvement comes from better-targeted retrieval and elimination of noise from unnecessary external knowledge.
Calibrated token-probability uncertainty consistently beats multi-call adaptive retrieval on single-hop tasks and matches performance on multi-hop, using a fraction of the LM and retriever calls. The model's self-knowledge proves more reliable than external heuristics for deciding when to retrieve.
The LOFT benchmark shows LCLMs match RAG on semantic retrieval without explicit training, but cannot execute relational queries requiring joins across structured tables. Context length alone cannot bridge this gap.
Unrestricted reasoning within single search turns consumes context needed for subsequent retrieval rounds, degrading the agent's ability to incorporate new evidence. Setting per-turn reasoning budgets, not just overall time limits, prevents this context erosion and maintains search quality across iterations.