Why do aggregation tasks degrade faster than multi-hop reasoning under sparsity?
This explores why tasks that combine or count many pieces of evidence (aggregation) collapse sooner than tasks that chain a few facts together (multi-hop) when you thin out the tokens or context a model can attend to.
This explores why aggregation tasks break down faster than multi-hop reasoning as you strip away tokens or context. The starting point is the corpus's most direct observation: sparsity tolerance is not a single number but a property of the task's shape How much sparsity can different reasoning tasks actually tolerate?. Single-QA tasks survive 95% sparsity because the answer lives in a handful of tokens — drop almost everything else and the load-bearing region stays intact. Multi-hop and aggregation both demand attention spread across many regions, but they spread it differently, and that difference is the whole story.
Multi-hop reasoning is shaped like a *path*: fact A leads to B leads to the answer. A path is surprisingly forgiving under sparsity because it has slack. There are often alternate routes to the same conclusion, intermediate hops can be skipped or inferred, and the structure can even be collapsed — HippoRAG shows multi-hop traversal compressed into a single retrieval step via graph PageRank Can knowledge graphs enable multi-hop reasoning in one retrieval step?, and Atom of Thoughts contracts a reasoning DAG so each state forgets its history without losing the answer Can reasoning systems forget history without losing coherence?. A chain can lose links and still reach the end.
Aggregation is shaped like a *set*: to sum, count, or compare across N items, you need all N simultaneously, and every one is load-bearing. There is no redundancy and no shortcut — drop one operand and the count is simply wrong. This is why the corpus's work on joint constraints matters here. Hypergraph memory exists precisely because aggregation-style relations bind three or more entities into a single relation that cannot be decomposed into pairwise pieces without losing the constraint Can hypergraphs capture multi-hop reasoning better than graphs?. Sparsity attacks exactly that joint binding: it removes members from the set the model needed held together at once.
The token-pruning work sharpens the mechanism. Models internally rank tokens by functional importance and preferentially preserve the symbolic-computation tokens that do the actual work Which tokens in reasoning chains actually matter most?. For a path, the surviving symbolic tokens still trace a route. For an aggregation, the 'important' tokens *are* the full set of operands — there is no subset that preserves the computation, so principled pruning has nowhere safe to cut. The task offers no compressible slack.
The practical upshot the corpus keeps circling: the fix for aggregation is not more compute but matching the structure to the task. StructRAG routes aggregation-flavored queries to tables and catalogues rather than chunks, grounding the choice in cognitive-fit theory Can routing queries to task-matched structures improve RAG reasoning?, and hierarchical architectures separate planning from synthesis so the combine step gets its own dedicated representation Do hierarchical retrieval architectures outperform flat ones on complex queries?. The quiet lesson is that 'reasoning difficulty' under sparsity is really about whether your task degrades like a chain — losing links but keeping its end — or like a sum, where one missing term poisons the whole result.
Sources 7 notes
Single-QA tasks tolerate 95% sparsity while multi-hop and aggregation tasks degrade substantially at 50-67% sparsity. This pattern reflects structural differences: single-QA concentrates reasoning in few tokens, while multi-hop and aggregation require distributed attention across multiple regions.
HippoRAG converts corpus into a knowledge graph, then uses Personalized PageRank seeded from query concepts to traverse multi-hop paths in one step. It matches iterative retrieval while being 10-20x cheaper and 6-13x faster, with 20% better accuracy on multi-hop QA.
Atom of Thoughts decomposes problems into DAGs and contracts them iteratively, ensuring each state depends only on the current problem—not prior steps. This memoryless approach eliminates historical baggage that bloats reasoning while maintaining answer equivalence.
HGMem organizes retrieved evidence as hyperedges rather than flat lists or binary graphs, allowing three or more entities to bind into single relations without decomposition. This structure accumulates coherent knowledge across retrieval steps, trading representational complexity for constraint expressiveness.
Greedy likelihood-preserving pruning reveals six functional token categories; symbolic computation tokens are preferentially preserved while grammar and meta-discourse are pruned first. Student models trained on these pruned chains outperform those trained on frontier-model compression.
StructRAG demonstrates that selecting knowledge structure type based on query demands—via DPO-trained router choosing among tables, graphs, algorithms, catalogues, and chunks—improves knowledge-intensive reasoning over standard retrieval. The approach grounds this in cognitive load and cognitive fit theory from cognitive science.
Separating query planning from answer synthesis into distinct components reduces interference and improves multi-hop query performance. This architectural principle mirrors documented benefits of separating planning from execution in agent design.