Can better tools fix LLM document editing errors?
Does giving LLMs agentic tool access—like diffing, re-reading, or structured editors—improve their reliability on long-horizon document workflows? Understanding whether the problem is tool limitations or decision-making quality matters for reliability engineering.
A natural intuition for fixing LLM document corruption: give the model better tools. Let it diff its own output, re-read the file, call a structured editor instead of regenerating prose. The DELEGATE-52 evaluation tests this directly and finds that agentic tool access does not improve performance on the benchmark.
The finding rules out a class of proposed fixes. Tool wrappers, ReAct loops, and structured editing affordances are not addressing the failure mechanism — they are downstream of it. The degradation comes from the model's own decisions about what to change and how, not from limitations of the editing interface. A model that decides to flip a numeric value will flip it through any tool you give it.
This also disambiguates two senses of "agent." The first sense — LLM-plus-tools, where capability is gated on tool affordances — predicts that tool access should improve document workflows. The second sense — LLM-as-decider, where the model's judgment about what to edit is the bottleneck — predicts that tool access should be roughly orthogonal. The DELEGATE-52 result favors the second.
The implication for workflow design: reliability gains on long delegated work probably come from changing what the model decides (better prompting, verification loops, decomposition into smaller reversible steps) rather than from upgrading what it can act through. Tool engineering helps when capability is interface-limited; it does not help when capability is judgment-limited.
Inquiring lines that read this note 25
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What mechanisms preserve shared understanding in evolving conversations?- What repair strategies work best at each level of Clark's ladder?
- How do insert-expansions and third position repair together cover full repair lifecycle?
- How do insert-expansions differ from third position repair in timing?
- Do architectural changes or training fixes better prevent agreement failures?
- Why do frontier models corrupt more documents than weaker models during workflows?
- Where does agent reliability come from if not better tools?
- How does model tier affect whether errors delete or corrupt document content?
- Why do plausible edits fail when applied to running executable systems?
- What makes draft-centric systems better anchors for coherence than feed-forward outputs?
- What detection mechanisms work best for corruption-style document errors?
- Can detection mechanisms like diff review catch corruption better than deletion?
- What causes silent document corruption in long LLM workflows?
- Why do LLMs choose incorrect edits despite understanding the task?
- How long does retrievability support error detection across repeated LLM use?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do frontier LLMs silently corrupt documents in long workflows?
DELEGATE-52 tests whether state-of-the-art language models reliably preserve document integrity across extended delegated tasks. Understanding this matters because single-step benchmarks may mask compounding failures that emerge only at workflow scale.
same paper, the parent finding
-
Why do language models fail to act on their own reasoning?
LLMs produce correct explanations far more often than they produce correct actions. What causes this knowing-doing gap, and can training methods close it?
adjacent: knowledge does not transfer to action
-
Where does agent reliability actually come from?
Exploring whether LLM agent performance depends on larger models or on thoughtful system design choices like memory, skills, and protocols that shift cognitive work outside the model.
partial counterpoint: harnessing helps elsewhere
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- LLMs Corrupt Your Documents When You Delegate
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- Bilevel Coordinated Reflection: A Game-Theoretic Approach to Multi-Agent LLM Systems
- Towards a Science of Scaling Agent Systems
- Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words
- Demystifying Agent Skills: Why They Work-Until They Don't
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Emergent Collusion in Long-Horizon LLM Agent Interaction
Original note title
agentic tool use does not improve llm document-editing reliability — tools are not a fix for long-horizon delegation drift