TOPIC
Agent Harness
A subject the collection covers, read through 16 synthesis notes.
View as
What are the three distinct layers of agent code?
Does separating agent code into model capabilities, system harness, and agent-created artifacts help explain why agentic systems fail and where to intervene for improvement?
Where does agent reliability actually come from? Does a single benchmark score actually predict agent readiness? What makes agent-authored code worth persisting and sharing? How can separating evaluation components make reward-hacking visible? Can brain structure guide how we design intelligent agents?
Agent reliability comes from externalizing cognitive burdens into system structures. Agent capability requires a vector across at least five separable axes Agent-authored persistent code remains the least studied harness layer Component decoupling surfaces reward-hacking through trajectory analysis. Brain-inspired modular decomposition unifies foundation agent architecture design
How does the agentic layer amplify individual agent failure modes? How do agentic systems recover when specialized models operate outside their scope? How do agent-created code artifacts become part of harness infrastructure? What makes agent-initiated artifacts the underexplored frontier in harness engineering? How do you verify agent code under incomplete feedback signals? What breaks when multiple agents share and revise the same artifacts? What makes harnesses more tangled than other types of agent code?
Should agent evaluation measure more than task success?
Current benchmarks reduce agents to a single success score, but agents emerge from multiple interacting systems. What dimensions of agent behavior should builders actually measure to predict deployment readiness?
Does a single benchmark score actually predict agent readiness? Does agent efficiency really break down into three distinct components? Do phone agents succeed at all three critical tasks equally? How should we evaluate agent behavior beyond final answers? Does raw token spending actually predict agent performance?
Agent capability requires a vector across at least five separable axes Agent efficiency decomposes into three orthogonal optimization axes Phone agents show independent success, privacy, and preference-reuse capabilities Agent evaluation must score trajectories not just final responses Agent performance scales with effective feedback not raw token count
Can standard accuracy metrics miss the real constraints on user consumption? When should you optimize agent behavior versus tool performance separately? Why do agents report success when they have actually failed at tasks? How does the execution layer constrain agent performance in tool use? How should domain-specific AI be evaluated differently from general benchmarks? How much does agent performance depend on demonstration quantity versus curation quality? What task characteristics determine whether humans or agents should handle work? Why do 85 percent of production agents avoid third-party frameworks? Can automated evaluation replace human judgment in agent testing? What tasks do AI agents still fail at most often? Why do current evaluation metrics fail to catch reasoning failures in persona agents? How do single-agent safety evaluations underestimate risks in deployed multi-agent systems?See all 56 inquiring lines on this note →
Where does agent reliability actually come from?
Exploring whether LLM agent performance depends on larger models or on thoughtful system design choices like memory, skills, and protocols that shift cognitive work outside the model.
Agent reliability emerges from harness scaffolding not model scale Agent code splits into three governed layers with different failure modes. Agentic tools do not improve LLM document editing reliability Coordination and governance become the binding constraint as agents scale economically Codified expert rules in agent scaffolding dissolves organizational specialist bottlenecks
Can persistent memory and identity files alone create genuine agent socialization? How does the agentic layer amplify individual agent failure modes? Does state persistence in AI systems create the same temporal presence as human waiting? How do multi-agent LLM systems fail at coordination and role consistency? How do LLM user simulators track and maintain consistent goal states across multi-turn interactions? Do emotion-driven actions in agent simulators capture genuine belief revision or just reactive behavior? Why do planning and grounding have opposing optimization requirements in agents? How should GUI agents remember patterns across different software environments? Can parallel agents or complementary mechanisms replace single-human interrogation of LLMs? Why do longer forecasting horizons degrade LLM accuracy in role-play? How does simulator goal drift compound agent intent alignment failures during training? Should user simulators be trained via RL like agents or decomposed into trackable state components?See all 229 inquiring lines on this note →
What makes agent-authored code worth persisting and sharing?
Agent-created artifacts like patches, tests, and skill libraries outlive single tasks, but we lack guidance on what should persist, how to maintain consistency across agents, and when persistence is worth the engineering effort.
Agents gain 24-51 percent by extracting and compounding reusable sub-task workflows Agents need two complementary adaptation timescales for continuous learning Agent code splits into three governed layers with different failure modes. Externalized skill libraries enable lifelong learning without catastrophic forgetting Harness evolution mostly memorizes task fixes rather than distilling strategies
How do standardized artifacts improve coordination between writing agents? When should agent-created code be promoted into permanent harness infrastructure? What prevents multiple agents from corrupting shared state in live artifacts? How do agents decide which created code should persist versus disappear? How should human oversight apply to persistent agent-authored code? What makes persistent, shared code artifacts from agents hard to manage at scale? What makes skills worth externalizing into a persistent harness? How do agent-created code artifacts become part of harness infrastructure? How do agents decide which created code deserves long-term persistence? What breaks when multiple agents share and revise the same artifacts? Are durable shared code artifacts better than per-task harness patches?
Why is finding distributed behavior code so hard?
When developers need to modify agent harnesses, they struggle to locate all the code implementing a target behavior because behaviors are scattered across files and stages while requests describe what to do, not where to look.
Behavior-centric harness organization lets weak planners match stronger model localization Agent code splits into three governed layers with different failure modes. Agent-authored persistent code remains the least studied harness layer Agent reliability emerges from harness scaffolding not model scale Harness updating is flat but benefiting peaks at mid-tier models
What makes harnesses more tangled than other types of agent code? How can we reorganize repositories to make behaviors easier to locate? Why does the harness layer accumulate distributed behaviors over time? What persistent failures remain unsolved despite harness evolution efforts? What makes behavior localization the bottleneck in agent harness evolution?
Can code serve as the operational substrate for agent reasoning?
Explores whether code functions not just as LLM output but as the executable medium through which agents reason, act, and verify progress. This reframing treats code as infrastructure rather than deliverable.
Should LLMs handle abstraction only in optimization? Can structured reasoning replace code execution for RL rewards? Can structured templates replace formal verification for code reasoning? How should agents split planning from visual grounding? Can structured templates make code reasoning more reliable than free-form thinking?
LLM optimization works best when restricted to abstraction only Structured reasoning reaches 93% accuracy for execution-free code verification Structured templates bridge formal verification and unstructured reasoning effectively Agents need an intermediate interface between planning and grounding layers. Structured reasoning templates force completeness and prevent unstructured reasoning failures
Why do planning and grounding have opposing optimization requirements in agents? How does credit assignment drive agents to write information into environments? What makes users willing to relinquish control to an agent? Why do workflow abstractions fail in embodied agent environments? What role do material artifacts play in solidifying AI relationships? Can deterministic function calls prevent agent failures better than protocol-mediated tool access? Can agentic reasoning outperform rigid rule-based systems for skill refinement? Can API-first interaction replace traditional UI-based agent interfaces? How do agentic systems recover when specialized models operate outside their scope? How should agents separate planning from perception grounding? Does the planning-grounding factoring principle apply to other agent tasks? What task characteristics determine whether humans or agents should handle work?See all 59 inquiring lines on this note →
Do harness edits learn strategies or just memorize fixes?
When a meta-agent evolves a harness over time, does it discover transferable principles that solve new classes of problems, or does it mainly cache task-specific workarounds that don't generalize?
Harness evolution gains need matched-budget baselines to isolate design Harness updating is flat but benefiting peaks at mid-tier models Agent-authored persistent code remains the least studied harness layer Agent reliability emerges from harness scaffolding not model scale Behavior-centric harness organization lets weak planners match stronger model localization
What makes evolving the benchmark different from evolving the optimizer itself? Can harness updates benefit agents equally across all model sizes? How should harness scaffolding be treated as a first-class object? What makes harnesses more tangled than other types of agent code? Why does the harness layer accumulate distributed behaviors over time? Can harness evolution be redirected toward distilling transferable procedures instead? What persistent failures remain unsolved despite harness evolution efforts? Why do evolved harness edits mostly memorize rather than generalize? What feedback signals matter most during harness evolution search? What makes behavior localization the bottleneck in agent harness evolution?
Are harness evolution gains really from better design?
When automatic harness evolution improves agent performance, how much of that gain comes from smarter design versus simply running more search? Understanding this distinction requires careful experimental controls.
Harness evolution mostly memorizes task fixes rather than distilling strategies Agent capability requires a vector across at least five separable axes Harness updating is flat but benefiting peaks at mid-tier models Agent reliability emerges from harness scaffolding not model scale Behavior-centric harness organization lets weak planners match stronger model localization
What makes evolving the benchmark different from evolving the optimizer itself? Can harness updates benefit agents equally across all model sizes? How should harness scaffolding be treated as a first-class object? What makes harnesses more tangled than other types of agent code? Why does the harness layer accumulate distributed behaviors over time? Why do mid-tier models benefit more from memorized harness shortcuts? Can harness evolution be redirected toward distilling transferable procedures instead? What persistent failures remain unsolved despite harness evolution efforts? Why do evolved harness edits mostly memorize rather than generalize? What feedback signals matter most during harness evolution search? What makes behavior localization the bottleneck in agent harness evolution?
Can skills work better as weights than as prompts?
Most agent systems store skills as text in prompts, but this inflates token costs and degrades model performance. Could compiling skills into trainable weight-space adapters instead offer a better trade-off between efficiency and capability?
Can models dynamically activate expert skills at inference time? Can skill documents be optimized like neural network weights? Can agents learn new skills without forgetting old ones? How should agents split planning from visual grounding? Can frozen models learn better by extracting context into skills?
Models compose task-specific experts at inference via singular value tuning Skill documents train like model weights using text-space optimization with validation gating Externalized skill libraries enable lifelong learning without catastrophic forgetting Agents need an intermediate interface between planning and grounding layers. Frozen models unlock hidden context knowledge through explicit skill extraction
Can person-grounded skills remain auditable without hidden prompt state?
Explores whether treating extracted expertise as versioned files—rather than persona prompts—enables meaningful accountability over person-grounded knowledge. Matters because audit trails determine whether captured skills can be corrected, rolled back, or safely withheld.
Can skill documents be optimized like neural network weights? Can codified expertise let non-experts match specialist output? Can agents learn new skills without forgetting old ones? What makes an AI system feel like a colleague rather than a chatbot? Can a separate trained curator improve skill libraries better than frozen agents?
Skill documents train like model weights using text-space optimization with validation gating Codified expert rules in agent scaffolding dissolves organizational specialist bottlenecks Externalized skill libraries enable lifelong learning without catastrophic forgetting Persistent architecture and reusable skills define colleague-like AI, not model scale Decoupled skill curators learn to evolve repositories toward strategic meta-skills
What makes skills worth externalizing into a persistent harness? Does inspectable skill artifacts guarantee the behavior matches the person it claims to ground? How do capability tracks and behavior tracks stay separable during skill deployment? What makes passive prompt transfer fail as a substitute for auditable expertise? How can post-training research become reproducible without releasing full interfaces?
How should agents route across thousands of skills?
As skill libraries grow, should routing focus on selecting one skill or composing many? This explores whether decomposition and chaining creates better task execution than single-skill selection.
What blocks skill retrieval in task decomposition? Does separating planning from execution improve reasoning accuracy? Can a separate trained curator improve skill libraries better than frozen agents? How should agents split planning from visual grounding? Can extreme task decomposition enable reliable execution at million-step scale?
Decomposition granularity is the primary retrieval bottleneck Separating decomposer from solver improves accuracy and generalizability Decoupled skill curators learn to evolve repositories toward strategic meta-skills Agents need an intermediate interface between planning and grounding layers. Extreme decomposition with subtask voting enables million-step error-free execution reliably
Can explicit behavior maps help weaker planners compete with stronger models?
Explores whether organizing harness repositories around runtime behavior—rather than relying on model inference—can narrow the capability gap between weaker and stronger planning models, and whether this reduces computational overhead.
Behavior localization is the real bottleneck in harness evolution Externalizing bookkeeping to a harness lets smaller search agents outperform larger ones Agent-authored persistent code remains the least studied harness layer Agent reliability emerges from harness scaffolding not model scale Agent reliability comes from externalizing cognitive burdens into system structures.
Can harness updates benefit agents equally across all model sizes? Can system-level engineering fixes replace hand-designed reward heuristics entirely? How should harness scaffolding be treated as a first-class object? What makes harnesses more tangled than other types of agent code? How can we reorganize repositories to make behaviors easier to locate? Why does the harness layer accumulate distributed behaviors over time? Can weaker planners match stronger models if behavior is reorganized? Can automated benchmarks accurately capture progress on real-world long-horizon tasks? Why do mid-tier models benefit more from memorized harness shortcuts? Can harness evolution be redirected toward distilling transferable procedures instead? What persistent failures remain unsolved despite harness evolution efforts? Are durable shared code artifacts better than per-task harness patches?See all 15 inquiring lines on this note →
Can externalized bookkeeping let smaller search agents beat larger ones?
Does offloading routine record-keeping to an environment harness free RL policies to focus on semantic search decisions, and can this approach outperform larger searchers with fewer parameters?
What are the three distinct layers of agent code? Where does agent reliability actually come from? Can agents fail from weak memory control rather than missing knowledge? Can explicit behavior maps help weaker planners compete with stronger models? Can direct corpus search beat embedding-based retrieval?
Agent code splits into three governed layers with different failure modes. Agent reliability comes from externalizing cognitive burdens into system structures. Multi-turn agent failure stems from weak memory control not knowledge gaps Behavior-centric harness organization lets weak planners match stronger model localization Direct corpus interaction recovers lexical precision where embeddings conflate entities
What role does retrieval mechanism design play in forecast accuracy? Why does externalizing bookkeeping raise effective feedback compute? Can external managers optimize context better than the model itself? How does external context control compare to agents managing their own state internally? How do search and reasoning workflows improve forecasting performance over base models? Can externalizing bookkeeping to a stateful harness replace internalized memory control? What specific bookkeeping tasks can environments maintain more reliably than policies? Do gains from harness-based agents transfer across different search benchmarks? Do information gathering and task execution require different incentive structures? How can harnesses externalize bookkeeping so models focus on semantic judgment?
What blocks skill retrieval in task decomposition?
When routing queries across a skill library, does the granularity of task decomposition determine whether retrieval can succeed? This explores whether fixing decomposition precision unlocks better skill matching.
How should agents route across thousands of skills? Does separating planning from execution improve reasoning accuracy? How should agents split planning from visual grounding? Can extreme task decomposition enable reliable execution at million-step scale? How should reasoning systems actually be architected?
Skill routing requires decomposing tasks and composing skills together Separating decomposer from solver improves accuracy and generalizability Agents need an intermediate interface between planning and grounding layers. Extreme decomposition with subtask voting enables million-step error-free execution reliably Reasoning systems should separate activation timing from execution capability.
Can task success alone reveal whether memory routing is working? Why do persistent, resynchronized artifacts compound harness capability gains? Why does task decomposition granularity become the bottleneck in skill routing? What makes skills suitable for retrieval and chaining in repositories? Can skill repositories evolve toward execution-oriented refinement over time? Why does fixing decomposition step count matter more than vocabulary alignment? Why should decomposition be diagnosed and fixed separately from solving? What cascading bottlenecks appear when skill routing is decomposed into stages?
Do stronger models always evolve harnesses better?
We explore whether base model capability predicts both the ability to write useful harness updates and the ability to benefit from them. The answer reshapes how we should allocate capability in self-evolving agent systems.
Agent reliability comes from externalizing cognitive burdens into system structures. Decoupled skill curators learn to evolve repositories toward strategic meta-skills Versioned resources with auditable lineage enable safe agent self-evolution Agent reliability emerges from harness scaffolding not model scale Harness evolution mostly memorizes task fixes rather than distilling strategies
Can mid-tier models benefit more from self-generated harness updates than others? Should we train the evolver or the executor when building self-improving agents? Can smaller models produce skill updates as useful as frontier model updates? What happens when different harnesses project the same model? Does harness benefit depend on which model tier you use? What makes skills worth externalizing into a persistent harness? How can agents evolve their own skills without human input? Does base model strength determine adapter usefulness across users? How do agent-created code artifacts become part of harness infrastructure? What makes agent-initiated artifacts the underexplored frontier in harness engineering? Can single-axis benchmarks measure across all three agent capability layers? Can agent-authored skill libraries compound autonomy gains over time?See all 34 inquiring lines on this note →
What makes agent memory quality better than storage capacity?
If agents need better memory, should we focus on adding storage or improving what gets kept? This explores why curation and selective forgetting matter more than raw capacity for reliable agent performance.
Agent memory splits between autonomous recognition and programmatic triggers Selective history retrieval outperforms full context inclusion in conversations Agent memory needs module-level evaluation not just end-to-end task scoring Agent memory should continuously adapt through execution-driven link creation and pruning Continuously consolidated agent memory follows an inverted-U utility curve
Should agents update memory after every turn or batch process sessions? Why do different agent memory architectures make incompatible granularity claims? How much does agent performance depend on demonstration quantity versus curation quality? What architectural changes would accelerate the cleanup phase? How does bottleneck automation differ from accessory work displacement? What tree depth is achievable before GPU memory becomes the bottleneck? What makes provenance infrastructure more critical than artifact quality? Can topology repair fix consolidation failures in agent memory? How does procedural memory granularity affect web agent performance? Which memory components trigger context-length problems in agents? Can pruning policies alone solve working memory bloat in agents? What is the right granularity level for agent memory to enable both reuse and composition?See all 30 inquiring lines on this note →