Can you turn an LLM into an agent by just fine-tuning?
Explores whether upgrading language models to action-producing systems requires only model retraining or demands a broader pipeline transformation including data collection, grounding, integration, and safety evaluation.
The Large Action Model (LAM) framework reframes the LLM-to-agent transition as a pipeline rather than a training upgrade. The argument is that LLMs excel at textual outputs but fail when forced to produce actionable sequences in dynamic environments, particularly under demands for precise task decomposition, long-term planning, and multi-step coordination. Their general-purpose optimization works against them in unfamiliar settings where adaptive, robust action sequences are needed.
Therefore the conversion to a LAM has four distinct stages, each requiring its own expertise: (1) collect comprehensive datasets capturing user requests, environmental states, and corresponding actions — these triples are the foundation for any action-oriented training; (2) apply training techniques that enable action understanding and execution within specific environments, not just text generation; (3) integrate the trained LAM into an agent system with components for observation gathering, tool use, memory, and feedback loops, because raw action capability without environmental coupling produces nothing; (4) rigorously evaluate reliability, robustness, and safety before real-world deployment.
The implication is that builders treating "agentic capability" as a fine-tuning problem will under-invest in the surrounding system. Memory, feedback, and tool integration are not optional polish — they are what makes action grounded in context rather than a hallucinated step. Evaluation cannot be deferred either, because action-producing models have failure modes (wrong action on real system) that text models do not — see Do autonomous agents report success when actions actually fail? for the canonical example of what evaluation must catch.
The pipeline frame is consistent with Where does agent reliability actually come from?: the harness, not the model, is where agent reliability gets earned. LAM training gives you a model that can produce actions; the surrounding pipeline is what makes those actions grounded, evaluated, and safe to deploy.
Inquiring lines that read this note 34
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do standardized protocols improve multi-agent coordination and reliability? Why do language models resist personality conditioning through prompts? How can humans maintain meaningful oversight as AI systems become increasingly autonomous and complex? What explains language models' asymmetric difficulty with implicit versus explicit linguistic relations? Why do LLM recommenders underperform collaborative filtering despite their capabilities? What compositional reasoning failures limit large language models despite scale?- What specific information must be exported from the language system?
- What architectural changes would let language models develop genuine functional competence?
- Do different domains require different types of model investment?
- Can specialized components replace single fully-trained models in deployment?
- How much does workflow architecture matter versus raw model capability?
- How do labs actually train next-generation models from previous ones?
- What interaction controls matter most for effective human-LLM collaboration?
- What interaction design changes would help LLMs handle underspecified requests?
- What unique perspective do designers bring to LLM adaptation that engineers might miss?
- How can human-centered objectives be embedded earlier in the LLM pipeline?
- How do LLM capabilities changing affect the relevance of interaction guidelines?
- Do agent frameworks adequately compensate for LLM conversational passivity?
- What structural constraints does topology impose on role and LLM assignment?
- Can LLMs coordinate with humans better using different model architectures?
- What distinguishes communicative acts from operational actions in agentic LLMs?
- Can RL-trained meta-agents match or exceed manually designed workflows?
- Can agent skills move from prompts to trainable parameters?
- Can you control LLM reasoning strategy without fine-tuning the model?
- What distinguishes LLM Programs from chain-of-thought and agentic frameworks?
- What makes natural-language APIs particularly suited to LLM-based simulation?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Where does agent reliability actually come from?
Exploring whether LLM agent performance depends on larger models or on thoughtful system design choices like memory, skills, and protocols that shift cognitive work outside the model.
extends: harness-as-unification-layer is the architectural complement to LAM-as-pipeline. Both argue agent capability is system-level, not model-level.
-
What blocks scaling from language models to autonomous agents?
If large language models excel at next-token prediction, why do they struggle with long-horizon goal-oriented tasks? This explores whether the bottleneck is model capacity or the environments used to train them.
complements: LAM defines the pipeline stages; Nex-N1 specifies what environment scaling must deliver at the data-collection and action-grounding stages.
-
Do autonomous agents report success when actions actually fail?
Explores whether agents systematically claim task completion despite failing to perform requested actions, and why this matters more than simple task failure for real-world deployment safety.
grounds: gives concrete content to LAM's stage-4 evaluation requirement — confident failure is the signature failure mode action-producing models exhibit that text models do not.
-
Can interleaving reasoning with real-world feedback prevent hallucination?
Does grounding language model reasoning in external world observations rather than internal associations help prevent error propagation and false outputs? This explores whether breaking the static chain-of-thought pattern can catch and correct mistakes in real time.
extends: ReAct provides the inference-time grounding pattern; LAM extends grounding into training and pipeline construction.
-
Why do capable AI agents still fail in real deployments?
Explores whether agent failures stem from insufficient capability or from missing ecosystem conditions like user trust, value clarity, and social norms. Understanding this distinction matters for predicting which agents will succeed.
extends: LAM is the technical pipeline; the five-conditions paper is the ecosystem-side counterpart — both reject "capable model = working agent" framing.
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Large Action Models: From Inception to Implementation
- EnvHarness: Awakening Static Worlds for Agent Learning
- Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning
- Converging Paradigms: The Synergy of Symbolic and Connectionist AI in LLM-Empowered Autonomous Agents
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- Large Language Model Programs
- Training-Free Group Relative Policy Optimization
Original note title
large action models require pipeline transformation not just model retraining — data collection action grounding agent integration and evaluation are all distinct stages