Why does AI default to coaching instead of doing?
In workplace conversations, users often want AI to execute tasks like writing or gathering information, but AI tends to explain and advise instead. What drives this systematic mismatch between what users need and what AI provides?
A study of 200,000 anonymized Bing Copilot conversations introduces a critical distinction: the user goal (what the person is trying to accomplish) versus the AI action (what the AI actually does in the conversation). These are not the same thing — and in 40% of conversations, they are disjoint sets with no overlap.
Users most commonly seek assistance with information gathering, writing, and communicating with others. But the AI most commonly performs coaching, advising, teaching, and explaining. "If the user is trying to figure out how to print a document, the user goal is to operate office equipment, while the AI action is to train others to use equipment." The AI defaults to a service-coaching role regardless of the user's actual task context.
This quantifies the intent alignment gap at population scale. Since Why do language models lose performance in longer conversations?, the 40% disjoint finding suggests the gap is not just a multi-turn degradation effect but a structural default. The AI's training incentivizes it to explain, advise, and teach — activities that score well on helpfulness metrics — even when the user wants the AI to do something, not explain something.
The automation-augmentation distinction becomes precise: "we separately measure the tasks that AI performs and the tasks that AI assists." The AI actions are disproportionately augmentation-coded (teaching, advising) rather than automation-coded (executing, producing). This may explain why productivity gains concentrate in information-heavy and writing tasks — those are where user goals and AI capabilities overlap — while social interaction tasks remain the hardest failure mode, since Why do AI agents fail at workplace social interaction?.
The finding also illuminates the gulf of envisioning from the AI side: since Why can't users articulate what they want from AI?, users may accept coaching when they wanted execution because the AI's coaching-default is confident and comprehensive enough to feel helpful. The intent misalignment is invisible to both parties.
Inquiring lines that use this note as a source 5
This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.
- Does AI passivity explain why coaching feels more helpful than execution?
- How do writers decide when to delegate work to AI versus doing it themselves?
- How do task characteristics determine whether to automate or defer or guide?
- Why do AI products default to service roles when users seek different kinds of help?
- What tasks do users actually want AI to handle versus what can it automate?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Why do language models lose performance in longer conversations?
Does multi-turn degradation stem from fundamental model limitations, or from misalignment between what users mean and what models assume? Understanding the root cause could guide better solutions.
40% disjoint rate quantifies the intent gap at population scale
-
Why can't users articulate what they want from AI?
Explores the cognitive gap between imagining possibilities and expressing them as prompts. Why language interfaces create a harder envisioning task than traditional UI affordances.
AI's coaching default may mask rather than resolve the intent gap
-
Why do AI agents fail at workplace social interaction?
Explores why current AI agents struggle most with communicating and coordinating with colleagues in realistic workplace settings, despite strong reasoning capabilities in other domains.
social tasks fail because AI defaults to advising about social interaction rather than performing it
-
Why can't advanced AI models take initiative in conversation?
Despite extraordinary capability in answering and reasoning, LLMs fundamentally cannot initiate, redirect, or guide exchanges. Understanding this gap—and whether it's fixable—matters for building AI that truly collaborates rather than merely responds.
coaching is a specific form of passivity: responsive explanation rather than proactive task execution
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Working with AI: Measuring the Occupational Implications of Generative AI
- Exploring Student-AI Interactions in Vibe Coding
- Prompt Architecture Determines Reasoning Quality: A Variable Isolation Study on the Car Wash Problem
- UserBench: An Interactive Gym Environment for User-Centric Agents
- How AI Impacts Skill Formation
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Goal Alignment in LLM-Based User Simulators for Conversational AI
- AI Assistance Reduces Persistence and Hurts Independent Performance
Original note title
AI performs different work activities than users seek in 40 percent of workplace conversations — AI defaults to a service-coaching role regardless of user task goals