Can AI agents learn when they have something worth saying?
What if AI proactivity came from modeling intrinsic motivation to participate rather than predicting who speaks next? This explores whether a framework based on human cognitive patterns—internal thought generation parallel to conversation—can make agents genuinely responsive rather than passively reactive.
The Inner Thoughts framework reverses the standard approach to AI proactivity. Instead of asking "who speaks next?" (next-speaker prediction, which fails to beat the "repeat last" baseline in social contexts), it asks "does this agent have something worth saying right now?"
The framework is inspired by cognitive psychology's distinction between covert responses (internal thoughts, feelings) and overt responses (verbal utterances, gestures). During human conversation, we process others' words, reflect on our experiences, and develop an internal train of thought. At some point we feel a strong urge to contribute — when we seek clarification, when someone mentions something we've experienced, when we detect a gap. The Inner Thoughts framework equips AI with this parallel covert stream.
Five stages structure the framework:
- Trigger — detecting a conversation moment worth processing
- Retrieval — accessing long-term and working memory for relevant knowledge
- Thought formation — generating a covert thought about the current exchange
- Evaluation — scoring intrinsic motivation to express this thought
- Participation — deciding whether to contribute based on the evaluation
The intrinsic motivation model draws from a think-aloud study with 24 participants across four group chats. Ten high-level themes emerged for how people decide to engage: relevance, information gap, emotional resonance, social obligation, etc. These are formalized into automatic evaluation criteria.
Technical evaluation shows agents driven by Inner Thoughts significantly outperform next-speaker prediction plus persona baselines across seven metrics: turn appropriateness, coherence, anthropomorphism, perceived engagement, intelligence, initiative, and adaptability. Participants preferred Inner Thoughts 82% of the time.
The distinction from CoT/ToT/o1 reasoning is important: those externalize intermediate steps for task reasoning. Inner Thoughts generate a parallel covert stream that models social motivation to participate — not task decomposition but interaction participation. Since Why can't conversational AI agents take the initiative?, this framework provides a concrete architecture for the missing proactivity.
Curiosity reward for personalization is a specific application of intrinsic motivation. While Inner Thoughts uses 10 general social motivation heuristics (from cognitive psychology think-aloud studies), the curiosity reward approach targets a specific type of intrinsic motivation: reducing uncertainty about the user's type. The agent is rewarded for each turn that improves its belief about who it's talking to — encouraging strategic questions and context-sensitive probes aimed at uncovering user preferences, personality, or attributes. This is personalization-specific proactivity rather than general social proactivity. The two approaches may be complementary: Inner Thoughts determines when to speak based on social motivation; curiosity reward determines what to say to learn about the user. See Can conversations themselves personalize without user profiles?.
Inquiring lines that use this note as a source 23
This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.
- What cognitive capabilities do agents need to internalize social feedback?
- What social patterns from human training data activate in agent context?
- What are the five specific conversation triggers where AI intervention adds value?
- How does the silent token approach compare to modeling intrinsic motivation for speaking?
- Can inner thoughts solve the importance recognition problem for agents?
- How do intrinsic motivation principles explain why generating novel challenges improves learning?
- How does intrinsic motivation drive conversational agents beyond passive responsiveness?
- How do intrinsic motivation mechanisms differ between social proactivity and personalization?
- Can AI learn to perform attention-seeking surface forms with genuine internal appeal?
- What are the ten intrinsic motivation heuristics that drive participation decisions?
- Does social scaffolding outperform purely intrinsic motivation for agent exploration?
- Why do chatbots default to external help instead of intrinsic motivation strategies?
- Could reward signals incentivize active intent discovery over passive response generation?
- Do behavioral cues enable proactive AI without event-triggered decision points?
- How can agents detect whether users are willing to follow their topic guidance?
- How does asymmetric information between users and agents relate to proactivity?
- How do agents decide when to abstain from contributing?
- Do people consciously notice social cues or respond automatically to them?
- What makes proactivity useful instead of intrusive in conversation?
- How do humans decide when to contribute to group conversations?
- Can AI learn intrinsic motivation to assess its own relevance?
- Can AI take initiative by questioning without being proactive in directive ways?
- How do agents decide when to pause and reflect on their strategy?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Why can't conversational AI agents take the initiative?
Explores whether current LLMs lack the structural ability to lead conversations, set goals, or anticipate user needs—and what architectural changes might enable proactive dialogue.
Inner Thoughts is the strongest architectural answer to the structural passivity diagnosis
-
How can proactive agents avoid feeling intrusive to users?
Explores why proactive conversational agents often feel annoying rather than helpful, and what design dimensions could prevent them from violating user expectations and autonomy.
intrinsic motivation scoring is how Inner Thoughts manages the civility dimension
-
Can models learn to ask clarifying questions instead of guessing?
Exploring whether large language models can be trained to detect incomplete queries and actively request missing information rather than hallucinating answers or refusing to respond. This matters because conversational agents today remain passive, responding only when prompted.
proactive critical thinking is one specific type of intrinsic motivation to speak
-
How should agents decide what memories to keep?
Agent memory management splits between agents autonomously recognizing important information versus programmatic triggers. Understanding this choice reveals why different memory architectures prioritize different information types.
inner thoughts could serve as the importance recognition layer for explicit hot-path memory: the agent's continuous covert thoughts identify what's worth remembering, solving the "what matters" problem
-
Why can't advanced AI models take initiative in conversation?
Despite extraordinary capability in answering and reasoning, LLMs fundamentally cannot initiate, redirect, or guide exchanges. Understanding this gap—and whether it's fixable—matters for building AI that truly collaborates rather than merely responds.
Inner Thoughts is the strongest architectural answer to the passivity problem: where the passivity diagnosis identifies next-turn reward optimization as the structural cause, Inner Thoughts provides a concrete mechanism (continuous covert motivation scoring) that operates independently of the turn-reward signal
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Proactive Conversational Agents with Inner Thoughts
- DiscussLLM: Teaching Large Language Models When to Speak
- Thinking LLMs: General Instruction Following with Thought Generation
- Building Machines that Learn and Think with People
- Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
- Towards Human-centered Proactive Conversational Agents
- Psychologically Enhanced AI Agents
- RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agents
Original note title
inner thoughts framework enables proactive AI by modeling intrinsic motivation through continuous covert thought generation parallel to conversation