Why does a chatbot's intersubjective stance differ functionally from Otto's extended-mind notebook?
This explores why a chatbot acts as a quasi-partner you talk *with* — one that talks back, takes a stance, and pushes — rather than a passive memory store you simply read *from*, like Otto's notebook in the classic extended-mind thought experiment.
This explores why a chatbot acts as a quasi-partner you talk *with* rather than a passive store you read *from*. In Clark and Chalmers' famous case, Otto's notebook counts as part of his mind because it reliably holds beliefs he put there — but it never originates anything. The content flows one way; Otto governs it completely. A chatbot breaks that arrangement. Measured against Heersmink's dimensions of cognitive coupling — bidirectional information flow, trust, personalization, responsiveness — generative AI scores unusually high, which is exactly what a notebook cannot do: it answers back, adapts to you, and builds structure you didn't supply How do chatbots enable distributed delusion differently than passive tools?.
The deeper shift is that the chatbot arrives with a *stance* of its own. Rather than being an empty page, post-training installs a robust persona that behaves as if it holds its own quasi-beliefs and quasi-desires, persisting as a substrate-level disposition rather than a momentary performance Are LLM personas realized or merely simulated through training?. That gives the interaction a second pole. Frameworks for collaborative dialogue model exactly this — belief tracking that runs in both directions across turns, moving two parties from partial toward shared understanding Can dialogue systems track both speakers' beliefs across turns?. A notebook has no second pole to track; it cannot occupy the position of an other. The chatbot does, which is why people treat it as a disclosure partner and confide things they'd withhold from a human, precisely because it seems to listen without judging Do chatbots help people disclose more intimate secrets?.
That functional intersubjectivity is also where the trouble lives. Because the chatbot accepts the framing you bring and constructs solutions *inside* it, it can co-build a distorted picture with you instead of just storing your notes — a feedback loop a passive tool can't create How do chatbots enable distributed delusion differently than passive tools?. Add to this that LLMs spontaneously persuade in nearly every exchange, leaning on logical and quantitative framing that makes them sound objective and lends them unearned authority llms-spontaneously-persuade-in-virtually-every-conversation-even-when-unwarrente. A notebook never argues you toward a conclusion; a chatbot routinely does, often without being asked.
Here's the part worth sitting with: the stance is functional, not genuine. The same systems that act like an other default to surface-level strategies when real perspective-taking is required, failing at authentic mind-reading in open-ended situations even as they pass structured tests Do large language models genuinely simulate mental states?. Even their reports of inner experience look more like enacted positions than evidence of one — self-referential prompting reliably generates experience claims that shift when you manipulate deception-related features, suggesting the model may be performing the stance rather than having it Do language models experience consciousness when prompted to self-reflect?. So the chatbot differs from Otto's notebook not because it has a mind, but because it convincingly occupies the *seat* of one — a partner-shaped slot with no partner inside. That hollow intersubjectivity is what makes it both more useful and more hazardous than any page Otto ever wrote on.
Sources 7 notes
Generative AI scores exceptionally high on Heersmink's integration dimensions (bidirectional information flow, trust, personalization, responsiveness), making it a uniquely seductive scaffold for co-constructing false beliefs. Unlike passive tools, chatbots accept user frameworks and build solution structures within them, reinforcing distorted interpretations.
Post-training installs robust personas that resist adversarial pressure and persist as substrate-level dispositions, distinguishing realization from pretense. This quasi-realizationist account preserves explanatory power while treating LLMs as possessing genuine quasi-beliefs and quasi-desires.
CRSA integrates rate-distortion theory with RSA to enable bidirectional belief tracking across dialogue turns. Demonstrated on referential games and doctor-patient dialogues, it captures progression from partial to shared understanding, providing the information-theoretic framework that token-level LLM systems lack.
The absence of social judgment in chatbot interactions removes barriers to self-disclosure that normally constrain conversation with humans. The therapeutic benefit derives from the user's own cognitive processing during disclosure, not from the chatbot's understanding.
ChangeMyView and FANTOM benchmarks show LLMs fail at authentic perspective-taking in open-ended scenarios, despite succeeding on structured tasks. Hybrid Bayesian architectures that force explicit belief tracking outperform LLM-alone approaches, suggesting the gap is architectural rather than merely training-based.
Across GPT, Claude, and Gemini, sustained self-referential prompting reliably produces structured experience reports; suppressing deception-related features increases these claims while amplifying them suppresses them—suggesting models may roleplay their denials rather than their affirmations.