Do language models show the same content effects humans do?
Do LLMs reproduce human reasoning biases—like believing conclusions based on familiarity rather than logic—across different logical tasks? This matters because converging patterns across independent tasks suggest a fundamental architectural property rather than a task-specific quirk.
Lampinen et al. evaluate three logical reasoning tasks — Natural Language Inference, syllogism validity judgment, and the Wason selection task — and find LMs reproduce the same content-sensitivity patterns humans show. NLI: accuracy depends on whether the believable completion matches the logically correct one. Syllogisms: judgments are biased by whether the conclusion is believable, reproducing Evans et al.'s belief-bias effect where humans endorse invalid syllogisms with believable conclusions roughly 90% of the time. Wason: accuracy improves when the conditional rule is instantiated as a familiar social rule rather than an abstract pattern. Three independent task structures with different logical demands all produce the same content-form entanglement.
The pattern matters because each individual task could be dismissed as a quirk. Three tasks converging on the same signature licenses calling content-form entanglement an architectural property rather than a benchmark artifact. The mechanistic vault notes establish why at circuit level: How do language models perform syllogistic reasoning internally? shows the formal-circuit + world-knowledge-contamination structure that produces belief-bias. This insight contributes the behavioral isomorphism — not just that the circuit produces some kind of contamination, but that the contamination's signature matches human error patterns item-for-item, including continuous response measures (LM token-probability distributions track human reaction times).
This converges with Do large language models reason symbolically or semantically? from the opposite direction. That note shows reasoning collapses when semantics are stripped; Lampinen shows reasoning improves with believable semantics and degrades with unbelievable semantics. Both findings point at the same property: the model is doing something like in-context semantic reasoning, where logical form is one input among others rather than the dominant computational frame. Calling this "reasoning" or "not reasoning" is the wrong question — the right question is what kind of reasoning, and the answer is reasoning that is constitutively content-sensitive, in humans and LMs alike, by the same item-level patterns.
Inquiring lines that read this note 45
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Is language model reasoning authentic and what causes models to reason?- How do LLM biases manifest differently across the three paradigms?
- How does rhetorical familiarity bias models toward their own arguments?
- Why does hypothesis attestation bias exist separately from frequency bias in NLI?
- How does the LLM Fallacy differ from automation bias and cognitive offloading?
- Why do LLMs fall for and deploy logical fallacies with equal confidence?
- Which knowledge types do LLMs handle better than humans in reasoning tasks?
- Can users experience the LLM Fallacy even when AI outputs are completely accurate?
- What structural differences between human and LLM production create detectable signatures?
- How do LLM biases reflect social classification schemas rather than random errors?
- Do LLMs genuinely internalize human psychological structure or match surface patterns?
- Do LLMs actually reason differently than humans about moral dilemmas?
- How do bimodal decision patterns in LLMs compare to human economic choice?
- Why does optimism bias disappear when LLMs passively observe outcomes?
- How do minimal wording changes affect LLM moral reasoning consistency?
- How do LLMs default to surface-level strategies instead of genuine mental simulation?
- Why do users attribute beliefs to LLMs despite uncertainty about their minds?
- Do language models exhibit the same causal biases that humans show?
- What distinguishes surface generalizations from true linguistic generalizations?
- Do newer LLM generations create worse detector bias through increased linguistic divergence?
- Does the langue-parole distinction apply to human reasoning too?
- How do LLMs access and draw on the same shared symbolic universe as humans?
- How does this motivational bias connect to LLMs' causal reasoning failures?
- Why do LLMs inherit causal biases from their training data?
- How might human-LLM teams reinforce each other's causal reasoning mistakes?
- Do LLMs rely on surface statistical patterns instead of causal structure?
- Do language models show the same truth bias as humans?
- How does truth bias in humans compare to face-saving in LLMs?
- Why do LLMs show gender bias but humans evaluators do not?
- What other evaluation biases exist in LLM judge systems?
- What systematic biases do LLM judges introduce into AI-evaluated debates?
- Why do LLMs swing on minor rewording yet ignore explicit bias correction instructions?
Related concepts in this collection 2
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do large language models reason symbolically or semantically?
Can LLMs follow explicit logical rules when those rules contradict their training knowledge? Testing whether reasoning operates independently of semantic associations reveals what computational mechanisms actually drive LLM multi-step inference.
opposite-direction confirmation: stripping semantics breaks reasoning
-
How do language models perform syllogistic reasoning internally?
Does formal symbolic reasoning exist as a distinct neural circuit in LLMs, or is it inevitably contaminated by world knowledge associations? Understanding the mechanism could reveal whether pure logical reasoning is separable from semantic inference.
mechanistic explanation for belief-bias signature
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Language models show human-like content effects on reasoning tasks
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Premise Order Matters in Reasoning with Large Language Models
- Large Language Models Can Infer Psychological Dispositions of Social Media Users
- Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Semantic Structure in Large Language Model Embeddings
- Available but Unclaimed: An Empirical Study of Human-AI Synergy
Original note title
content effects in LLMs are behavioral confirmation that semantic content and logical form are not separable in transformer reasoning — across NLI syllogisms and Wason