Do language models show the same content effects humans do?
Do LLMs reproduce human reasoning biases—like believing conclusions based on familiarity rather than logic—across different logical tasks? This matters because converging patterns across independent tasks suggest a fundamental architectural property rather than a task-specific quirk.
Lampinen et al. evaluate three logical reasoning tasks — Natural Language Inference, syllogism validity judgment, and the Wason selection task — and find LMs reproduce the same content-sensitivity patterns humans show. NLI: accuracy depends on whether the believable completion matches the logically correct one. Syllogisms: judgments are biased by whether the conclusion is believable, reproducing Evans et al.'s belief-bias effect where humans endorse invalid syllogisms with believable conclusions roughly 90% of the time. Wason: accuracy improves when the conditional rule is instantiated as a familiar social rule rather than an abstract pattern. Three independent task structures with different logical demands all produce the same content-form entanglement.
The pattern matters because each individual task could be dismissed as a quirk. Three tasks converging on the same signature licenses calling content-form entanglement an architectural property rather than a benchmark artifact. The mechanistic vault notes establish why at circuit level: How do language models perform syllogistic reasoning internally? shows the formal-circuit + world-knowledge-contamination structure that produces belief-bias. This insight contributes the behavioral isomorphism — not just that the circuit produces some kind of contamination, but that the contamination's signature matches human error patterns item-for-item, including continuous response measures (LM token-probability distributions track human reaction times).
This converges with Do large language models reason symbolically or semantically? from the opposite direction. That note shows reasoning collapses when semantics are stripped; Lampinen shows reasoning improves with believable semantics and degrades with unbelievable semantics. Both findings point at the same property: the model is doing something like in-context semantic reasoning, where logical form is one input among others rather than the dominant computational frame. Calling this "reasoning" or "not reasoning" is the wrong question — the right question is what kind of reasoning, and the answer is reasoning that is constitutively content-sensitive, in humans and LMs alike, by the same item-level patterns.
Inquiring lines that use this note as a source 41
This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.
- How do LLM biases manifest differently across the three paradigms?
- Why do LLMs fall for and deploy logical fallacies with equal confidence?
- How do different LLM integration paradigms affect inheritance of pretraining biases?
- Does epistemic drift operate the same way across all languages?
- How do LLM biases reflect social classification schemas rather than random errors?
- Do LLMs genuinely internalize human psychological structure or match surface patterns?
- Do language models exhibit the same causal biases that humans show?
- What circuit mechanisms produce belief bias in syllogistic reasoning?
- Does the langue-parole distinction apply to human reasoning too?
- Where do humans and language models actually diverge in reasoning ability?
- How does rhetorical familiarity bias models toward their own arguments?
- How does tone sensitivity create systematic informational bias in model responses?
- Why does hypothesis attestation bias exist separately from frequency bias in NLI?
- How do LLMs access and draw on the same shared symbolic universe as humans?
- Do LLMs actually reason differently than humans about moral dilemmas?
- How do bimodal decision patterns in LLMs compare to human economic choice?
- Why does optimism bias disappear when LLMs passively observe outcomes?
- How does this motivational bias connect to LLMs' causal reasoning failures?
- Do language models show the same truth bias as humans?
- Why do transformer attention patterns show positional and sequential bias across tasks?
- How does truth bias in humans compare to face-saving in LLMs?
- Why do LLMs show gender bias but humans evaluators do not?
- Why do LLMs inherit causal biases from their training data?
- How might human-LLM teams reinforce each other's causal reasoning mistakes?
- Do LLMs rely on surface statistical patterns instead of causal structure?
- Which knowledge types do LLMs handle better than humans in reasoning tasks?
- What distinguishes surface generalizations from true linguistic generalizations?
- How do minimal wording changes affect LLM moral reasoning consistency?
- How do humans and R1 models differ in information gain patterns?
- How do LLMs default to surface-level strategies instead of genuine mental simulation?
- Why do users attribute beliefs to LLMs despite uncertainty about their minds?
- Do reasoning models become more vulnerable to persona-induced bias than standard models?
- How do we verify that stated beliefs actually follow from underlying motifs?
- What role does inductive bias play versus model capacity in practice?
- Can users experience the LLM Fallacy even when AI outputs are completely accurate?
- How does the LLM Fallacy differ from automation bias and cognitive offloading?
- What other evaluation biases exist in LLM judge systems?
- Can implicit association tests reveal LLM biases beneath trained responses?
- Do newer LLM generations create worse detector bias through increased linguistic divergence?
- What structural differences between human and LLM production create detectable signatures?
- How does typicality bias in human annotation affect downstream model behavior?
Related concepts in this collection 2
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do large language models reason symbolically or semantically?
Can LLMs follow explicit logical rules when those rules contradict their training knowledge? Testing whether reasoning operates independently of semantic associations reveals what computational mechanisms actually drive LLM multi-step inference.
opposite-direction confirmation: stripping semantics breaks reasoning
-
How do language models perform syllogistic reasoning internally?
Does formal symbolic reasoning exist as a distinct neural circuit in LLMs, or is it inevitably contaminated by world knowledge associations? Understanding the mechanism could reveal whether pure logical reasoning is separable from semantic inference.
mechanistic explanation for belief-bias signature
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Language models show human-like content effects on reasoning tasks
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Premise Order Matters in Reasoning with Large Language Models
- Large Language Models Can Infer Psychological Dispositions of Social Media Users
- Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Semantic Structure in Large Language Model Embeddings
- Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners
Original note title
content effects in LLMs are behavioral confirmation that semantic content and logical form are not separable in transformer reasoning — across NLI syllogisms and Wason