Can breaking down visual reasoning into three stages improve model performance?
This explores whether structuring visual reasoning through perception, situation, and norm stages—grounded in cognitive science—helps language models reason about socially complex scenes better than flat chain-of-thought approaches.
Cognitive Chain-of-Thought (CoCoT) introduces a three-stage prompting strategy for visual language models that mirrors how humans process socially complex scenes. Unlike flat CoT that treats reasoning as linear, CoCoT structures reasoning through progressively abstract interpretation stages grounded in cognitive science.
The three stages:
Perception (Embodied) — "Based on the image, describe what is directly observable." Anchors reasoning in concrete perceptual evidence. The model actively interprets rather than passively processing visual features.
Situation (Embedded + Enactive) — "Based on the identified elements, determine the relationships or context among them." Captures social dynamics and contextual cues from lived interaction. Infers situational meaning beyond surface perception.
Norm (Extended) — "Based on the above reasoning stages, infer the most socially plausible interpretation." Reasons over socially constructed values and expectations that transcend the immediate context but remain grounded in prior interpretation.
The theoretical grounding is 4E cognition (Newen et al. 2018): cognition is Embodied (shaped by bodily interactions), Embedded (situated in environmental context), Enactive (emerging through action and interaction), and Extended (augmented by external tools and social structures). CoCoT maps each stage to a different cognitive dimension.
Results: +8% average improvement over flat CoT and direct prompting across intent disambiguation, commonsense reasoning, and safety benchmarks. The improvement is specifically on socially complex visual tasks where bridging perception with norm-grounded judgment is essential.
Since Can reasoning topologies be formally classified as graph types?, CoCoT represents a different structural principle: not branching or graph traversal, but progressive abstraction within a linear chain. Each stage constrains the next — norms must be grounded in situations, which must be grounded in perceptions. This enforces interpretive coherence that flat CoT does not.
The connection to social reasoning is direct. Since Why do reasoning models struggle with theory of mind tasks?, social tasks may specifically benefit from cognitively structured scaffolding rather than more reasoning tokens. CoCoT's success on social benchmarks supports this: the structure matters more than the volume.
Inquiring lines that use this note as a source 9
This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.
- What scaffolding tools help users specify implicit contextual boundaries to models?
- Can parsing screens into structured elements before acting improve vision models?
- What fine-grained distinctions matter most for human situated action in categories?
- How should visual content be connected to text within a unified knowledge representation?
- How does factoring perception from reasoning improve sparse-label learning?
- What cognitive structures do realistic belief models need to include?
- Should GUI perception happen inside or outside the foundation model?
- Why must world models be nested rather than flat and uniform?
- Why does chain-of-thought fail to improve multimodal model perception performance?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can reasoning topologies be formally classified as graph types?
This explores whether Chain of Thought, Tree of Thought, and Graph of Thought represent distinct formal graph structures with different computational properties. Understanding this matters because the topology itself determines what reasoning strategies are possible.
CoCoT is a different structural principle: progressive abstraction rather than branching
-
Why do reasoning models struggle with theory of mind tasks?
Extended reasoning training helps with math and coding but not social cognition. We explore whether reasoning models can track mental states the way they solve formal problems, and what that reveals about the structure of social reasoning.
social tasks benefit from cognitive structure over volume
-
Can modular cognitive tools unlock reasoning without training?
Can reasoning capabilities be elicited by structuring LLM calls as isolated cognitive operations—understanding, recalling, examining, and backtracking—rather than through reinforcement learning?
CoCoT provides an alternative: cognitive scaffolding through prompting rather than tool calls
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Cognitive Chain-of-Thought: Structured Multimodal Reasoning about Social Situations
- Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models
- A Comment On "The Illusion of Thinking": Reframing the Reasoning Cliff as an Agentic Gap
- Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
- Large Language Model Reasoning Failures
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
- Decoupling Knowledge and Reasoning in LLMs: An Exploration Using Cognitive Dual-System Theory
Original note title
cognitive chain-of-thought scaffolds visual reasoning through three cognitively grounded stages — perception situation and norm