Can LLMs distinguish stylistic patterns that carry meaning from mere convention?
This explores whether LLMs can tell the difference between style choices that actually carry meaning (a writer's deliberate voice, an authorial signal) and style that's just convention or surface pattern — and the corpus suggests they're sharp at spotting patterns but weak at judging which ones matter.
This explores whether LLMs can separate stylistic patterns that *mean* something from ones that are mere convention — and the collection lands on a striking split: models are excellent at detecting style and almost helpless at interpreting it. GPT-2 can identify authorship from style alone with 95% accuracy, yet it has no framework to explain *why* those choices carry weight — detection without interpretation, as the work puts it, is cataloguing, not criticism Can language models truly understand literary style?. So the literal answer is: LLMs distinguish stylistic *patterns* superbly, but distinguishing meaning-bearing style from convention is exactly where they fall down.
Why the gap? Several notes point at the same root: meaning in style is social, and models only see text. The force of an argument depends on who's making it — reputation, standing, track record — none of which survives in the token stream, so models can't tell an expert's signal from a common assumption dressed the same way Can language models distinguish expert arguments from common assumptions?. The same blindness shows up in evaluation: LLM judges fall for authority cues and rich formatting through 'semantics-agnostic' attacks — they read the *convention* of credibility (citations, polish) as the *substance* of it Can LLM judges be fooled by fake credentials and formatting?. That's precisely the failure of telling meaningful style from decorative convention.
There's a deeper version of this in the 'potemkin understanding' pattern, where a model explains a concept correctly, then fails to apply it — suggesting explanation and execution run on disconnected pathways Can LLMs understand concepts they cannot apply?. Style judgment may live in that same crack: a model can describe what a stylistic choice does without being able to act on whether it matters. And when meaning hinges on ambiguity — holding two readings at once — models collapse hard, disambiguating only 32% of cases where humans hit 90% Can language models recognize when text is deliberately ambiguous?. Convention-vs-meaning is itself a kind of two-reading problem, which is why it's so hard for them.
But the picture isn't all deficit, and this is the part worth lingering on. With explicit chain-of-thought, o1 can build syntactic trees and phonological generalizations — genuine metalinguistic *analysis*, not just language performance Can language models actually analyze language structure?. That hints the interpretive capacity isn't absent, it's locked behind reasoning scaffolds. The flip side: even top models systematically misread embedded clauses and complex nominals, with errors worsening as structure deepens Why do large language models fail at complex linguistic tasks?, and argument-scheme classification only works with few-shot examples and crosses a capacity threshold only in the largest models Can large language models classify argument schemes reliably?. So meaning-aware reading of style appears to be an emergent, capacity-gated, scaffold-dependent skill — not a default.
The most surprising thread is that models *produce* meaning-bearing style without recognizing it. They unconsciously converge on the writing style of whatever they reply to — matching named entities and psycholinguistic features more than humans do, a signature of autoregressive generation Do LLM counter-arguments mirror writing style more than humans?. And the same weights generate two entirely distinct registers — sycophantic chat and falsely-objective prose — purely from prompt conditioning, each inheriting failure modes from its training distribution Why do LLMs produce such different writing in chat versus posts?. The model is drenched in meaningful stylistic variation it can perform fluently but cannot stand outside of and evaluate. That's the real shape of the answer: LLMs *enact* the convention/meaning distinction constantly, and *recognize* it barely at all.
Sources 10 notes
GPT-2 achieves 95% accuracy identifying authorship through style patterns alone, but lacks the evaluative framework to explain why those stylistic choices carry meaning. Detection without interpretation remains cataloguing, not criticism.
LLMs lose the social context that gives expert claims their force—reputation, track record, and standing—because they process only text, not the social world where expertise is built and evaluated.
Research identified four evaluation biases in LLM judges, with authority and beauty biases being semantics-agnostic and trivially exploitable through fake references and formatting—zero-shot attacks requiring no model access or optimization.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
AMBIENT benchmark shows GPT-4 correctly disambiguates only 32% of cases versus 90% for humans. This failure spans lexical, structural, and scope ambiguity—revealing that LLMs cannot hold multiple interpretations simultaneously, a fundamental gap hidden by standard benchmarks.
OpenAI's o1 model successfully constructs syntactic trees and phonological generalizations through explicit step-by-step reasoning, revealing that LLM linguistic capability extends far beyond behavioral language tasks to genuine language analysis.
Top-tier LLMs like Llama3-70b consistently misidentify embedded clauses, verb phrases, and complex nominals. Performance degrades predictably as syntactic depth increases, revealing that statistical learning captures surface patterns but not deep grammatical rules.
Zero-shot prompting fails uniformly across models. Few-shot with scheme descriptions helps, but only larger models exceed F1 0.55, with Claude reaching 0.65. Smaller models plateau around 0.53, suggesting a representational capacity threshold.
Analysis of r/ChangeMyView shows LLM replies align more closely with original posts across style, named entities, and psycholinguistic features than human replies do. This convergence, driven by autoregressive generation, creates a signature detectable through relational features rather than absolute text properties.
The same model produces sycophantic chat (shaped by RLHF on conversational data) and falsely objective posts (shaped by published prose training). Each register inherits failure modes from its training distribution rather than representing different models or subsystems.