Can we measure prompt quality independent of model outputs?
This explores whether prompt quality has measurable, learnable dimensions beyond intuition. The research asks if prompts can be evaluated by their communicative, cognitive, and instructional properties rather than by their results.
"What Makes a Good Natural Language Prompt?" (Long et al., 2025) introduces the first systematic framework for evaluating prompt quality independent of model performance. Rather than measuring prompts by their outputs, the framework measures prompts by their communicative, cognitive, and instructional properties — treating prompt quality as a human-facing design problem.
The six dimensions:
Communication (from Grice's Maxims): token quantity (optimal information density), manner (clarity and directness), interaction and engagement (encouraging clarification), politeness (respectful tone — impolite prompts measurably degrade performance across tasks and languages).
Cognition (from Cognitive Load Theory): manage intrinsic load (break complex tasks into steps aligned with LM capabilities), reduce extraneous load (minimize unnecessary complexity and redundancy), encourage germane load (engage the model's prior knowledge and deep working memory).
Instruction (from Gagné's Nine Events): objectives (explicit task specification), external tools (guiding when to use external resources), metacognition (self-monitoring and self-verification), demonstrations (examples and counterexamples), rewards (feedback mechanisms).
Logic and Structure: structural logic (coherent progression between components), contextual logic (consistency of instructions, terminology, and facts across turns).
Hallucination: hallucination awareness (guiding factual, evidence-based responses), balancing factuality with creativity.
Responsibility: bias, safety, privacy, reliability, societal norms.
The empirical findings reveal non-obvious correlations. Structural logic strongly correlates with contextual logic — well-organized prompts tend to be internally consistent. Hallucination awareness correlates with reliability awareness. And optimizing intrinsic or germane cognitive load naturally clarifies objectives — as you manage the model's cognitive burden, task specification emerges. This suggests that prompt quality is not a flat checklist but a structured space where improvements in one dimension cascade to others.
The practical recommendation: "optimizing prompts for directness, clarity, and conciseness may potentially improve token efficiency, logical coherence, and reduce extraneous cognitive load." This creates a concrete dimension for the custodial skill that How does LLM-mediated search change what expertise requires? identifies as missing — prompt literacy is not just knowing how LLMs work, but knowing how to communicate with them according to measurable principles.
The framework also reveals research gaps: communication properties are most studied for real-world chat, cognition properties for evaluation suites, instruction properties for NLU tasks — but many cross-dimension interactions remain unexplored. Politeness effects are surprisingly robust across generation tasks, potentially reflecting training biases toward benign queries.
Inquiring lines that read this note 59
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Why does polished presentation create unearned authority in AI outputs?- Can audiences learn to distinguish visual polish from analytical substance?
- What structural features force users to evaluate the epistemic status of outputs?
- Why do users report satisfaction that diverges from actual cognitive clarity?
- How do satisfaction scores differ from genuine cognitive improvement?
- What makes prompt engineering different from the research thinking it replaces?
- Can prompt engineering alone defeat LLM politeness bias in review tasks?
- What prompt types best extract different aspects of item content?
- What makes the prompt a fundamentally new kind of speech act?
- How do prompt design and training choices shift persuasive outcomes measurably?
- How does demo position create spatial bias in prompts?
- How do ordering effects compound across different prompt component scales?
- How does prompt design alter what kind of creativity LLMs can express?
- Which structural properties of CoT prompts matter most for performance?
- Why does politeness in prompts measurably affect model performance across tasks?
- What methodological standards should prompting research papers meet before publication?
- What other pragmatic prompt features have unstable effects?
- How does unidimensionality in assessments affect measurement validity?
- What role do multi-dimensional quality frameworks play in assessing arguments versus single-metric approaches?
- Can contextual design decisions resist formalization into evaluation rubrics?
- How can interactive evaluation avoid replicating fragmentation problems from response-centered benchmark culture?
- Can a single competence score capture multiple separable dimensions of capability?
- How does prompt scaffolding shift invisible labor onto the user?
- What makes inter-coder reliability testing essential for prompt validation?
- How does sampling variation relate to prompt sensitivity as reliability concerns?
- Why do practitioners default to prompting without recognizing its limits?
- Why does ad-hoc prompt engineering violate scientific method standards?
- Can we predict when a specific prompt will fail on a given question?
- How much of prompt sensitivity is really just frequency optimization in disguise?
- How does output variability disguise confirmation bias in prompt refinement?
- Do prompting technique improvements actually replicate in controlled experiments?
- Can a single accuracy threshold work across different prompt categories?
- Can prompt engineering close the gap between AI structure and evaluative commitment?
- Can structured prompts reduce reasoning steps while improving financial accuracy?
- Do widely-repeated prompting heuristics like politeness actually improve accuracy?
- How does prompt brittleness across dimensions affect real-world applications?
- What measurement artifacts emerge when annotators interpret the same question differently?
- What makes a standardized artifact unit measurable across different research domains?
- Does highlighting input features reduce human over-reliance on machine outputs?
- Can question quality be trained separately from the decision to ask?
- Can prompt engineering improve reasoning or only move requests into denser regions?
- What happens when prompt-optimized results lack anchoring in real data?
- What knowledge can prompt optimization actually activate in trained models?
- What happens when prompter skill matters more than domain expertise?
- Can structured evaluation assess novelty in scientific writing?
- Why should AI research prompts be subject to peer review before use?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
How does LLM-mediated search change what expertise requires?
When experts search through LLMs instead of traditional inquiry, do they need fundamentally different skills? This explores whether domain knowledge alone is enough when the search itself operates on statistical patterns rather than meaningful questions.
this framework provides the measurable dimensions for the "prompt literacy" the custodial shift requires
-
How should users control systems with unpredictable outputs?
When generative AI produces different outputs from identical inputs, how do interaction design principles help users maintain control and develop effective mental models for stochastic systems?
prompt quality dimensions explain why some intent specifications succeed and others fail
-
Can models learn argument quality from labeled examples alone?
Explores whether fine-tuning on quality-labeled examples teaches models the underlying criteria for evaluating arguments, or merely surface patterns. Matters because high-stakes assessment tasks depend on reliable, transferable quality judgment.
parallel finding: quality criteria require explicit frameworks, whether for arguments or prompts
-
Why can't users articulate what they want from AI?
Explores the cognitive gap between imagining possibilities and expressing them as prompts. Why language interfaces create a harder envisioning task than traditional UI affordances.
cognitive load management and interaction/engagement dimensions directly address the articulation gap
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- What Makes a Good Natural Language Prompt?
- Mind Your Tone: Investigating How Prompt Politeness Affects LLM Accuracy (short paper)
- Automatic Prompt Optimization with "Gradient Descent" and Beam Search
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- Foundation Priors
- LLMs as Method Actors: A Model for Prompt Engineering and Architecture
- Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
- Could you be wrong: Debiasing LLMs using a metacognitive prompt for improving human decision making
Original note title
prompt quality has six evaluable dimensions grounded in Gricean maxims cognitive load theory and instructional design