Can AI generate assessment questions as good as human experts?

A rigorous psychometric evaluation comparing ChatGPT-generated formative assessment questions to published Creative Commons textbook questions finds no statistically significant differences on the properties that matter for measurement quality.

Using Item Response Theory (IRT) with a linking methodology to ensure comparability, the study (N=207) tested 15 ChatGPT-generated items against 15 human-authored items from the same lesson content. Results:

• Difficulty parameters — no significant difference between pools
• Discrimination parameters — no significant difference, with some evidence ChatGPT items were marginally better at differentiating respondent abilities
• Response time — no significant difference
• Unidimensionality — ChatGPT items showed evidence of measuring a single construct and did not disrupt the unidimensionality of the original set when tested together

This is notable because psychometric quality is a higher bar than surface-level plausibility. Difficulty and discrimination are the core parameters in educational measurement — they determine whether a question is appropriately challenging and whether it distinguishes students who understand the material from those who don't. Matching human experts on these parameters means the generation is functionally equivalent for formative assessment purposes.

However, the scope is constrained: one lesson summary, one textbook, formative (not summative) assessment. The generalization to diverse subjects, higher-stakes testing, or open-ended question formats remains untested.

The finding connects to a broader pattern in LLM generation quality: since Can LLMs generate more novel ideas than human experts?, structured generation tasks with clear constraints (like assessment items from a lesson summary) may represent a sweet spot where LLMs match or exceed human quality — while open-ended evaluative tasks remain a weakness.