Can an AI sort thousands of public comments into policy themes as reliably as trained human reviewers do?
Can AI systems generate policy themes as well as humans can map them?
This explores whether AI can do the job of reading thousands of public consultation responses and sorting them into policy themes as reliably as trained human reviewers, and what 'as well as' should even mean when humans don't fully agree with each other.
This explores whether AI can do the job of reading public consultation responses and sorting them into policy themes about as reliably as trained human reviewers. The corpus has one direct test, and its answer is a qualified yes. The UK government's Consult tool was scored against expert reviewers. It disagreed with them only slightly more than two human reviewers disagreed with each other (an F1 score, a standard overlap measure, of 0.76 for AI versus 0.81 between humans). Those disagreements rarely changed which themes came out on top Does AI theme-mapping perform as well as human reviewers?. The surprising lesson is about the benchmark. Theme-mapping has no single right answer, so the useful test isn't 'did the AI match the truth' but 'does the AI fall within the range where reasonable humans already disagree.'
There's a catch: matching human accuracy on average doesn't mean making human-like mistakes. In a study of social norms, GPT-4.5 judged what counts as appropriate behavior better than every individual human tested. But all the AI models got the same unwritten norms wrong in the same way Can AI learn social norms better than humans?. For consultations, that matters. Human reviewers' errors are scattered, so they tend to cancel out. An AI's errors may be consistent, so the same kind of response could be misread every time. A good overall score can hide a blind spot that always lands on the same side.
A second clue comes from creative writing. AI-generated stories spell out their themes and prefer tidy, single-track plots, while human writing leaves room for ambiguity Do AI stories explain their themes more than human stories do?. That's a different task, but it hints at a tendency worth watching in policy work: AI may pull messy, mixed-feeling responses into neat categories. Some public submissions are valuable precisely because they don't fit the existing themes.
Finally, mapping themes is different from deciding what to do about them. Levine argues that even if AI could compute answers to 'wicked' policy problems, it wouldn't have the political standing to decide whose values count. That role belongs to democratic institutions Can AI systems legitimately resolve wicked policy problems?. This is why the form of the output matters. Work on formal argumentation shows that structured outputs let people point to the exact claim they reject, while ordinary LLM prose doesn't Can formal argumentation make AI decisions truly contestable?. A theme map that links each theme back to the responses behind it can be checked and challenged. A plain summary can't.
To be clear about the limits: the corpus has a single head-to-head evaluation of policy theme-mapping, and the rest is adjacent evidence. The fair reading is that AI is already about as good as an extra human reviewer at sorting responses. The open questions are whether its errors are systematic, and whether anyone can trace and contest how it grouped the public's voice.
Sources 5 notes
UK government's Consult tool achieved F1 0.76 against expert reviewers, compared to F1 0.81 between two human reviewers. Differences rarely affected which themes ranked top, suggesting AI performance was competitive despite inherent subjectivity in theme assignment.
GPT-4.5 outperformed every individual human at judging social appropriateness across 555 scenarios, challenging the theory that embodied cultural experience is necessary. However, all AI models share identical systematic errors on unwritten norms.
Analysis of 304 narrative features reduced to 30 core signals shows AI fiction systematically over-explains themes, uses tidy single-track plots, and avoids moral ambiguity, while human stories employ temporal complexity and nonlinear structure. This pattern holds across all five major LLM models tested.
Levine argues AI's constraint on value questions is a policy choice by designers, not a technical impossibility. Even if AI could compute answers to wicked problems, it would lack the political standing to settle whose values count—a role exclusive to legitimate democratic institutions.
Dung-style argumentation structures AI outputs as traversable attack/defense graphs, allowing users to identify and contest specific premises. Standard LLM outputs lack this structure, making it impossible to pinpoint which claims users actually reject.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- AI Models Exceed Individual Human Accuracy in Predicting Everyday Social Norms
- Argumentative Large Language Models for Explainable and Contestable Decision-Making
- StoryScope: Investigating idiosyncrasies in AI fiction
- Will AI Solve Our Wicked Problems?
- The Moral Turing Test: Evaluating Human-LLM Alignment in Moral Decision-Making
- The human-authorship halo: attribution bias in literary style evaluation by humans and AI
- Consult Evaluation: Scottish Government's Non-surgical Cosmetic Procedures Consultation
- Can Large Language Models Understand Argument Schemes?