03/02 · Essay
The first thing I learned about systematic reviews is that the screening is where the review is made or lost. Two reviewers, thousands of abstracts, a calibration exercise, and a kappa statistic that everyone pretends to care about until the deadline. It is tedious work and it is also where you discover what the question actually is, because the borderline abstracts are the ones that reveal the boundary of your inclusion criteria.
Language models are now good enough at this task to be used, and the evidence on how good is instructive. A 2025 comparison found that performance varied by model, prompt, and threshold, with some models erring toward sensitivity and others toward specificity. That is a polite way of saying that your screening result depends on choices you may not have thought to report. The same year, the major evidence-synthesis organisations issued a joint position on AI, and a set of recommendations with the memorable acronym RAISE reached its fourth version by early this year. The common thread is disclosure, human oversight, and author accountability.
What the standards get right
They are good standards. The position that a human remains accountable for every inclusion decision is correct, and the requirement to report model, version, prompt, and threshold is the minimum that reproducibility demands. I would add a requirement to report the date, because the same model name can refer to a different system a few months apart, and a screening that cannot be re-run is not a method.
What the standards cannot protect
What I worry about is the habit underneath the method. A second human reviewer can be embarrassed. When she disagrees with me on an abstract, we talk, and the talk is where the review’s criteria become precise. A model does not get embarrassed. It will cheerfully apply a slightly wrong reading of the inclusion criteria to four thousand abstracts with perfect consistency, and consistency is the quality that makes the error invisible.
The evidence-synthesis literature has known for decades that the disagreements are the data. The point of dual screening was never to catch the model’s errors. It was to catch the criteria’s errors, and the criteria’s errors show up as disagreements between people who both read the protocol in good faith.
How I would use it
Use the model as a third screener, not a second. Let it screen everything, and then sample its disagreements with each human for discussion, exactly as you would with a new team member. Report the model’s sensitivity against the human consensus on a held-out set, and report it by the kind of study, because models are reliably worse on the designs that matter most in implementation research: the messy pragmatic trial, the mixed-methods evaluation, the protocol paper that reads like a results paper. Keep the calibration exercise. It was never about the kappa.
There is a version of AI-assisted review that makes evidence synthesis faster and more transparent. There is another that makes it faster and uniformly, silently wrong. The reporting standards distinguish the two on paper. The habit of arguing about borderline abstracts distinguishes them in practice.
Sources
- Performance of large language models for title and abstract screening: a comparison across models, prompts, and thresholds. Research Synthesis Methods, 2025.
- Cochrane, Campbell Collaboration, JBI, and the Collaboration for Environmental Evidence. Joint position statement on responsible AI use in evidence synthesis. 2025.
- Responsible AI in evidence synthesis (RAISE) recommendations, version 4. 2026.
- Than N, et al. Qualitative coding with generative large language models. Sociological Methods and Research, 2025. https://journals.sagepub.com/doi/10.1177/00491241251339188