The Chatbot That Went 0 for 104
A team in Nice decided to test whether ChatGPT and Bard could do the single most mechanical part of academic writing: finding papers that already exist and citing them correctly. Retrieval, the…
The Chatbot That Went 0 for 104
A team in Nice decided to test whether ChatGPT and Bard could do the single most mechanical part of academic writing: finding papers that already exist and citing them correctly. Retrieval, the boring part everyone assumes a computer is good at.
They published the results in the Journal of Medical Internet Research as "Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis". One number from that paper is worth sitting with before anything else: Bard's precision rate was 0%. Zero, out of 104 references it produced. A perfect record, aimed the wrong way.
Who ran it
The study is led by Mikaël Chelli, MSc, MD, of the Institute for Sports and Reconstructive Bone and Joint Surgery at Groupe Kantys in Nice, France, with co-authors Jules Descamps, Vincent Lavoué, Christophe Trojani, Michel Azar, Marcel Deckert, Jean-Luc Raynier, Gilles Clowez, Pascal Boileau and Caroline Ruetsch-Chelli. This is peer-reviewed, published in a serious medical informatics journal, on a subject specific enough that there's nowhere for a model to hide behind vagueness.
The setup
They took 11 real systematic reviews of randomized clinical trials, all published in 2020, all found via a PubMed search run on July 27, 2023. Each review already had a real, known-correct reference list, assembled the slow way, by actual reviewers doing actual reading. Two reviewers on the study, MC and PB, screened titles, abstracts and full texts, with a third, JD, brought in to settle any disagreement.
Then they asked ChatGPT (both GPT-3.5 and GPT-4) and Bard, since rebranded Gemini, to reproduce the reference lists for those same reviews. One prompt per review per model. That gave 33 prompts across 11 reviews and 3 models. 471 references analyzed.
The choice of 2020 was deliberate. It sits neatly before ChatGPT's training cutoff of September 2021, meaning these were exactly the papers a model should have had its best shot at knowing. Nobody was asking it to reach into next year's literature. This was the easy version of the test, run on home turf, and it still went badly.
The scorecard
The study's definition of "hallucinated" was precise: a reference counted if two or more of three things were wrong, title, first author, or year of publication. The paper's own identity, in other words.
"Papers were considered 'hallucinated' if any 2 of the following information were wrong: title, first author, or year of publication."
Against that bar: hallucination rates stood at 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4 and 91.4% (95/104) for Bard (P<.001). GPT-4, the best performer in the room, still invented more than a quarter of its citations. Bard invented more than nine in ten.
Precision, recall and the polite word for "made it up"
Hallucination rate alone is generous to a model, because it only tells you how many citations were outright fabricated. It says nothing about whether the ones that survived were actually useful, the right paper for the right claim. So the study also measured precision (of everything a model handed back, how much was correct and relevant) and recall (of everything that should have been found, how much the model actually surfaced), alongside a combined F1-score to capture both at once.
Precision came in at 9.4% (13/139) for GPT-3.5, 13.4% (16/119) for GPT-4 and 0% (0/104) for Bard (P<.001). Recall was 11.9% (13/109) for GPT-3.5 and 13.7% (15/109) for GPT-4. Bard's recall matched its precision: the study reports it as failing to retrieve any relevant papers at all (P<.001).
Read that last one twice. Given 104 opportunities to name a real, correct citation, Bard did not manage it once. Zero for 104.
Why this happens
A language model doesn't have a library card. It has never looked anything up. Ask it to name the studies underneath a systematic review of shoulder surgery trials and it produces something with the shape of a citation, because it has read an enormous number of real citations and knows precisely what one is supposed to look like, a plausible author and a year that fits the decade you asked for. Whether that particular paper exists is nobody's question inside the process. The model is completing a pattern, and "citation" happens to be an extremely easy pattern to fake convincingly.
That's why GPT-4 beat Bard so badly and still failed badly. The retrieval stayed fake. A better liar is still a liar. It's just a harder one to catch, which is arguably worse.
The study caught something else worth keeping in mind: even the citations that turned out to be real skewed toward papers that were easy to find on the open internet, with a noticeable geographical and open-access bias in what the models surfaced. They failed in a pattern that happened to match what's freely indexed online, a quiet distortion sitting underneath the more dramatic failure of outright invention.
What the authors actually said
Chelli and colleagues stated the conclusion plainly:
"Given their current performance, it is not recommended for LLMs to be deployed as the primary or exclusive tool for conducting systematic reviews. Any references generated by such models warrant thorough validation by researchers."
That's the polite, peer-reviewed way of saying: check every citation by hand, against the real paper, every single time. More than a quarter of the time for the best model tested. Nine times out of ten for the worst one, which never once got it right.
The 2am version
Here's the part that should sit with you longer than the percentages. This was a fair question, the easiest version of the test, a narrow, well-documented medical topic, papers safely before the training cutoff and reference lists that already existed in published, peer-reviewed reviews. GPT-4, the best model in the study, still got more than a quarter of its citations wrong. Bard got nine out of ten wrong and never produced one usable reference across 104 tries.
The reference list is the part of a paper nobody double-checks under deadline, because it's supposed to be the boring, mechanical part, the part you trust precisely because it looks like formatting. That's exactly what makes it dangerous. A hallucinated argument tends to read wrong; something about it snags. A hallucinated citation reads perfectly. Correct-looking title and author, a year that fits your argument. You find out it's fiction when someone downstream goes looking for the paper and it isn't there.
That's the boring, unglamorous fix worth having on hand: every citation you're about to submit checked against the live record before it goes anywhere near a supervisor or a journal editor. An actual check that the paper exists, that it says what you think it says and that it hasn't since been retracted out from under you. A real check against a real paper, the chatbot doesn't get to grade its own homework. Boring is the goal. Boring is not being the twelfth review in somebody's next hallucination study.