Several of the models were strong. In each case what failed was the ability to check what came out. Sometimes the citation did not exist. Sometimes a number had no inspectable chain back to its data. Sometimes accuracy held only under the conditions it was measured in, or "new" meant new to the model.
The cases are chosen for primary documentation: peer-reviewed papers, journal corrections and institutional statements. Where a case is contested or only partly verifiable, the chapter says so.
Citations that do not exist
- Measured rates. Walters and Wilder had GPT-3.5 and GPT-4 write 84 short literature reviews and checked all 636 citations. "55% of the GPT-3.5 citations but just 18% of the GPT-4 citations are fabricated." Among the real citations, a further 43% and 24% had substantive errors (Sci. Rep. 2023).
- A science model's demo. In November 2022, Meta's Galactica was presented as a language model for science. Its public demo was paused after about two days, once users found it "made up fake papers (sometimes attributing them to real authors)" (MIT Technology Review).
- Peer review. A scan of 4,841 papers accepted at NeurIPS 2025 found 100 hallucinated citations in about 51 of them, each confirmed by a human. The conference said the papers' content was "not necessarily invalidated" (GPTZero, Fortune). The checker's maker sells it, which is worth knowing. The examples were checked by hand.
- A deployed tool. FDA staff told CNN in July 2025 that the agency's internal assistant had cited nonexistent studies or misrepresented real ones. Its click-through citations worked only for internal documents (CNN).
A fabrication rate of "only" 18% still fails at scale. Asking a model to be careful does not change that. The fix is a tool that can emit only citations it holds as records (R-8).
Numbers with no chain to their data
An institution withdrew support for a headline AI-in-science study
A widely discussed 2024 preprint reported that an AI tool for materials discovery led researchers to "discover 44% more materials", with 39% more patent filings. In May 2025, after an internal review, MIT stated that it "has no confidence in the provenance, reliability or validity of the data" and asked for the paper's withdrawal. arXiv withdrew it, citing "concerns about the validity of the data" (MIT, arXiv).
A plausible number with no inspectable chain back to its data cannot be trusted, however good the story (R-1, R-7).
None of 62 COVID-19 imaging models was fit for clinical use
A systematic review screened 2,212 studies of machine learning for COVID-19 diagnosis and prognosis from chest images. It found that none of the 62 that passed quality review was of potential clinical use. Much of the cause was data lineage:
- Child controls. 16 papers used a pneumonia dataset of children aged one to five as controls, often without saying so.
- "Frankenstein" datasets. Some datasets had been "assembled from other datasets and redistributed under a new name", sometimes containing one another.
(Roberts et al., Nat. Mach. Intell. 2021).
Dataset identity, version and cohort have to travel with every result (R-7, R-9).
Accuracy that held only where it was measured
A sepsis model deployed at hundreds of hospitals failed external validation
Epic's sepsis model reported an AUC of 0.76 to 0.83 in internal documentation. An independent validation at Michigan Medicine covered 38,455 hospitalizations:
- AUC. The model scored 0.63.
- Missed cases. It missed 67% of sepsis cases.
- Alert burden. It alerted on 18% of all hospitalizations.
The authors note that per-prediction AUCs "are misleading because they treat each prediction as independent" (Wong et al., JAMA Intern. Med. 2021). Epic later revised the model and now recommends training on a hospital's own data before clinical use.
Measure on the user's conditions, next to the vendor's number, and state the unit of evaluation (R-12).
A screening model met real clinics, and a fifth of the images were rejected
Google Health's diabetic-retinopathy model had shown more than 90% sensitivity and specificity. Google's own team then studied it in 11 clinics in Thailand:
- Rejected images. 393 of 1,838 images (21%) did not meet the system's standard for grading.
- Slow uploads. Uploads took 60 to 90 seconds on clinic connections.
- Patients put off. Because an ungradable image meant automatic referral, some nurses discouraged patients from taking part, for fear of the travel costs.
The protocol was changed in response (Beede et al., CHI 2020). This one deserves attention, and the deployer deserves credit for publishing it.
Lab accuracy has preconditions, so an eval has to run under realistic ones (R-12).
"New", relative to nothing in particular
A 2023 paper reported that an autonomous lab "realized 41 novel compounds". Independent chemists then found that about two-thirds of the claimed successes were "likely to be known compositionally disordered versions" of the predicted compounds (Leeman et al., PRX Energy 2024). In January 2026 the journal published an author correction:
- Novelty restated. "New to the prediction platform, not necessarily new to science."
- Fewer confirmed results. The correction confirms 36 of the 40 successes.
- A leaked target. One compound that "was mistakenly included in the training data" was removed.
"New" means nothing without the reference set and its version. Checking novelty has to include the model's own training data (R-1, R-12).
Benchmarks, and how they get quoted
Even the evidence about agents in biology needs its provenance stated.
- Two versions, two numbers. BixBench reported 17% open-answer accuracy for its best agent in one version of its paper, on 296 questions. A later version reports 21%, on 205 questions. The arXiv abstract still shows 17% (arXiv 2503.00096). Quote the version.
- The harness matters. BiomniBench found that swapping only the agent harness moved one model from 55.19 to 68.69. That is "more than the gap between successive model generations". It also found that agents "reliably ground claims in real sources yet consistently fall short on method selection, biological interpretation, and scientific reasoning" (bioRxiv 2026). Early notes for this project had cited the paper for agents fabricating citations. That line is an illustration in its introduction. The paper does not report it as a finding.
- A label's basis. GPT-4 annotations of cell types fully or partly matched manual labels in over 75% of cell types in most studies. The authors themselves warn that the model's "undisclosed" training corpus makes "verifying the basis of its annotations challenging" (Hou & Ji, Nat. Methods 2024). A label is checkable only with what it rests on: markers, taxonomy, level and version (R-3).
What this means for the design
| Mechanism | Prevented by | Requirement |
|---|---|---|
| Citations that do not exist | citations only from the source catalogue | R-8 |
| Numbers with no lineage | a provenance record on every answer: releases, sources, code | R-1, R-7 |
| Accuracy only where measured | an eval under realistic conditions, failures first, intervals on every difference | R-12 |
| Novelty relative to nothing | every claim names its reference set and version | R-1 |
| Labels with no stated basis | taxonomy, level and confidence with every cell-type answer | R-3 |