Originally published January 4, 2024. Updated October 16, 2026.
Published online on January 2, 2024, a JAMA Pediatrics study reported an 83% diagnostic error rate for ChatGPT 3.5. Read the research letter.
Editorial correction: Our earlier article identified the tested model incorrectly. This update corrects that attribution and explains how to read the headline. We have not conducted a new model test.
Original AI-generated editorial illustration of exam papers and a checklist; not study material, clinical evidence or a model test.
What was tested
Researchers submitted text from 100 pediatric cases on July 19–22, 2023. Published during 2013–2023, these comprised 60 JAMA Pediatrics challenges and 40 randomly selected NEJM Massachusetts General Hospital pediatric case records. They requested possible diagnoses and a final diagnosis; two physician researchers compared responses with the published answers.
What 83% included
- 17 cases: correct final diagnosis
- 72 cases: incorrect diagnosis
- 11 cases: related answer that was too broad to count as correct
The headline combines the last two groups. The denominator is this set of cases. See the methods and results.
Keep the scope attached
In their April 29, 2024 response to readers, the authors described an exploratory baseline using the free version, a simple prompt, and no figures or multiple-choice options. That later explanation is correspondence, not a new experiment.
Our reading of this design is narrow: the findings concern a particular version and task setup. They do not establish the accuracy of current models, the results of a live consultation, or performance after adding images or changing prompts.
The practical reading lesson is to separate three questions: what the system produced, how the researchers scored it, and which real-world claim the headline invites you to infer. Each needs its own evidence. This article examines reporting and study design; it does not offer medical guidance.
A nonmedical example
Consider an invented test about identifying building materials. The reference answer is “oak.” A response of “wood” is relevant but lacks the required detail. A response of “steel” is incorrect. A list containing “oak, pine or maple” includes the answer, but it has not selected oak.
Those responses can earn different scores under different rules. If an evaluator asks for one material, credit for mentioning it among several guesses answers another question. Before comparing two headline percentages, check whether both evaluators rewarded the same behavior. This fictional example illustrates scoring only; it contains no study data or model output.
A quick reading checklist
- Identify the task. Was the system choosing from supplied options, producing an open-ended answer, or carrying out a real workflow?
- Locate the denominator. Ask what each counted item represents and how it entered the sample.
- Read the scoring rule. Check how broad answers, partial credit and lists of possibilities were handled.
- Check the comparison. Look for the same inputs, conditions and success criteria before ranking people or systems.
- Attach the date. Record when the test happened as well as when the article appeared.
- Follow later updates. Distinguish a correction, an author reply and a new study; they serve different purposes.
For another historical example, read our guide to the GPT-4 neurology study. Keep its exam-question setting separate from this case-report evaluation.