Originally published August 30, 2023. Updated October 15, 2026. This documentation-only correction revisits the original research and institutional announcement. We did not test a chatbot or conduct a clinical study.
Original AI-generated editorial illustration for GPTOnline; it does not depict the study materials, a patient, or a clinical test.
The Mass General Brigham announcement described roughly 72% accuracy across clinical-decision questions, including diagnoses and management. Our original headline shortened that to diagnostic accuracy, overstating what the number represented. This revision corrects that framing and withdraws the unsupported suggestion of cost savings.
What was tested
The original paper used GPT-3.5-based ChatGPT, January 9, 2023 version, with 36 hypothetical MSD Manual vignettes. Information arrived sequentially; image questions were excluded. Each case had three runs. Two scorers checked answers against the manual, averaging replicate proportions of correct decisions. The overall score also included general medical-knowledge questions.
The reported scores
The abstract reports these estimates and 95% confidence intervals:
- Overall: 71.7% (69.3%–74.1%)
- Initial differential diagnosis: 60.3% (54.2%–66.6%)
- Final diagnosis: 76.9% (67.8%–86.1%)
Results and Discussion instead report 71.8% overall. Both figures round to 72%; the discrepancy is in the paper.
The intervals describe uncertainty around the study estimates. They are not confidence ratings for individual chatbot answers.
Why the stages differ
In the research team’s account, initial differential diagnosis concerns possible explanations; final-diagnosis questions follow additional information, including test results. The stages therefore involve different questions and different available evidence.
A useful reading habit is to ask what information a system had when it answered. A final answer supplied after a structured case has unfolded should not be treated as a measure of what the system can infer from an opening question.
A numerical example
Consider two equally weighted, fictional quiz questions. A learner scores 50% on the first and 100% on the second. The mean score is 75%, while only one of the two questions was answered completely correctly. Those are different summaries of the same marks.
This is an invented arithmetic example, not a reconstruction of the research data or a patient case. It illustrates why an average score cannot simply be turned into a count of fully correct cases. In particular, reading the headline as “72 out of every 100 patients correctly diagnosed” would change both the unit being counted and the setting.
What remains unproven
The paper leaves possible training overlap unresolved: 34 of 36 vignettes predated the model’s cutoff. Resource use was untested, and erroneous outputs were reported. It describes no patient-outcome or clinician-comparator trial.
The institutional release presented the intern/resident comparison as an estimate without established benchmarks. It called for further research before clinical integration.
Our interpretation: these findings cannot establish clinical safety, cost savings or doctor equivalence. This historical account makes no assessment of current-model performance and provides no diagnosis or treatment advice.
Read beyond the headline
For another AI study, first identify the model and version, the source of the test material, the information supplied, and what earned a point. Then look for the comparison group and the outcome actually measured. Leave missing information marked as unknown rather than filling it in from a product’s reputation.
A compact note might read: “Version: __. Test material: __. Scoring unit: __. Comparator: __. Outcome: __.” This is an editorial reading aid, not a clinical assessment tool. It helps keep a research result attached to the conditions that produced it.
For a broader discussion of test design outside healthcare, see our guide to interpreting AI benchmark scores. It is a general evaluation guide, not medical evidence.