Updated October 15, 2026

About this update: This is a historical, document-based review. We checked the published paper, its correction and its supplement. We did not run the models or conduct a patient study. The original publication date is retained.

Image: Original AI-generated editorial illustration of exam papers and a checklist; not study material, clinical evidence or a model test.

How should a reader interpret an AI exam headline? Start with the task definition. A percentage needs a question source, comparison group, grading rule and test date before it can support a useful conclusion. This archive entry keeps those pieces visible and provides a practical way to read the evidence.

The correction belongs with the headline

JAMA Network Open published a correction on January 29, 2024, clarifying the source of the examination questions. For anyone revisiting the original story, that notice is an important part of the publication record. Read the corrected version before repeating an early headline.

What the historical study measured

The paper was published December 7, 2023; testing ran May 17–31, 2023. GPT-3.5 and GPT-4 were accessed through the API, without neurology-specific fine-tuning. BoardVitals supplied 2,036 multiple-choice questions; 80 image/video or linked items were excluded. These were practice questions, not official board-examination items.

  • GPT-4: 1,662/1,956 correct (85.0%).
  • GPT-3.5: 1,306/1,956 (66.8%).
  • Human comparison: 73.8%, averaging question-level platform-user results. Users’ qualifications were unknown; repeat practice could affect scores.

The prompt assigned a doctor/exam role and requested JSON: answer, explanation, confidence, question classification and classification confidence.

Seventy percent was an approximate pass threshold tied to the question bank’s CME credits; official ABPN/EBN criterion-referenced scoring was not applied.

Both models scored lower on higher-order questions and expressed confidence when wrong. A separate repeatability check queried 100 questions 50 times each. Consistency correlated with correctness, without guaranteeing it. No patients were studied. Read the primary paper.

The additional checks had their own scope

A separate set of 19 European Board of Neurology sample questions yielded 14/19 (73.7%) for GPT-4 and 10/19 (52.6%) for GPT-3.5; the adjusted P value was .31. These items sit outside the 1,956-question result. This small comparison did not establish a statistically significant difference.

For memorization checks, the authors asked each model to continue partial question texts at temperature 0. Neither exactly completed any of 1,956 items; mean ordered-token matching was .14 for both. See Supplement 1, eTable 1 and eMethods, available with the paper.

Our interpretation: that test targets a particular form of detectable memorization. Treating failure to reproduce exact wording as proof of no training exposure would be a stronger conclusion. It does not expose a model’s training corpus.

How far can a benchmark claim travel?

Keep a claim within the task that produced it. Selecting an answer from supplied options, writing an open-ended response and taking responsibility for a real-world decision are separate evaluation targets. Evidence for one target should not silently become evidence for all three.

For a comparison with doctors, ask which people participated, what information they received and whether the test conditions matched. For a claim about practical usefulness, ask what outcome was measured and whose judgment determined success. If the report does not answer those questions, leave them open rather than filling the gaps with a confident headline.

Our conclusion: this historical result cannot establish clinical competence, doctor equivalence or current-model performance.

Example: a document-sorting score

Imagine a report about an assistant sorting documents into folders. It assigns 42 of 50 sample documents to the expected folder. A review site then says it is ready to manage an organization’s records.

The 42/50 result answers a defined classification question. The broader claim introduces other requirements: handling unfamiliar documents, following access rules, recovering from mistakes and noticing when a choice needs review. Those requirements need their own evidence.

Now suppose another test uses 12 different documents and produces 8 correct placements. Keep that result separate until you know whether the document selection and scoring rules are comparable. Combining the totals immediately could hide an important difference between the tests.

This example is invented for evidence-reading practice. Its numbers are unrelated to the neurology research, and it contains no medical instructions.

A short checklist for reading AI exam stories

  1. Trace the item source. Locate the original paper, then check for corrections. Identify who wrote the questions and why.
  2. Write down the denominator. Keep questions, people, repeat runs and validation sets in separate columns.
  3. Inspect the comparison. Find the human group, the information each side received and any differences in conditions.
  4. Check the scoring rule. Ask who set the threshold and what passing would actually signify.
  5. Look beyond the average. Read the errors, exclusions, consistency checks and uncertainty before judging the result.
  6. Date the claim. Record the model and test period. Carry unresolved questions into your summary instead of turning assumptions into facts.

For another example of how task definitions affect interpretation, see our historical review of a clinical-vignette benchmark. Compare the evaluation designs before comparing their percentages.

This page is an evidence-reading explainer and does not provide diagnosis or treatment guidance.