Stop underestimating ChatGPT: it beat 50 doctors in a real test. 92% vs 74%.
ChatGPT beat 50 doctors in a real diagnosis test, 92% to 74%.
We didn’t run this. Goh E, et al. Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial. JAMA Netw Open. 2024;7(10):e2440969 did. ModelSignal read the published work and checked every number below against it — the sources and the dates we read them are at the end.
ChatGPT vs 50 doctors
92%
ChatGPT's score on
diagnosis. Working alone.
The doctors scored 74%. JAMA Network Open, 2024.
Doctors vs ChatGPT
50 real doctors
74%
Their usual tools
With ChatGPT76%
ChatGPT alone
92%
No doctor involved
vs the doctors+16 points
Read it twice.
Giving doctors ChatGPT barely helped (+2 points). Letting it work alone won.
JAMA Network Open · 50 doctors · 6 cases
The proof
Published in JAMA Network Open, Oct 2024. 50 doctors, 6 cases. Read 13 Oct 2026.
The full breakdown
Somebody else ran this trial; we read it. Every number below is quoted from the paper, with the date we read it.
The trial
Goh E, et al. Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial. JAMA Network Open, October 2024; 7(10): e2440969. Conducted 29 Nov – 29 Dec 2023 at three institutions.
| Who | 50 US doctors (26 attendings, 24 residents), 88% internal medicine |
| Task | 6 clinical cases in a one-hour session; diagnostic reasoning scored by rubric |
| Tool | GPT-4 via ChatGPT Plus; no training in using it |
Results
| Group | Median score |
|---|---|
| ChatGPT (GPT-4) alone | 92% (IQR 82–97) |
| Doctors + ChatGPT | 76% (IQR 66–87) |
| Doctors, usual tools | 74% (IQR 63–84) |
- Doctors with vs without ChatGPT: +2 points (95% CI −4 to 8; P = .60), not significant.
- ChatGPT alone vs doctors alone: +16 points (95% CI 2–30; P = .03).
Authors' conclusion: "the availability of an LLM to physicians as a diagnostic aid did not significantly improve clinical reasoning compared with conventional resources."
Source
- https://pmc.ncbi.nlm.nih.gov/articles/PMC11519755/ (read 13 Oct 2026)
The catches
- Six written cases in one hour; the authors note no sample of cases covers the field.
- GPT-4 as of late 2023; doctors received no training in using it.
- The outcome is a reasoning score on paper, not patient outcomes.
Every number, and where it’s quoted from
- 50 physicians (26 attendings, 24 residents), 88% internal medicine, 3 institutions“50 US-licensed physicians (26 attendings, 24 residents) from three institutions, predominantly internal medicine (88%)”JAMA Netw Open 2024 (PMC11519755)
- 6 clinical vignettes in a 1-hour session“6 cases completed by participants within a 1-hour session”JAMA Netw Open 2024
- doctors with GPT-4: 76% median“Physicians + LLM: 76% (IQR, 66%-87%)”JAMA Netw Open 2024
- doctors alone: 74% median“Physicians + conventional resources only: 74% (IQR, 63%-84%)”JAMA Netw Open 2024
- +2 points, not significant, P=.60“Adjusted difference: 2 percentage points (95% CI, −4 to 8; P = .60)”JAMA Netw Open 2024
- GPT-4 alone 92%; +16 points above doctors, P=.03“LLM alone: 92% (IQR, 82%-97%), difference of 16 percentage points (95% CI, 2-30; P = .03)”JAMA Netw Open 2024
- model GPT-4 via ChatGPT Plus (2023 version)“ChatGPT Plus (GPT-4) by OpenAI”JAMA Netw Open 2024
- trial dates Nov 29 to Dec 29, 2023“November 29 to December 29, 2023”JAMA Netw Open 2024
- no prompt training given“Participants were given access to the chatbot without explicit training in prompt engineering techniques”JAMA Netw Open 2024 (limitations)
- citation JAMA Netw Open 2024;7(10):e2440969, PMC11519755“JAMA Netw Open. 2024;7(10):e2440969 · PMCID: PMC11519755”PMC record
