Why does AI diagnosis still miss what doctors catch?
An AI system just diagnosed 304 hard cases at 83.2 percent while doctors averaged 36.1 percent. Yet you still cannot safely hand it a real patient, and the reason is the test itself.
The numbers that started the debate
Microsoft's AI Diagnostic Orchestrator, MAI-DxO, diagnosed 83.2 percent of 304 complex New England Journal of Medicine cases correctly, per TechTimes. Generalist physicians with full reference access averaged 36.1 percent, and 22.3 percent without references, Chosun Ilbo adds. On 325 emergency department cases, the AI scored 80.3 percent against 46.8 percent for clinicians, IBTimes UK reports. MAI-DxO beat doctors by more than doubling their accuracy on the hardest written cases in medicine.
- 83.2% vs 36.1% on 304 NEJM cases, per TechTimes
- 80.3% vs 46.8% on 325 emergency cases, per IBTimes UK
Why the test is not your hospital
Whether that benchmark advantage translates into real-world clinical gains is a separate, open question that a new hospital partnership is only now beginning to test. The cases came from the New England Journal of Medicine: curated written vignettes with a known right answer. A real patient is vague, inconsistent, and has no answer key. TechTimes states plainly whether that benchmark advantage translates into real-world clinical gains is a separate, open question that a new hospital partnership is only now beginning to test. A puzzle with a clean answer rewards exhaustive reasoning. Medicine rarely offers one.
What the AI does differently
It reasons through a case step by step, like a physician, rather than answering in one pass. MAI-DxO is a multi-agent system, meaning several AI programs each handle part of the job and check each other's work. It reasons through a case step by step, like a physician, rather than answering in one pass, TechTimes explains. It starts with a brief vignette, the same few lines a doctor gets at first contact. Chosun Ilbo notes it is designed to collect information, choosing which tests to order, not just label a disease. That simulated curiosity is why it wins on paper puzzles. It can afford thoroughness in a way a time-starved generalist cannot.
What this does not solve
The 36.1 percent doctor score is itself a warning about the benchmark, not just the doctors. These cases were selected because they are notoriously difficult, per TechTimes. No source reports accuracy on ordinary, messy, multi-condition patients. No source reports what happens when the AI is wrong and a human defers to it. Accuracy on written cases tells you nothing about trust, liability, or integration.
If an AI is right 80 percent of the time on hard cases, who owns the 20 percent it gets wrong in your organization?
Which of your decisions resemble textbook puzzles with clean answers, and which depend on context no vignette captures?
Take one decision you make weekly and write down whether the written record alone contains enough information to make it. That is the gap
Sources
Recommended for you
- Health
Why does FDA approval still leave medical AI unsafe?
Regulators clear AI devices fast, then lose sight of them. Approval was the easy part; the hard part starts the day the device meets real patients.
- Health
What do mental health chatbots actually solve?
Millions now talk to chatbots before therapists. The evidence says they help with access and first-step support, but fail exactly where care gets hard.
- Health
Should patients see what the AI heard in their visit?
A court says ambient AI recordings are not medical records, even as vendors push AI deeper into clinical decisions.
- Health
How do you kill a failing drug before it costs billions?
Novartis lost $30 billion in market value when a late-stage trial missed. An AI start-up had predicted the failure months earlier using virtual patients.
Enjoyed this article?
Subscribe for new articles. No spam. Unsubscribe anytime.