Back to blog

Why does AI diagnosis still miss what doctors catch?

Thắng Đoàn
Thắng Đoàn

An AI system just diagnosed 304 hard cases at 83.2 percent while doctors averaged 36.1 percent. Yet you still cannot safely hand it a real patient, and the reason is the test itself.

The numbers that started the debate

Microsoft's AI Diagnostic Orchestrator, MAI-DxO, diagnosed 83.2 percent of 304 complex New England Journal of Medicine cases correctly, per TechTimes. Generalist physicians with full reference access averaged 36.1 percent, and 22.3 percent without references, Chosun Ilbo adds. On 325 emergency department cases, the AI scored 80.3 percent against 46.8 percent for clinicians, IBTimes UK reports. MAI-DxO beat doctors by more than doubling their accuracy on the hardest written cases in medicine.

  • 83.2% vs 36.1% on 304 NEJM cases, per TechTimes
  • 80.3% vs 46.8% on 325 emergency cases, per IBTimes UK

Why the test is not your hospital

Whether that benchmark advantage translates into real-world clinical gains is a separate, open question that a new hospital partnership is only now beginning to test. The cases came from the New England Journal of Medicine: curated written vignettes with a known right answer. A real patient is vague, inconsistent, and has no answer key. TechTimes states plainly whether that benchmark advantage translates into real-world clinical gains is a separate, open question that a new hospital partnership is only now beginning to test. A puzzle with a clean answer rewards exhaustive reasoning. Medicine rarely offers one.

What the AI does differently

It reasons through a case step by step, like a physician, rather than answering in one pass. MAI-DxO is a multi-agent system, meaning several AI programs each handle part of the job and check each other's work. It reasons through a case step by step, like a physician, rather than answering in one pass, TechTimes explains. It starts with a brief vignette, the same few lines a doctor gets at first contact. Chosun Ilbo notes it is designed to collect information, choosing which tests to order, not just label a disease. That simulated curiosity is why it wins on paper puzzles. It can afford thoroughness in a way a time-starved generalist cannot.

What this does not solve

The 36.1 percent doctor score is itself a warning about the benchmark, not just the doctors. These cases were selected because they are notoriously difficult, per TechTimes. No source reports accuracy on ordinary, messy, multi-condition patients. No source reports what happens when the AI is wrong and a human defers to it. Accuracy on written cases tells you nothing about trust, liability, or integration.

If an AI is right 80 percent of the time on hard cases, who owns the 20 percent it gets wrong in your organization?
Which of your decisions resemble textbook puzzles with clean answers, and which depend on context no vignette captures?

Take one decision you make weekly and write down whether the written record alone contains enough information to make it. That is the gap

Sources

IBTimes UK

TechTimes

Chosun Ilbo

Share:

Recommended for you

Enjoyed this article?

Subscribe for new articles. No spam. Unsubscribe anytime.

By subscribing you agree to receive the newsletter. No spam, and you can unsubscribe anytime.