A Question About Microsoft’s AI Doctor Claim: What Was Tested?

🔍 Read the full analysis: A Question About Microsoft’s AI Doctor Claim: What Was Tested? on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic AI reached 83.2% accuracy on 304 difficult medical case records, compared with 36.1% for generalist physicians using textbooks and the internet, according to a peer-reviewed Science paper published Oct. 22. The result is based on historical, text-only cases in a simulator, not tests of the system treating patients in a hospital.

The study, titled Sequential diagnosis with language models, evaluated diagnosis as a series of decisions rather than a single answer to a completed case. Each case began with a short summary. The AI or physician had to request additional information, such as questions or tests, from a simulated gatekeeper holding the rest of the case file. The researchers scored both the final diagnosis and the estimated cost of obtaining more information.

Microsoft reported that its system scored 80.3% on 325 emergency department cases, versus 46.8% for physicians. On the 304 complex cases, the physician score was 22.3% without reference materials and 36.1% with them. The company’s figures are its own study results; publication in Science means the paper underwent peer review, but does not establish that independent researchers have replicated the results.

MAI-DxO coordinates a group of virtual physician agents that suggest diagnoses, request information and challenge one another. Yet the underlying language model appears to account for much of the performance: Microsoft’s chart gives GPT-5 alone a 78.9% score on the complex cases. The source report says the orchestration layer added about four percentage points and reduced estimated testing costs. Those cost figures are modeled, not observed hospital spending.

Why the Doctor Comparison Matters

The study offers evidence that language models can perform well on a defined diagnostic benchmark, particularly when they can choose what information to request. That is relevant to developers and health systems considering AI support for clinical work. But the score does not show that MAI-DxO improves diagnosis, outcomes or costs in routine care. No patients received care from the system in this benchmark, and it did not make decisions alongside clinicians treating them.

The comparison group also affects how readers should interpret the headline. The doctors were generalists working individually, while the AI system was designed as a panel of agents. The study did not include specialists or clinician teams, although difficult cases in practice may be referred to specialists or discussed among multiple doctors. This makes the results a comparison between particular setups, not a direct measure of AI against the full range of medical practice.

The case selection creates another limit. The complex records came from published cases chosen for diagnostic difficulty. According to the source account, these cases included no healthy patients or harmless explanations, so the benchmark could not measure how often the system would raise a false alarm. High accuracy on selected hard cases cannot, by itself, establish how the tool would balance missed diagnoses against unnecessary tests in a general patient population.

How the Diagnostic Benchmark Worked

The benchmark, called the Sequential Diagnosis Benchmark, was intended to reflect the step-by-step nature of diagnosis: a clinician begins with limited information, decides what to ask or test, and chooses when to stop. That differs from an exam-style task in which all case details are provided up front. The scoring included estimated costs for tests, using billing codes and a 2023 U.S. hospital price list, plus a fixed physician-visit charge, according to the source report. The authors described those prices as approximations.

Some methodological details in the source account come from a 2025 preprint, rather than the full Science article, which is behind a paywall. The preprint says that when a requested test result was absent from a case record, a gatekeeper model generated a synthetic result. It also describes a separate model grading diagnoses against the published final answer; physician review of a sample showed good, but imperfect, agreement with that grader.

The study’s doctor arm included primary care physicians and hospital generalists, not specialists. The source account says the earlier preprint involved 21 U.S. and UK generalists and that physicians were tested on a subset of cases, while the AI was scored across the full 304-case set. The Science paper’s reported comparison includes an open-book physician result, but the precise final-study procedures should be checked against the published paper rather than assumed to match every detail in the preprint.

“Building Medical Superintelligence”

— Microsoft, in its blog post about the system

What the Study Cannot Establish

The results do not establish whether MAI-DxO is safer or more effective than doctors when used with actual patients. The benchmark involved historical records and simulated information requests; it included no physical examinations, real-time imaging or live clinical decisions. The available source material also does not provide an independent replication of Microsoft’s results.

It remains unclear how the system would perform on ordinary cases, where common and benign explanations matter, or how often it might prompt unnecessary investigations. The source account says the authors could not measure false alarms using the NEJM cases because they lacked healthy patients and harmless explanations. It is also not clear from the supplied material how the final paper’s full methods compare with all details reported in the earlier preprint.

Finally, the reported cost estimates should not be treated as hospital savings. They rely on approximate test prices and a modeled process. Actual costs could depend on local prices, clinician time, follow-up and whether AI recommendations lead to additional testing.

The Planned Patient Study

The next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. The source says consenting patients are expected to interact with MAI-DxO alongside their normal care. That would move the evaluation beyond historical cases, although the supplied material does not specify the study’s start date, size, endpoints or timetable for results.

Those details will matter in judging whether the system can support clinicians safely and usefully. Readers should look for results on diagnostic accuracy across a broader mix of patients, false alarms, unnecessary testing, costs and patient outcomes, as well as clarity about how clinicians oversee the tool. Until such evidence is reported, Microsoft’s figures describe performance on a selected simulation benchmark, not proof that the system is ready to diagnose patients in routine care.

Key Questions

What did Microsoft test?

Microsoft tested MAI-DxO on historical medical case records. The system had to request additional information through a simulator before arriving at a diagnosis; it did not examine or treat live patients in this evaluation.

What does the 83.2% accuracy figure refer to?

It is Microsoft’s reported score for MAI-DxO on 304 complex NEJM cases. The study also reported 36.1% for generalists using reference materials and 22.3% without them.

Were specialists included in the comparison?

The source account describes the physician comparison group as primary care doctors and hospital generalists. It says the study did not compare MAI-DxO with specialists or physician teams.

Does the study show that the AI is ready for hospitals?

No. The benchmark measured performance on simulated, historical cases. Whether the system can improve care or do so safely for live patients remains unproven by this study.

What is expected next?

The source reports a planned prospective study at Beth Israel Deaconess Medical Center, where consenting patients would interact with the system alongside their usual care. Its schedule and detailed evaluation measures were not provided.

Source: ThorstenMeyerAI.com