Microsoft AI tool outperforms physicians on complex cases, study says

The latest:

A Microsoft AI model that works through a case the way a clinician does — asking questions, ordering tests, revising its hypothesis — was more than twice as accurate as general physicians on more than 300 unusually difficult medical cases, according to a study published Thursday in the journal Science. The physicians in the comparison worked with textbooks and internet access.

Details:

• How it works: According to the report, the model is coordinated by what the researchers call an orchestrator, which organizes how the system approaches a problem by running a simulated panel of clinical experts that collaborate on the case rather than producing a single answer in one pass.
• The sequence: The researchers said the model first decides what to ask about the patient’s history and what matters in the physical exam. Based on what it learns, it selects which tests to order, weighs the results and orders further testing as needed, updating its working hypotheses and plans throughout.
• The contrast: Existing medical AI models typically take an all-at-once approach, examining a patient’s symptoms, lab results and imaging studies together to generate a diagnosis, according to the report. The new design breaks that into sequential decisions instead.
• The results: Tasked with more than 300 highly unusual and confusing medical dilemmas, the orchestrator-driven model was more than twice as accurate as general physicians at working up the patient and reaching a diagnosis, the researchers said.
• Second test set: The tool also outperformed human physicians on scenarios drawn from a major database of emergency room admissions, according to the researchers. The database was not named, and the size of that comparison was not specified.
• On cost: The researchers said diagnostic processes cost less when the orchestrator designed the work-up than when human doctors did. Harsha Nori, head of AI at Microsoft AI Health, said the team measured not only accuracy but how judiciously and carefully the system moves through a diagnostic work-up.
• The benchmark critique: Dr. Dominic King, corporate vice president of the health unit at Microsoft AI, said assessments of older systems typically rely on multiple-choice questions, where AI already scores near-perfect results. “No patient ever came to see me with a nice summary.”
• Next trial: In a collaboration with Beth Israel Deaconess Medical Center in Boston, the researchers will next test the tool’s ability to triage patients in emergency department waiting rooms, according to Reuters.
• The limits: The comparison was run against general physicians rather than specialists, and the cases were selected for being highly unusual. Microsoft did not announce a timeline for clinical deployment or regulatory submission.

Between the lines:

The framing of the claim matters as much as the number. The benchmark was not a multiple-choice exam but a sequential work-up, and the comparison group was general physicians working from textbooks and the internet — not specialists on their own cases. Microsoft is also presenting cost per diagnosis alongside accuracy, which points the pitch at hospital budgets, not only at clinical performance.

What’s next

The Beth Israel Deaconess collaboration on emergency department triage is the next concrete test, moving the tool from curated case files to live patient flow. Independent replication of the accuracy and cost findings published in Science is the other signal to watch.