Israeli Study Finds Medical AI Usually Avoids Unnecessary Tests, with Two Persistent Trouble Spots

Six commercial artificial intelligence models usually recognized when a medical test or treatment was unnecessary in an Israeli study published on October 20 in Annals of Internal Medicine. Their mistakes clustered around two decisions: ordering ultrasound for small incidental thyroid nodules and performing routine electrocardiograms in adults at low risk. The study measured answers to written clinical scenarios; no patients were treated or enrolled.

An Israeli team tested restraint and action

The research brought together Dean Ariel of Clalit Health Services and Tel Aviv University, Guy Laban of Ben-Gurion University of the Negev, and Stav Endelman of Rabin Medical Center's Beilinson Hospital and Tel Aviv University. The paper examined a practical problem for medical AI: recognizing when further intervention adds little value while still recommending care when it is needed.

That balance was built into the experiment. A system that simply rejected every proposed test would perform well on some questions and dangerously on others. The researchers therefore assessed unnecessary intervention and inappropriate withholding of care as separate outcomes, giving the models cases that required different decisions within the same area of medicine.

How 225 scenarios produced 4,050 responses

The team selected 15 recommendations from Choosing Wisely, an initiative created by the American Board of Internal Medicine Foundation to encourage conversations about unnecessary care. For each recommendation, researchers constructed 10 scenarios in which restraint was appropriate and five in which action was appropriate. That produced 225 scenarios.

Six models from OpenAI, Anthropic and Google answered each scenario three times through their production programming interfaces in February and March 2026. The resulting 4,050 responses were evaluated for both types of error. Prompts adopted a clinician's perspective. Responses were classified using an AI judge with physician adjudication, a design choice relevant to how the benchmark's results were assessed.

Most errors clustered in two recommendations

The overall rate of recommended overuse was 5.4%. In eight of the 15 recommendation groups, the models proposed no unnecessary intervention. False restraint, meaning a recommendation to withhold an appropriate intervention, occurred in 0.6% of responses and was absent in 11 groups. The highest false restraint rate was 3.3% for prostate specific antigen screening.

The larger weaknesses appeared on the overuse side. Models recommended unnecessary thyroid ultrasound in 42.8% of relevant responses and routine electrocardiography in 28.3%. The concentration matters clinically: a reassuring overall average can coexist with unreliable recommendations on a particular question that arises in a consulting room.

The difficult cases reflect disagreement in medicine

Ben-Gurion University said the thyroid results reflect different approaches taken by the American Thyroid Association and the American College of Radiology to small nodules found incidentally. For the electrocardiogram scenarios, mentions of palpitations complicated the decision because that symptom can justify further investigation in an appropriate clinical setting.

Laban also described large differences between individual models facing identical cases. In some disputed situations, one recommended unnecessary testing less than 10% of the time while another did so 80% of the time. The university presented this variation as evidence that a fluent answer from a chatbot can conceal substantial disagreement among systems trained on medical information.

Why the benchmark used Choosing Wisely

The ABIM Foundation launched Choosing Wisely in 2012 around recommendations supplied by medical specialty societies. Its central purpose was to help clinicians and patients discuss whether a proposed intervention offered enough benefit in that person's circumstances. The initiative addressed unnecessary testing as a quality of care problem, including the possibility of harm from further procedures prompted by incidental findings.

That history explains the study's paired cases. The same type of scan, prescription or screening test can be appropriate when a patient's history changes. Testing that switch gives a more demanding assessment than asking a model to recite a recommendation in isolation. It also exposes places where clinical guidance itself leaves room for competing judgments.

The next test is a real conversation

In an October 31 interview with The Jerusalem Post, Ariel said the models had been tested only in English and that further work was required before clinical use could be validated. The study used single prompts without additional exchanges, so it did not measure whether an assistant would maintain a sound recommendation when a worried patient insisted on another test.

The researchers identified that interaction as an important next question. Prospective clinical research would also need to assess patient outcomes and practical safeguards, rather than judge success only by the number of tests avoided. For now, the Israeli team's contribution is a detailed account of where the tested models showed restraint, where they withheld necessary care, and which medical decisions remained the most difficult.