Before a trainee doctor can obtain a General Practitioner (GP) license from the UK’s Royal College of General Practitioners, the final MRCGP exam must be passed. The exam, to a large extent, has been designed to test trainee doctors’ ability to diagnose medical conditions, demonstrating competence and clinical skills are at a sufficient level for independent practice. The exam was therefore a fitting test for a new AI system designed to aid doctors with triage and diagnosis.
When provided with a representative sample of questions from the paper, Babylon Health’s AI system passed the exam with flying colors, achieving a pass mark of 81%. The average score for trainee doctors is 72%. Since the AI system learns from past mistakes, it is expected to get better over time.
The exam only covers a limited number of scenarios that doctors are likely to face in independent practice so, as an additional test, the AI system was tested using 100 independently-devised symptom sets (vignettes) created with assistance from the Royal College of Physicians, Dr Arnold DoRosario, Chief Population Health Officer at Yale New Haven Health, and Dr. Megan Mahoney, Chief of General Primary Care, Division of Primary Care and Population Health, at Stanford University. The 100 vignettes were also given to seven highly-experienced primary care physicians.
The physicians scored 64% to 94% on the test, with the AI system scoring 80%. When tested against the conditions most commonly experienced in primary care medicine, the AI system scored 98% compared to the physicians who scored 52%-99%. When the answers were assessed for safety, the doctors scored and average of 93.1% whereas the AI system scored 97%.
Babylon Health is not the only company to test the effectiveness of an AI system on medical licensing exams. A iFlytex Co Ltd system took the national medical licensing exam in China and passed with a score of 456 points, exceeding the pass mark by 96 points.
Babylon Health was careful to point out that its AI system has not been developed to diagnose medical conditions, instead it is a tool that can help doctors with diagnoses. The system also has potential to be used in online systems that assess patients who do not have easy access to physicians –in a similar capacity to the UK’s 111 service, which connects patients with non-medical staff for triage over the phone to determine the level of urgency of medical complaints to ensure they get the right care.
“The World Health Organisation estimates that there is a shortage of over 5 million doctors globally, leaving more than half the world’s population without access to even the most basic healthcare services. Even in the richest nations, primary care is becoming increasingly unaffordable and inconvenient, often with waiting times that make it not readily accessible.” said Dr Ali Parsa, Babylon’s Founder and CEO. “Babylon’s latest artificial intelligence capabilities show that it is possible for anyone, irrespective of their geography, wealth or circumstances, to have free access to health advice that is on-par with top-rated practicing clinicians.”