ChatGPT takes on the tough US medical licensing exam

  • AI
  • Friday, 10 Feb 2023

The AI system was tested on 350 of the 376 public questions on the June 2022 version of the exam, the study said, and the chatbot was not given any specialised training ahead of time. — AP

WASHINGTON: Dr ChatGPT will see you soon.

The artificial intelligence system scored passing or near passing results on the US medical licensing exam, according to a study published on Thursday.

"Reaching the passing score for this notoriously difficult expert exam, and doing so without any human reinforcement, marks a notable milestone in clinical AI maturation," said the authors of the study published in the journal PLOS Digital Health.

"These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making," they said.

ChatGPT, which is able to produce essays, poems and programming code within seconds, was developed by OpenAI, a California-based startup founded in 2015 with early funding from Elon Musk among others.

Microsoft invested US$1bil (RM4.33bil) in OpenAI in 2019 and just inked a new multi-billion deal with the firm.

For the study, researchers at California-based AnsibleHealth tested ChatGPT's performance on a three-part licensing exam taken by medical students and physicians-in-training in the United States.

The standardised exam tests knowledge in multiple medical disciplines from basic science to biochemistry to diagnostic reasoning to bioethics.

The AI system was tested on 350 of the 376 public questions on the June 2022 version of the exam, the study said, and the chatbot was not given any specialised training ahead of time.

Image-based questions were removed.

ChatGPT scored between 52.4% and 75% across the three parts of the exam.

A passing grade is around 60%.

According to the study, the first part of the exam, which focuses on basic science and pharmacology, is typically taken by medical students who have put in 300-400 hours of dedicated study time.

The second part is generally taken by fourth-year medical students and emphasises clinical reasoning, medical management and bioethics.

The final section is for physicians who have completed at least six months to a year of postgraduate medical education.

Dr Google and Nurse Bing

The questions were presented to ChatGPT in various formats including open-ended prompting such as "What would be the patient's diagnosis based on the information provided?"

There were also multiple choice questions such as: "The patient's condition is mostly caused by which of the following pathogens?"

Two physician adjudicators who were blinded to each other reviewed the responses to come up with the final grades, the study said.

An outside expert, Simon McCallum, a senior lecturer in software engineering at Victoria University of Wellington, New Zealand, noted that Google has received encouraging results with an AI medical tool known as Med-PaLM.

"ChatGPT may pass the exam, but Med-PaLM is able to give advice to patients that is as good as a professional GP," McCallum said. "And both of these systems are improving.

"Society is about to change, and instead of warning about the hypochondria of randomly searching the internet for symptoms, we may soon get our medical advice from Doctor Google or Nurse Bing."

ChatGPT also proved useful to the authors of the medical exam study in another way.

They used the chatbot to help write it, said co-author Tiffany Kung. – AFP

Subscribe now to our Premium Plan for an ad-free and unlimited reading experience!

Next In Tech News

In Latin America, data centre plans fuel water worries
‘Not stupid’: China girl determined to find husband who doesn’t expect her to do housework in contrast to situation at home, delights social media
Elon Musk's X Corp in another legal fight over unpaid rent, this time as plaintiff
Forget Harry Styles. Riot Games is one of LA’s biggest stars headlining the Kia Forum�
Game on: The couple that games together, stays together
Robinhood sees $100 million costs tied to regulatory issues in third quarter
Video game buddies, banter and battles: Casual couch co-op chaos
Valkyrie Funds halts ethereum purchases for exchange-traded fund
EU examines Nvidia-dominated AI chip market's abuses - Bloomberg News
US Supreme Court to hear Warner Music case on limits to copyright damages

Others Also Read