BERLIN: Artificial intelligence (AI) has an inbuilt "sycophancy" that makes chatbots inclined to come across as "excessively helpful and agreeable" rather than give appropriate or accurate answers to medical queries.
In a series of tests, large language model (LLM) AI bots displayed "potentially harmful" behaviour and "complied with requests for misinformation," according to a team of doctors at Mass Brigham General – a combination of Massachusetts General Hospital and Brigham and Women's Hospital.
The LLMs "overwhelmingly" failed "to appropriately challenge illogical medical queries," the team found, meaning that people who miscue their medical prompts or questions posed to the bots could end up getting wildly dangerous advice.
The medics and researchers said they came up with "a series of simple queries about drug safety" which they then used to "assess the logical reasoning capabilities" of three GPT bots and two Meta Llama variants.
"These models do not reason like humans do, and this study shows how LLMs designed for general uses tend to prioritise helpfulness over critical thinking in their responses," says Mass Brigham's Danielle Bitterman, whose team’s research was published in npj Digital Medicine, part of the Nature network of science journals.
The bots "exhibit a vulnerability arising from being trained to be helpful," the team say in their journal paper.
The LLMs "comply with illogical requests that would generate false information" despite having "the knowledge" that should enable them "to identify the request as illogical," the researchers add.
"In healthcare, we need a much greater emphasis on harmlessness even if it comes at the expense of helpfulness," Bitterman says. – dpa
