Asking AI about medical symptoms does not help patients make better decisions about their health than other methods, such as a standard internet search, according to a new study published in Nature Medicine.<\/p>\n\n\n\n
The authors said the study was important as people were increasingly turning to AI and chatbots for advice on their health, but without evidence that this was necessarily the best and safest approach.<\/p>\n\n\n\n
Researchers led by the University of Oxford\u2019s Internet Institute worked alongside a group of doctors to draw up 10 different medical scenarios, ranging from a common cold to a life-threatening hemorrhage causing bleeding on the brain.<\/p>\n\n\n\n
When tested without human participants, three large-language models \u2013 Open AI’s Chat GPT-4o, \u200cMeta’s Llama \u200c3 and Cohere’s Command R+ \u2013 identified the conditions in \u200c94.9% \u2060of cases, \u200cand chose the correct course of action, like calling an ambulance or going to the doctor, in an average of 56.3% of cases. The companies did not respond to requests for comment.<\/p>\n\n\n\n
‘HUGE GAP’ BETWEEN AI’S POTENTIAL AND ACTUAL PERFORMANCE<\/p>\n\n\n\n
The researchers then recruited 1,298 participants in Britain to either use AI, or their usual resources like an internet search, or their experience, or the National Health Service website to \u2060investigate the symptoms and decide their next step.<\/p>\n\n\n\n
When the participants did this, relevant conditions were identified in \u200cless than 34.5% of cases, and the right course of action was given in less than 44.2%, no better than the control group using more traditional tools.<\/p>\n\n\n\n
Adam Mahdi, co-author of the paper and associate professor at Oxford, said the study showed the \u201chuge gap\u201d between the potential of AI and the pitfalls when it was used by people.<\/p>\n\n\n\n
\u201cThe knowledge may be in those bots; however, this knowledge doesn\u2019t always translate when interacting with humans,\u201d he said, meaning that more work was needed to identify why this was happening.<\/p>\n\n\n\n
HUMANS OFTEN GIVING INCOMPLETE INFORMATION<\/p>\n\n\n\n
The \u2060team studied around 30 of the interactions in detail, and concluded that often humans were providing incomplete or wrong information, but the LLMs were also sometimes generating misleading or incorrect responses.<\/p>\n\n\n\n
For example, one patient reporting the symptoms of a subarachnoid hemorrhage \u2013 a life-threatening condition causing bleeding on the brain \u2013 was correctly told by AI to go to hospital after describing a stiff neck, light sensitivity and the “worst headache ever”. The other described the same symptoms but a “terrible” headache, and was told to lie down in a darkened room.<\/p>\n\n\n\n
The team now plans a similar study in different countries and languages, and over time, to test if that impacts AI\u2019s performance.<\/p>\n\n\n\n
The \u200cstudy was supported by the data company Prolific, the German non-profit Dieter Schwarz Stiftung, and the UK and US governments.<\/p>\n","protected":false},"excerpt":{"rendered":"
Asking AI about medical symptoms does not help patients make better decisions about their health than other methods, such as a standard internet search, according…<\/p>\n","protected":false},"author":2,"featured_media":3809,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[44,4,45,161,36],"tags":[42,258,140],"class_list":["post-3808","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-health","category-international","category-medical","category-science","category-technology","tag-ai","tag-medical","tag-openai"],"yoast_head":"\n