BACKGROUND Patients with hepatitis B virus(HBV)infection require chronic and personalized care to improve outcomes.Large language models(LLMs)can potentially provide medical information for patients.AIM To examine the...BACKGROUND Patients with hepatitis B virus(HBV)infection require chronic and personalized care to improve outcomes.Large language models(LLMs)can potentially provide medical information for patients.AIM To examine the performance of three LLMs,ChatGPT-3.5,ChatGPT-4.0,and Google Gemini,in answering HBV-related questions.METHODS LLMs’responses to HBV-related questions were independently graded by two medical professionals using a four-point accuracy scale,and disagreements were resolved by a third reviewer.Each question was run three times using three LLMs.Readability was assessed via the Gunning Fog index and Flesch-Kincaid grade level.RESULTS Overall,all three LLM chatbots achieved high average accuracy scores for subjective questions(ChatGPT-3.5:3.50;ChatGPT-4.0:3.69;Google Gemini:3.53,out of a maximum score of 4).With respect to objective questions,ChatGPT-4.0 achieved an 80.8%accuracy rate,compared with 62.9%for ChatGPT-3.5 and 73.1%for Google Gemini.Across the six domains,ChatGPT-4.0 performed better in terms of diagnosis,whereas Google Gemini demonstrated excellent clinical manifestations.Notably,in the readability analysis,the mean Gunning Fog index and Flesch-Kincaid grade level scores of the three LLM chatbots were significantly higher than the standard level eight,far exceeding the reading level of the normal population.CONCLUSION Our results highlight the potential of LLMs,especially ChatGPT-4.0,for delivering responses to HBV-related questions.LLMs may be an adjunctive informational tool for patients and physicians to improve outcomes.Nevertheless,current LLMs should not replace personalized treatment recommendations from physicians in the management of HBV infection.展开更多
AIM:To assess the possibility of using different large language models(LLMs)in ocular surface diseases by selecting five different LLMS to test their accuracy in answering specialized questions related to ocular surfa...AIM:To assess the possibility of using different large language models(LLMs)in ocular surface diseases by selecting five different LLMS to test their accuracy in answering specialized questions related to ocular surface diseases:ChatGPT-4,ChatGPT-3.5,Claude 2,PaLM2,and SenseNova.METHODS:A group of experienced ophthalmology professors were asked to develop a 100-question singlechoice question on ocular surface diseases designed to assess the performance of LLMs and human participants in answering ophthalmology specialty exam questions.The exam includes questions on the following topics:keratitis disease(20 questions),keratoconus,keratomalaciac,corneal dystrophy,corneal degeneration,erosive corneal ulcers,and corneal lesions associated with systemic diseases(20 questions),conjunctivitis disease(20 questions),trachoma,pterygoid and conjunctival tumor diseases(20 questions),and dry eye disease(20 questions).Then the total score of each LLMs and compared their mean score,mean correlation,variance,and confidence were calculated.RESULTS:GPT-4 exhibited the highest performance in terms of LLMs.Comparing the average scores of the LLMs group with the four human groups,chief physician,attending physician,regular trainee,and graduate student,it was found that except for ChatGPT-4,the total score of the rest of the LLMs is lower than that of the graduate student group,which had the lowest score in the human group.Both ChatGPT-4 and PaLM2 were more likely to give exact and correct answers,giving very little chance of an incorrect answer.ChatGPT-4 showed higher credibility when answering questions,with a success rate of 59%,but gave the wrong answer to the question 28% of the time.CONCLUSION:GPT-4 model exhibits excellent performance in both answer relevance and confidence.PaLM2 shows a positive correlation(up to 0.8)in terms of answer accuracy during the exam.In terms of answer confidence,PaLM2 is second only to GPT4 and surpasses Claude 2,SenseNova,and GPT-3.5.Despite the fact that ocular surface disease is a highly specialized discipline,GPT-4 still exhibits superior performance,suggesting that its potential and ability to be applied in this field is enormous,perhaps with the potential to be a valuable resource for medical students and clinicians in the future.展开更多
基金Supported by National Natural Science Foundation of China,No.82260133the Key Laboratory Project of Digestive Diseases in Jiangxi Province,No.2024SSY06101Jiangxi Clinical Research Center for Gastroenterology,No.20223BCG74011.
文摘BACKGROUND Patients with hepatitis B virus(HBV)infection require chronic and personalized care to improve outcomes.Large language models(LLMs)can potentially provide medical information for patients.AIM To examine the performance of three LLMs,ChatGPT-3.5,ChatGPT-4.0,and Google Gemini,in answering HBV-related questions.METHODS LLMs’responses to HBV-related questions were independently graded by two medical professionals using a four-point accuracy scale,and disagreements were resolved by a third reviewer.Each question was run three times using three LLMs.Readability was assessed via the Gunning Fog index and Flesch-Kincaid grade level.RESULTS Overall,all three LLM chatbots achieved high average accuracy scores for subjective questions(ChatGPT-3.5:3.50;ChatGPT-4.0:3.69;Google Gemini:3.53,out of a maximum score of 4).With respect to objective questions,ChatGPT-4.0 achieved an 80.8%accuracy rate,compared with 62.9%for ChatGPT-3.5 and 73.1%for Google Gemini.Across the six domains,ChatGPT-4.0 performed better in terms of diagnosis,whereas Google Gemini demonstrated excellent clinical manifestations.Notably,in the readability analysis,the mean Gunning Fog index and Flesch-Kincaid grade level scores of the three LLM chatbots were significantly higher than the standard level eight,far exceeding the reading level of the normal population.CONCLUSION Our results highlight the potential of LLMs,especially ChatGPT-4.0,for delivering responses to HBV-related questions.LLMs may be an adjunctive informational tool for patients and physicians to improve outcomes.Nevertheless,current LLMs should not replace personalized treatment recommendations from physicians in the management of HBV infection.
基金Supported by National Natural Science Foundation of China(No.82160195,No.82460203)Degree and Postgraduate Education Teaching Reform Project of Jiangxi Province(No.JXYJG-2020-026).
文摘AIM:To assess the possibility of using different large language models(LLMs)in ocular surface diseases by selecting five different LLMS to test their accuracy in answering specialized questions related to ocular surface diseases:ChatGPT-4,ChatGPT-3.5,Claude 2,PaLM2,and SenseNova.METHODS:A group of experienced ophthalmology professors were asked to develop a 100-question singlechoice question on ocular surface diseases designed to assess the performance of LLMs and human participants in answering ophthalmology specialty exam questions.The exam includes questions on the following topics:keratitis disease(20 questions),keratoconus,keratomalaciac,corneal dystrophy,corneal degeneration,erosive corneal ulcers,and corneal lesions associated with systemic diseases(20 questions),conjunctivitis disease(20 questions),trachoma,pterygoid and conjunctival tumor diseases(20 questions),and dry eye disease(20 questions).Then the total score of each LLMs and compared their mean score,mean correlation,variance,and confidence were calculated.RESULTS:GPT-4 exhibited the highest performance in terms of LLMs.Comparing the average scores of the LLMs group with the four human groups,chief physician,attending physician,regular trainee,and graduate student,it was found that except for ChatGPT-4,the total score of the rest of the LLMs is lower than that of the graduate student group,which had the lowest score in the human group.Both ChatGPT-4 and PaLM2 were more likely to give exact and correct answers,giving very little chance of an incorrect answer.ChatGPT-4 showed higher credibility when answering questions,with a success rate of 59%,but gave the wrong answer to the question 28% of the time.CONCLUSION:GPT-4 model exhibits excellent performance in both answer relevance and confidence.PaLM2 shows a positive correlation(up to 0.8)in terms of answer accuracy during the exam.In terms of answer confidence,PaLM2 is second only to GPT4 and surpasses Claude 2,SenseNova,and GPT-3.5.Despite the fact that ocular surface disease is a highly specialized discipline,GPT-4 still exhibits superior performance,suggesting that its potential and ability to be applied in this field is enormous,perhaps with the potential to be a valuable resource for medical students and clinicians in the future.
