摘要
目的评估不同大语言模型(LLM)所提供医疗保健信息的准确性,以探讨其在口腔辅助诊疗和健康咨询中应用的可行性及局限性。方法设计8项共47个有关口腔疾病诊疗的问题[评估LLM作为人工智能(AI)医疗助手的表现]和5项共35个口腔健康咨询方面的问题(评估LLM作为模拟医师的表现),并由5种LLM(文心一言、华佗GPT、通义千问、讯飞星火、ChatGPT)分别作答。由两名工作5年以上的主治医师使用3C标准(正确性、清晰度、简洁性)对答案进行独立评分,并采用Spearman等级相关系数评估评分者间一致性,采用Kruskal-Wallis检验和Dunn事后检验评估各模型间差异。使用600道2023年口腔执业医师资格考试试题,评估各模型的答题耗时、得分及准确率。结果作为AI医疗助手,LLM可辅助医师诊疗决策,评价者间Spearman系数为0.505(P<0.01);作为模拟医师,LLM可对患者进行科普,评价者间Spearman系数为0.533(P<0.01)。各模型作为AI医疗助手和模拟医师的3C评分分别为文心一言2.00(1.00,3.00)、2.00(2.00,3.00)分,华佗GPT 1.00(1.00,2.00)、2.00(1.00,2.00)分,通义千问2.00(1.00,2.00)、2.00(1.00,3.00)分、讯飞星火2.00(1.00,2.00)、2.00(1.75,2.25)分、ChatGPT 3.00(2.00,3.00)、3.00(2.00,3.00)分(满分4分);Kruskal-Wallis检验结果显示,作为AI医疗助手或模拟医师,5种LLM的3C评分差异均有统计学意义(均P<0.001)。5种LLM在口腔执业医师资格考试中的平均得分为370.2分、准确率61.7%(370.2/600)、耗时94.6 min,其中文心一言耗时115 min、得分363分、准确率60.5%(363/600),华佗GPT耗时224 min、得分305分、准确率50.8%(305/600),通义千问耗时43 min、得分438分、准确率73.0%(438/600),讯飞星火耗时32 min、得分364分、准确率60.7%(364/600),ChatGPT耗时59 min、得分381分、准确率63.5%(381/600)。结论基于LLM作为AI医疗助手和模拟医师的双重角色评估,ChatGPT表现最佳,回答基本正确、清晰、简洁,文心一言、通义千问、讯飞星火次之,华佗GPT表现显著落后。口腔执业医师资格考试中,除华佗GPT外其余4个LLM均达到合格水平,且5个LLM的作答时间相较于考试规定的8 h均明显下降。LLM在口腔辅助诊疗和健康咨询中具备应用可行性,可帮助医患双方快速获取医疗信息,但其输出存在错误风险(3C评分未达满分),使用时应审慎判断。
Objective To evaluate the accuracy of the oral healthcare information provided by different large language models(LLM)to explore their feasibility and limitations in the application of oral auxiliary,treatment and health consultation.MethodsThis study designed eight items comprising 47 questions in total related to the diagnosis and treatment of oral diseases[to assess the performance of LLM as an artificial intelligence(AI)medical assistant],and five items comprising 35 questions in total about oral health consultations(to assess the performance of LLM as a simulated doctor).These questions were answered individually by the five LLM models(Erine Bot,HuatuoGPT,Tongyi Qianwen,iFlytek Spark,ChatGPT).Two attending physicians with more than 5 years of experience independently rated the responses using the 3C criteria(correct,clear,concise),and the consistency between the raters was assessed using the Spearman rank correlation coefficient,and the Kruskal-Wallis test and Dunn post hoc test were used to assess the statistical differences between the models.Additionally,this study used 600 questions from the 2023 dental licensing examination to evaluate the time taken to answer,scores,and accuracy of each model.ResultsAs an AI medical assistant,LLM can assist doctors in diagnosis and treatment decision-making,with an inter-evaluator Spearman coefficient of 0.505(P<0.01).As a simulated doctor,LLM can carry out patient popularization,with an inter-evaluator Spearman coefficient of 0.533(P<0.01).The 3C scores of each model as an AI medical assistant and a simulated doctor were respectively:2.00(1.00,3.00)and 2.00(2.00,3.00)points of Erine Bot,1.00(1.00,2.00)and 2.00(1.00,2.00)points of HuatuoGPT,2.00(1.00,2.00)and 2.00(1.00,3.00)points of Tongyi Qianwen,2.00(1.00,2.00)and 2.00(1.75,2.25)points of iFlytek Spark,3.00(2.00,3.00)and 3.00(2.00,3.00)points of ChatGPT(full score of 4 points).The Kruskal-Wallis test results showed that,as an AI medical assistant or a simulated doctor,there were statistically differences in the 3C scores among the five large language models(all P<0.001).The average score of the 5 LLMs on the dental licensing examination was 370.2,with an accuracy rate of 61.7%(370.2/600)and a time consumption of 94.6 min.Specifically,Erine Bot took 115 min,scored 363 points with an accuracy rate of 60.5%(363/600),HuatuoGPT took 224 min and scored 305 points with an accuracy rate of 50.8%(305/600),Tongyi Qianwen took 43 min,scored 438 points with an accuracy rate of 73.0%(438/600),iFlytek Spark took 32 min,scored 364 points with an accuracy rate of 60.7%(364/600),and ChatGPT took 59 min,scored 381 points with an accuracy rate of 63.5%(381/600).ConclusionsBased on the evaluation of LLM′s dual roles as an AI medical assistant and a simulated doctor,ChatGPT performes the best,with basically correct,clear and concise answers,followed by Erine Bot,Tongyi Qianwen and iFlytek Spark,with HuatuoGPT lagging behind significantly.In the dental licensing examination,all the 4 LLM,except for HuatuoGPT,reach the passing level,and the time consumpution for answering is significantly reduced compared to the 8 h required for the exam regulations in all of the five models.LLM has the feasibility of application in oral auxiliary,treatment and health consultation,and it can help both doctors and patients obtain medical information quickly.Howere,their outputs carry a risk of errors(since the 3C scoring results do not reach the full marks),so prudent judgment should be exercised when using them.
作者
韩彩玲
白石柱
张婷民
刘琛
刘昱晨
胡祥翔
赵铱民
Han Cailing;Bai Shizhu;Zhang Tingmin;Liu Chen;Liu Yuchen;Hu Xiangxiang;Zhao Yimin(Digital Center,School of Stomatology,The Fourth Military Medical University,State Key Laboratory of Oral&Maxillofacial Reconstruction and Regeneration,National Clinical Research Center for Oral Diseases,Shaanxi Key Laboratory of Stomatology,Xi′an 710032,China)
出处
《中华口腔医学杂志》
北大核心
2025年第8期871-878,共8页
Chinese Journal of Stomatology
基金
国家自然科学基金(82471035)
陕西省创新能力支撑计划(2023-CX-PT-27)
口颌系统重建与再生全国重点实验室自主研究课题(2021ZA02)。
关键词
人工智能
口腔医学
咨询
口腔保健
大语言模型
数字化口腔医学
Artificial intelligence
Oral medicine
Counseling
Oral health
Large language models
Digital dentistry
