摘要
学术文本深度理解能力已成为情报工作重要支撑,大模型在此类工作中展现了巨大的潜力。大模型可以从推理端和调优端两个方向提升模型的知识挖掘和利用能力。当前,在领域深度相关的学术文本挖掘任务上,推理端的各类指令工程技术仍难以充分发挥大模型的深度语义理解能力,因此,在调优端使用参数高效微调技术面向领域任务对模型参数进行适配,成为大模型赋能学术文本挖掘的关键。目前尚未形成对模型应用不同调优方法的性能和效益的系统性探索。本研究构建了面向学术文本挖掘的参数高效微调框架和性能效益评测体系,通过对7类指令调优模型应用8项调优方法后的性能指标与成本效益进行评估,对参数高效微调策略与调优模型在学术文本挖掘任务上的能力边界进行探索。研究结果表明,在各类调优方法中,全量微调性能最优,但其领先优势并不显著;QLoRA(quantized low-rank adaptation)的计算成本最低,成为综合效益最高的调优方法。不同规模和架构的大模型调优后的性能差异不大,Mistral-7B-Instruct-v0.1等规模较小的模型使用QLoRA调优后可取得与百亿级模型相当的性能指标。调优后的大模型在引文功能识别、科技实体抽取、科技文本推理3类任务上的性能指标均大幅领先于其在指令端的表现;相比于传统深度学习模型,大模型在学术文本推理任务上全面领先,在科技实体抽取和引文功能识别任务上与小模型性能相近。由此可见,大模型在难度较高的复杂任务上表现更好,而对于简单的序列标注任务和分类任务,使用小模型的收益更高。
The ability to deeply understand academic texts has become a crucial support in intelligence work,and large language models(LLMs)have shown great potential in this area.LLMs can enhance knowledge extraction and utilization capabilities from both the inference end and tuning end.Currently,in academic text mining,various instruction engineering techniques at the inference end struggle to fully leverage the deep semantic understanding capabilities of LLMs.Therefore,adapting model parameters for domain-specific tasks using techniques such as parameter-efficient fine-tuning(PEFT)at the tuning end has become the key for LLMs to empower academic text mining.The performance and efficiency of applying different PEFT methods to LLMs have not yet been systematically explored.This study constructs a PEFT framework and evaluation system for academic text mining.It evaluates the performance metrics and cost-efficiency of seven instruction-tuned LLMs after applying seven PEFT methods,exploring the capability boundaries of PEFT strategies and instruction-tuned LLMs in academic text mining.The experiments demonstrate that,among the various tuning methods,finetuning achieves the best performance.However,its advantage is not significantly pronounced.By contrast,quantized lowrank adaptation(QLoRA)incurs the lowest computational cost,making it the most efficient PEFT method in terms of overall benefits.The performance differences following tuning across LLMs of varying sizes and architectures are minimal.Mistral-7B-Instruct-v0.1,which is smaller in scale,can achieve performance metrics comparable to those of models with 70B parameters when tuned with QLoRA.The LLMs show substantial improvements in performance across tasks such as citation function identification,scientific entity extraction,and scientific text reasoning,surpassing their performance on the instruction end by a significant margin.Compared with traditional deep learning models,LLMs in the tuning end comprehensively outperform in academic text reasoning tasks and perform similarly to smaller models in scientific entity extraction and citation function identification tasks.Therefore,LLMs perform better in tasks with higher difficulty,whereas small models are more beneficial for simpler sequence labeling and classification tasks.
作者
刘寅鹏
陆伟
石湘
刘家伟
程齐凯
黄永
Liu Yinpeng;Lu Wei;Shi Xiang;Liu Jiawei;Cheng Qikai;Huang Yong(School of Information Management,Wuhan University,Wuhan 430072;Institute of Intelligence and Innovation Governance,Wuhan University,Wuhan 430072)
出处
《情报学报》
北大核心
2025年第9期1159-1172,共14页
Journal of the China Society for Scientific and Technical Information
基金
国家自然科学基金重点项目“数智赋能的科技信息资源与知识管理理论变革”(72234005)
国家自然科学基金面上项目“基于机器阅读理解的科学命题文本论证逻辑识别”(72174157)。
关键词
大模型
学术文本挖掘
参数高效微调策略
能力评估
large language models
academic text mining
parameter-efficient fine-tuning(PEFT)
capability evaluation
