Activity cliffs(ACs)are generally defined as pairs of similar compounds that only differ by a minor structural modification but exhibit a large difference in their binding affinity for a given target.ACs offer crucial...Activity cliffs(ACs)are generally defined as pairs of similar compounds that only differ by a minor structural modification but exhibit a large difference in their binding affinity for a given target.ACs offer crucial insights that aid medicinal chemists in optimizing molecular structures.Nonetheless,they also form a major source of prediction error in structure-activity relationship(SAR)models.To date,several studies have demonstrated that deep neural networks based on molecular images or graphs might need to be improved further in predicting the potency of ACs.In this paper,we integrated the triplet loss in face recognition with pre-training strategy to develop a prediction model ACtriplet,tailored for ACs.Through extensive comparison with multiple baseline models on 30 benchmark datasets,the results showed that ACtriplet was significantly better than those deep learning(DL)models without pretraining.In addition,we explored the effect of pre-training on data representation.Finally,the case study demonstrated that our model's interpretability module could explain the prediction results reasonably.In the dilemma that the amount of data could not be increased rapidly,this innovative framework would better make use of the existing data,which would propel the potential of DL in the early stage of drug discovery and optimization.展开更多
针对非侵入式负荷分解方法负荷特征捕捉不足、负荷分解精度不够等问题,文章提出一种基于改进BERT(bidirectional encoder representations from transformers)模型的多头自注意力非侵入式负荷分解方法(frequency and temporal attention...针对非侵入式负荷分解方法负荷特征捕捉不足、负荷分解精度不够等问题,文章提出一种基于改进BERT(bidirectional encoder representations from transformers)模型的多头自注意力非侵入式负荷分解方法(frequency and temporal attention-BERT, FAT-BERT)。首先通过傅里叶变换将时域数据转换为频域数据,采用多尺度卷积全面捕捉负荷信号的时域和频域特征,从而增强模型对多样化负荷信号的表达能力;其次,在多头自注意力机制中引入频率注意力机制,从而增强模型对时序数据中频率成分的感知能力,进一步改善复杂负荷模式的表示,改进BERT模型中增加局部自注意力从而减少不必要的全局计算,提升模型的运行速度;接着将残差连接和正则化技术结合使模型在训练过程中更加稳定,并且能够更好地避免过拟合,最后在REDD和UK-DALE数据集上对提出的方法进行实验,实验结果验证了所提方法的有效性。展开更多
方面级情感分析旨在识别文本中针对特定方面的情感倾向,然而现有研究仍面临多重挑战:基于BERT的方面级情感分析研究存在语义过拟合、低层级语义利用不足的问题;自注意力机制存在局部信息丢失的问题;多编码层和多粒度语义的结构存在信息...方面级情感分析旨在识别文本中针对特定方面的情感倾向,然而现有研究仍面临多重挑战:基于BERT的方面级情感分析研究存在语义过拟合、低层级语义利用不足的问题;自注意力机制存在局部信息丢失的问题;多编码层和多粒度语义的结构存在信息冗余问题。为此,提出一种融合BERT编码层的多粒度语义方面级情感分析模型(multi-granular semantic aspect-based sentiment analysis model with fusion of BERT encoding layers,MSBEL)。具体地,引入金字塔注意力机制,利用各个编码层的语义特征,并结合低层编码器以降低过拟合;通过多尺度门控卷积增强模型处理局部信息丢失的能力;使用余弦注意力突出与方面词相关的情感特征,从而减少信息冗余。t-SNE的可视化分析表明,MSBEL的情感表示聚类效果优于BERT。此外,在多个基准数据集上将本文模型与主流模型的性能进行了对比,结果显示:与LCF-BERT相比,本文模型在5个数据集上的F1分别提升了1.53%、3.94%、1.39%、6.68%、5.97%;与SenticGCN相比,本文模型的F1平均提升0.94%,最大提升2.12%;与ABSA-DeBERTa相比,本文模型的F1平均提升1.16%,最大提升4.20%,验证了本文模型在方面级情感分析任务上的有效性和优越性。展开更多
Dialectal Arabic text classifcation(DA-TC)provides a mechanism for performing sentiment analysis on recent Arabic social media leading to many challenges owing to the natural morphology of the Arabic language and its ...Dialectal Arabic text classifcation(DA-TC)provides a mechanism for performing sentiment analysis on recent Arabic social media leading to many challenges owing to the natural morphology of the Arabic language and its wide range of dialect variations.Te availability of annotated datasets is limited,and preprocessing of the noisy content is even more challenging,sometimes resulting in the removal of important cues of sentiment from the input.To overcome such problems,this study investigates the applicability of using transfer learning based on pre-trained transformer models to classify sentiment in Arabic texts with high accuracy.Specifcally,it uses the CAMeLBERT model fnetuned for the Multi-Domain Arabic Resources for Sentiment Analysis(MARSA)dataset containing more than 56,000 manually annotated tweets annotated across political,social,sports,and technology domains.Te proposed method avoids extensive use of preprocessing and shows that raw data provides better results because they tend to retain more linguistic features.Te fne-tuned CAMeLBERT model produces state-of-the-art accuracy of 92%,precision of 91.7%,recall of 92.3%,and F1-score of 91.5%,outperforming standard machine learning models and ensemble-based/deep learning techniques.Our performance comparisons against other pre-trained models,namely AraBERTv02-twitter and MARBERT,show that transformer-based architectures are consistently the best suited when dealing with noisy Arabic texts.Tis work leads to a strong remedy for the problems in Arabic sentiment analysis and provides recommendations on easy tuning of the pre-trained models to adapt to challenging linguistic features and domain-specifc tasks.展开更多
Spam emails remain one of the most persistent threats to digital communication,necessitating effective detection solutions that safeguard both individuals and organisations.We propose a spam email classification frame...Spam emails remain one of the most persistent threats to digital communication,necessitating effective detection solutions that safeguard both individuals and organisations.We propose a spam email classification frame-work that uses Bidirectional Encoder Representations from Transformers(BERT)for contextual feature extraction and a multiple-window Convolutional Neural Network(CNN)for classification.To identify semantic nuances in email content,BERT embeddings are used,and CNN filters extract discriminative n-gram patterns at various levels of detail,enabling accurate spam identification.The proposed model outperformed Word2Vec-based baselines on a sample of 5728 labelled emails,achieving an accuracy of 98.69%,AUC of 0.9981,F1 Score of 0.9724,and MCC of 0.9639.With a medium kernel size of(6,9)and compact multi-window CNN architectures,it improves performance.Cross-validation illustrates stability and generalization across folds.By balancing high recall with minimal false positives,our method provides a reliable and scalable solution for current spam detection in advanced deep learning.By combining contextual embedding and a neural architecture,this study develops a security analysis method.展开更多
针对火电机组故障预测准确率低的问题,提出基于BERT-STAM1DCNN的火电机组故障预测模型。在预处理阶段,利用随机森林算法对原始数据进行处理,筛选重要的特征,提高故障预测准确率;利用BERT(bidirectional encoder representations from tr...针对火电机组故障预测准确率低的问题,提出基于BERT-STAM1DCNN的火电机组故障预测模型。在预处理阶段,利用随机森林算法对原始数据进行处理,筛选重要的特征,提高故障预测准确率;利用BERT(bidirectional encoder representations from transformers)模型对特征数据进行训练,并提出一种融合二次加权时空注意力机制的一维卷积神经网络(STAM1DCNN)模型,提高关键信息对预测结果的影响程度。以火电机组实际运行数据作为数据集,与其他模型相比,所提出的模型具有良好的性能和稳定性。展开更多
Entity relation extraction(ERE)is an important task in the field of information extraction.With the wide application of pre-training language model(PLM)in natural language processing(NLP),using PLM has become a brand ...Entity relation extraction(ERE)is an important task in the field of information extraction.With the wide application of pre-training language model(PLM)in natural language processing(NLP),using PLM has become a brand new research direction of ERE.In this paper,BERT is used to extracting entityrelations,and a separated pipeline architecture is proposed.ERE was decomposed into entity-relation classification sub-task and entity-pair annotation sub-task.Both sub-tasks conduct the pre-training and fine-tuning independently.Combining dynamic and static masking,newVerb-MLM and Entity-MLM BERT pre-training tasks were put forward to enhance the correlation between BERT pre-training and TargetedNLPdownstream task-ERE.Inter-layer sharing attentionmechanismwas added to the model,sharing the attention parameters according to the similarity of the attention matrix.Contrast experiment on the SemEavl 2010 Task8 dataset demonstrates that the new MLM task and inter-layer sharing attention mechanism improve the performance of BERT on the entity relation extraction effectively.展开更多
基金supported by the National Natural Science Foundation of China(Grant Nos.:U23A20530,82273858,and 82173746)the National Key Research and Development Programof China(Grant No.:2023YFF1204904)Shanghai Frontiers Science Center of Optogenetic Techniques for Cell Metabolism(Shanghai Municipal Education Commission,China).
文摘Activity cliffs(ACs)are generally defined as pairs of similar compounds that only differ by a minor structural modification but exhibit a large difference in their binding affinity for a given target.ACs offer crucial insights that aid medicinal chemists in optimizing molecular structures.Nonetheless,they also form a major source of prediction error in structure-activity relationship(SAR)models.To date,several studies have demonstrated that deep neural networks based on molecular images or graphs might need to be improved further in predicting the potency of ACs.In this paper,we integrated the triplet loss in face recognition with pre-training strategy to develop a prediction model ACtriplet,tailored for ACs.Through extensive comparison with multiple baseline models on 30 benchmark datasets,the results showed that ACtriplet was significantly better than those deep learning(DL)models without pretraining.In addition,we explored the effect of pre-training on data representation.Finally,the case study demonstrated that our model's interpretability module could explain the prediction results reasonably.In the dilemma that the amount of data could not be increased rapidly,this innovative framework would better make use of the existing data,which would propel the potential of DL in the early stage of drug discovery and optimization.
文摘针对非侵入式负荷分解方法负荷特征捕捉不足、负荷分解精度不够等问题,文章提出一种基于改进BERT(bidirectional encoder representations from transformers)模型的多头自注意力非侵入式负荷分解方法(frequency and temporal attention-BERT, FAT-BERT)。首先通过傅里叶变换将时域数据转换为频域数据,采用多尺度卷积全面捕捉负荷信号的时域和频域特征,从而增强模型对多样化负荷信号的表达能力;其次,在多头自注意力机制中引入频率注意力机制,从而增强模型对时序数据中频率成分的感知能力,进一步改善复杂负荷模式的表示,改进BERT模型中增加局部自注意力从而减少不必要的全局计算,提升模型的运行速度;接着将残差连接和正则化技术结合使模型在训练过程中更加稳定,并且能够更好地避免过拟合,最后在REDD和UK-DALE数据集上对提出的方法进行实验,实验结果验证了所提方法的有效性。
文摘方面级情感分析旨在识别文本中针对特定方面的情感倾向,然而现有研究仍面临多重挑战:基于BERT的方面级情感分析研究存在语义过拟合、低层级语义利用不足的问题;自注意力机制存在局部信息丢失的问题;多编码层和多粒度语义的结构存在信息冗余问题。为此,提出一种融合BERT编码层的多粒度语义方面级情感分析模型(multi-granular semantic aspect-based sentiment analysis model with fusion of BERT encoding layers,MSBEL)。具体地,引入金字塔注意力机制,利用各个编码层的语义特征,并结合低层编码器以降低过拟合;通过多尺度门控卷积增强模型处理局部信息丢失的能力;使用余弦注意力突出与方面词相关的情感特征,从而减少信息冗余。t-SNE的可视化分析表明,MSBEL的情感表示聚类效果优于BERT。此外,在多个基准数据集上将本文模型与主流模型的性能进行了对比,结果显示:与LCF-BERT相比,本文模型在5个数据集上的F1分别提升了1.53%、3.94%、1.39%、6.68%、5.97%;与SenticGCN相比,本文模型的F1平均提升0.94%,最大提升2.12%;与ABSA-DeBERTa相比,本文模型的F1平均提升1.16%,最大提升4.20%,验证了本文模型在方面级情感分析任务上的有效性和优越性。
基金funded by the Deanship of Scientific Research at Imam Mohammad Ibn Saud Islamic University(IMSIU)(grant number IMSIU-DDRSP2504).
文摘Dialectal Arabic text classifcation(DA-TC)provides a mechanism for performing sentiment analysis on recent Arabic social media leading to many challenges owing to the natural morphology of the Arabic language and its wide range of dialect variations.Te availability of annotated datasets is limited,and preprocessing of the noisy content is even more challenging,sometimes resulting in the removal of important cues of sentiment from the input.To overcome such problems,this study investigates the applicability of using transfer learning based on pre-trained transformer models to classify sentiment in Arabic texts with high accuracy.Specifcally,it uses the CAMeLBERT model fnetuned for the Multi-Domain Arabic Resources for Sentiment Analysis(MARSA)dataset containing more than 56,000 manually annotated tweets annotated across political,social,sports,and technology domains.Te proposed method avoids extensive use of preprocessing and shows that raw data provides better results because they tend to retain more linguistic features.Te fne-tuned CAMeLBERT model produces state-of-the-art accuracy of 92%,precision of 91.7%,recall of 92.3%,and F1-score of 91.5%,outperforming standard machine learning models and ensemble-based/deep learning techniques.Our performance comparisons against other pre-trained models,namely AraBERTv02-twitter and MARBERT,show that transformer-based architectures are consistently the best suited when dealing with noisy Arabic texts.Tis work leads to a strong remedy for the problems in Arabic sentiment analysis and provides recommendations on easy tuning of the pre-trained models to adapt to challenging linguistic features and domain-specifc tasks.
基金funded by Princess Nourah bint Abdulrahman University Researchers Supporting Project number(PNURSP2026R234)Princess Nourah bint Abdulrahman University,Riyadh,Saudi Arabia.
文摘Spam emails remain one of the most persistent threats to digital communication,necessitating effective detection solutions that safeguard both individuals and organisations.We propose a spam email classification frame-work that uses Bidirectional Encoder Representations from Transformers(BERT)for contextual feature extraction and a multiple-window Convolutional Neural Network(CNN)for classification.To identify semantic nuances in email content,BERT embeddings are used,and CNN filters extract discriminative n-gram patterns at various levels of detail,enabling accurate spam identification.The proposed model outperformed Word2Vec-based baselines on a sample of 5728 labelled emails,achieving an accuracy of 98.69%,AUC of 0.9981,F1 Score of 0.9724,and MCC of 0.9639.With a medium kernel size of(6,9)and compact multi-window CNN architectures,it improves performance.Cross-validation illustrates stability and generalization across folds.By balancing high recall with minimal false positives,our method provides a reliable and scalable solution for current spam detection in advanced deep learning.By combining contextual embedding and a neural architecture,this study develops a security analysis method.
文摘针对火电机组故障预测准确率低的问题,提出基于BERT-STAM1DCNN的火电机组故障预测模型。在预处理阶段,利用随机森林算法对原始数据进行处理,筛选重要的特征,提高故障预测准确率;利用BERT(bidirectional encoder representations from transformers)模型对特征数据进行训练,并提出一种融合二次加权时空注意力机制的一维卷积神经网络(STAM1DCNN)模型,提高关键信息对预测结果的影响程度。以火电机组实际运行数据作为数据集,与其他模型相比,所提出的模型具有良好的性能和稳定性。
基金Hainan Province High level talent project of basic and applied basic research plan(Natural Science Field)in 2019(No.2019RC100)Haikou City Key Science and Technology Plan Project(2020–049)Hainan Province Key Research and Development Project(ZDYF2020018).
文摘Entity relation extraction(ERE)is an important task in the field of information extraction.With the wide application of pre-training language model(PLM)in natural language processing(NLP),using PLM has become a brand new research direction of ERE.In this paper,BERT is used to extracting entityrelations,and a separated pipeline architecture is proposed.ERE was decomposed into entity-relation classification sub-task and entity-pair annotation sub-task.Both sub-tasks conduct the pre-training and fine-tuning independently.Combining dynamic and static masking,newVerb-MLM and Entity-MLM BERT pre-training tasks were put forward to enhance the correlation between BERT pre-training and TargetedNLPdownstream task-ERE.Inter-layer sharing attentionmechanismwas added to the model,sharing the attention parameters according to the similarity of the attention matrix.Contrast experiment on the SemEavl 2010 Task8 dataset demonstrates that the new MLM task and inter-layer sharing attention mechanism improve the performance of BERT on the entity relation extraction effectively.
