In this paper,the small-signal modeling of the Indium Phosphide High Electron Mobility Transistor(InP HEMT)based on the Transformer neural network model is investigated.The AC S-parameters of the HEMT device are train...In this paper,the small-signal modeling of the Indium Phosphide High Electron Mobility Transistor(InP HEMT)based on the Transformer neural network model is investigated.The AC S-parameters of the HEMT device are trained and validated using the Transformer model.In the proposed model,the eight-layer transformer encoders are connected in series and the encoder layer of each Transformer consists of the multi-head attention layer and the feed-forward neural network layer.The experimental results show that the measured and modeled S-parameters of the HEMT device match well in the frequency range of 0.5-40 GHz,with the errors versus frequency less than 1%.Compared with other models,good accuracy can be achieved to verify the effectiveness of the proposed model.展开更多
Cyberbullying on social media poses significant psychological risks,yet most detection systems over-simplify the task by focusing on binary classification,ignoring nuanced categories like passive-aggressive remarks or...Cyberbullying on social media poses significant psychological risks,yet most detection systems over-simplify the task by focusing on binary classification,ignoring nuanced categories like passive-aggressive remarks or indirect slurs.To address this gap,we propose a hybrid framework combining Term Frequency-Inverse Document Frequency(TF-IDF),word-to-vector(Word2Vec),and Bidirectional Encoder Representations from Transformers(BERT)based models for multi-class cyberbullying detection.Our approach integrates TF-IDF for lexical specificity and Word2Vec for semantic relationships,fused with BERT’s contextual embeddings to capture syntactic and semantic complexities.We evaluate the framework on a publicly available dataset of 47,000 annotated social media posts across five cyberbullying categories:age,ethnicity,gender,religion,and indirect aggression.Among BERT variants tested,BERT Base Un-Cased achieved the highest performance with 93%accuracy(standard deviation across±1%5-fold cross-validation)and an average AUC of 0.96,outperforming standalone TF-IDF(78%)and Word2Vec(82%)models.Notably,it achieved near-perfect AUC scores(0.99)for age and ethnicity-based bullying.A comparative analysis with state-of-the-art benchmarks,including Generative Pre-trained Transformer 2(GPT-2)and Text-to-Text Transfer Transformer(T5)models highlights BERT’s superiority in handling ambiguous language.This work advances cyberbullying detection by demonstrating how hybrid feature extraction and transformer models improve multi-class classification,offering a scalable solution for moderating nuanced harmful content.展开更多
AlphaPanda(AlphaFold2[1]inspired protein-specific antibody design in a diffusional manner)is an advanced algorithm for designing complementary determining regions(CDRs)of the antibody targeted the specific epitope,com...AlphaPanda(AlphaFold2[1]inspired protein-specific antibody design in a diffusional manner)is an advanced algorithm for designing complementary determining regions(CDRs)of the antibody targeted the specific epitope,combining transformer[2]models,3DCNN[3],and diffusion[4]generative models.展开更多
Short⁃term traffic flow prediction plays a crucial role in the planning of intelligent transportation systems.Nowadays,there is a large amount of traffic flow data generated from the monitoring devices of urban road n...Short⁃term traffic flow prediction plays a crucial role in the planning of intelligent transportation systems.Nowadays,there is a large amount of traffic flow data generated from the monitoring devices of urban road networks,which contains road network traffic information with high application value.In this study,an improved spatio⁃temporal attention transformer model(ISTA⁃transformer model)is proposed to provide a more accurate method for predicting multi⁃step short⁃term traffic flow based on monitoring data.By embedding a temporal attention layer and a spatial attention layer in the model,the model learns the relationship between traffic flows at different time intervals and different geographic locations,and realizes more accurate multi⁃step short⁃time flow prediction.Finally,we validate the superiority of the model with monitoring data spanning 15 days from 620 monitoring points in Qingdao,China.In the four time steps of prediction,the MAPE(Mean Absolute Percentage Error)values of ISTA⁃transformers prediction results are 0.22,0.29,0.37,and 0.38,respectively,and its prediction accuracy is usually better than that of six baseline models(Transformer,GRU,CNN,LSTM,Seq2Seq and LightGBM),which indicates that the proposed model in this paper always has a better ability to explain the prediction results with the time steps in the multi⁃step prediction.展开更多
Deep learning(DL)has become a crucial technique for predicting the El Niño-Southern Oscillation(ENSO)and evaluating its predictability.While various DL-based models have been developed for ENSO predictions,many f...Deep learning(DL)has become a crucial technique for predicting the El Niño-Southern Oscillation(ENSO)and evaluating its predictability.While various DL-based models have been developed for ENSO predictions,many fail to capture the coherent multivariate evolution within the coupled ocean-atmosphere system of the tropical Pacific.To address this three-dimensional(3D)limitation and represent ENSO-related ocean-atmosphere interactions more accurately,a novel this 3D multivariate prediction model was proposed based on a Transformer architecture,which incorporates a spatiotemporal self-attention mechanism.This model,named 3D-Geoformer,offers several advantages,enabling accurate ENSO predictions up to one and a half years in advance.Furthermore,an integrated gradient method was introduced into the model to identify the sources of predictability for sea surface temperature(SST)variability in the eastern equatorial Pacific.Results reveal that the 3D-Geoformer effectively captures ENSO-related precursors during the evolution of ENSO events,particularly the thermocline feedback processes and ocean temperature anomaly pathways on and off the equator.By extending DL-based ENSO predictions from one-dimensional Niño time series to 3D multivariate fields,the 3D-Geoformer represents a significant advancement in ENSO prediction.This study provides details in the model formulation,analysis procedures,sensitivity experiments,and illustrative examples,offering practical guidance for the application of the model in ENSO research.展开更多
针对现有深度学习算法在壁画修复时,存在全局语义一致性约束不足及局部特征提取不充分,导致修复后的壁画易出现边界效应和细节模糊等问题,提出一种双向自回归Transformer与快速傅里叶卷积增强的壁画修复方法.首先,设计基于Transformer...针对现有深度学习算法在壁画修复时,存在全局语义一致性约束不足及局部特征提取不充分,导致修复后的壁画易出现边界效应和细节模糊等问题,提出一种双向自回归Transformer与快速傅里叶卷积增强的壁画修复方法.首先,设计基于Transformer结构的全局语义特征修复模块,利用双向自回归机制与掩码语言模型(masked language modeling,MLM),提出改进的多头注意力全局语义壁画修复模块,提高对全局语义特征的修复能力.然后,构建了由门控卷积和残差模块组成的全局语义增强模块,增强全局语义特征一致性约束.最后,设计局部细节修复模块,采用大核注意力机制(large kernel attention,LKA)与快速傅里叶卷积提高细节特征的捕获能力,同时减少局部细节信息的丢失,提升修复壁画局部和整体特征的一致性.通过对敦煌壁画数字化修复实验,结果表明,所提算法修复性能更优,客观评价指标均优于比较算法.展开更多
基金Supported by the National Natural Science Foundation of China(62201293,62034003)the Open-Foundation of State Key Laboratory of Millimeter-Waves(K202313)the Jiangsu Province Youth Science and Technology Talent Support Project(JSTJ-2024-040)。
文摘In this paper,the small-signal modeling of the Indium Phosphide High Electron Mobility Transistor(InP HEMT)based on the Transformer neural network model is investigated.The AC S-parameters of the HEMT device are trained and validated using the Transformer model.In the proposed model,the eight-layer transformer encoders are connected in series and the encoder layer of each Transformer consists of the multi-head attention layer and the feed-forward neural network layer.The experimental results show that the measured and modeled S-parameters of the HEMT device match well in the frequency range of 0.5-40 GHz,with the errors versus frequency less than 1%.Compared with other models,good accuracy can be achieved to verify the effectiveness of the proposed model.
基金funded by Scientific Research Deanship at University of Hail-Saudi Arabia through Project Number RG-23092.
文摘Cyberbullying on social media poses significant psychological risks,yet most detection systems over-simplify the task by focusing on binary classification,ignoring nuanced categories like passive-aggressive remarks or indirect slurs.To address this gap,we propose a hybrid framework combining Term Frequency-Inverse Document Frequency(TF-IDF),word-to-vector(Word2Vec),and Bidirectional Encoder Representations from Transformers(BERT)based models for multi-class cyberbullying detection.Our approach integrates TF-IDF for lexical specificity and Word2Vec for semantic relationships,fused with BERT’s contextual embeddings to capture syntactic and semantic complexities.We evaluate the framework on a publicly available dataset of 47,000 annotated social media posts across five cyberbullying categories:age,ethnicity,gender,religion,and indirect aggression.Among BERT variants tested,BERT Base Un-Cased achieved the highest performance with 93%accuracy(standard deviation across±1%5-fold cross-validation)and an average AUC of 0.96,outperforming standalone TF-IDF(78%)and Word2Vec(82%)models.Notably,it achieved near-perfect AUC scores(0.99)for age and ethnicity-based bullying.A comparative analysis with state-of-the-art benchmarks,including Generative Pre-trained Transformer 2(GPT-2)and Text-to-Text Transfer Transformer(T5)models highlights BERT’s superiority in handling ambiguous language.This work advances cyberbullying detection by demonstrating how hybrid feature extraction and transformer models improve multi-class classification,offering a scalable solution for moderating nuanced harmful content.
基金supported by the Key Project of International Cooperation of Qilu University of Technology(Grant No.:QLUTGJHZ2018008)Shandong Provincial Natural Science Foundation Committee,China(Grant No.:ZR2016HB54)Shandong Provincial Key Laboratory of Microbial Engineering(SME).
文摘AlphaPanda(AlphaFold2[1]inspired protein-specific antibody design in a diffusional manner)is an advanced algorithm for designing complementary determining regions(CDRs)of the antibody targeted the specific epitope,combining transformer[2]models,3DCNN[3],and diffusion[4]generative models.
基金Sponsored by National Key Research and Development Program of China(Grant No.2020YEB1600500).
文摘Short⁃term traffic flow prediction plays a crucial role in the planning of intelligent transportation systems.Nowadays,there is a large amount of traffic flow data generated from the monitoring devices of urban road networks,which contains road network traffic information with high application value.In this study,an improved spatio⁃temporal attention transformer model(ISTA⁃transformer model)is proposed to provide a more accurate method for predicting multi⁃step short⁃term traffic flow based on monitoring data.By embedding a temporal attention layer and a spatial attention layer in the model,the model learns the relationship between traffic flows at different time intervals and different geographic locations,and realizes more accurate multi⁃step short⁃time flow prediction.Finally,we validate the superiority of the model with monitoring data spanning 15 days from 620 monitoring points in Qingdao,China.In the four time steps of prediction,the MAPE(Mean Absolute Percentage Error)values of ISTA⁃transformers prediction results are 0.22,0.29,0.37,and 0.38,respectively,and its prediction accuracy is usually better than that of six baseline models(Transformer,GRU,CNN,LSTM,Seq2Seq and LightGBM),which indicates that the proposed model in this paper always has a better ability to explain the prediction results with the time steps in the multi⁃step prediction.
基金Supported by the Laoshan Laboratory(No.LSKJ202202402)the National Natural Science Foundation of China(No.42030410)+2 种基金the Startup Foundation for Introducing Talent of Nanjing University of Information Science&Technology,and Jiangsu Innovation Research Group(No.JSSCTD 202346)supported by the China National Postdoctoral Program for Innovative Talents(No.BX20240169)the China Postdoctoral Science Foundation(No.2141062400101)。
文摘Deep learning(DL)has become a crucial technique for predicting the El Niño-Southern Oscillation(ENSO)and evaluating its predictability.While various DL-based models have been developed for ENSO predictions,many fail to capture the coherent multivariate evolution within the coupled ocean-atmosphere system of the tropical Pacific.To address this three-dimensional(3D)limitation and represent ENSO-related ocean-atmosphere interactions more accurately,a novel this 3D multivariate prediction model was proposed based on a Transformer architecture,which incorporates a spatiotemporal self-attention mechanism.This model,named 3D-Geoformer,offers several advantages,enabling accurate ENSO predictions up to one and a half years in advance.Furthermore,an integrated gradient method was introduced into the model to identify the sources of predictability for sea surface temperature(SST)variability in the eastern equatorial Pacific.Results reveal that the 3D-Geoformer effectively captures ENSO-related precursors during the evolution of ENSO events,particularly the thermocline feedback processes and ocean temperature anomaly pathways on and off the equator.By extending DL-based ENSO predictions from one-dimensional Niño time series to 3D multivariate fields,the 3D-Geoformer represents a significant advancement in ENSO prediction.This study provides details in the model formulation,analysis procedures,sensitivity experiments,and illustrative examples,offering practical guidance for the application of the model in ENSO research.
文摘针对现有深度学习算法在壁画修复时,存在全局语义一致性约束不足及局部特征提取不充分,导致修复后的壁画易出现边界效应和细节模糊等问题,提出一种双向自回归Transformer与快速傅里叶卷积增强的壁画修复方法.首先,设计基于Transformer结构的全局语义特征修复模块,利用双向自回归机制与掩码语言模型(masked language modeling,MLM),提出改进的多头注意力全局语义壁画修复模块,提高对全局语义特征的修复能力.然后,构建了由门控卷积和残差模块组成的全局语义增强模块,增强全局语义特征一致性约束.最后,设计局部细节修复模块,采用大核注意力机制(large kernel attention,LKA)与快速傅里叶卷积提高细节特征的捕获能力,同时减少局部细节信息的丢失,提升修复壁画局部和整体特征的一致性.通过对敦煌壁画数字化修复实验,结果表明,所提算法修复性能更优,客观评价指标均优于比较算法.
