The rapid growth of digital data necessitates advanced natural language processing(NLP)models like BERT(Bidi-rectional Encoder Representations from Transformers),known for its superior performance in text classificati...The rapid growth of digital data necessitates advanced natural language processing(NLP)models like BERT(Bidi-rectional Encoder Representations from Transformers),known for its superior performance in text classification.However,BERT’s size and computational demands limit its practicality,especially in resource-constrained settings.This research compresses the BERT base model for Bengali emotion classification through knowledge distillation(KD),pruning,and quantization techniques.Despite Bengali being the sixth most spoken language globally,NLP research in this area is limited.Our approach addresses this gap by creating an efficient BERT-based model for Bengali text.We have explored 20 combinations for KD,quantization,and pruning,resulting in improved speedup,fewer parameters,and reduced memory size.Our best results demonstrate significant improvements in both speed and efficiency.For instance,in the case of mBERT,we achieved a 3.87×speedup and 4×compression ratio with a combination of Distil+Prune+Quant that reduced parameters from 178 to 46 M,while the memory size decreased from 711 to 178 MB.These results offer scalable solutions for NLP tasks in various languages and advance the field of model compression,making these models suitable for real-world applications in resource-limited environments.展开更多
Electroencephalographic studies using graph theoretic analysis have found aberrations in functional connectivity in children with developmental dyslexia.However,how the training with visual tasks can change the functi...Electroencephalographic studies using graph theoretic analysis have found aberrations in functional connectivity in children with developmental dyslexia.However,how the training with visual tasks can change the functional connectivity of the semantic network in developmental dyslexia is still unclear.We looked for differences in local and global topological properties of functional networks between 21 healthy controls and 22 dyslexic children(8–9 years old)before and after training with visual tasks in this prospective case-control study.The minimum spanning tree method was used to construct the subjects’brain networks in multiple electroencephalographic frequency ranges during a visual word/pseudoword discrimination task.We found group differences in the theta,alpha,beta and gamma bands for four graph measures suggesting a more integrated network topology in dyslexics before the training compared to controls.After training,the network topology of dyslexic children had become more segregated and similar to that of the controls.In theθ,αandβ1-frequency bands,compared to the controls,the pre-training dyslexics exhibited a reduced degree and betweenness centrality of the left anterior temporal and parietal regions.The simultaneous appearance in the left hemisphere of hubs in temporal and parietal(α,β1),temporal and superior frontal cortex(θ,α),parietal and occipitotemporal cortices(β1),identified in the networks of normally developing children was not present in the brain networks of dyslexics.After training,the hub distribution for dyslexics in the theta and beta1 bands had become similar to that of the controls.In summary,our findings point to a less efficient network configuration in dyslexics compared to a more optimal global organization in the controls.This is the first study to investigate the topological organization of functional brain networks of Bulgarian dyslexic children.Approval for the study was obtained from the Ethics Committee of the Institute of Neurobiology and the Institute for Population and Human Studies,Bulgarian Academy of Sciences(approval No.02-41/12.07.2019)on March 28,2017,and the State Logopedic Center and the Ministry of Education and Science(approval No.09-69/14.03.2017)on July 12,2019.展开更多
Transformers have demonstrated considerable success across various domains but are constrained by their significant computational and memory requirements.This poses challenges for deployment on resource-constrained de...Transformers have demonstrated considerable success across various domains but are constrained by their significant computational and memory requirements.This poses challenges for deployment on resource-constrained devices.Quantization,as an effective model compression method,can significantly reduce the operational time of Transformers on edge devices.Notably,Transformers display more substantial outliers than convolutional neural networks,leading to uneven feature distribution among different channels and tokens.To address this issue,we propose an adaptive outlier correction quantization(AOCQ)method for Transformers,which significantly alleviates the adverse effects of these outliers.AOCQ adjusts the notable discrepancies in channels and tokens across three levels:operator level,framework level,and loss level.We introduce a new operator that equivalently balances the activations across different channels and insert an extra stage to optimize the activation quantization step on the framework level.Additionally,we transfer the imbalanced activations across tokens and channels to the optimization of model weights on the loss level.Based on the theoretical study,our method can reduce the quantization error.The effectiveness of the proposed method is verified on various benchmark models and tasks.Surprisingly,DeiT-Base with 8-bit post-training quantization(PTQ)can achieve 81.57%accuracy with a 0.28 percentage point drop while enjoying 4×faster runtime.Furthermore,the weights of Swin and DeiT on several tasks,including classification and object detection,can be post-quantized to ultra-low 4 bits,with a minimal accuracy loss of 2%,while requiring nearly 8×less memory.展开更多
文摘The rapid growth of digital data necessitates advanced natural language processing(NLP)models like BERT(Bidi-rectional Encoder Representations from Transformers),known for its superior performance in text classification.However,BERT’s size and computational demands limit its practicality,especially in resource-constrained settings.This research compresses the BERT base model for Bengali emotion classification through knowledge distillation(KD),pruning,and quantization techniques.Despite Bengali being the sixth most spoken language globally,NLP research in this area is limited.Our approach addresses this gap by creating an efficient BERT-based model for Bengali text.We have explored 20 combinations for KD,quantization,and pruning,resulting in improved speedup,fewer parameters,and reduced memory size.Our best results demonstrate significant improvements in both speed and efficiency.For instance,in the case of mBERT,we achieved a 3.87×speedup and 4×compression ratio with a combination of Distil+Prune+Quant that reduced parameters from 178 to 46 M,while the memory size decreased from 711 to 178 MB.These results offer scalable solutions for NLP tasks in various languages and advance the field of model compression,making these models suitable for real-world applications in resource-limited environments.
基金The study was supported by the National Science Fund of the Ministry of Education and Science(project DN05/14-2016,to JAD).
文摘Electroencephalographic studies using graph theoretic analysis have found aberrations in functional connectivity in children with developmental dyslexia.However,how the training with visual tasks can change the functional connectivity of the semantic network in developmental dyslexia is still unclear.We looked for differences in local and global topological properties of functional networks between 21 healthy controls and 22 dyslexic children(8–9 years old)before and after training with visual tasks in this prospective case-control study.The minimum spanning tree method was used to construct the subjects’brain networks in multiple electroencephalographic frequency ranges during a visual word/pseudoword discrimination task.We found group differences in the theta,alpha,beta and gamma bands for four graph measures suggesting a more integrated network topology in dyslexics before the training compared to controls.After training,the network topology of dyslexic children had become more segregated and similar to that of the controls.In theθ,αandβ1-frequency bands,compared to the controls,the pre-training dyslexics exhibited a reduced degree and betweenness centrality of the left anterior temporal and parietal regions.The simultaneous appearance in the left hemisphere of hubs in temporal and parietal(α,β1),temporal and superior frontal cortex(θ,α),parietal and occipitotemporal cortices(β1),identified in the networks of normally developing children was not present in the brain networks of dyslexics.After training,the hub distribution for dyslexics in the theta and beta1 bands had become similar to that of the controls.In summary,our findings point to a less efficient network configuration in dyslexics compared to a more optimal global organization in the controls.This is the first study to investigate the topological organization of functional brain networks of Bulgarian dyslexic children.Approval for the study was obtained from the Ethics Committee of the Institute of Neurobiology and the Institute for Population and Human Studies,Bulgarian Academy of Sciences(approval No.02-41/12.07.2019)on March 28,2017,and the State Logopedic Center and the Ministry of Education and Science(approval No.09-69/14.03.2017)on July 12,2019.
文摘Transformers have demonstrated considerable success across various domains but are constrained by their significant computational and memory requirements.This poses challenges for deployment on resource-constrained devices.Quantization,as an effective model compression method,can significantly reduce the operational time of Transformers on edge devices.Notably,Transformers display more substantial outliers than convolutional neural networks,leading to uneven feature distribution among different channels and tokens.To address this issue,we propose an adaptive outlier correction quantization(AOCQ)method for Transformers,which significantly alleviates the adverse effects of these outliers.AOCQ adjusts the notable discrepancies in channels and tokens across three levels:operator level,framework level,and loss level.We introduce a new operator that equivalently balances the activations across different channels and insert an extra stage to optimize the activation quantization step on the framework level.Additionally,we transfer the imbalanced activations across tokens and channels to the optimization of model weights on the loss level.Based on the theoretical study,our method can reduce the quantization error.The effectiveness of the proposed method is verified on various benchmark models and tasks.Surprisingly,DeiT-Base with 8-bit post-training quantization(PTQ)can achieve 81.57%accuracy with a 0.28 percentage point drop while enjoying 4×faster runtime.Furthermore,the weights of Swin and DeiT on several tasks,including classification and object detection,can be post-quantized to ultra-low 4 bits,with a minimal accuracy loss of 2%,while requiring nearly 8×less memory.
