Large-scale Language Models(LLMs)have achieved significant breakthroughs in Natural Language Processing(NLP),driven by the pre-training and fine-tuning paradigm.While this approach allows models to specialize in speci...Large-scale Language Models(LLMs)have achieved significant breakthroughs in Natural Language Processing(NLP),driven by the pre-training and fine-tuning paradigm.While this approach allows models to specialize in specific tasks with reduced training costs,the substantial memory requirements during fine-tuning present a barrier to broader deployment.Parameter-Efficient Fine-Tuning(PEFT)techniques,such as Low-Rank Adaptation(LoRA),and parameter quantization methods have emerged as solutions to address these challenges by optimizing memory usage and computational efficiency.Among these,QLoRA,which combines PEFT and quantization,has demonstrated notable success in reducing memory footprints during fine-tuning,prompting the development of various QLoRA variants.Despite these advancements,the quantitative impact of key variables on the fine-tuning performance of quantized LLMs remains underexplored.This study presents a comprehensive analysis of these key variables,focusing on their influence across different layer types and depths within LLM architectures.Our investigation uncovers several critical findings:(1)Larger layers,such as MLP layers,can maintain performance despite reductions in adapter rank,while smaller layers,like self-attention layers,aremore sensitive to such changes;(2)The effectiveness of balancing factors depends more on specific values rather than layer type or depth;(3)In quantization-aware fine-tuning,larger layers can effectively utilize smaller adapters,whereas smaller layers struggle to do so.These insights suggest that layer type is a more significant determinant of fine-tuning success than layer depth when optimizing quantized LLMs.Moreover,for the same discount of trainable parameters,reducing the trainable parameters in a larger layer is more effective in preserving fine-tuning accuracy than in a smaller one.This study provides valuable guidance for more efficient fine-tuning strategies and opens avenues for further research into optimizing LLM fine-tuning in resource-constrained environments.展开更多
This paper proposes to develop a data-driven via＇s depth estimator of the deep reactive ion etching process based on statistical identification of key variables.Several feature extraction algorithms are presented to ...This paper proposes to develop a data-driven via＇s depth estimator of the deep reactive ion etching process based on statistical identification of key variables.Several feature extraction algorithms are presented to reduce the high-dimensional data and effectively undertake the subsequent virtual metrology（VM） model building process.With the available on-line VM model,the model-based controller is hence readily applicable to improve the quality of a via＇s depth.Real operational data taken from a industrial manufacturing process are used to verify the effectiveness of the proposed method.The results demonstrate that the proposed method can decrease the MSE from 2.2×10^（-2） to 9×10^（-4） and has great potential in improving the existing DRIE process.展开更多
Atmospheric effects have significant influence on the performance of a free-space optical continuous variable quantum key distribution(CVQKD)system.In this paper,we investigate how the transmittance,excess noise and i...Atmospheric effects have significant influence on the performance of a free-space optical continuous variable quantum key distribution(CVQKD)system.In this paper,we investigate how the transmittance,excess noise and interruption probability caused by atmospheric effects affect the secret-key rate(SKR)of the CVQKD.Three signal wavelengths,two weather conditions,two detection schemes,and two types of attacks are considered in our investigation.An expression aims at calculating the interruption probability is proposed based on the Kolmogorov spectrum model.The results show that a signal using long working wavelength can propagate much further than that of using short wavelength.Moreover,as the wavelength increases,the influence of interruption probability on the SKR becomes more significant,especially within a certain transmission distance.Therefore,interruption probability must be considered for CVQKD by using long-signal wavelengths.Furthermore,different detection schemes used by the receiver will result in different transmission distances when subjected to individual attacks and collective attacks,respectively.展开更多
In order to address the issues of complex system structure and variable selection difficulty for the current heavy haul railway line status evaluation system, a three-category and three-layer heavy-haul line status ev...In order to address the issues of complex system structure and variable selection difficulty for the current heavy haul railway line status evaluation system, a three-category and three-layer heavy-haul line status evaluation variable set construction and reduction optimization method is proposed. Firstly, the status of heavy haul railway line is analyzed, and an initial set of evaluation variables affecting the line status is constructed. Then, based on the association rule and the principal component analysis method, key variables are extracted from the initial variable set to establish the evaluation system. Finally, this method is verified with actual data of a line. The results show that the service performance of heavy haul railway line can still be evaluated accurately when the evaluation variables are reduced by 60% in the proposed method.展开更多
Continuous-variable quantum key distribution(CV QKD)using optical coherent detectors is practically favorable due to its low implementation cost,flexibility of wavelength division multiplexing,and compatibility with s...Continuous-variable quantum key distribution(CV QKD)using optical coherent detectors is practically favorable due to its low implementation cost,flexibility of wavelength division multiplexing,and compatibility with standard coherent communication technologies.However,the security analysis and parameter estimation of CV QKD are complicated due to the infinite-dimensional latent Hilbert space.Also,the transmission of strong reference pulses undermines the security and complicates the experiments.In this work,we tackle these two problems by presenting a time-bin-encoding CV protocol with a simple phase-error-based security analysis valid under general coherent attacks.With the key encoded into the relative intensity between two optical modes,the need for global references is removed.Furthermore,phase randomization can be introduced to decouple the security analysis of different photon-number components.We can hence tag the photon number for each round,effectively estimate the associated privacy using a carefully designed coherent-detection method,and independently extract encryption keys from each component.Simulations manifest that the protocol using multi-photon components increases the key rate by two orders of magnitude compared to the one using only the single-photon component.Meanwhile,the protocol with four-intensity decoy analysis is sufficient to yield tight parameter estimation with a short-distance key-rate performance comparable to the best Bennett-Brassard-1984 implementation.展开更多
This study describes a classification methodology based on support vector machines(SVMs),which offer superior classification performance for fault diagnosis in chemical process engineering.The method incorporates an e...This study describes a classification methodology based on support vector machines(SVMs),which offer superior classification performance for fault diagnosis in chemical process engineering.The method incorporates an efficient parameter tuning procedure(based on minimization of radius/margin bound for SVM's leave-one-out errors)into a multi-class classification strategy using a fuzzy decision factor,which is named fuzzy support vector machine(FSVM).The datasets generated from the Tennessee Eastman process(TEP)simulator were used to evaluate the clas-sification performance.To decrease the negative influence of the auto-correlated and irrelevant variables,a key vari-able identification procedure using recursive feature elimination,based on the SVM is implemented,with time lags incorporated,before every classifier is trained,and the number of relatively important variables to every classifier is basically determined by 10-fold cross-validation.Performance comparisons are implemented among several kinds of multi-class decision machines,by which the effectiveness of the proposed approach is proved.展开更多
基金supported by the National Key R&D Program of China(No.2021YFB0301200)National Natural Science Foundation of China(No.62025208).
文摘Large-scale Language Models(LLMs)have achieved significant breakthroughs in Natural Language Processing(NLP),driven by the pre-training and fine-tuning paradigm.While this approach allows models to specialize in specific tasks with reduced training costs,the substantial memory requirements during fine-tuning present a barrier to broader deployment.Parameter-Efficient Fine-Tuning(PEFT)techniques,such as Low-Rank Adaptation(LoRA),and parameter quantization methods have emerged as solutions to address these challenges by optimizing memory usage and computational efficiency.Among these,QLoRA,which combines PEFT and quantization,has demonstrated notable success in reducing memory footprints during fine-tuning,prompting the development of various QLoRA variants.Despite these advancements,the quantitative impact of key variables on the fine-tuning performance of quantized LLMs remains underexplored.This study presents a comprehensive analysis of these key variables,focusing on their influence across different layer types and depths within LLM architectures.Our investigation uncovers several critical findings:(1)Larger layers,such as MLP layers,can maintain performance despite reductions in adapter rank,while smaller layers,like self-attention layers,aremore sensitive to such changes;(2)The effectiveness of balancing factors depends more on specific values rather than layer type or depth;(3)In quantization-aware fine-tuning,larger layers can effectively utilize smaller adapters,whereas smaller layers struggle to do so.These insights suggest that layer type is a more significant determinant of fine-tuning success than layer depth when optimizing quantized LLMs.Moreover,for the same discount of trainable parameters,reducing the trainable parameters in a larger layer is more effective in preserving fine-tuning accuracy than in a smaller one.This study provides valuable guidance for more efficient fine-tuning strategies and opens avenues for further research into optimizing LLM fine-tuning in resource-constrained environments.
基金supported by the National Natural Science Foundation of China(No.60904053)the Natural Science Foundation of Jiangsu(No. SBK201123307)the Priority Academic Program Development of Jiangsu Higher Education Institutions(PAPD)
文摘This paper proposes to develop a data-driven via＇s depth estimator of the deep reactive ion etching process based on statistical identification of key variables.Several feature extraction algorithms are presented to reduce the high-dimensional data and effectively undertake the subsequent virtual metrology（VM） model building process.With the available on-line VM model,the model-based controller is hence readily applicable to improve the quality of a via＇s depth.Real operational data taken from a industrial manufacturing process are used to verify the effectiveness of the proposed method.The results demonstrate that the proposed method can decrease the MSE from 2.2×10^（-2） to 9×10^（-4） and has great potential in improving the existing DRIE process.
基金Project supported by the National Natural Science Foundation of China(Grant No.62071180)Fundamental Research Funds for the Central Universities,China(Grant No.2020MS099)。
文摘Atmospheric effects have significant influence on the performance of a free-space optical continuous variable quantum key distribution(CVQKD)system.In this paper,we investigate how the transmittance,excess noise and interruption probability caused by atmospheric effects affect the secret-key rate(SKR)of the CVQKD.Three signal wavelengths,two weather conditions,two detection schemes,and two types of attacks are considered in our investigation.An expression aims at calculating the interruption probability is proposed based on the Kolmogorov spectrum model.The results show that a signal using long working wavelength can propagate much further than that of using short wavelength.Moreover,as the wavelength increases,the influence of interruption probability on the SKR becomes more significant,especially within a certain transmission distance.Therefore,interruption probability must be considered for CVQKD by using long-signal wavelengths.Furthermore,different detection schemes used by the receiver will result in different transmission distances when subjected to individual attacks and collective attacks,respectively.
文摘In order to address the issues of complex system structure and variable selection difficulty for the current heavy haul railway line status evaluation system, a three-category and three-layer heavy-haul line status evaluation variable set construction and reduction optimization method is proposed. Firstly, the status of heavy haul railway line is analyzed, and an initial set of evaluation variables affecting the line status is constructed. Then, based on the association rule and the principal component analysis method, key variables are extracted from the initial variable set to establish the evaluation system. Finally, this method is verified with actual data of a line. The results show that the service performance of heavy haul railway line can still be evaluated accurately when the evaluation variables are reduced by 60% in the proposed method.
基金Engineering and Physical Sciences Research Council(project EP/T001011/1)Shenzhen-Hong Kong Cooperation Zone for Technology and Innovation(HZQB-KCZYB-2020050)+7 种基金Hong Kong Research Grant Council(R7035-21)Army Research Office(W911NF-23-1-0077)Multidisciplinary University Research Initiative(W911NF-21-1-0325)Air Force Office of Scientific Research(FA9550-19-1-0399,FA9550-21-1-0209)National Science Foundation(OMA-1936118,ERC-1941583,OMA-2137642)NTT ResearchDavid and Lucile Packard Foundation(2020-71479)Marshall and Arlene Bennett Family Research Program。
文摘Continuous-variable quantum key distribution(CV QKD)using optical coherent detectors is practically favorable due to its low implementation cost,flexibility of wavelength division multiplexing,and compatibility with standard coherent communication technologies.However,the security analysis and parameter estimation of CV QKD are complicated due to the infinite-dimensional latent Hilbert space.Also,the transmission of strong reference pulses undermines the security and complicates the experiments.In this work,we tackle these two problems by presenting a time-bin-encoding CV protocol with a simple phase-error-based security analysis valid under general coherent attacks.With the key encoded into the relative intensity between two optical modes,the need for global references is removed.Furthermore,phase randomization can be introduced to decouple the security analysis of different photon-number components.We can hence tag the photon number for each round,effectively estimate the associated privacy using a carefully designed coherent-detection method,and independently extract encryption keys from each component.Simulations manifest that the protocol using multi-photon components increases the key rate by two orders of magnitude compared to the one using only the single-photon component.Meanwhile,the protocol with four-intensity decoy analysis is sufficient to yield tight parameter estimation with a short-distance key-rate performance comparable to the best Bennett-Brassard-1984 implementation.
文摘In this paper, we obtain an explicit formula of general solution for a class of the homogeneous recurrence of variable coefficients with two indices.
基金Supported by the Special Funds for Major State Basic Research Program of China (973 Program,No.2002CB312200)the Na-tional Natural Science Foundation of China (No.60574019,No.60474045)+1 种基金the Key Technologies R&D Program of Zhejiang Province (No.2005C21087)the Academician Foundation of Zhejiang Province (No.2005A1001-13).
文摘This study describes a classification methodology based on support vector machines(SVMs),which offer superior classification performance for fault diagnosis in chemical process engineering.The method incorporates an efficient parameter tuning procedure(based on minimization of radius/margin bound for SVM's leave-one-out errors)into a multi-class classification strategy using a fuzzy decision factor,which is named fuzzy support vector machine(FSVM).The datasets generated from the Tennessee Eastman process(TEP)simulator were used to evaluate the clas-sification performance.To decrease the negative influence of the auto-correlated and irrelevant variables,a key vari-able identification procedure using recursive feature elimination,based on the SVM is implemented,with time lags incorporated,before every classifier is trained,and the number of relatively important variables to every classifier is basically determined by 10-fold cross-validation.Performance comparisons are implemented among several kinds of multi-class decision machines,by which the effectiveness of the proposed approach is proved.
