Online kernel selection is a fundamental problem of online kernel methods.In this paper,we study online kernel selection with memory constraint in which the memory of kernel selection and online prediction procedures ...Online kernel selection is a fundamental problem of online kernel methods.In this paper,we study online kernel selection with memory constraint in which the memory of kernel selection and online prediction procedures is limited to a fixed budget.An essential question is what is the intrinsic relationship among online learnability,memory constraint,and data complexity.To answer the question,it is necessary to show the trade-offs between regret and memory budget.Previous work gives a worst-case lower bound depending on the data size,and shows learning is impossible within a small memory budget.In contrast,we present distinct results by offering data-dependent upper bounds that rely on two data complexities:kernel alignment and the cumulative losses of competitive hypothesis.We propose an algorithmic framework giving data-dependent upper bounds for two types of loss functions.For the hinge loss function,our algorithm achieves an expected upper bound depending on kernel alignment.For the smooth loss functions,our algorithm achieves a high-probability upper bound depending on the cumulative losses of competitive hypothesis.We also prove a matching lower bound for smooth loss functions.Our results show that if the two data complexities are sub-linear,then learning is possible within a small memory budget.Our algorithmic framework depends on a new buffer maintaining framework and a reduction from online kernel selection to prediction with expert advice.Finally,we empirically verify the prediction performance of our algorithms on benchmark datasets.展开更多
This paper studies variable selection using the penalized likelihood method for dis-tributed sparse regression with large sample size n under a limited memory constraint.This is a much needed research problem to be so...This paper studies variable selection using the penalized likelihood method for dis-tributed sparse regression with large sample size n under a limited memory constraint.This is a much needed research problem to be solved in the big data era.A naive divide-and-conquer method solving this problem is to split the whole data into N parts and run each part on one of N machines,aggregate the results from all machines via averaging,andfinally obtain the selected variables.However,it tends to select more noise variables,and the false discovery rate may not be well controlled.We improve it by a special designed weighted average in aggregation.Although the alternating direction method of multiplier can be used to deal with massive data in the literature,our proposed method reduces the computational burden a lot and performs better by mean square error in most cases.Theoretically,we establish asymptotic properties of the resulting estimators for the likelihood models with a diverging number of parame-ters.Under some regularity conditions,we establish oracle properties in the sense that our distributed estimator shares the same asymptotic efficiency as the estimator based on the full sample.Computationally,a distributed penalized likelihood algorithm is proposed to refine the results in the context of general likelihoods.Furthermore,the proposed method is evaluated by simulations and a real example.展开更多
The rapid advancement of Large Language Models(LLMs)has revolutionized artificial intelligence,yet their deployment remains hindered by computational inefficiency,memory constraints,and latency challenges.This vip e...The rapid advancement of Large Language Models(LLMs)has revolutionized artificial intelligence,yet their deployment remains hindered by computational inefficiency,memory constraints,and latency challenges.This vip editorial highlights three pioneering studies that address these issues through innovative co-optimization of algorithms and hardware,offering scalable solutions for resource-constrained environments.“Enhancing LLM Inference Performance on ARM CPUs through Software and Hardware Co-optimization Strategies”coauthored by Zhang et al.展开更多
基金supported by the National Natural Science Foundation of China under Grant No.62076181.
文摘Online kernel selection is a fundamental problem of online kernel methods.In this paper,we study online kernel selection with memory constraint in which the memory of kernel selection and online prediction procedures is limited to a fixed budget.An essential question is what is the intrinsic relationship among online learnability,memory constraint,and data complexity.To answer the question,it is necessary to show the trade-offs between regret and memory budget.Previous work gives a worst-case lower bound depending on the data size,and shows learning is impossible within a small memory budget.In contrast,we present distinct results by offering data-dependent upper bounds that rely on two data complexities:kernel alignment and the cumulative losses of competitive hypothesis.We propose an algorithmic framework giving data-dependent upper bounds for two types of loss functions.For the hinge loss function,our algorithm achieves an expected upper bound depending on kernel alignment.For the smooth loss functions,our algorithm achieves a high-probability upper bound depending on the cumulative losses of competitive hypothesis.We also prove a matching lower bound for smooth loss functions.Our results show that if the two data complexities are sub-linear,then learning is possible within a small memory budget.Our algorithmic framework depends on a new buffer maintaining framework and a reduction from online kernel selection to prediction with expert advice.Finally,we empirically verify the prediction performance of our algorithms on benchmark datasets.
基金supported by NSFC(11871263)NSF grant of Guangdong Province of China(No.2017A030313012).
文摘This paper studies variable selection using the penalized likelihood method for dis-tributed sparse regression with large sample size n under a limited memory constraint.This is a much needed research problem to be solved in the big data era.A naive divide-and-conquer method solving this problem is to split the whole data into N parts and run each part on one of N machines,aggregate the results from all machines via averaging,andfinally obtain the selected variables.However,it tends to select more noise variables,and the false discovery rate may not be well controlled.We improve it by a special designed weighted average in aggregation.Although the alternating direction method of multiplier can be used to deal with massive data in the literature,our proposed method reduces the computational burden a lot and performs better by mean square error in most cases.Theoretically,we establish asymptotic properties of the resulting estimators for the likelihood models with a diverging number of parame-ters.Under some regularity conditions,we establish oracle properties in the sense that our distributed estimator shares the same asymptotic efficiency as the estimator based on the full sample.Computationally,a distributed penalized likelihood algorithm is proposed to refine the results in the context of general likelihoods.Furthermore,the proposed method is evaluated by simulations and a real example.
文摘The rapid advancement of Large Language Models(LLMs)has revolutionized artificial intelligence,yet their deployment remains hindered by computational inefficiency,memory constraints,and latency challenges.This vip editorial highlights three pioneering studies that address these issues through innovative co-optimization of algorithms and hardware,offering scalable solutions for resource-constrained environments.“Enhancing LLM Inference Performance on ARM CPUs through Software and Hardware Co-optimization Strategies”coauthored by Zhang et al.
