In this paper,we study the geometric ergodicity of continuous time Markov pro-cesses in general state space.For the geometric ergodic continuous time Markov processes,the condition π(f^(p))<∞,p>1 is added.Usin...In this paper,we study the geometric ergodicity of continuous time Markov pro-cesses in general state space.For the geometric ergodic continuous time Markov processes,the condition π(f^(p))<∞,p>1 is added.Using the coupling method,we obtain the existence of a full absorbing set on which continuous time Markov processes are f-geometric ergodic.展开更多
Let X=(X) be a two-parameter *-Markov process with a transition function (p1, p2, p), where X, takes values in the state space (Er,), T=[0,)2. For each r T, let f, be a measurable transformation of (E,) into the state...Let X=(X) be a two-parameter *-Markov process with a transition function (p1, p2, p), where X, takes values in the state space (Er,), T=[0,)2. For each r T, let f, be a measurable transformation of (E,) into the state space (E’r, ). Set Y,=f,(X,), r T. A sufficient condition is given for the process Y=(Yr) still to be a two-parameter *-Markov process with a transition function in terms of transition function (p1, p2, p) and fr. For *-Markov families of two-parameter processes with a transition function, a similar problem is also discussed.展开更多
I. INTRODUCTION AND DEFINITIONS In this report, we shall give a simple counterexample to negative Theorem 1 and Proposition 3 (c)(ii)in [1] and explain the difference between the large-past Markov property and *-Marko...I. INTRODUCTION AND DEFINITIONS In this report, we shall give a simple counterexample to negative Theorem 1 and Proposition 3 (c)(ii)in [1] and explain the difference between the large-past Markov property and *-Markov property. Thereby some mistakes are cleared up.展开更多
We establish the Hausdorff dimension of the graph of general Markov processes on Rd based on some probability estimates of the processes staying or leaving small balls in small time.In particular,our results indicate ...We establish the Hausdorff dimension of the graph of general Markov processes on Rd based on some probability estimates of the processes staying or leaving small balls in small time.In particular,our results indicate that,for symmetric diffusion processes(withα=2)or symmetricα-stable-like processes(withα∈(0,2))on Rd,it holds almost surely that dimH GrX([0,1])=1{α<1}+(2−1/α)1{α≥1,d=1}+(d∧α)1{α≥1,d≥2}.We also systematically prove the corresponding results about the Hausdorff dimension of the range of the processes.展开更多
We investigate the approximating capability of Markov modulated Poisson processes (MMPP) for modeling multifractal Internet traffic. The choice of MMPP is motivated by its ability to capture the variability and correl...We investigate the approximating capability of Markov modulated Poisson processes (MMPP) for modeling multifractal Internet traffic. The choice of MMPP is motivated by its ability to capture the variability and correlation in moderate time scales while being analytically tractable. Important statistics of traffic burstiness are described and a customized moment-based fitting procedure of MMPP to traffic traces is presented. Our methodology of doing this is to examine whether the MMPP can be used to predict the performance of a queue to which MMPP sample paths and measured traffic traces are fed for comparison respectively, in addition to the goodness-of-fit test of MMPP. Numerical results and simulations show that the fitted MMPP can approximate multifractal traffic quite well, i.e. accurately predict the queueing performance.展开更多
A real-time pricing system of electricity is a system that charges different electricity prices for different hours of the day and for different days, and is effective for reducing the peak and flattening the load cur...A real-time pricing system of electricity is a system that charges different electricity prices for different hours of the day and for different days, and is effective for reducing the peak and flattening the load curve. In this paper, using a Markov decision process (MDP), we propose a modeling method and an optimal control method for real-time pricing systems. First, the outline of real-time pricing systems is explained. Next, a model of a set of customers is derived as a multi-agent MDP. Furthermore, the optimal control problem is formulated, and is reduced to a quadratic programming problem. Finally, a numerical simulation is presented.展开更多
This paper presents a small perturbation Cramer method for obtaining the large deviation principle of a family of measures (β,ε> 0) on a topological vector space. As an application, we obtain the moderate deviati...This paper presents a small perturbation Cramer method for obtaining the large deviation principle of a family of measures (β,ε> 0) on a topological vector space. As an application, we obtain the moderate deviation estimations for uniformly ergodic Markov processes.展开更多
Let Q be the Q-matrix of an irreducible, positive recurrent Markov process on a countable state space. We show that, under a number of conditions, the stationary distributions of the n × n north-west corner augme...Let Q be the Q-matrix of an irreducible, positive recurrent Markov process on a countable state space. We show that, under a number of conditions, the stationary distributions of the n × n north-west corner augmentations of Q converge in total variation to the stationary distribution of the process. Two conditions guaranteeing such convergence include exponential ergodicity and stochastic monotonicity of the process. The same also holds for processes dominated by a stochastically monotone Markov process. In addition, we shall show that finite perturbations of stochastically monotone processes may be viewed as being dominated by a stochastically monotone process, thus extending the scope of these results to a larger class of processes. Consequently, the augmentation method provides an attractive, intuitive method for approximating the stationary distributions of a large class of Markov processes on countably infinite state spaces from a finite amount of known information.展开更多
In this paper, we consider a general form of the increments for a two-parameter Wiener process. Both the Csorgo-Revesz's increments and a class of the lag increments are the special cases of this general form of i...In this paper, we consider a general form of the increments for a two-parameter Wiener process. Both the Csorgo-Revesz's increments and a class of the lag increments are the special cases of this general form of increments. Our results imply the theorem that have been given by Csorgo and Revesz (1978), and some of their conditions are removed.展开更多
Markov decision processes （MDPs） and their variants are widely studied in the theory of controls for stochastic discrete- event systems driven by Markov chains. Much of the literature focusses on the risk-neutral cr...Markov decision processes （MDPs） and their variants are widely studied in the theory of controls for stochastic discrete- event systems driven by Markov chains. Much of the literature focusses on the risk-neutral criterion in which the expected rewards, either average or discounted, are maximized. There exists some literature on MDPs that takes risks into account. Much of this addresses the exponential utility （EU） function and mechanisms to penalize different forms of variance of the rewards. EU functions have some numerical deficiencies, while variance measures variability both above and below the mean rewards; the variability above mean rewards is usually beneficial and should not be penalized/avoided. As such, risk metrics that account for pre-specified targets （thresholds） for rewards have been considered in the literature, where the goal is to penalize the risks of revenues falling below those targets. Existing work on MDPs that takes targets into account seeks to minimize risks of this nature. Minimizing risks can lead to poor solutions where the risk is zero or near zero, but the average rewards are also rather low. In this paper, hence, we study a risk-averse criterion, in particular the so-called downside risk, which equals the probability of the revenues falling below a given target, where, in contrast to minimizing such risks, we only reduce this risk at the cost of slightly lowered average rewards. A solution where the risk is low and the average reward is quite high, although not at its maximum attainable value, is very attractive in practice. To be more specific, in our formulation, the objective function is the expected value of the rewards minus a scalar times the downside risk. In this setting, we analyze the infinite horizon MDP, the finite horizon MDP, and the infinite horizon semi-MDP （SMDP）. We develop dynamic programming and reinforcement learning algorithms for the finite and infinite horizon. The algorithms are tested in numerical studies and show encouraging performance.展开更多
The anthem investigate the hitting probability, polarity and the relationship between the polarity and Hausdorff dimension for self-similar Markov processes with state space (0, infinity) and increasing path.
Optimal policies in Markov decision problems may be quite sensitive with regard to transition probabilities.In practice,some transition probabilities may be uncertain.The goals of the present study are to find the rob...Optimal policies in Markov decision problems may be quite sensitive with regard to transition probabilities.In practice,some transition probabilities may be uncertain.The goals of the present study are to find the robust range for a certain optimal policy and to obtain value intervals of exact transition probabilities.Our research yields powerful contributions for Markov decision processes(MDPs)with uncertain transition probabilities.We first propose a method for estimating unknown transition probabilities based on maximum likelihood.Since the estimation may be far from accurate,and the highest expected total reward of the MDP may be sensitive to these transition probabilities,we analyze the robustness of an optimal policy and propose an approach for robust analysis.After giving the definition of a robust optimal policy with uncertain transition probabilities represented as sets of numbers,we formulate a model to obtain the optimal policy.Finally,we define the value intervals of the exact transition probabilities and construct models to determine the lower and upper bounds.Numerical examples are given to show the practicability of our methods.展开更多
This paper studies the limit average variance criterion for continuous-time Markov decision processes in Polish spaces. Based on two approaches, this paper proves not only the existence of solutions to the variance mi...This paper studies the limit average variance criterion for continuous-time Markov decision processes in Polish spaces. Based on two approaches, this paper proves not only the existence of solutions to the variance minimization optimality equation and the existence of a variance minimal policy that is canonical, but also the existence of solutions to the two variance minimization optimality inequalities and the existence of a variance minimal policy which may not be canonical. An example is given to illustrate all of our conditions.展开更多
The simulation of wind power time series is a key process in renewable power allocation planning,operation mode calculation,and safety assessment.Traditional single-point modeling methods discretely generate wind powe...The simulation of wind power time series is a key process in renewable power allocation planning,operation mode calculation,and safety assessment.Traditional single-point modeling methods discretely generate wind power at each moment;however,they ignore the daily output characteristics and are unable to consider both modeling accuracy and efficiency.To resolve this problem,a wind power time series simulation model based on typical daily output processes and Markov algorithm is proposed.First,a typical daily output process classification method based on time series similarity and modified K-means clustering algorithm is presented.Second,considering the typical daily output processes as status variables,a wind power time series simulation model based on Markov algorithm is constructed.Finally,a case is analyzed based on the measured data of a wind farm in China.The proposed model is then compared with traditional methods to verify its effectiveness and applicability.The comparison results indicate that the statistical characteristics,probability distributions,and autocorrelation characteristics of the wind power time series generated by the proposed model are better than those of the traditional methods.Moreover,modeling efficiency considerably improves.展开更多
Let X=(Ω,■,■,X_(t),θ_(t),P^(x))be a jump Markov process with q-pair q(x)-q(x,A).In this paper,the equilibrium principle is established and equilibrium functions,energy,capacity and related problems is investigated...Let X=(Ω,■,■,X_(t),θ_(t),P^(x))be a jump Markov process with q-pair q(x)-q(x,A).In this paper,the equilibrium principle is established and equilibrium functions,energy,capacity and related problems is investigated in terms of the q-pair q(x)-q(x,A).展开更多
Markov decision process(MDP)offers a general framework for modelling sequential decision making where outcomes are random.In particular,it serves as a mathematical framework for reinforcement learning.This paper intro...Markov decision process(MDP)offers a general framework for modelling sequential decision making where outcomes are random.In particular,it serves as a mathematical framework for reinforcement learning.This paper introduces an extension of MDP,namely quantum MDP(q MDP),that can serve as a mathematical model of decision making about quantum systems.We develop dynamic programming algorithms for policy evaluation and finding optimal policies for q MDPs in the case of finite-horizon.The results obtained in this paper provide some useful mathematical tools for reinforcement learning techniques applied to the quantum world.展开更多
This article explores controllable Borel spaces, stationary, homogeneous Markov processes, discrete time with infinite horizon, with bounded cost functions and using the expected total discounted cost criterion. The p...This article explores controllable Borel spaces, stationary, homogeneous Markov processes, discrete time with infinite horizon, with bounded cost functions and using the expected total discounted cost criterion. The problem of the estimation of stability for this type of process is set. The central objective is to obtain a bounded stability index expressed in terms of the Lévy-Prokhorov metric;likewise, sufficient conditions are provided for the existence of such inequalities.展开更多
We prove that non-recursive base conversion can always be implemented by using a deterministic Markov process. Our paper discusses the pros and cons of recursive and non-recursive methods, in general. And we include a...We prove that non-recursive base conversion can always be implemented by using a deterministic Markov process. Our paper discusses the pros and cons of recursive and non-recursive methods, in general. And we include a comparison between non-recursion and a deterministic Markov process, proving that the Markov process is twice as efficient.展开更多
基金Supported by the Natural Science Foundation of Hubei Province(2021CFB275)National Natural Science Foundation of China(12301667).
文摘In this paper,we study the geometric ergodicity of continuous time Markov pro-cesses in general state space.For the geometric ergodic continuous time Markov processes,the condition π(f^(p))<∞,p>1 is added.Using the coupling method,we obtain the existence of a full absorbing set on which continuous time Markov processes are f-geometric ergodic.
基金Project supported by the National Natural Science Foundation of Chinathe Guangdong Provincial Natural Science Foundation of China the Foundation of Zhongshan University Advanced Research Center.
文摘Let X=(X) be a two-parameter *-Markov process with a transition function (p1, p2, p), where X, takes values in the state space (Er,), T=[0,)2. For each r T, let f, be a measurable transformation of (E,) into the state space (E’r, ). Set Y,=f,(X,), r T. A sufficient condition is given for the process Y=(Yr) still to be a two-parameter *-Markov process with a transition function in terms of transition function (p1, p2, p) and fr. For *-Markov families of two-parameter processes with a transition function, a similar problem is also discussed.
文摘I. INTRODUCTION AND DEFINITIONS In this report, we shall give a simple counterexample to negative Theorem 1 and Proposition 3 (c)(ii)in [1] and explain the difference between the large-past Markov property and *-Markov property. Thereby some mistakes are cleared up.
基金supported by Leshan Normal University Scientific Research Start-up Project for Introducing High-level Talents(Grand No.RC2024001).
文摘We establish the Hausdorff dimension of the graph of general Markov processes on Rd based on some probability estimates of the processes staying or leaving small balls in small time.In particular,our results indicate that,for symmetric diffusion processes(withα=2)or symmetricα-stable-like processes(withα∈(0,2))on Rd,it holds almost surely that dimH GrX([0,1])=1{α<1}+(2−1/α)1{α≥1,d=1}+(d∧α)1{α≥1,d≥2}.We also systematically prove the corresponding results about the Hausdorff dimension of the range of the processes.
文摘We investigate the approximating capability of Markov modulated Poisson processes (MMPP) for modeling multifractal Internet traffic. The choice of MMPP is motivated by its ability to capture the variability and correlation in moderate time scales while being analytically tractable. Important statistics of traffic burstiness are described and a customized moment-based fitting procedure of MMPP to traffic traces is presented. Our methodology of doing this is to examine whether the MMPP can be used to predict the performance of a queue to which MMPP sample paths and measured traffic traces are fed for comparison respectively, in addition to the goodness-of-fit test of MMPP. Numerical results and simulations show that the fitted MMPP can approximate multifractal traffic quite well, i.e. accurately predict the queueing performance.
文摘A real-time pricing system of electricity is a system that charges different electricity prices for different hours of the day and for different days, and is effective for reducing the peak and flattening the load curve. In this paper, using a Markov decision process (MDP), we propose a modeling method and an optimal control method for real-time pricing systems. First, the outline of real-time pricing systems is explained. Next, a model of a set of customers is derived as a multi-agent MDP. Furthermore, the optimal control problem is formulated, and is reduced to a quadratic programming problem. Finally, a numerical simulation is presented.
文摘This paper presents a small perturbation Cramer method for obtaining the large deviation principle of a family of measures (β,ε> 0) on a topological vector space. As an application, we obtain the moderate deviation estimations for uniformly ergodic Markov processes.
文摘Let Q be the Q-matrix of an irreducible, positive recurrent Markov process on a countable state space. We show that, under a number of conditions, the stationary distributions of the n × n north-west corner augmentations of Q converge in total variation to the stationary distribution of the process. Two conditions guaranteeing such convergence include exponential ergodicity and stochastic monotonicity of the process. The same also holds for processes dominated by a stochastically monotone Markov process. In addition, we shall show that finite perturbations of stochastically monotone processes may be viewed as being dominated by a stochastically monotone process, thus extending the scope of these results to a larger class of processes. Consequently, the augmentation method provides an attractive, intuitive method for approximating the stationary distributions of a large class of Markov processes on countably infinite state spaces from a finite amount of known information.
基金Supported by the National Natural Science Foundation of ChinaZhejiang Province Natural Science Fund
文摘In this paper, we consider a general form of the increments for a two-parameter Wiener process. Both the Csorgo-Revesz's increments and a class of the lag increments are the special cases of this general form of increments. Our results imply the theorem that have been given by Csorgo and Revesz (1978), and some of their conditions are removed.
文摘Markov decision processes （MDPs） and their variants are widely studied in the theory of controls for stochastic discrete- event systems driven by Markov chains. Much of the literature focusses on the risk-neutral criterion in which the expected rewards, either average or discounted, are maximized. There exists some literature on MDPs that takes risks into account. Much of this addresses the exponential utility （EU） function and mechanisms to penalize different forms of variance of the rewards. EU functions have some numerical deficiencies, while variance measures variability both above and below the mean rewards; the variability above mean rewards is usually beneficial and should not be penalized/avoided. As such, risk metrics that account for pre-specified targets （thresholds） for rewards have been considered in the literature, where the goal is to penalize the risks of revenues falling below those targets. Existing work on MDPs that takes targets into account seeks to minimize risks of this nature. Minimizing risks can lead to poor solutions where the risk is zero or near zero, but the average rewards are also rather low. In this paper, hence, we study a risk-averse criterion, in particular the so-called downside risk, which equals the probability of the revenues falling below a given target, where, in contrast to minimizing such risks, we only reduce this risk at the cost of slightly lowered average rewards. A solution where the risk is low and the average reward is quite high, although not at its maximum attainable value, is very attractive in practice. To be more specific, in our formulation, the objective function is the expected value of the rewards minus a scalar times the downside risk. In this setting, we analyze the infinite horizon MDP, the finite horizon MDP, and the infinite horizon semi-MDP （SMDP）. We develop dynamic programming and reinforcement learning algorithms for the finite and infinite horizon. The algorithms are tested in numerical studies and show encouraging performance.
基金the National Natural Science Foundation of China and the StateEducation of Commission Ph.D. Station Foundation
文摘The anthem investigate the hitting probability, polarity and the relationship between the polarity and Hausdorff dimension for self-similar Markov processes with state space (0, infinity) and increasing path.
基金Supported by the National Natural Science Foundation of China(71571019).
文摘Optimal policies in Markov decision problems may be quite sensitive with regard to transition probabilities.In practice,some transition probabilities may be uncertain.The goals of the present study are to find the robust range for a certain optimal policy and to obtain value intervals of exact transition probabilities.Our research yields powerful contributions for Markov decision processes(MDPs)with uncertain transition probabilities.We first propose a method for estimating unknown transition probabilities based on maximum likelihood.Since the estimation may be far from accurate,and the highest expected total reward of the MDP may be sensitive to these transition probabilities,we analyze the robustness of an optimal policy and propose an approach for robust analysis.After giving the definition of a robust optimal policy with uncertain transition probabilities represented as sets of numbers,we formulate a model to obtain the optimal policy.Finally,we define the value intervals of the exact transition probabilities and construct models to determine the lower and upper bounds.Numerical examples are given to show the practicability of our methods.
基金supported by the National Natural Science Foundation of China(10801056)the Natural Science Foundation of Ningbo(2010A610094)
文摘This paper studies the limit average variance criterion for continuous-time Markov decision processes in Polish spaces. Based on two approaches, this paper proves not only the existence of solutions to the variance minimization optimality equation and the existence of a variance minimal policy that is canonical, but also the existence of solutions to the two variance minimization optimality inequalities and the existence of a variance minimal policy which may not be canonical. An example is given to illustrate all of our conditions.
基金supported by the China Datang Corporation project“Study on the performance improvement scheme of in-service wind farms”,the Fundamental Research Funds for the Central Universities(2020MS021)the Foundation of State Key Laboratory“Real-time prediction of offshore wind power and load reduction control method”(LAPS2020-07).
文摘The simulation of wind power time series is a key process in renewable power allocation planning,operation mode calculation,and safety assessment.Traditional single-point modeling methods discretely generate wind power at each moment;however,they ignore the daily output characteristics and are unable to consider both modeling accuracy and efficiency.To resolve this problem,a wind power time series simulation model based on typical daily output processes and Markov algorithm is proposed.First,a typical daily output process classification method based on time series similarity and modified K-means clustering algorithm is presented.Second,considering the typical daily output processes as status variables,a wind power time series simulation model based on Markov algorithm is constructed.Finally,a case is analyzed based on the measured data of a wind farm in China.The proposed model is then compared with traditional methods to verify its effectiveness and applicability.The comparison results indicate that the statistical characteristics,probability distributions,and autocorrelation characteristics of the wind power time series generated by the proposed model are better than those of the traditional methods.Moreover,modeling efficiency considerably improves.
文摘Let X=(Ω,■,■,X_(t),θ_(t),P^(x))be a jump Markov process with q-pair q(x)-q(x,A).In this paper,the equilibrium principle is established and equilibrium functions,energy,capacity and related problems is investigated in terms of the q-pair q(x)-q(x,A).
基金partly supported by National Key R&D Program of China(No.2018YFA0306701)the Australian Research Council(Nos.DP160101652 and DP180100691)+1 种基金National Natural Science Foundation of China(No.61832015)the Key Research Program of Frontier Sciences,Chinese Academy of Sciences。
文摘Markov decision process(MDP)offers a general framework for modelling sequential decision making where outcomes are random.In particular,it serves as a mathematical framework for reinforcement learning.This paper introduces an extension of MDP,namely quantum MDP(q MDP),that can serve as a mathematical model of decision making about quantum systems.We develop dynamic programming algorithms for policy evaluation and finding optimal policies for q MDPs in the case of finite-horizon.The results obtained in this paper provide some useful mathematical tools for reinforcement learning techniques applied to the quantum world.
文摘This article explores controllable Borel spaces, stationary, homogeneous Markov processes, discrete time with infinite horizon, with bounded cost functions and using the expected total discounted cost criterion. The problem of the estimation of stability for this type of process is set. The central objective is to obtain a bounded stability index expressed in terms of the Lévy-Prokhorov metric;likewise, sufficient conditions are provided for the existence of such inequalities.
文摘We prove that non-recursive base conversion can always be implemented by using a deterministic Markov process. Our paper discusses the pros and cons of recursive and non-recursive methods, in general. And we include a comparison between non-recursion and a deterministic Markov process, proving that the Markov process is twice as efficient.
