Probability and Statistics with Reliability, Queuing, and Computer Science Applications, 2nd Edition

Chapter 7Discrete-Time Markov Chains

7.1 Introduction

A Markov process is a stochastic process whose dynamic behavior is such that probability distributions for its future development depend only on the present state and not on how the process arrived in that state. If we assume that the state space, I, is discrete (finite or countably infinite), then the Markov process is known as a Markov chain. If we further assume that the parameter space, T, is also discrete, then we have a discrete-time Markov chain (DTMC). Such processes are the subject of this chapter. Since the parameter space is discrete, we will let without loss of generality.

We choose to observe the state of a system at a discrete set of time points. The successive observations define the random variables X₀, X₁, X₂, , X_n, , at time steps , respectively. If , then the state of the system at time step n is j. X₀ is the initial state of the system. The Markov property can then be succinctly stated as

7.1

Intuitively, equation ...