Markov chains and the transition matrix

Markov chains were introduced in 1906 by Andrei Andreyevich Markov (1856–1922) and were named in his honour.

An example and some interesting questions#

Example 1.1. A frog hops about on 7 lily pads. The number next to each arrow of Figure 1 is the probability with which, at the next jump, the frog moves along that arrow to a neighbouring pad. When the probabilities of the arrows leaving a pad sum to less than 1, the frog stays where it is with the remaining probability, drawn as a loop.

Figure 1. The 7 lily pads of Example 1.1 and the probabilities of the jumps between them.

There are 7 states, the lily pads. The probabilities of the jumps form the matrix

$$P = \begin{pmatrix} 0 & 1 & 0 & 0 & 0 & 0 & 0 \\[4pt] 0 & \frac12 & \frac12 & 0 & 0 & 0 & 0 \\[4pt] \frac12 & 0 & \frac12 & 0 & 0 & 0 & 0 \\[4pt] 0 & 0 & \frac14 & \frac12 & \frac14 & 0 & 0 \\[4pt] 0 & 0 & 0 & 0 & 0 & \frac12 & \frac12 \\[4pt] 0 & 0 & 0 & 1 & 0 & 0 & 0 \\[4pt] 0 & 0 & 0 & 0 & 0 & 0 & 1 \end{pmatrix}.$$

The element ${p_{57} = \frac12}$ of $P$ is the probability that, when starting in state 5, the next jump takes the frog to state 7. We would like to know where the frog goes, how long it takes to get there, and what happens in the long run. Write $p^{(n)}_{11}$ for the probability that the frog, starting in state 1, is in state 1 after $n$ jumps. Specifically:
(a) Starting in state 1, what is the probability that the frog is still in state 1 after 3 jumps, after 5 jumps, after 1000 jumps? The answers are ${p^{(3)}_{11} = \frac14,}$ ${p^{(5)}_{11} = \frac{3}{16},}$ and about $\frac15$ after 1000 jumps, because ${\lim_{n \to \infty} p^{(n)}_{11} = \frac15.}$
(b) Starting in state 4, what is the probability that the frog ever reaches state 7? The answer is $\frac13.$
(c) Starting in state 4, how long on average does it take to reach either state 3 or state 7? The answer is $\frac{11}{3}.$
(d) Starting in state 2, what is the long-run proportion of time spent in state 3? The answer is $\frac25.$

Markov chain models and methods are useful in answering questions such as the following. How long does it take to shuffle a deck of cards? How likely is a queue to overflow its buffer? How long does it take for a knight making random moves on a chessboard to return to its initial square? The answer is 168 if it starts in a corner and 42 if it starts near the centre. What do the hyperlinks between web pages say about their relative popularity?

Definitions#

Probability spaces and random variables#

Definition (Probability space). A probability space is a triple $(\Omega, \mathscr{F}, P).$ The sample space $\Omega$ is the set of all possible outcomes of a random process. The event space $\mathscr{F}$ is a set of events, where an event is a subset of $\Omega.$ The probability $P$ is a function that assigns a number between 0 and 1 to every event in $\mathscr{F}.$

Definition (Random variable). Let $E$ be a measurable space. A random variable with values in $E$ is a measurable function ${X \colon \Omega \to E.}$

Definition (Distribution). The probability distribution of a random variable $X$ describes how probability is assigned to the values of $X.$

Example (The outcome of 10 jumps). A sequence of positions of the frog is an experiment. The set of all experiments is the sample space $\Omega,$ $\mathscr{F}$ is a set of subsets of $\Omega,$ and $P$ is a probability. Take the experiment of exactly 11 positions, that is, 10 jumps,

$$\omega = (1, 2, 3, 2, 3, 4, 3, 4, 2, 1, 2).$$

The random variable $X_n$ returns the position after $n$ jumps, ${X_n(\omega) = \omega_n,}$ so ${X_0(\omega) = 1,}$ ${X_1(\omega) = 2}$ and ${X_2(\omega) = 3.}$ A random variable takes as input an element of $\Omega,$ an outcome and not an event, and returns a value.

Markov chains#

Let $I$ be a countable set, with elements denoted $i, j, k, \dots.$ Each ${i \in I}$ is called a state and $I$ is called the state space. We work in a probability space ${(\Omega, \mathscr{F}, P).}$ The object of study is a sequence of random variables ${X_0, X_1, \dots}$ with values in $I$ whose joint distribution is determined by simple rules.

A row vector ${\lambda = (\lambda_i : i \in I)}$ is called a measure if ${\lambda_i \ge 0}$ for all ${i \in I.}$ If moreover ${\sum_i \lambda_i = 1,}$ it is a distribution, or probability measure. We start with an initial distribution $\lambda$ over $I,$ which is the initial condition of the chain. The special case in which the chain starts in state $i$ with probability 1 is denoted

$$\lambda = \delta_i = (0, \dots, 0, 1, 0, \dots),$$

with the entry 1 in position $i.$ We also have a transition matrix ${P = (p_{ij} : i, j \in I).}$ It is a stochastic matrix, meaning that ${p_{ij} \ge 0}$ for all ${i, j \in I}$ and ${\sum_{j} p_{ij} = 1}$ for all ${i \in I,}$ that is, each row of $P$ is a distribution over $I.$

Definition 1.2 (Markov chain). A sequence $(X_n)_{n \ge 0}$ of random variables with values in $I$ is a Markov chain with initial distribution $\lambda$ and transition matrix $P$ if, for all ${n \ge 0}$ and ${i_0, \dots, i_{n+1} \in I,}$ the following two conditions hold.
(i) ${P(X_0 = i_0) = \lambda_{i_0}.}$
(ii) Whenever ${P(X_0 = i_0, \dots, X_n = i_n) > 0,}$

$$P(X_{n+1} = i_{n+1} \mid X_0 = i_0, \dots, X_n = i_n) = P(X_{n+1} = i_{n+1} \mid X_n = i_n) = p_{i_n i_{n+1}}.$$

For short, we say that $(X_n)_{n \ge 0}$ is $\mathrm{Markov}(\lambda, P).$ Checking conditions (i) and (ii) is usually the most helpful way to determine whether or not a given random process $(X_n)_{n \ge 0}$ is a Markov chain. It can also be helpful to have the alternative description provided by Theorem 1.3.

Theorem 1.3. A sequence $(X_n)_{n \ge 0}$ of random variables with values in $I$ is $\mathrm{Markov}(\lambda, P)$ if and only if, for all ${n \ge 0}$ and ${i_0, \dots, i_n \in I,}$

$$P(X_0 = i_0, \dots, X_n = i_n) = \lambda_{i_0} p_{i_0 i_1} \cdots p_{i_{n-1} i_n}. \tag{1.1}$$

Proof (sketch). Suppose that $(X_n)_{n \ge 0}$ is $\mathrm{Markov}(\lambda, P).$ Then, for ${n \ge 1,}$

$$\begin{aligned} &P(X_0 = i_0, \dots, X_n = i_n) \\ &\quad = P(X_n = i_n \mid X_0 = i_0, \dots, X_{n-1} = i_{n-1}) \, P(X_0 = i_0, \dots, X_{n-1} = i_{n-1}) \\ &\quad = P(X_0 = i_0) \, P(X_1 = i_1 \mid X_0 = i_0) \cdots P(X_n = i_n \mid X_0 = i_0, \dots, X_{n-1} = i_{n-1}) \\ &\quad = \lambda_{i_0} p_{i_0 i_1} \cdots p_{i_{n-1} i_n}, \end{aligned}$$

and for ${n = 0,}$ (1.1) is condition (i). On the other hand, if (1.1) holds, summing it over all ${i_1, \dots, i_n}$ gives ${P(X_0 = i_0) = \lambda_{i_0},}$ that is, (i). Then summing (1.1) over $i_n$ gives ${P(X_0 = i_0, \dots, X_{n-1} = i_{n-1}) = \lambda_{i_0} p_{i_0 i_1} \cdots p_{i_{n-2} i_{n-1}}.}$ Hence

$$P(X_n = i_n \mid X_0 = i_0, \dots, X_{n-1} = i_{n-1}) = \frac{P(X_0 = i_0, \dots, X_n = i_n)}{P(X_0 = i_0, \dots, X_{n-1} = i_{n-1})} = p_{i_{n-1} i_n},$$

which establishes (ii). $\blacksquare$

Where do Markov chains come from?#

At each time we apply some new randomness to determine the next step, in a way that is a function only of the current state. We might take ${U_1, U_2, \dots}$ as independent and identically distributed (i.i.d.) random variables with values in a set $E,$ and a function ${F \colon I \times E \to I}$ such that ${u \mapsto F(j, u)}$ is measurable for every ${j \in I.}$ Take ${i \in I.}$ Set ${X_0 = i}$ and define recursively

$$X_{n+1} = F(X_n, U_{n+1}), \qquad n \ge 0.$$

Then $(X_n)_{n \ge 0}$ is $\mathrm{Markov}(\delta_i, P),$ where ${p_{jk} = P(F(j, U) = k)}$ for all ${j, k \in I}$ and $U$ has the common distribution of the $U_n.$ Indeed, ${X_0, \dots, X_n}$ are functions of ${U_1, \dots, U_n,}$ so $U_{n+1}$ is independent of them, and on the event ${\{ X_n = j \}}$ we have ${X_{n+1} = F(j, U_{n+1}).}$ Hence both conditional probabilities in (ii) equal $p_{jk}.$

How can we simulate them?#

Label the states ${1, 2, \dots.}$ Use a computer to simulate ${U_1, U_2, \dots}$ as i.i.d. random variables uniformly distributed on $[0, 1],$ written $U[0, 1].$ Define ${F(i, U) = J}$ by the rules

$$\begin{aligned} U &\in [0, p_{i1}) & &\Longrightarrow \quad J = 1, \\ U &\in \left[ \sum_{k=1}^{j-1} p_{ik}, \ \sum_{k=1}^{j} p_{ik} \right), \quad j \ge 2 & &\Longrightarrow \quad J = j. \end{aligned}$$

Figure 2. The rules above on the chain of Example 1.1. The row of the current state $X_n$ splits $[0, 1)$ into one interval for each state $j$ with ${p_{X_n j} > 0,}$ and the interval containing the mark $U_{n+1}$ gives $X_{n+1}.$ The slider sets $X_0$ and the button draws new uniform variables.

The $n$-step transition matrix#

Let $A$ be an event. A convenient notation is

$$P_i(A) = P(A \mid X_0 = i),$$

for every state $i$ with ${\lambda_i > 0;}$ here and below a term $\lambda_i P_i(\cdot)$ with ${\lambda_i = 0}$ is read as 0. For example, ${P_i(X_1 = j) = p_{ij}.}$ Given the initial distribution $\lambda,$ we treat it as a row vector. Then

$$P(X_1 = j) = \sum_{i \in I} \lambda_i P_i(X_1 = j) = \sum_{i \in I} \lambda_i p_{ij}.$$

Similarly,

$$\begin{aligned} P_i(X_2 = j) &= \sum_{k} P_i(X_1 = k, X_2 = j) = \sum_{k} p_{ik} p_{kj} = (P^2)_{ij}, \\ P(X_2 = j) &= \sum_{i, k} \lambda_i P_i(X_1 = k, X_2 = j) = \sum_{i, k} \lambda_i p_{ik} p_{kj} = (\lambda P^2)_j. \end{aligned}$$

Continuing in this way,

$$\begin{aligned} P_i(X_n = j) &= (\delta_i P^n)_j = (P^n)_{ij} = p^{(n)}_{ij}, \\ P(X_n = j) &= \sum_{i_0, \dots, i_{n-1}} \lambda_{i_0} p_{i_0 i_1} \cdots p_{i_{n-1} j} = (\lambda P^n)_j. \end{aligned}$$

Here $\delta_i$ is a special case of $\lambda.$ Thus ${P^{(n)} = (p^{(n)}_{ij}),}$ the $n$-step transition matrix, is simply $P^n,$ the matrix $P$ raised to the power $n.$ Also, for all ${i, j \in I}$ and ${n, m \ge 0,}$ the (obvious) Chapman–Kolmogorov equations hold:

$$p^{(n+m)}_{ij} = \sum_{k \in I} p^{(n)}_{ik} p^{(m)}_{kj}.$$

They are named for their independent formulation by Chapman (a Trinity College graduate, 1880–1970) and Kolmogorov (1903–1987).

In Example 1.1, as ${n \to \infty,}$

$$P^{(n)} \to \begin{pmatrix} \frac15 & \frac25 & \frac25 & 0 & 0 & 0 & 0 \\[6pt] \frac15 & \frac25 & \frac25 & 0 & 0 & 0 & 0 \\[6pt] \frac15 & \frac25 & \frac25 & 0 & 0 & 0 & 0 \\[6pt] \frac{2}{15} & \frac{4}{15} & \frac{4}{15} & 0 & 0 & 0 & \frac13 \\[6pt] \frac{1}{15} & \frac{2}{15} & \frac{2}{15} & 0 & 0 & 0 & \frac23 \\[6pt] \frac{2}{15} & \frac{4}{15} & \frac{4}{15} & 0 & 0 & 0 & \frac13 \\[6pt] 0 & 0 & 0 & 0 & 0 & 0 & 1 \end{pmatrix}.$$

The matrix $P^{(n)}$ of a two-state chain#

Example 1.4 (A two-state Markov chain). Let ${\alpha, \beta \in [0, 1]}$ with ${\alpha + \beta > 0,}$ and consider the chain on ${I = \{1, 2\}}$ with transition matrix

$$P = \begin{pmatrix} 1 - \alpha & \alpha \\ \beta & 1 - \beta \end{pmatrix}.$$

The eigenvalues of $P$ are 1 and ${1 - \alpha - \beta,}$ which are distinct because ${\alpha + \beta > 0.}$ So we can write

$$P = U \begin{pmatrix} 1 & 0 \\ 0 & 1 - \alpha - \beta \end{pmatrix} U^{-1} \quad \Longrightarrow \quad P^n = U \begin{pmatrix} 1 & 0 \\ 0 & (1 - \alpha - \beta)^n \end{pmatrix} U^{-1}.$$

Hence ${p^{(n)}_{11} = A + B (1 - \alpha - \beta)^n}$ for some constants $A$ and $B.$ Since ${p^{(0)}_{11} = 1 = A + B}$ and ${p^{(1)}_{11} = 1 - \alpha = A + B (1 - \alpha - \beta),}$ we get ${(A, B) = (\beta, \alpha)/(\alpha + \beta),}$ that is,

$$P_1(X_n = 1) = p^{(n)}_{11} = \frac{\beta}{\alpha + \beta} + \frac{\alpha}{\alpha + \beta} (1 - \alpha - \beta)^n.$$

When ${\alpha + \beta < 2,}$ this tends exponentially fast to a limit of ${\beta/(\alpha + \beta).}$ The other components of $P^n$ can be computed similarly, and

$$P^n = \begin{pmatrix} \frac{\beta}{\alpha + \beta} + \frac{\alpha}{\alpha + \beta} (1 - \alpha - \beta)^n & \frac{\alpha}{\alpha + \beta} - \frac{\alpha}{\alpha + \beta} (1 - \alpha - \beta)^n \\[6pt] \frac{\beta}{\alpha + \beta} - \frac{\beta}{\alpha + \beta} (1 - \alpha - \beta)^n & \frac{\alpha}{\alpha + \beta} + \frac{\beta}{\alpha + \beta} (1 - \alpha - \beta)^n \end{pmatrix}.$$

α

0.30

β

0.20

Figure 3. The chain of Example 1.4 for the values of $\alpha$ and $\beta$ set by the sliders, and $p^{(n)}_{11}$ for ${n = 0, 1, \dots, 30}$ with the level ${\beta/(\alpha + \beta).}$

Note. We might have reached the same answer by arguing that

$$p^{(n)}_{11} = p^{(n-1)}_{11} p_{11} + p^{(n-1)}_{12} p_{21} = p^{(n-1)}_{11} (1 - \alpha) + \Bigl( 1 - p^{(n-1)}_{11} \Bigr) \beta.$$

This gives the linear recurrence relation, an equation that expresses each element of a sequence as a function of the preceding ones,

$$p^{(n)}_{11} = \beta + (1 - \alpha - \beta) \, p^{(n-1)}_{11},$$

to which the solution is of the form ${p^{(n)}_{11} = A + B (1 - \alpha - \beta)^n.}$