Lecture 1, Markov chains and the transition matrix

Markov chains were introduced in 1906 by Andrei Andreyevich Markov (1856–1922) and were named in his honour. This lecture defines them, describes where they come from and how to simulate them, and computes the probabilities of moving between states in $n$ steps.

An example and some interesting questions

Example 1.1. A frog hops about on 7 lily pads. The number next to each arrow of Figure 1 is the probability with which, at the next jump, the frog moves along that arrow to a neighbouring pad. When the probabilities of the arrows leaving a pad sum to less than 1, the frog stays where it is with the remaining probability, drawn as a loop.

Figure 1. The 7 lily pads of Example 1.1 and the probabilities of the jumps between them.

There are 7 states, the lily pads. The probabilities of the jumps form the matrix

$$P = \begin{pmatrix} 0 & 1 & 0 & 0 & 0 & 0 & 0 \\[4pt] 0 & \frac12 & \frac12 & 0 & 0 & 0 & 0 \\[4pt] \frac12 & 0 & \frac12 & 0 & 0 & 0 & 0 \\[4pt] 0 & 0 & \frac14 & \frac12 & \frac14 & 0 & 0 \\[4pt] 0 & 0 & 0 & 0 & 0 & \frac12 & \frac12 \\[4pt] 0 & 0 & 0 & 1 & 0 & 0 & 0 \\[4pt] 0 & 0 & 0 & 0 & 0 & 0 & 1 \end{pmatrix}.$$

The element $p_{57} = \frac12$ of $P$ is the probability that, when starting in state 5, the next jump takes the frog to state 7. We would like to know where the frog goes, how long it takes to get there, and what happens in the long run. Write $p^{(n)}_{11}$ for the probability that the frog, starting in state 1, is in state 1 after $n$ jumps. Specifically:
(a) Starting in state 1, what is the probability that the frog is still in state 1 after 3 jumps, after 5 jumps, after 1000 jumps? The answers are ${p^{(3)}_{11} = \frac14,}$ ${p^{(5)}_{11} = \frac{3}{16},}$ and about $\frac15$ after 1000 jumps, because ${\lim_{n \to \infty} p^{(n)}_{11} = \frac15.}$
(b) Starting in state 4, what is the probability that the frog ever reaches state 7? The answer is $\frac13.$
(c) Starting in state 4, how long on average does it take to reach either state 3 or state 7? The answer is $\frac{11}{3}.$
(d) Starting in state 2, what is the long-run proportion of time spent in state 3? The answer is $\frac25.$

Markov chain models and methods are useful in answering questions such as the following. How long does it take to shuffle a deck of cards? How likely is a queue to overflow its buffer? How long does it take for a knight making random moves on a chessboard to return to its initial square? The answer is 168 if it starts in a corner and 42 if it starts near the centre. What do the hyperlinks between web pages say about their relative popularity?

Definitions

Probability spaces and random variables

A sequence of positions of the frog can be read as the outcome of a random process, or experiment.

Definition (Probability space). A probability space is a triple $(\Omega, \mathscr{F}, P).$ The sample space $\Omega$ is the set of all possible outcomes of a random process. The event space $\mathscr{F}$ is a set of events, where an event is a subset of $\Omega.$ The probability $P$ is a function that assigns a number between 0 and 1 to every event in $\mathscr{F}.$

Definition (Random variable). Let $(E, \mathscr{E})$ be a measurable space. A random variable with values in $E$ is a measurable function ${X \colon \Omega \to E,}$ that is, a function such that, for every ${B \in \mathscr{E},}$

$$X^{-1}(B) = \{ \omega \in \Omega : X(\omega) \in B \} \in \mathscr{F}.$$

Definition (Distribution). The probability distribution of a random variable $X$ describes how probability is assigned to the values of $X{:}$ it assigns to every ${B \in \mathscr{E}}$ the number ${P(X \in B).}$

Example (The outcome of 10 jumps). An experiment of exactly 11 positions of the frog, that is, 10 jumps, is an outcome ${\omega \in \Omega = \{1, \dots, 7\}^{11},}$ and $\mathscr{F}$ is a set of subsets of $\Omega.$ Take

$$\omega = (1, 2, 3, 2, 3, 4, 3, 4, 2, 1, 2).$$

The random variable $X_n$ returns the position after $n$ jumps, ${X_n(\omega) = \omega_n,}$ so ${X_0(\omega) = 1,}$ ${X_1(\omega) = 2}$ and ${X_2(\omega) = 3.}$ A random variable takes as input an element of $\Omega,$ an outcome and not an event, and returns a value. Figure 2 draws this $\omega.$

n

3

Figure 2. The outcome $\omega$ of the example, position against $n.$ The slider chooses $n$ and the figure reads off $X_n(\omega).$

Markov chains

Let $I$ be a countable set, with elements denoted $i, j, k, \dots.$ Each ${i \in I}$ is called a state and $I$ is called the state space. We work in a probability space ${(\Omega, \mathscr{F}, P).}$ The object of study is a sequence of random variables ${X_0, X_1, \dots}$ with values in $I$ whose joint distribution is determined by simple rules.

A row vector ${\lambda = (\lambda_i : i \in I)}$ is called a measure if ${\lambda_i \ge 0}$ for all ${i \in I.}$ If moreover ${\sum_i \lambda_i = 1,}$ it is a distribution, or probability measure. We start with an initial distribution $\lambda$ over $I,$ which is the initial condition of the chain. The special case in which the chain starts in state $i$ with probability 1 is denoted

$$\lambda = \delta_i = (0, \dots, 0, 1, 0, \dots),$$

with the entry 1 in position $i.$ We also have a transition matrix ${P = (p_{ij} : i, j \in I).}$ It is a stochastic matrix, meaning that ${p_{ij} \ge 0}$ for all ${i, j \in I}$ and ${\sum_{j} p_{ij} = 1}$ for all ${i \in I,}$ that is, each row of $P$ is a distribution over $I.$

Definition 1.2 (Markov chain). A sequence $(X_n)_{n \ge 0}$ of random variables with values in $I$ is a Markov chain with initial distribution $\lambda$ and transition matrix $P$ if, for all ${n \ge 0}$ and ${i_0, \dots, i_{n+1} \in I,}$ the following two conditions hold.
(i) ${P(X_0 = i_0) = \lambda_{i_0}.}$
(ii) Whenever ${P(X_0 = i_0, \dots, X_n = i_n) > 0,}$

$$P(X_{n+1} = i_{n+1} \mid X_0 = i_0, \dots, X_n = i_n) = P(X_{n+1} = i_{n+1} \mid X_n = i_n) = p_{i_n i_{n+1}}.$$

For short, we say that $(X_n)_{n \ge 0}$ is $\mathrm{Markov}(\lambda, P).$ Checking conditions (i) and (ii) is usually the most helpful way to determine whether or not a given random process $(X_n)_{n \ge 0}$ is a Markov chain. It can also be helpful to have the alternative description provided by Theorem 1.3.

Theorem 1.3. A sequence $(X_n)_{n \ge 0}$ of random variables with values in $I$ is $\mathrm{Markov}(\lambda, P)$ if and only if, for all ${n \ge 0}$ and ${i_0, \dots, i_n \in I,}$

$$P(X_0 = i_0, \dots, X_n = i_n) = \lambda_{i_0} p_{i_0 i_1} \cdots p_{i_{n-1} i_n}. \tag{1.1}$$

Proof (sketch). Suppose that $(X_n)_{n \ge 0}$ is $\mathrm{Markov}(\lambda, P).$ Then, for ${n \ge 1,}$

$$\begin{aligned} &P(X_0 = i_0, \dots, X_n = i_n) \\ &\quad = P(X_n = i_n \mid X_0 = i_0, \dots, X_{n-1} = i_{n-1}) \, P(X_0 = i_0, \dots, X_{n-1} = i_{n-1}) \\ &\quad = P(X_0 = i_0) \, P(X_1 = i_1 \mid X_0 = i_0) \cdots P(X_n = i_n \mid X_0 = i_0, \dots, X_{n-1} = i_{n-1}) \\ &\quad = \lambda_{i_0} p_{i_0 i_1} \cdots p_{i_{n-1} i_n}, \end{aligned}$$

and for ${n = 0}$ (1.1) is condition (i). On the other hand, if (1.1) holds, summing it over all ${i_1, \dots, i_n}$ gives ${P(X_0 = i_0) = \lambda_{i_0},}$ that is, (i). Then summing (1.1) over $i_n$ gives ${P(X_0 = i_0, \dots, X_{n-1} = i_{n-1}) = \lambda_{i_0} p_{i_0 i_1} \cdots p_{i_{n-2} i_{n-1}}.}$ Hence

$$P(X_n = i_n \mid X_0 = i_0, \dots, X_{n-1} = i_{n-1}) = \frac{P(X_0 = i_0, \dots, X_n = i_n)}{P(X_0 = i_0, \dots, X_{n-1} = i_{n-1})} = p_{i_{n-1} i_n},$$

which establishes (ii). $\blacksquare$

Details. The first computation divides by ${P(X_0 = i_0, \dots, X_{n-1} = i_{n-1}),}$ so it needs this probability to be positive. When it is zero, both sides of (1.1) are zero: the left-hand side is at most this probability, and if $m$ is the smallest index with ${P(X_0 = i_0, \dots, X_m = i_m) = 0,}$ then ${\lambda_{i_0} = 0}$ when ${m = 0}$ and ${p_{i_{m-1} i_m} = 0}$ when ${m \ge 1,}$ by the same computation up to time $m.$ In the converse, the middle term of (ii) equals $p_{i_{n-1} i_n}$ as well, because summing (1.1) over ${i_0, \dots, i_{n-2}}$ gives ${P(X_{n-1} = i_{n-1}, X_n = i_n) = P(X_{n-1} = i_{n-1}) \, p_{i_{n-1} i_n}.}$

By (1.1), the probability of a sequence of jumps is a product of entries of $P$ along it. For question (a) of Example 1.1, the only sequence of 3 jumps from state 1 back to state 1 with positive probability visits the states 1, 2, 3, 1, and its probability is ${1 \cdot \frac12 \cdot \frac12 = \frac14 = p^{(3)}_{11}.}$ Figure 3 computes the product for any sequence of states.

Path: 1, 2, 3, 1

P(X₀ = 1, X₁ = 2, X₂ = 3, X₃ = 1) = λ₁p₁₂p₂₃p₃₁ = 1· 1· ½· ½ = 1/4

Restart from 1 Path 1, 2, 3, 1

Figure 3. Formula (1.1) on the chain of Example 1.1 with ${\lambda = \delta_{i_0}.}$ Clicking a pad appends it to the sequence, and the product of the entries of $P$ along the sequence is updated; it is 0 as soon as one jump has probability 0.

Where do Markov chains come from?

At each time we apply some new randomness to determine the next step, in a way that is a function only of the current state. We might take ${U_1, U_2, \dots}$ as independent and identically distributed (i.i.d.) random variables with values in a set $E,$ and a function ${F \colon I \times E \to I}$ such that ${u \mapsto F(j, u)}$ is measurable for every ${j \in I.}$ Take ${i \in I.}$ Set ${X_0 = i}$ and define recursively

$$X_{n+1} = F(X_n, U_{n+1}), \qquad n \ge 0.$$

Then $(X_n)_{n \ge 0}$ is $\mathrm{Markov}(\delta_i, P),$ where ${p_{jk} = P(F(j, U) = k)}$ for all ${j, k \in I}$ and $U$ has the common distribution of the $U_n.$ Indeed, ${X_0, \dots, X_n}$ are functions of ${U_1, \dots, U_n,}$ so $U_{n+1}$ is independent of them, and on the event ${\{ X_n = j \}}$ we have ${X_{n+1} = F(j, U_{n+1}).}$ Hence both conditional probabilities in (ii) equal $p_{jk}.$

How can we simulate them?

Label the states ${1, 2, \dots.}$ Use a computer to simulate ${U_1, U_2, \dots}$ as i.i.d. random variables uniformly distributed on $[0, 1],$ written $U[0, 1].$ Define ${F(i, U) = J}$ by the rules

$$\begin{aligned} U &\in [0, p_{i1}) & &\Longrightarrow \quad J = 1, \\ U &\in \Big[ \sum_{k=1}^{j-1} p_{ik}, \ \sum_{k=1}^{j} p_{ik} \Big), \quad j \ge 2 & &\Longrightarrow \quad J = j. \end{aligned}$$

For fixed $i$ these intervals are disjoint, the interval of ${J = j}$ has length $p_{ij},$ and their union is $[0, 1)$ because ${\sum_j p_{ij} = 1.}$ Hence ${P(F(i, U) = j) = p_{ij},}$ and the construction of the previous section produces a chain that is $\mathrm{Markov}(\delta_i, P).$

Figure 4. The rules above on the chain of Example 1.1. The row of the current state $X_n$ splits $[0, 1)$ into one interval for each state $j$ with ${p_{X_n j} > 0,}$ and the interval containing the mark $U_{n+1}$ gives $X_{n+1}.$ The slider sets $X_0$ and the button draws new uniform variables.

The $n$-step transition matrix

Let $A$ be an event. A convenient notation is

$$P_i(A) = P(A \mid X_0 = i),$$

for every state $i$ with ${\lambda_i > 0;}$ here and below a term $\lambda_i P_i(\cdot)$ with ${\lambda_i = 0}$ is read as 0. For example, ${P_i(X_1 = j) = p_{ij}.}$ Given the initial distribution $\lambda,$ we treat it as a row vector. Then

$$P(X_1 = j) = \sum_{i \in I} \lambda_i P_i(X_1 = j) = \sum_{i \in I} \lambda_i p_{ij}.$$

Similarly,

$$\begin{aligned} P_i(X_2 = j) &= \sum_{k} P_i(X_1 = k, X_2 = j) = \sum_{k} p_{ik} p_{kj} = (P^2)_{ij}, \\ P(X_2 = j) &= \sum_{i, k} \lambda_i P_i(X_1 = k, X_2 = j) = \sum_{i, k} \lambda_i p_{ik} p_{kj} = (\lambda P^2)_j. \end{aligned}$$

Continuing in this way,

$$\begin{aligned} P_i(X_n = j) &= (\delta_i P^n)_j = (P^n)_{ij} = p^{(n)}_{ij}, \\ P(X_n = j) &= \sum_{i_0, \dots, i_{n-1}} \lambda_{i_0} p_{i_0 i_1} \cdots p_{i_{n-1} j} = (\lambda P^n)_j. \end{aligned}$$

Here $\delta_i$ is a special case of $\lambda.$ Thus ${P^{(n)} = (p^{(n)}_{ij}),}$ the $n$-step transition matrix, is simply $P^n,$ the matrix $P$ raised to the power $n.$ Also, for all ${i, j \in I}$ and ${n, m \ge 0,}$ the (obvious) Chapman–Kolmogorov equations hold:

$$p^{(n+m)}_{ij} = \sum_{k \in I} p^{(n)}_{ik} p^{(m)}_{kj}.$$

They are obvious because the right-hand side is the entry $(i, j)$ of ${P^n P^m = P^{n+m}.}$ They are named for their independent formulation by Chapman (a Trinity College graduate, 1880–1970) and Kolmogorov (1903–1987).

∑_k p_1k⁽²⁾p_k3⁽³⁾ = 0.3750, p₁₃⁽⁵⁾ = 0.3750

i

1

j

3

n

2

m

3

Figure 5. The Chapman–Kolmogorov equations on the chain of Example 1.1. A line through state $k$ at time $n$ is drawn thicker when ${p^{(n)}_{ik} p^{(m)}_{kj}}$ is larger, and the sliders set $i,$ $j,$ $n$ and $m.$ The sum over $k$ equals $p^{(n+m)}_{ij}.$

In Example 1.1, as ${n \to \infty,}$

$$P^{(n)} \to \begin{pmatrix} \frac15 & \frac25 & \frac25 & 0 & 0 & 0 & 0 \\[6pt] \frac15 & \frac25 & \frac25 & 0 & 0 & 0 & 0 \\[6pt] \frac15 & \frac25 & \frac25 & 0 & 0 & 0 & 0 \\[6pt] \frac{2}{15} & \frac{4}{15} & \frac{4}{15} & 0 & 0 & 0 & \frac13 \\[6pt] \frac{1}{15} & \frac{2}{15} & \frac{2}{15} & 0 & 0 & 0 & \frac23 \\[6pt] \frac{2}{15} & \frac{4}{15} & \frac{4}{15} & 0 & 0 & 0 & \frac13 \\[6pt] 0 & 0 & 0 & 0 & 0 & 0 & 1 \end{pmatrix}.$$

n

3

Figure 6. The matrix $P^n$ of Example 1.1, each entry shaded by its value. The slider sets $n$ from 0 to 60, and $p^{(n)}_{11}$ is written above the grid: ${n = 3}$ and ${n = 5}$ give the values $\frac14$ and $\frac{3}{16}$ of question (a).

The matrix $P^{(n)}$ of a two-state chain

Example 1.4 (A two-state Markov chain). Let ${\alpha, \beta \in [0, 1]}$ with ${\alpha + \beta > 0,}$ and consider the chain on ${I = \{1, 2\}}$ with transition matrix

$$P = \begin{pmatrix} 1 - \alpha & \alpha \\ \beta & 1 - \beta \end{pmatrix}.$$

The eigenvalues of $P$ are 1 and ${1 - \alpha - \beta,}$ which are distinct because ${\alpha + \beta > 0.}$ So we can write

$$P = U \begin{pmatrix} 1 & 0 \\ 0 & 1 - \alpha - \beta \end{pmatrix} U^{-1} \quad \Longrightarrow \quad P^n = U \begin{pmatrix} 1 & 0 \\ 0 & (1 - \alpha - \beta)^n \end{pmatrix} U^{-1}.$$

Hence ${p^{(n)}_{11} = A + B (1 - \alpha - \beta)^n}$ for some constants $A$ and $B.$ Since ${p^{(0)}_{11} = 1 = A + B}$ and ${p^{(1)}_{11} = 1 - \alpha = A + B (1 - \alpha - \beta),}$ we get ${(A, B) = (\beta, \alpha)/(\alpha + \beta),}$ that is,

$$P_1(X_n = 1) = p^{(n)}_{11} = \frac{\beta}{\alpha + \beta} + \frac{\alpha}{\alpha + \beta} (1 - \alpha - \beta)^n.$$

When ${\alpha + \beta < 2,}$ this tends exponentially fast to a limit of ${\beta/(\alpha + \beta).}$ The other components of $P^n$ can be computed similarly, and

$$P^n = \begin{pmatrix} \frac{\beta}{\alpha + \beta} + \frac{\alpha}{\alpha + \beta} (1 - \alpha - \beta)^n & \frac{\alpha}{\alpha + \beta} - \frac{\alpha}{\alpha + \beta} (1 - \alpha - \beta)^n \\[6pt] \frac{\beta}{\alpha + \beta} - \frac{\beta}{\alpha + \beta} (1 - \alpha - \beta)^n & \frac{\alpha}{\alpha + \beta} + \frac{\beta}{\alpha + \beta} (1 - \alpha - \beta)^n \end{pmatrix}.$$

α

0.30

β

0.20

Figure 7. The chain of Example 1.4 for the values of $\alpha$ and $\beta$ set by the sliders, and $p^{(n)}_{11}$ for ${n = 0, 1, \dots, 30}$ with the level ${\beta/(\alpha + \beta).}$

Note. We might have reached the same answer by arguing that

$$p^{(n)}_{11} = p^{(n-1)}_{11} p_{11} + p^{(n-1)}_{12} p_{21} = p^{(n-1)}_{11} (1 - \alpha) + \big( 1 - p^{(n-1)}_{11} \big) \beta.$$

This gives the linear recurrence relation, an equation that expresses each element of a sequence as a function of the preceding ones,

$$p^{(n)}_{11} = \beta + (1 - \alpha - \beta) \, p^{(n-1)}_{11},$$

to which the solution is of the form ${p^{(n)}_{11} = A + B (1 - \alpha - \beta)^n.}$