A Markov chain is a probability model for a system that occupies one state at a time and moves to another state at each discrete step. Its defining rule—the Markov property—says that once the current state is known, the probabilities for the next state do not depend on the earlier history. A transition matrix records those one-step probabilities, matrix powers describe several steps, and a stationary distribution describes a probability mix that remains unchanged after a transition.
What is a Markov chain?
Let X0, X1, X2, ... denote the states of a system at successive times. A discrete-time Markov chain is a stochastic process whose next state satisfies
As an Amazon Associate I earn from qualifying purchases.
P(Xn+1=j | Xn=i, Xn-1, ..., X0) = P(Xn+1=j | Xn=i).
In words, the present state contains all the information needed to calculate the distribution of the next state. The chain does not claim that the past is irrelevant to the system’s actual history; it says that the past adds no information about the next step once the current state is specified.
#1 Best Overall
The state space can be finite, countably infinite, or more general. This introduction uses finite state spaces and discrete time, as in introductory treatments from Stanford Information Retrieval, the University of Illinois CS 357, and standard stochastic-modeling texts.
A concrete illustrative example
Consider a hypothetical daily weather model with three states: Sunny (S), Cloudy (C), and Rainy (R). The following row-stochastic transition matrix is illustrative only; it is not estimated from a real weather record.
| Today Tomorrow | Sunny (S) | Cloudy (C) | Rainy (R) |
|---|---|---|---|
| Sunny (S) | 0.60 | 0.30 | 0.10 |
| Cloudy (C) | 0.20 | 0.50 | 0.30 |
| Rainy (R) | 0.10 | 0.40 | 0.50 |
Rows represent the current state and columns represent the next state. For example, if today is Sunny, the model assigns a 0.30 probability to Cloudy tomorrow. Every row sums to 1 because one of the listed next states must occur.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesHow to read a transition matrix
One-step probabilities
With the row convention, entry Pij means
Pij = P(Xn+1=j | Xn=i).
In the example, PSR=0.10 means a 10% modeled chance of Rainy tomorrow given Sunny today. It does not mean that 10% of all rainy days follow sunny days; that would be a different conditional probability.
Rank #2
Checking a valid matrix
- Each entry must be between 0 and 1.
- Each row must sum to 1 under the row-stochastic convention.
- The state order must be documented and kept consistent in every calculation.
Row-vector and column-vector conventions
Courses and software use two equivalent notational conventions. In the row-stochastic convention above, a probability distribution is a row vector and advances by multiplying on the right:
pn+1 = pnP.
If the current distribution is entirely Sunny, p0 = (1,0,0), then
p1 = (1,0,0)P = (0.60, 0.30, 0.10).
Some texts instead use column probability vectors, place probabilities in a column-stochastic matrix, and calculate pn+1 = Ppn. The numerical process is the same after transposing the matrix, but mixing a row vector with a column-stochastic matrix (or vice versa) produces incorrect results. State your convention before multiplying.
Recommended Free Tools
Several steps and matrix powers
For a time-homogeneous chain, the same transition matrix applies at every step. The probability of moving from state i to state j in k steps is the (i,j) entry of Pk:
Rank #3
pn+k = pnPk for row vectors.
For example, the two-step probability of Sunny followed by Rainy is obtained by summing over the possible intermediate states:
(P2)SR = (0.60)(0.10) + (0.30)(0.30) + (0.10)(0.50) = 0.20.
Thus this illustrative model gives a 0.20 probability of being Rainy two days after a Sunny day. The multiplication automatically accounts for every possible weather state on the intervening day.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →What is a stationary distribution?
A stationary (or invariant) distribution is a probability vector π that is unchanged by one transition. Under the row convention it satisfies
Rank #4
πP = π,
together with πi ≥ 0 and Σπi = 1. It describes a balance of probabilities: if the chain starts with distribution π, every later step has that same distribution.
How to find one
- Write the equations from
πP = π. - Add the normalization equation
π1+...+πm=1. - Solve the resulting linear system.
- Check that all components are nonnegative and that the product
πPreproducesπ.
For the illustrative weather matrix, solving these equations gives approximately π = (0.291, 0.403, 0.306) in the order Sunny, Cloudy, Rainy. This is a stationary mix for the example, not a claim about actual climate frequencies.
Stationary does not automatically mean eventual convergence
A chain can have a stationary distribution without every starting distribution converging to it. Reducibility can split the state space into separate closed classes, producing multiple stationary distributions. Periodicity can cause distributions to cycle instead of settling at each individual time step. Stronger conditions—commonly irreducibility and aperiodicity for a finite chain—are used when claiming convergence to a unique stationary distribution. The assumptions must accompany any long-run statement.
Absorbing states and absorbing chains
State i is absorbing when
Pii = 1.
Once entered, it cannot be left on the next step (and therefore cannot be left at any later step). A self-loop with probability less than 1 is not an absorbing state; the chain can still escape.
Example
Suppose a process has states Working, Failed, and Retired, and the Retired row is (0,0,1). Retired is absorbing. Working or Failed may eventually lead there, but whether they do so with probability 1 depends on the other transition probabilities and on reachability.
An absorbing chain normally means more than “the matrix contains an absorbing state”: every state can reach at least one absorbing state, usually in a finite number of steps with positive probability. This distinction matters when calculating eventual absorption probabilities or expected time to absorption.
Common mistakes
- Reversing the conditional probability: “from Sunny to Rainy” is not the same as “Sunny given Rainy.”
- Mixing conventions: check whether states are rows or columns and which side the matrix multiplies.
- Forgetting time homogeneity: a single matrix for every step is an additional assumption; a changing environment may require matrices
P0, P1, .... - Calling any self-loop absorbing: absorption requires a self-transition probability of exactly 1.
- Promising a unique long-run limit without conditions: inspect communication between states and periodic cycles first.
- Treating an illustrative matrix as measured data: a useful teaching example does not establish real-world frequencies.
Where Markov chains are useful
The model is appropriate when a system can be represented by a manageable state description and the next-step uncertainty is reasonably summarized by that current state. Examples include queue or inventory states, board-game positions, reliability modes, text-generation states, and simplified weather or preference models. If the past has an important effect that is not captured in the current state, enlarge the state representation or use a model with memory rather than forcing the Markov assumption.
Further study
For worked exercises, look for a probability textbook with a dedicated Markov-chains chapter. Grinstead and Snell’s Probability: An Introduction, second edition, includes such a chapter. More advanced subjects—continuous-time chains, Markov chain Monte Carlo, mixing times, and general state spaces—require definitions beyond this discrete-time introduction.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
