An n-gram language model estimates the next word from counts of short word sequences in a training corpus. A bigram uses one preceding word as context; a trigram uses two. This makes the approach comparatively simple to calculate, but also limits it: unseen sequences get zero probability in a raw-count model, and even a smoothed model cannot use context beyond its fixed window.
What is an n-gram language model?
An n-gram is a sequence of n adjacent items. In a word-based language model, those items are usually words; in other applications, they can be characters. The model uses how often sequences occur in a corpus to estimate the probability of a word appearing next.
As an Amazon Associate I earn from qualifying purchases.
The order names describe the number of items in the sequence. For next-word prediction, the context is one item shorter than the n-gram itself:
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errors| Model | Sequence size | Preceding-word context for next-word prediction |
|---|---|---|
| Unigram | One word | No preceding word |
| Bigram | Two words | One preceding word |
| Trigram | Three words | Two preceding words |
So “n-gram” refers to the size of the counted sequence, not the number of words in a generated answer. A trigram model may generate a long passage one word at a time, while conditioning each prediction on only the most recent two words.
#1 Best Overall
How does the model estimate the next word?
For a current word wᵢ, an order-n model estimates its probability given the previous n−1 words: P(wᵢ | wᵢ₋ₙ₊₁ … wᵢ₋₁). A basic maximum-likelihood estimate divides the count of the complete sequence by the count of its preceding context.
For example, a trigram model estimating the word after “I want to” looks at corpus counts for sequences such as “I want to leave” and “I want to eat,” then compares those counts with the number of times the context “I want to” occurs. Which continuation receives the highest estimate depends on the corpus, its tokenization, and the model’s treatment of rare or unseen sequences; the phrase has no universally most probable completion.
This is a Markov-style approximation: the model conditions on a selected recent history rather than the entire document. It trades long-range context for a finite set of counts that is easier to estimate and use.
Free tools Windows power users keep installed
One-click scans. No signup required.
Why do raw n-gram counts fail on unseen sequences?
A training corpus is only a sample of language. Even a sequence that is plausible may never appear in that sample. If its count is zero, the basic count-based estimate assigns it zero probability. Rare sequences can also receive unreliable estimates because a small change in the corpus can substantially affect their counts.
Rank #3
- Language fundamentals grade 1
- Language skills
- Grammar practice
This is the sparse-data problem. Increasing the order gives the model more context, but it also creates more possible sequences to count. As a result, the training data are spread across a larger set of combinations, and many combinations may be rare or absent.
How does smoothing help?
Smoothing adjusts probability estimates so that observed counts do not monopolize all probability mass. Some methods discount counts and redistribute probability to unseen events; backoff methods can consult a shorter context when a longer sequence has little or no evidence; interpolation methods combine estimates from multiple orders. These are related responses to sparse counts, not interchangeable recipes.
Stanley F. Chen and Joshua Goodman’s 1998 Harvard technical report surveys smoothing methods and compares them across training-data sizes, corpora, count cutoffs, and bigram versus trigram models. In their experiments, training-data size was the most significant factor among those considered. Their results support evaluating smoothing choices under the conditions of the intended application—not assuming one method always wins. The report also describes a Kneser–Ney variation motivated by its empirical analysis. The authors’ findings, including reported speech-recognition improvements associated with improved smoothing, apply to their experimental settings rather than guaranteeing the same result elsewhere. Read the Harvard technical report; the related ACL proceedings publication is listed by Microsoft Research.
How should n-gram models be compared?
A higher order can capture a longer local pattern, but it also requires more data to estimate its counts reliably. Neither “use the largest n” nor “use the most sophisticated smoothing” is a sound rule without evaluation on the relevant corpus and task.
- Order and context: Compare unigram, bigram, trigram, or higher-order models against the amount and type of available training data.
- Smoothing: Compare discounting, backoff, and interpolation on the same data rather than presuming a universal ranking.
- Training conditions: Keep corpus, vocabulary, tokenization, training-set size, and count cutoffs in view; each can change the observed performance.
- Evaluation: Cross-entropy or perplexity can assess probability estimates, while task measures—such as speech-recognition performance—test usefulness in an application. Results on one measure do not automatically establish gains on another.
Where do n-grams fit in language-model history?
N-gram models are part of statistical language modeling: estimating how likely a sequence of language is, or what item may follow a given context. Their intellectual background includes Markov-chain methods and Claude Shannon’s information-theoretic treatment of statistical sources. Those foundations are relevant to the lineage, but they should not be mistaken for the design of today’s NLP systems or for use of the modern phrase “large language model.”
N-grams were an important practical approach to language modeling before neural methods became prominent. Their fixed context and dependence on observed counts make them useful for explaining both statistical prediction and the limits of sparse, local evidence. Later neural language models use different modeling approaches and can generalize across patterns in ways a table of n-gram counts cannot; smoothing reduces zero-count failures but does not remove the fixed-window constraint. For a fuller treatment of the topic and its place in NLP, see Stanford’s Speech and Language Processing, third edition, n-gram chapter.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →
