BM25 ranks documents by weighing how often a query term appears, how rare that term is across the collection, and how long the document is. Repeated matches help, but their contribution levels off; a match in a long document is also treated differently from the same count in a short one. The result is a ranking score—not a calibrated probability that a document is relevant.
What BM25 is
BM25, often called Okapi BM25, is a lexical ranking function in the probabilistic relevance framework. “Lexical” means it works with query terms and terms found in documents: it can reward a document for containing the words a person searched for, but it does not by itself establish that two differently worded passages mean the same thing.
As an Amazon Associate I earn from qualifying purchases.
For each query, BM25 supplies a score that a search system can use to order candidate documents. The score is useful for comparing results within a search, but should not be read as a percentage chance of relevance.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How BM25 builds a score
BM25 combines three kinds of evidence. Their effects interact: a query term’s contribution depends both on its frequency in a document and on how that document compares in length with the collection.
#1 Best Overall
Term frequency: matches help, then saturate
Term frequency is the number of times a query term occurs in a document. More occurrences can raise its contribution, but with diminishing returns. The first few occurrences can matter more than many additional repetitions, so simply repeating a keyword does not increase the score without limit.
Inverse document frequency: rare terms can distinguish results
Inverse document frequency reflects how widely a term appears across the document collection. A term found in relatively few documents can provide more distinguishing evidence than a very common term. The collection matters: a word’s rarity, and therefore its usefulness as a discriminator, depends on which documents are indexed.
Rank #2
Length normalization: counts are read in context
BM25 adjusts term-frequency contribution in relation to document length and the collection’s average document length. This helps prevent raw counts from being interpreted identically in a short document and a much longer one. It does not mean long documents are automatically penalized out of the results; it changes how term matches contribute to the score.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →What the BM25 settings k1 and b mean
Implementations expose parameters that adjust the scoring behavior. In Elasticsearch’s similarity reference, k1 controls nonlinear term-frequency saturation, while b controls the degree of document-length normalization. That reference lists defaults of k1 = 1.2 and b = 0.75; these are Elasticsearch documentation defaults, not universal constants or evidence that those values are best for every collection.
Rank #3
Changing k1 changes how quickly the benefit from repeated term occurrences levels off. Changing b changes how strongly document length affects the term-frequency contribution. The practical effect depends on the corpus, fields, analyzer, and queries, so parameter changes should be judged against relevance judgments for the intended search task rather than assumed to improve results.
BM25 compared with TF-IDF
Both BM25 and TF-IDF use term frequency and term rarity as signals. BM25’s characteristic treatment includes diminishing returns for repeated occurrences and document-length normalization. That distinction makes BM25 more than a direct count of matching words, but it does not make it a universal replacement for every retrieval method. A useful comparison depends on the implementation and on evaluation against the documents and queries the system actually serves.
Where BM25 fits in a modern search pipeline
BM25 remains useful for full-text retrieval: it can find and rank candidates with matching terms. In Elasticsearch and Lucene, documentation identifies BM25 as the default similarity; Elastic describes its default algorithm as “The algorithm used by default in Elasticsearch and Lucene.” Defaults and available settings should be checked against the version actually deployed.
BM25 need not be the only retrieval signal. Search systems can combine lexical results with vector-search results in a hybrid approach, then apply a later reranking stage. These stages serve different roles: lexical retrieval rewards term matches, vector retrieval can surface semantic similarity, and reranking can reorder a candidate set. None is established as the winner for all queries or collections; evaluate the full pipeline on the target task.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How BM25F handles documents with multiple fields
When documents have distinct fields—such as a title and body—a single undifferentiated text representation may not express how much each field should matter. BM25F is a conceptual extension for structured or multi-field documents. It can accumulate term evidence across fields while applying field-specific weights and length normalization. The specific API and implementation details vary, so consult the documentation for the search library in use.
Quick Recap
How to decide whether to tune BM25
- Define the search task. Identify the queries, documents, and relevance criteria that matter to users; a parameter setting cannot be judged without a target.
- Establish a baseline. Record results with the implementation’s current settings and evaluate them on representative queries and relevance judgments.
- Change one factor at a time. Test adjustments to term-frequency saturation or length normalization separately, so their effects can be interpreted.
- Compare the outcomes. Use a suitable relevance metric and inspect failures as well as aggregate results. A gain on one query type can coincide with a regression on another.
- Evaluate the whole pipeline. If hybrid retrieval or reranking is involved, assess those stages together with BM25 rather than attributing every result change to the lexical score alone.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
