Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11What does cross-entropy actually measure, and why is it connected to compression? It measures how costly a model’s probability predictions are on data from a particular source: each observed outcome costs more when the model assigned it a lower probability. Averaged across the data, that cost is the cross-entropy. Because those same probabilities can drive a lossless code, better predictions can mean shorter idealized codes—but that does not make compression a complete measure of intelligence.
How surprising outcomes become a score
Suppose a model assigns an outcome probability of 1/8. Its idealized binary code length under that probability is −log₂(1/8), or 3 bits. An outcome assigned probability 1/2 costs 1 bit; one assigned probability 1/16 costs 4 bits. In general, the cost is −log₂ Q(x), where Q is the model’s probability distribution and x is the outcome that occurred.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
Information Theory, Inference and Learning Algorithms | $75.24 | Buy on Amazon |
| 2 |
|
Elements of Information Theory | $71.94 | Buy on Amazon |
| 3 |
|
Information Theory: A Tutorial Introduction (2nd Edition) | $27.91 | Buy on Amazon |
| 4 |
|
Information Theory (Dover Books on Mathematics) | $16.95 | Buy on Amazon |
| 5 |
|
Information Theory: From Coding to Learning | $66.92 | Buy on Amazon |
As an Amazon Associate I earn from qualifying purchases.
Cross-entropy averages those costs over examples generated by a source distribution P:
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →H(P, Q) = Ex∼P[−log₂ Q(x)]
With base-2 logarithms, the result is measured in bits. Using natural logarithms gives nats instead. The score depends on both the data source P and the model Q: it is not simply an intrinsic property of a piece of text. (See Stanford CS324’s introduction and Takata, Kaji, and Utsuro’s 2020 paper in Entropy.)
#1 Best Overall
Why model mismatch adds cost
Cross-entropy has a useful decomposition:
H(P, Q) = H(P) + DKL(P || Q)
H(P) is the source’s entropy: its average uncertainty under its own distribution. DKL(P || Q), the Kullback–Leibler divergence, is the extra expected cost of using Q rather than P to assign probabilities. Since KL divergence is nonnegative, cross-entropy cannot be lower than source entropy. It reaches that minimum when Q matches P, apart from outcomes that have zero probability under the source.
In practice, the true source distribution is usually unknown, so its entropy is not directly available. A measured cross-entropy therefore combines the source’s uncertainty with the model’s mismatch. Calling a model’s score “the entropy of the text” loses that distinction.
How predictive loss becomes a compression cost
In language modeling, a sequence is assigned a probability by multiplying the model’s conditional next-token probabilities. Taking the negative logarithm turns that product into a sum: each token contributes a cost based on the probability the model gave it in context. Summing gives the sequence’s idealized model-based code length; averaging gives a per-token or per-character score, depending on the evaluation convention.
Rank #2
Those probabilities can also guide an entropy coder such as arithmetic coding. The encoder and decoder share the probability model and coding procedure, and the decoder reconstructs the original sequence exactly. This is lossless compression—not paraphrasing, summarizing, or discarding meaning. With a suitable coder, bitstream length can approach the summed negative log-probability, subject to implementation and message-termination overhead. (The COLM 2024 paper “Lossless Compression of Language” describes this connection through source coding and arithmetic coding.)
That idealized length is not automatically the final file size. A practical system may also need framing, coding overhead, and model side information; if the decoder does not already have the model, transmitting it can matter substantially. So lower cross-entropy implies shorter idealized codes for the same data and setup, not a fixed reduction in bytes for every compression system.
What language-model cross-entropy tells you
Training and evaluation
For classification, a common cross-entropy loss is the negative logarithm of the probability assigned to the correct class. For an autoregressive language model, the same idea is applied repeatedly to the observed next token. Training minimizes this negative-log-likelihood objective; evaluation on held-out examples estimates how well the model assigns probability to data from that evaluation distribution.
A lower held-out loss means better average probability assignment on that distribution under that evaluation protocol. It does not guarantee the same ranking on different data, and it does not by itself measure every capability a model might have.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsUnits and comparison
Cross-entropy values are interpretable only with their logarithm base and averaging convention. Per-token values also depend on tokenization: different tokenizers divide the same text into different numbers of units, so bits per token are not a universal, tokenizer-independent scale. Bits per character or byte require a clearly specified text representation.
Perplexity is another expression of average log loss: it is commonly calculated by exponentiating average natural-log loss, or by raising 2 to average loss in bits. Comparisons need to use the same convention. To compare model scores responsibly, align the evaluation corpus and split, tokenization or character unit, context protocol, log base, normalization, and sequence-boundary handling. Also distinguish in-distribution evaluation from shifted data.
OpenAI’s January 23, 2020 paper on scaling laws for neural language models studies empirical relationships between cross-entropy loss and model size, dataset size, and training compute. That makes cross-entropy useful as a performance metric in scaling research; it does not establish that loss alone captures intelligence.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What published entropy estimates can—and cannot—show
Takata, Kaji, and Utsuro reported an English entropy-rate estimate of 1.12 bits per character in their 2020 paper. It was obtained by extrapolating the effects of training-data size and context length toward infinity using neural language models. It is the authors’ model-based extrapolation, not a settled universal constant or a directly measured file-compression rate.
In the same paper’s specific model and dataset experiments, the authors reported observed minimum cross-entropies of 1.21 bits per character for English and 4.43 bits per character for Chinese. Those results belong to that historical experimental setting; they should not be treated as universal figures or compared casually with scores measured using a different corpus, model, or unit.
Best Value
When the compression–intelligence link is useful
The link is rigorous but conditional: on the same data, tokenization, context, and probability convention, a model that predicts observed outcomes better has lower cross-entropy and assigns them shorter idealized codes on average. This makes compression a helpful interpretation of predictive fit. It does not show that a compressed representation has understood the data, or that one loss number settles how intelligent a system is.
To assess broader capability, cross-entropy needs to sit alongside other relevant evidence and tests. It answers a narrower question: how much probability did this model assign to the outcomes that occurred, on this distribution and under this evaluation setup?
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
