Entropy measures the average uncertainty in a probability distribution. Cross-entropy measures how costly it is to score outcomes from one distribution using probabilities from another. Their key relationship is H(p,q) = H(p) + DKL(p‖q): cross-entropy is the source’s entropy plus a nonnegative penalty for using a mismatched probability model.
A small example: how much probability did the model give the right answer?
Suppose the true class is known to be A. Represent that target distribution as p(A)=1, with probability zero for other classes. A model predicts a distribution q.
As an Amazon Associate I earn from qualifying purchases.
- If
q(A)=0.8, the cross-entropy for this example is−log2(0.8) ≈ 0.322bits. - If
q(A)=0.1, it is−log2(0.1) ≈ 3.322bits.
These values follow directly from the definition. The second model receives a larger loss because it assigned less probability to the known target. For a one-hot target, only the probability assigned to the correct class contributes to that example’s cross-entropy.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteWhat entropy measures
For a discrete random variable X with probability mass function p(x), Shannon entropy is the expected surprisal of its outcomes:
#1 Best Overall
H(X) = −Σx p(x) log2 p(x)
The surprisal, or self-information, of one outcome x is −log2 p(x). An unlikely outcome has greater surprisal than a likely one. Entropy averages those values over outcomes according to p; it is not the information content of one particular observation.
For a variable with m possible outcomes, entropy is zero if one outcome is certain and is no greater than log2(m). It reaches that maximum for a uniform distribution. With base-2 logarithms, the unit is bits; using natural logarithms gives nats instead. Carnegie Mellon’s Information Theory I lecture notes discuss these properties and entropy’s coding interpretation.
Rank #2
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
What cross-entropy measures
For two distributions over the same outcomes, cross-entropy is
Free tools Windows power users keep installed
One-click scans. No signup required.
H(p,q) = −Σx p(x) log2 q(x)
Here p supplies the outcomes and the weights used in the average, while q supplies the probabilities being scored. In plain terms, imagine outcomes generated according to p, but evaluated using the probabilities assigned by q. If the distributions match, cross-entropy equals entropy: H(p,p)=H(p).
Rank #3
The order matters. H(p,q) averages according to p; switching the distributions generally changes the result. If q(x)=0 for any outcome with p(x)>0, the cross-entropy is infinite under the standard discrete definition. Terms where p(x)=0 contribute zero by the limiting convention 0 log(0/q)=0.
The bridge to KL divergence
The Kullback–Leibler divergence from p to q is
DKL(p‖q) = Σx p(x) log2[p(x)/q(x)]
Rearranging the definitions gives the central identity:
Rank #4
H(p,q) = H(p) + DKL(p‖q)
So cross-entropy consists of two parts: the uncertainty inherent in the source distribution p, and the extra cost caused by representing it with q. KL divergence is nonnegative, so cross-entropy cannot be less than H(p); the mismatch penalty is zero when the distributions match. When q assigns zero probability to an outcome that has positive probability under p, the divergence is infinite as well.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
KL divergence is generally asymmetric: DKL(p‖q) need not equal DKL(q‖p). It is therefore more precise to call it a divergence or mismatch measure than a distance. Carnegie Mellon’s lecture notes define cross-entropy and KL divergence and explain their relationship and asymmetry.
Best Value
Why the distinction matters in classification
A classifier predicts probabilities over classes. For a one-hot target with correct class y, the per-example cross-entropy reduces to the negative log probability assigned to that class:
−log q(y)
Across a fixed dataset, the mean of these per-example losses is the average negative log-likelihood of the observed labels. Minimizing it is therefore equivalent to maximizing their likelihood under the model’s predicted probabilities.
Targets can also be probability distributions rather than one-hot labels. For target probabilities p(x), the loss is −Σ p(x)log q(x). If the target distribution is fixed, H(p) is constant, so minimizing cross-entropy over q also minimizes DKL(p‖q). This equivalence concerns these distributions and fixed target data; it does not make every loss function or training setup interchangeable. The LMU Statistical Learning and Data Science chapter on cross-entropy and KL treats this connection in machine learning.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Why cross-entropy also appears in coding
Entropy gives the lower bound on the expected code length for a source when its probabilities are known. If a code is optimized for q but used on outcomes drawn from p, the expected description length corresponds to cross-entropy H(p,q). The overhead compared with a code based on the true distribution is DKL(p‖q). Carnegie Mellon’s lecture notes explain this coding interpretation.
A note about continuous variables
For continuous variables, entropy is defined using a probability density and an integral rather than a sum. This differential entropy can be negative, unlike the nonnegative discrete entropy described above, so discrete intuitions do not all carry over unchanged. MIT’s 6.441 Information Theory lecture notes distinguish differential entropy from discrete entropy.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
