Concepts and theorems · Intermediate

Entropy: quantify uncertainty in a probability distribution

Calculate entropy for fair, biased and dependent variables, then distinguish uncertainty, lossless compression, cross-entropy and prediction accuracy.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Separate an event from a distribution
  2. Check units and zero-probability terms
  3. Compute three coin distributions
  4. Outcome count gives an upper bound
  5. Do not add dependent variables blindly
  6. Separate average coding bounds from finite files
  7. Confidence is not prediction accuracy
  8. Recognize uncertainty in estimated entropy
  9. Ask Logeia about this articleCheck your understanding

Separate an event from a distribution

Entropy is a property of a probability distribution: which outcomes occur and with what probabilities. The self-information of an outcome with probability p is −log2(p); averaging this quantity under the distribution gives H(X)=−Σp(x)log2(p(x)). An outcome of probability one-half carries one bit of self-information, while an outcome of probability one-eighth carries three bits. Observing one rare event does not mean that the underlying distribution’s entropy increased on that particular trial. Specify the distribution first, then distinguish information assigned to one outcome from the average across possible outcomes. This prevents a surprising anecdote from being mistaken for an estimate of the uncertainty of the full source.

Check units and zero-probability terms

Base-two logarithms give bits, whereas natural logarithms give nats. Values calculated with different bases need conversion before numerical comparison. A zero-probability term contributes zero by the limit of −p log p as p approaches zero; implementations can skip it rather than evaluating log(0) directly. Validate nonnegative probabilities and a total of one. Raw counts are not probabilities until normalized, and a percentage such as fifty must become 0.5 rather than fifty. Floating-point rounding also differs from supplying an invalid distribution. These small input and unit checks matter because an expression can look like the entropy formula while operating on numbers that do not represent the stated probability model at all.

Compute three coin distributions

A fair coin gives H=−2×(1/2)log2(1/2)=1 bit. A deterministic coin that always produces heads gives zero. For probabilities 0.9 and 0.1, the formula gives approximately 0.469 bits. The outcome labels can be identical across all three examples while the uncertainty changes with the probabilities. The fair distribution maximizes entropy among binary distributions. The fractional value does not mean that a single physical record can literally contain 0.469 binary digits. Turning an entropy value into a statement about average representation length requires a coding model, often involving many outcomes encoded together. Keep the probability calculation distinct from the additional engineering needed to store or transmit a finite sequence.

Outcome count gives an upper bound

A discrete distribution with at most M possible values has entropy at most log2(M), attained when all M values are equally likely. Four equally probable outcomes give two bits, but four available outcomes with one overwhelmingly likely result give less. Counting possible labels alone therefore does not determine the entropy of the actual distribution. The definition of an outcome also matters. Classifying user activity only as success or failure defines a different random variable from retaining separate failure categories. Both can be derived from the same underlying records. When comparing two entropy measurements, align the classification and probabilities as well as the logarithm base. Otherwise a change in category granularity can be mistaken for a change in behavior.

Do not add dependent variables blindly

For two independent fair coins, the pair has four equally probable outcomes and joint entropy two bits. If Y always equals X instead, the pair can only be heads-heads or tails-tails, so its joint entropy is one bit. Each marginal variable still has entropy one. The sum of marginal entropies therefore overcounts the joint uncertainty in the dependent example. The chain rule is H(X,Y)=H(X)+H(Y|X). When Y=X, knowing X leaves zero uncertainty about Y. This matters when estimating the information in multiple fields or logs: separately counting every field can count repeated information more than once. If an independent model is used as an approximation, state why it is reasonable and what dependencies it leaves out.

Separate average coding bounds from finite files

Entropy under a source model is fundamental to average lengths in lossless coding. It does not guarantee that every short file compresses by a fixed ratio. Finite codes have constraints such as integer bit lengths, and real formats can require headers or a description of the model. A model treating successive characters as independent can also give a different estimate from one conditioned on previous characters. If observed compression is poor, investigate modeling assumptions and dependence as well as implementation details. Distinguish the theoretical limit, the selected coding scheme and the format’s extra costs. A measured file size is an operational observation, whereas an entropy bound applies under a specified probability and coding framework.

Confidence is not prediction accuracy

A model that places almost all probability on one answer has low predictive entropy, even if that answer is wrong. The entropy of its own predicted distribution, −Σq log q, is not the same quantity as cross-entropy evaluated under a reference distribution p, −Σp log q. The latter penalizes assigning too little probability to outcomes that occur under p. If q gives zero probability to an outcome with positive p, the theoretical cross-entropy is infinite. Numerical clipping can avoid infinite computed values, but it changes the evaluated expression and needs to be documented. Assess accuracy or calibration with appropriate observations rather than assuming that a concentrated distribution demonstrates knowledge or trustworthy reasoning.

Recognize uncertainty in estimated entropy

In practice the distribution is often unknown and probabilities are estimated from observed frequencies. A small sample can miss rare outcomes entirely, assigning them estimated probability zero. The resulting entropy estimate can understate uncertainty, but a universal constant correction does not solve every estimation problem. Report sample size, category definitions, smoothing and changes in the source over time. Comparing values obtained under different procedures can be misleading. This article concerns discrete variables. Differential entropy for continuous variables has different properties, including different behavior under changes of units and transformations. Check the relevant definition before carrying over the discrete intuition. Estimated entropy describes a model derived from observations; it is not automatically a complete or exact measure of the underlying process.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 10

Choose a learning path