Study guides · Intermediate

Likelihood and estimation with concrete data

Construct a likelihood from coin observations, find its maximum, and distinguish an estimate from uncertainty or model validation.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Fix the observations
  2. Likelihood is not a posterior
  3. Optimize the log likelihood
  4. Check boundary solutions
  5. Inspect uncertainty and misspecification
  6. Verify a finite comparison
  7. Ask Logeia about this articleCheck your understanding

Fix the observations

A likelihood holds the observed data fixed and varies a parameter. Suppose ten independent coin tosses share an unknown head probability p, and seven heads are observed. For one particular ordered sequence, the probability is p to the seventh power times one minus p to the third. If only the head count is recorded, a binomial coefficient appears as well. That coefficient does not depend on p, so it does not change the maximizing value. Specify the observation being modeled before deciding which probability expression is appropriate.

Likelihood is not a posterior

For a probability distribution, the parameter is fixed and probabilities sum over possible datasets. A likelihood instead varies the parameter for the one observed dataset; it need not integrate to one over parameter values. A larger likelihood means that candidate parameter assigns greater probability to the observations. It is not directly the probability that the parameter is correct. A posterior distribution additionally requires a prior and normalization. Compare likelihood values only under a consistently specified dataset and observation model; otherwise differences may reflect a changed experiment rather than a better explanation.

Optimize the log likelihood

Taking logarithms converts the product into seven log p plus three log one minus p, up to a constant. Differentiating inside the unit interval gives seven divided by p minus three divided by one minus p. Setting this to zero yields p=0.7. The second derivative is negative, confirming an interior maximum. Because logarithms preserve ordering, the likelihood and log likelihood share the maximum. Independence justifies multiplying observation probabilities; the shared-distribution assumption justifies using the same parameter for all tosses. Both are modeling choices rather than consequences of differentiation.

Check boundary solutions

If every toss is a head, the likelihood is p to the tenth power and its maximum over the closed interval is p=1. An equation obtained by setting an interior derivative to zero would miss this boundary solution. The estimate does not prove that tails are impossible; it is a point estimate from a finite sample. Zero probability for an unobserved event can have serious consequences in subsequent prediction. Smoothing or introducing a prior may address that issue, but changes the estimation procedure and should be stated explicitly rather than silently labeled maximum likelihood.

Inspect uncertainty and misspecification

Seven heads in ten tosses and seven hundred in one thousand share the maximum-likelihood estimate 0.7, but their sampling variability differs under the independent, identically distributed model. The point estimate alone does not describe precision or future prediction error. Confidence intervals, held-out prediction, or repeated sampling answer additional questions. If the coin or environment changes halfway through observation, a single parameter may be inappropriate. More observations do not automatically repair a misspecified model. Preserve order and subgroup information so you can investigate changes that a pooled count would conceal.

Verify a finite comparison

Evaluate the log likelihood at 0.5,0.7, and 0.9, and verify that 0.7 wins among these candidates. Then change the observations to six heads and four tails; the same derivative argument gives 0.6. For large samples, multiplying many small probabilities may underflow, whereas summing logarithms is more practical. Endpoints still require care because log zero is undefined as a finite real number. Record the parameter domain, numerical search method, and final constraint checks. A successful optimization establishes a maximum for the chosen model; it does not validate the model’s assumptions.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 11

Choose a learning path