Ask what a probability means
Calibration asks whether cases assigned probabilities near 0.8 exhibit positive outcomes at roughly that frequency. Binary positive-class calibration and calibration of a multiclass model’s maximum confidence are related but different evaluations. Fix the discussion here to binary positive probabilities. One individual outcome cannot establish whether 0.8 was an appropriate frequency estimate; groups of observations are required. State the population and observation period to which probabilities refer. High predicted probability is not itself good calibration, and a negative outcome for one high-probability case does not alone establish miscalibration.
Compute a small discrepancy
Suppose ten cases each receive probability 0.8 and six are actually positive. Mean predicted probability is 0.8, observed positive frequency is 0.6, and their difference is 0.2. This describes a small sample, not a proven population-wide discrepancy of exactly 0.2. A threshold 0.5 predicts every case positive and achieves sixty-percent accuracy, but that accuracy does not explain the meaning of 0.8. Changing the probabilities to 0.6 preserves this group’s ordering and thresholded predictions while changing agreement with its observed frequency. The example separates probability interpretation from a single classification score.
Inspect reliability groups
A reliability diagram groups predictions into bins and compares each bin’s mean probability with its observed positive frequency. Include counts: ten cases and one thousand cases have different sampling uncertainty. Boundaries and bin counts influence the result, so a single aggregate expected calibration error cannot tell the whole story. Record weighting, empty-bin handling, and which probability is evaluated. A pooled curve can also hide discrepancies for a subgroup. Examine relevant groups and operational conditions rather than assuming an attractive overall diagram guarantees equally reliable probabilities for everyone.
Use probability-sensitive scores
Binary Brier score is the mean of(p−y) squared. In the ten-case example, six positives contribute 0.04 each and four negatives contribute 0.64 each, giving 0.28. Predicting 0.6 for all cases gives 0.24. Brier score and log loss evaluate probability values, but they reflect more than calibration alone, including information useful for discrimination. Do not attribute every improvement in one such score solely to calibration. Confidently wrong endpoint probabilities can be severely penalized by log loss. If numerical clipping is used, include its rule in the reproducible evaluation procedure.
Fit calibration without leakage
Temperature scaling, sigmoid mappings, and isotonic methods learn a transformation using observed outcomes. Use appropriately separated data or cross-validation, and do not call data used to fit calibration an untouched final test. A shared positive temperature generally leaves the argmax unchanged while modifying probabilities, so classification accuracy alone cannot assess its intended benefit. Decide which probability behavior matters before fitting a correction. Distribution shifts at deployment can invalidate a previously useful mapping. Evaluate the transformation under the operational population rather than assuming that a fitted calibrator remains valid indefinitely.
Connect probabilities to decisions
Operational decisions also require error costs, populations, thresholds, and capacity constraints. A constant predictor equal to the overall positive rate may be well calibrated in aggregate yet provide little case-level discrimination. Recalculate the example’s Brier scores and explain why probabilities 0.8 and 0.6 can yield the same thresholded predictions but different probability-sensitive scores. Completion means separating frequency agreement, ranking, and action selection, while avoiding certainty from tiny bins. Return to the logit article if the transformation from model scores to probabilities is unclear before assessing whether those probabilities are empirically useful.