Study guides · Intermediate

Generalization, Overfitting, and Evaluation Metrics

Use an explicit confusion matrix to connect unseen-data evaluation, thresholds, precision and recall.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Ask about error on unseen data
  2. Split before fitting preprocessing
  3. Calculate precision and recall from counts
  4. Evaluate the threshold and deployment conditions
  5. Ask Logeia about this articleCheck your understanding

Ask about error on unseen data

Generalization asks whether a fitted rule remains useful on data not used to fit it. Falling training error accompanied by worsening validation error can indicate fitting accidental details of the training set. A mismatch between training and validation conditions can produce a similar gap, so investigate the data rather than naming every gap overfitting. Repeated records, shared users and time periods may create dependent observations. Specify which future population the evaluation is intended to represent. One score is interpretable only in relation to that data-generating and splitting procedure.

Split before fitting preprocessing

Computing preprocessing statistics on the whole dataset before splitting lets test information influence training. Feature selection and missing-value imputation can leak in the same way as standardization. A pipeline fitted only inside each training fold helps keep cross-validation honest. Time-ordered tasks may require chronological splits, while shared subjects may require grouped splits. Repeatedly choosing models with the same validation set also adapts decisions to that set. Reserve an appropriately isolated final test for the selected procedure, and do not mistake a reusable tuning score for an untouched estimate of deployment performance.

Calculate precision and recall from counts

Suppose 80 of 100 positive cases are detected, while 90 of 900 negative cases are incorrectly flagged. The counts are TP=80, FN=20, FP=90 and TN=810. Precision is 80 divided by 170, approximately 47 percent; recall is 80 percent; accuracy is 890 divided by 1000, or 89 percent. Different metrics emphasize different outcomes for the same predictions. When positives are rare, a majority-only predictor can achieve high accuracy without useful detection. Show the underlying counts so that false alarms and missed cases remain visible.

Evaluate the threshold and deployment conditions

Lowering a threshold on a fixed score typically flags more cases, increasing both detections and false alarms on that sample. The appropriate operating point depends on missed-case costs, false-alarm costs and available review capacity. Choosing the threshold on the test set and then reporting that same set’s performance creates optimistic evaluation. Calibration, subgroup error and distribution shift require further checks. An improved metric and a useful deployment are related but distinct claims. Evaluate the full decision procedure, including threshold selection and operational constraints, rather than only a model’s raw score ordering.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Prepare for AI systems · STEP 15

Choose a learning path