Ask about error on unseen data
Generalization asks whether a fitted rule remains useful on data not used to fit it. Falling training error accompanied by worsening validation error can indicate fitting accidental details of the training set. A mismatch between training and validation conditions can produce a similar gap, so investigate the data rather than naming every gap overfitting. Repeated records, shared users and time periods may create dependent observations. Specify which future population the evaluation is intended to represent. One score is interpretable only in relation to that data-generating and splitting procedure.
Split before fitting preprocessing
Computing preprocessing statistics on the whole dataset before splitting lets test information influence training. Feature selection and missing-value imputation can leak in the same way as standardization. A pipeline fitted only inside each training fold helps keep cross-validation honest. Time-ordered tasks may require chronological splits, while shared subjects may require grouped splits. Repeatedly choosing models with the same validation set also adapts decisions to that set. Reserve an appropriately isolated final test for the selected procedure, and do not mistake a reusable tuning score for an untouched estimate of deployment performance.
Calculate precision and recall from counts
Suppose 80 of 100 positive cases are detected, while 90 of 900 negative cases are incorrectly flagged. The counts are TP=80, FN=20, FP=90 and TN=810. Precision is 80 divided by 170, approximately 47 percent; recall is 80 percent; accuracy is 890 divided by 1000, or 89 percent. Different metrics emphasize different outcomes for the same predictions. When positives are rare, a majority-only predictor can achieve high accuracy without useful detection. Show the underlying counts so that false alarms and missed cases remain visible.
Evaluate the threshold and deployment conditions
Lowering a threshold on a fixed score typically flags more cases, increasing both detections and false alarms on that sample. The appropriate operating point depends on missed-case costs, false-alarm costs and available review capacity. Choosing the threshold on the test set and then reporting that same set’s performance creates optimistic evaluation. Calibration, subgroup error and distribution shift require further checks. An improved metric and a useful deployment are related but distinct claims. Evaluate the full decision procedure, including threshold selection and operational constraints, rather than only a model’s raw score ordering.