When two percent of events are the ones that matter

Out-of-specification events in a stability program are rare and that rarity breaks an initially intuitive metric.

Accuracy looks great by ignoring the phenomenon

At a two percent event rate, a model that predicts “no event, ever” will be 98 percent accurate and has learned nothing. Any metric that can be maximized by ignoring the rare class is the wrong metric for studying the rare class.

A screening test for a hypothetical disease affecting 1 in 100 patients makes the predicament uncomfortably clear: by declaring everyone healthy, it is right 99 times in 100 and useless exactly when it counts.

Two questions worth asking

Everything useful comes out of the confusion matrix, a four-way tally of what the model flagged against what actually happened. It reduces the complexity to two questions:

Precision: of the units the model flagged, how many really went out of specification? The cost of a false alarm.
Recall: of the units that really went out of specification, how many did the model flag? The cost of a miss.

Moving the decision threshold trades one for the other, no threshold maximizes both. Thus, reporting a single metric without saying which one was optimized hides part of the decision.

The precision-recall curve tells the truth

The ROC curve plots the true positive rate against the false positive rate. That false positive rate has all the negatives in its denominator. Under heavy imbalance, the denominator is enormous, so a couple of thousand false alarms move it by little more than a tenth. The curve can look excellent while the flagged list is mostly noise.

The precision-recall curve puts precision on the vertical axis for false alarms show up immediately as a drop. Its baseline is informative too: an entirely random model scores 0.5 on ROC whatever the balance, while on precision-recall it scores the event rate itself. At a two percent rate, a PR-AUC of 0.2 constitutes a tenfold improvement over chance.

The same simulated model on 20,000 units with a two percent event rate, scored two ways. The ROC curve reaches an AUC of 0.92, and the threshold that recovers 80% of the real events sits at a false positive rate of only 0.13. On the precision-recall curve that identical threshold shows what the flagged list looks like: 2,523 false alarms next to 320 real events, a precision of 0.11.

Picking a threshold is not a pure modelling decision

In a stability setting that employs any kind of predictive analytics, a missed risk may translate to a spicy surprise at the twelve-month pull. On the other hand, a false alarm may result in an extra condition or an extra pull point. Those costs are not symmetric and differ between programs and settings, plus they are not necessarily at the modeller’s discretion. The honest deliverable is a calibrated probability and a curve, followed by a conversation about where to cut.

An initial reflex worth resisting

Resampling the training set, whether by duplicating the rare class or synthesizing neighbours, changes the class balance the model believes in. The probabilities stop corresponding to reality, which matters a lot when a calibrated probability is the deliverable, and the ranking is no better for it.

Class weights deserve a warning of their own, because they look like the careful alternative and are not. Reweighting the loss moves the intercept exactly as duplicating rows does. On a simulated set with a 6.7% event rate, the plain fit predicts 6.7% on average, while both a balanced-weight fit and a 50/50 oversampled fit predict around 40%, with identical ranking in all three. What actually keeps the output interpretable is leaving the balance alone and moving the threshold instead. If you do reweight, the prior has to be corrected afterwards.

The rule: choose a metric that cannot be satisfied by ignoring the rare class, report precision and recall together with the threshold that produced them, and discuss that threshold with the person who pays for both kinds of error.

The companion note, When a 0.976 ROC-AUC means nothing, takes up the other half of the question and portrays a scenario where it is still wise to remain cautious.