When a 0.976 ROC-AUC means nothing

AssayVault ships a risk model that scores 19,016 synthetic formulation-condition-attribute units for the probability of an out-of-specification event. It reports a ROC-AUC of 0.976. I designed the validation, but currently I still would not lend that number any weight about real stability behavior.

Three pieces of vocabulary carry that sentence. A unit is one combination of formulation, lot, storage condition and quality attribute, so a formulation with six lots on stability contributes six rows per condition and attribute. ROC-AUC is the area under a curve that scores how well a model ranks. The validation is out-of-fold, meaning every prediction came from a model fitted without the row it is predicting. The words every model note assumes takes a closer look at those essential terms.

Validation across groups

The model is a deliberately regularized logistic regression that mostly sees what is knowable at study design time: the recipe, the storage condition, the attribute and the starting value. It never sees the out-of-specification outcome or anything derived directly from it.

One family of features deserves to be named rather than glossed over. The accelerated-kinetics terms are Arrhenius aggregates fitted across a lot’s own storage arms, so they do carry information from the observed trajectory. The raw per-series slopes sit on an explicit leakage denylist; their aggregates do not, which is a judgement call rather than a clean separation. Their individual separation is weak, and the score is driven by storage temperature and attribute instead, but calling the feature set free of hindsight would overstate it.

Validation is out-of-fold with a GroupKFold approach over molecules. As a result, every prediction for a molecule comes from a model that never saw that molecule. With random splits, rows from the same molecule land on both sides and the model gets graded on memorizing its own neighbors. Grouped splits are the difference between “predicts unseen molecules” and “recognizes molecules it was trained on”, only the first claim holds up to scrutiny.

What the number is a property of

The reference build runs on synthetic data. So the honest reading goes like this: 0.976 measures how learnable the simulated degradation process is. It says nothing about real formulations, because the generator’s regularities are not chemistry’s regularities.

A high score in a flight simulator proves skill at flying the simulator. Whether it transfers to a real cockpit depends entirely on how faithfully the simulator models reality, a kind of fidelity which has not been measured yet. Therefore, the headline AUC is a simulator score and it is labeled as one. The model card states this; the demo states this. The metric is a property of the data-generating process, and pretending otherwise would be the most sophisticated form of leakage there is.

None of this makes the model pointless. The pipeline, the leakage guards, and the grouped validation are the deliverable: plug in real history and the same machinery produces a number you may actually believe. Building the machinery honestly on synthetic data is precisely what makes the future number trustworthy.

A ROC-AUC hides a second thing as well. Under a rare outcome it can look excellent while the flagged list is mostly false alarms, which is the subject of When two percent of events are the ones that matter.