Your random split is lying to you

What happens when information the model should not have at prediction time sneaks into training? Data leakage. The result is a model that looks brilliant in validation and useless in production, it cheated on the exam and nobody noticed because the cheating was structural.

The three usual suspects

Target leakage: a feature secretly contains the answer, like scoring the risk of an out-of-specification event using measurements taken after the event. Temporal leakage: training on Tuesday’s data to predict Monday’s. Both are famous and reasonably easy to police at the feature level.

The third one hides in the split itself. Group leakage: your rows are not independent samples but come in families. Think of patients with many visits, customers with many orders, molecules with many formulation-condition series. Split randomly, and members of the same family land on both sides of the fence. The model does not learn the phenomenon; it learns to recognize the family, and the validation score rewards the memorization.

GroupKFold splits by family, not by row

To see why this matters, forget molecules for a moment. Picture a model that reads handwriting, trained on 10,000 scanned letters from 100 writers. Split the letters randomly into a training set and test set and every writer will end up on both sides of the split. In the test set the model keeps meeting handwriting it already studied during training, so of course it will score brilliantly. Hand it a letter from writer number 101 and watch the performance collapse. The underlying problem: our validation never measured reading, it measured recognizing familiar writers.

The same rows, split two ways. Held out at random, a validation row's own group is still represented in the training set, so the model has met that group before. Held out by group, a validation row belongs to a group the model has never seen.

GroupKFold fixes the split, not the model: you declare which rows belong together (all letters by one writer, all visits by one patient, all series for one molecule), the splitter deals out whole groups. How? The data is divided into several folds, every group lands entirely in one of them. Each fold then takes one turn as the test set, so a prediction only comes from a model that trained without a single row from the groups it is being tested on. The resulting score honestly answers the question most relevant for a deployed model: how does it perform on a family it has never met?

Out-of-fold ROC-AUC on 900 simulated rows from 60 groups, where part of each group's effect can only be memorised, never derived. The flexible model looks clearly better under the random split and lands at chance once whole groups are held out; the linear model, which cannot isolate individual groups, was never flattered as much and keeps most of its score.

In AssayVault, the family is the molecule. The OOS risk model scores 19,016 units of synthetic stability data, one per lot, storage condition and quality attribute. However, we should keep in mind that those units cluster under a much smaller number of molecules and that everything about a molecule correlates across its rows. Validation therefore runs GroupKFold over molecules: no molecule ever informs its own score. On the feature side the model sees the recipe, the storage condition, the attribute and the starting value, and never the outcome it predicts. The accelerated-kinetics features are the one place where the line is drawn by judgement rather than by rule, since they summarize a lot’s own storage arms.

The rule: before trusting any validation score, ask what the rows have in common. If the answer names an entity (a molecule, a patient, a customer), two choices present themselves: split by that entity or accept that your metric measures recognition instead of prediction.

The pattern transfers: clinical models split by patient, churn models by customer, manufacturing models by batch. Random splits are only honest when rows are strangers to each other. In real data, rows almost never are.

We have just seen that the grouping approach is one way a validation score may flatter a model. Tuning on the test set without noticing is another one worth knowing about.