Tuning on the test set without noticing

AssayVault’s risk model has a penalty strength to choose. It is a hyperparameter, a number set before fitting rather than learned from the data. The usual procedure is to try a range of values, score each by cross-validation, keep the best one, and report that best score. Such a standardized approach would be referred to as hyperparameter tuning.

However, applied to our case, the resulting estimate would stop being honest while nothing in the code looks wrong.

The maximum of noisy estimates is biased upward

Let’s start by taking a closer look at the problem. Each candidate’s cross-validated score is an estimate with noise in it. Taking the best of twenty such estimates does not give the true performance of the best candidate. Instead, it gives the largest of twenty draws, and the largest draw sits systematically above the truth, because a value that got lucky on those particular folds is exactly the kind of value that wins a maximum.

The folds have now been used to make a choice. Whatever they report afterwards no longer qualifies as a measurement on unseen data.

A more intuitive framing of the same problem: imagine taking a practice exam twenty times and reporting the best attempt as your expected grade. The number might be real but still won’t tell you much about how you will do in the upcoming exam.

Trying to understand the effect size

It grows with the number of things being tuned and with the size of the search. On the other hand, it shrinks with the amount of independent data. Two hyperparameters over thousands of independent rows: probably negligible. A grid search over a dataset whose effective sample size is the number of molecule groups rather than the number of rows: not negligible at all. If you want to take a closer look, this is the situation described in Your random split is lying to you.

Nested cross-validation puts the wall back

An outer loop splits the data into training and test folds. Inside each outer training set, an inner loop runs the entire tuning procedure and picks a hyperparameter. The chosen model is then scored once on the outer test fold, which took part in no decision at all.

Two ways to spend five folds. In the single loop, the candidates are chosen and scored on the same validation folds, so the reported score is a best-of-many. In the nested version, one outer fold is held back, the remaining four are split again for the choice, and the chosen model meets the outer fold exactly once.

Two properties are worth noticing. Every outer fold may select a different hyperparameter. This is not a defect: what is being evaluated is the procedure, not one fitted model. Secondly, the spread across outer folds is in itself a result, because a wide spread will tell us that the selection is unstable and a single number would have been misleading in either direction.

The computational cost is the reason it might get skipped, since the number of fits multiplies. Where that is unaffordable, a three-way split into training, validation and test, with the test set touched exactly once at the end, is weaker but still preferable.

Count every decision made by looking at the data

Hyperparameters are the obvious case and the smallest one. Selecting features by their correlation with the target, dropping an outlier because it hurt the score, picking a target transformation, choosing the model class, setting the threshold: each of those consumes information from the data it was chosen on.

The generalized rule: any choice informed by a score should sit inside the fold that score came from. If a decision cannot be moved inside, the number it produced describes the data you have rather than predicting the data you do not.

The companion note on how that model selection turned out is Why the boring model won.