The words every model note assumes

The other notes in this category use words like feature, fold and held-out data as though everyone is familiar with them. Nothing here is specific to my projects and this note is meant to provide a brief overview on some key concepts.

Rows, features, targets

A row is one thing you observed. In AssayVault a row is one combination of formulation, lot, storage condition and quality attribute: this recipe, this lot, stored at 25 °C, measured for aggregation.

The features are the columns describing that thing, the inputs a model is allowed to look at: what is in the recipe, which temperature, what the value was at the start.

The target is the column you want predicted. Here it is whether that unit ended up outside its specification. While training a model, the target must never be among the features.

Everything a model can learn lives in the features, which is why How a recipe becomes a row is about that translation and nothing else. A feature that was never encoded cannot be learned, however good the model. Thus, both choosing and encoding features deserve deeper considerations.

Why training data gets split

Models are good at memorizing. Grade it on the same rows it learned from and you measure memory rather than skill. Imagine an exam you’ll eventually have to take is always created from a fixed set of 50 questions. Knowing each potential question will easily lead to achieving the best score. Now, consider an alternative scenario. Same fifty questions in the pool, but you only have 35 of them and the remaining questions will be the exam. Memorizing will not do anymore, you will need to develop subject matter expertise to perform well. The same principle applies to training a model, the rows are split. The model is fitted on the training set and graded on a test set it has never seen. A driving examiner who only ever tests on the route the learner practised finds out very little about their driving.

Folds and cross-validation

A single split has two drawbacks. It spends a chunk of scarce data on grading instead of learning. The score also depends on which rows happened to land in the test set.

k-fold cross-validation fixes both. Cut the rows into k equal parts, called folds. Each fold takes one turn as the test set while the other k-1 are used for training. After k rounds every row has been predicted exactly once, by a model that did not see it. Those are the out-of-fold predictions, and a score computed from them averages k grades rather than resting on a single lucky or unlucky split. Five folds is the usual default: each round trains on 80 percent of the rows and grades on the remaining 20. The downside: five times the required compute as you are training five independent models, each on a different training set.

One assumption sits underneath all of it: the rows are independent of each other. Whenever they are not, cross-validation overestimates the performance of your model which is the subject of Your random split is lying to you.

Validation and test do different jobs

Two different things happen on held-out data which is why those sets should not be merged:

• A validation set chooses something: which model, which setting, which threshold.
• A test set is used to report how well the chosen thing works.

Once a set has been used to choose it can no longer report honestly because the choice was steered toward it. Tuning on the test set without noticing is about how that happens while everyone follows the recipe correctly.

Coefficients are learned while hyperparameters are chosen

A coefficient (also called a parameter or a weight) is a number the model fitting procedure works out autonomously from the data: how strongly this feature shifts the prediction.

A hyperparameter is a number you set before fitting and it governs how the fitting behaves. A few examples: the learning rate, the strength of a ridge penalty, the number of trees in a forest, the depth of a tree.

The distinction matters: coefficients come from the training data, hyperparameters come from a person looking at scores. Every look conveys an opinion and spends a little of the honesty of those scores.

For the model-side vocabulary, from capacity and shrinkage to calibration and proper scoring rules, the companion glossary is What lasso, ridge, and boosting actually do.