The penalty needs a ruler

Ridge regression adds a penalty on the sum of squared coefficients which is what keeps AssayVault’s risk model stable when its inputs move together. The penalty measures coefficients which carry units. One step down the road, routine preprocessing turns into part of the model.

Units decide who gets punished

Think about a risk model for a second. It sees a starting value in percent with numbers around 95. A storage temperature in degrees Celsius, from 5 to 40. An excipient concentration often sits in the millimolar range, from 0 to about 50. An acceleration factor will be expressed as a small ratio.

What follows: to shift the prediction by the same amount, the coefficient on a feature ranging over 90 units would have to be roughly a hundredth of the coefficient on a feature ranging over 1. Without employing any initial transformation step, squaring and summing those coefficients therefore penalizes features according to their measurement unit. Express the same concentration in molar rather than millimolar and the fitted model would change entirely even though the underlying chemistry did not.

It is the mistake of ranking two medicines purely by the number on the box and concluding that 500 mg of paracetamol must be a far stronger drug than 20 micrograms of vitamin D.

Standardization is essential

Subtracting the mean and dividing by the standard deviation puts every feature into units of its own spread. After doing so, the penalty applies comparably across features and the coefficients become comparable to each other. As an added benefit, this also makes them readable to a domain expert (see Why the boring model won).

However, standardization should not be considered as a magic bullet and there are a few things it does not do. It does not make a skewed feature symmetric which would call for a transformation such as a logarithm. Potency data is the standard example: ChEMBL publishes pChEMBL, the negative log of a molar potency, and SARVault derives the same kind of value for cell-line records that arrive without one. It also does not handle outliers which still contribute to stretching the very standard deviation they are measured against.

Preprocessing carries a subtle risk

There’s a catch: both the mean and standard deviation required for the standardization step have to come from somewhere. Computed over the whole dataset before splitting into a training set and test set, they would carry a summary of the held-out rows into training. The effect is usually small which is why it may survive unnoticed for long. Luckily, avoiding the catch is very straightforward: put the scaler and the model in one pipeline so that the scaler is fitted inside each training fold and then just applied to the held-out fold.

When none of this applies

Tree-based models split on thresholds. Any transformation that preserves order leaves every possible split unchanged, so scaling does nothing for them at all. The advice to always scale features relates to models where either a penalty or a distance is involved: most linear models, nearest-neighbor methods, anything evaluating vectors by cosine or Euclidean distance.

From this, we can derive a short test of whether scaling matters. If the objective contains a penalty or a distance, scaling is part of the model and belongs in the pipeline. If it contains neither, you may very well consider skipping it.