What lasso, ridge, and boosting actually do

Why the boring model won argued for a regularized logistic regression over gradient boosting while leaning on a handful of terms that carry a lot of weight in one sentence each. This note unpacks them and tries to paint a picture that makes each one stick.

Capacity: how much a model can bend

Irrespective of the chosen model, it will sit somewhere on a scale of flexibility. A linear model draws a single straight line through the feature space. A deep tree ensemble can carve that space into thousands of small regions with as many individual answers.

Consider drawing a line through a cloud of scattered points. A ruler gives you one straight line: no matter how you draw it, it will miss individual points. However, the line describes a trend you can state in words. A flexible curve can pass through every point exactly. At this point, it has stopped describing the trend and started tracing the noise. High capacity is not better or worse in itself. It can only add value if there is enough data to tell trend and noise apart.

Fourteen noisy observations. The straight line misses individual points but states a trend; the flexible curve passes close to every point, including the noise. Both were fitted to the same data.

Logistic regression: a weighted sum followed by a squash

Logistic regression computes a weighted sum of the inputs and passes it through a sigmoidal function that maps any input number to an output between 0 and 1. The weighted sum lives in log-odds space, which is why the coefficients may be read as sentences: “this excipient shifts the log-odds of an out-of-specification event by +0.4” translates to “the odds multiply by roughly 1.5” under the assumption that all other parameters remain fixed.

That readability is the reason the model can be checked. A domain expert can look at a coefficient’s sign and say whether it agrees with what is known. When it does not, that disagreement may turn into a lead worth following.

Gradient boosting: think of a committee of proofreaders

Boosting builds its answer in rounds. The first shallow tree makes a crude prediction. The second tree is trained not on the original target but on what the first one got wrong, the third on what the first two still get wrong, and so on for hundreds of rounds. Each contribution will be scaled down by a learning rate so that no single tree dominates.

The proofreading analogy describes the approach nicely: a manuscript passes down a line of readers, each looking only for the mistakes missed by everyone before them. The result catches subtleties no single reader would, including interactions between features that nobody thought to specify. That mechanism could be considered boosting’s real superpower on tabular data.

However, unfortunately there is no free lunch and the costs follow from the same mechanism. Hundreds of trees have no readable summary, the ensemble can fit small groups closely enough to memorize them, the raw output is a score optimized for the ensemble’s own objective.

Regularization: paying for complexity

Regularization is an elegant approach which adds a penalty for large coefficients to whatever the model is already optimizing. The fit now has to trade accuracy against that penalty, a coefficient will only grow if the data really insists. Pulling coefficients toward zero this way is called shrinkage.

Lasso (L1) penalizes the sum of absolute values, and that geometry lets coefficients hit exactly zero. As a consequence, it also performs feature selection as a side effect of fitting which can be quite valuable when you think most of the current inputs might be irrelevant. Among correlated features, it tends to keep one and zero out the rest. The kept feature may fluctuate between refits which should be kept in mind.

Ridge (L2) penalizes the sum of squared coefficients. It shrinks everything toward zero without reaching it. In contrast to Lasso, all features will stay in the model albeit with reduced influence. With correlated features, ridge shares the weight among them rather than picking a winner which is exactly what you want when starting value, margin to specification, and accelerated kinetics all carry overlapping information.

Elastic net mixes both penalties, keeping lasso’s ability to drop features while retaining ridge’s stability across correlated groups.

Coefficient paths for three features as the penalty grows, features A and B strongly correlated. Lasso drives each coefficient to exactly zero and leaves it there; ridge shrinks all three smoothly and keeps every one of them.

Let’s look at regularization from a different perspective, thinking of a budget makes the distinction a bit more palpable. Lasso is a spending review: some line items are cut entirely so others get to keep their funding. Ridge is an across-the-board trim: nobody has to go but everyone receives a salary cut. In both cases the strength of the penalty is a knob, tuned by cross-validation rather than chosen by taste. However, choosing it carefully is a trap of its own. Due to the penalty measuring coefficients carrying the units of their features, standardizing those features is a part of the model. The penalty needs a ruler elucidates why this is more than a mere preprocessing detail.

Calibration: aligning probability and reality

A model can be considered calibrated when its stated probabilities match observed realities: among all cases it scores at 8%, close to 8% should actually occur. A reliability diagram shows this directly by bucketing held-out predictions and plotting the predicted rate against the observed one with the diagonal as the target.

Calibration is what separates a risk number a scientist can act on from a score that only ranks. It matters most when the number triggers a decision with downstream consequences.

Proper scoring rules are the reason the choice of loss function is not cosmetic. Log loss and the Brier score are strictly proper, meaning that in expectation they reach their optimum only when the model reports the probability it actually believes.

Logistic regression optimizes log loss, so calibration comes close to free. Only close, though: a penalty pulls predictions toward the base rate, class weights and resampling push them away from it, and a misspecified model can be calibrated on average while being wrong inside every subgroup. The reliability diagram is worth drawing rather than assuming.

Which is an argument for reporting the measurement alongside the ranking scores. Two numbers do it cheaply: the mean predicted probability against the observed rate, and a Brier score to catch what that average hides. AssayVault’s risk model reports 20.6% predicted against 20.3% observed, and it took exactly that comparison to notice that an earlier build was inflating every probability by a constant while its AUC stayed put.

A common claim is that tree ensembles simply optimize something else. That is true of random forests, which average votes, and of AdaBoost with its exponential loss, but not of gradient boosting, which minimizes log loss by default in every mainstream implementation. Its miscalibration comes from overfitting and from the stagewise fit pushing scores toward the extremes, not from a different objective. Either way the remedy is the same: a separate calibration step such as Platt scaling or isotonic regression fitted on held-out data, which can be tricky for a small dataset.

AUC ranks but does not calibrate

Worth stating plainly since the two are easily confused: ROC-AUC, the area under the ROC curve, is the probability that a randomly chosen positive case outscores a randomly chosen negative one. It says nothing about whether the scores match observed frequencies. Square every predicted probability and the AUC does not move, because the ranking does not move, while the calibration is now badly wrong. A model can rank cases well and still give badly misleading probabilities.

A reliability diagram over 12,000 simulated cases in ten quantile bins. Squaring every prediction keeps the ranking and the bin membership, so ROC-AUC and the observed frequencies stay identical, while every stated probability now understates the risk.

As a practical consequence one should report both while being explicit about which property a use case needs. Ranking a shortlist of formulations to test next requires discrimination. Telling someone the probability that a formulation will fail specification requires calibration.

Effective sample size: how much data you really have?

Row counts tend to overestimate the size of a dataset. When rows come in groups (repeated measurements of one molecule, several visits by one patient), the information overlaps heavily and the amount of independent evidence sits closer to the number of groups than to the number of rows. That is why grouped validation matters and why a high-capacity model might be the wrong tool for a table that initially looks large.

How to use the vocabulary: ask what the model has to deliver, is it a ranking or a number someone will act on? Coefficients a domain expert can challenge or the best possible fit? How many independent groups stand behind the training set? Those answers will usually assist in narrowing the choice before any model is fitted which then confirms or refutes the reasoning.