Why the boring model won

AssayVault scores the probability that a given unit, meaning one lot of one recipe at one storage condition measured for one quality attribute, will produce an out-of-specification result. Nineteen thousand units, a tabular feature set, a binary outcome: sounds like textbook gradient boosting territory. However, the shipped model is a regularized logistic regression. This note attempts a breakdown of my reasoning process, because “I picked the simple one” only translates to a virtue if you can state why. The vocabulary underneath this note, from capacity to calibration, is unpacked in What lasso, ridge, and boosting actually do.

Row count does not equal sample size

The first correction to make is to the number 19,016: we need to consider that those units cluster under a much smaller number of molecules. Rows from the same molecule are anything but independent - same protein, same degradation chemistry, same handling. For learning purposes the effective sample size sits a lot closer to the molecule count than to the row count.

That perspective changes which models should be considered appropriate. While gradient boosting has enough capacity to carve up a few dozen molecules into memorized regions and call it a decision boundary, it would probably score beautifully in a random split and still fall apart on a molecule it had never seen. Under grouped validation (every prediction comes from a model that never met that molecule) the advantage of the flexible model largely evaporates and it is the grouped number that predicts deployment.

Regularization amounts to an opinion about the world

The deployed model is ridge-penalized, which is not a formality. The chosen penalty encodes a design perspective: many features contribute a little, no single feature should dominate, coefficients stay in a range you can read. With correlated inputs (starting value, margin to specification, accelerated kinetics) an unpenalized fit would distribute weights erratically, producing coefficients that flip sign between refits. Shrinkage buys stability which makes the coefficients interpretable at all.

Feature selection followed the same principle. The model sees the recipe, the storage condition, the attribute, the starting value and, where available, accelerated kinetics. It never sees the outcome it is predicting.

The kinetics terms are the honest exception. They are fitted across a lot’s own storage arms, so they summarize the observed trajectory rather than sitting strictly before it; the raw slopes are denylisted, their aggregates are not. Their contribution is small enough that the score does not rest on them, but the boundary there is a judgement rather than a wall, and it is worth saying so.

The model should produce a probability instead of a score

This requirement eliminates most alternatives as the deliverable is not a ranking of risky formulations but a number a formulation scientist should be able to act on: “given this composition, the chance of an OOS event before the first scheduled sampling is about 8%.” For that sentence to be honest, the model has to be calibrated - among all units scored at 8%, roughly 8% should actually fail.

Logistic regression optimizes log loss, a proper scoring rule, which means the objective is only at its optimum if the model reports the probability it actually believes. Calibration therefore comes close to free with one honest caveat: the penalty that buys the auditability tends to pull predictions toward the base rate. As a result, the fit tends to be a little under-confident.

Close to free is not free, so the property is measured rather than assumed. Across the scored rows the mean predicted risk is 20.6% against an observed rate of 20.3%, and every risk band’s observed rate falls inside its own predicted range. That number is reported next to the AUC precisely because the AUC cannot see it: an earlier build of this model carried class weights, which inflated every probability while leaving the ranking, and therefore both headline scores, untouched.

Auditability is a functional requirement

In a regulated context, a model that cannot explain itself creates work instead of saving it. A logistic model’s coefficients are a readable statement: this excipient combination shifts the log-odds this much, in this direction. A domain expert can check whether the direction agrees with what is known about the chemistry. If the observed direction aligns with the expectation, great. If it does not, disagreement is still informative as it may point at a data artifact or at something else worth investigating. Such a workflow would be impossible with a thousand trees.

One fundamental principle underlines all arguments: pick the simplest model whose failure modes you can name. Where the answer has to survive a question from an auditor, capacity that cannot be audited probably buys less than it costs.

None of this should be taken as an argument that boosting itself is bad. However, it should be emphasized that it answers a different question. Given a hundred molecules of real history instead of a synthetic cohort, the emerging picture could be very different. Thus, the AssayVault pipeline is built in such a fashion that swapping the estimator is a small change, the validation discipline around it turned out to be the harder part.

The penalty needs a ruler takes a closer look at why scaling of features should be considered as part of the model instead of a mere preprocessing detail.