Regularization

You run an ice-cream stand. From ten recorded days, a computer builds a rule to predict sales from temperature. You let it bend a curve until it hits every observation. Unfortunately, between the recorded temperatures, the curve jumps sharply. You want to predict tomorrow’s stock needs, and reducing errors in the notebook alone cannot tell you which bends describe demand and which describe a chance event on one day.

You can change the rules of learning: fitting the data still matters, but not every improvement in the notebook is worth extra flexibility. Regularization is a family of changes to the learning process intended to reduce errors on new data, even at the cost of a worse fit to the data used for learning (Goodfellow, Bengio and Courville, Deep Learning, §5.2.2). This is the method’s aim, not a guarantee of improvement.

Change what the rule is rewarded for

In our example, coefficients describe the curve: numbers that determine its shape. During fitting, you can add a penalty for large values of these numbers: square each one and add the results. Squaring means large negative values cost something too. With an appropriately chosen way of representing the curve, this penalty restrains sharp bends.

We then choose the smaller value of “error on recorded days + penalty strength × penalty”, rather than error alone. This change to the learning objective is described in Deep Learning, §7.1, equation 7.1.

Consider two illustrative candidates: curve A has an error score of 0 and a penalty of 100; curve B has an error of 4 and a penalty of 4. With a penalty strength of 0.1, their total costs are 0 + 0.1 × 100 = 10 and 4 + 0.1 × 4 = 4.4. B wins. These numbers are an invented example of comparing costs, not sales measurements or results from the demonstration. We know why B was chosen; we do not yet know whether it will predict tomorrow better.

Check whether the restraint helps

Too strong a penalty may also flatten a genuine increase in demand with temperature. You choose its strength using a Validation Set: separate days used to select settings, rather than fit the coefficients. The assumption about the desirable shape of the rule may be wrong.

• Overfitting describes excessive fitting to past days. In its experiment, select a flexible curve, reveal new days and increase the penalty: compare both errors. Also check when further increases start to hurt. The demonstration fits an actual curve to simulated data; it does not train a language model.
• Weight decay pulls weights, the numbers controlling a model’s calculations, towards zero. It is one of the techniques used for this purpose; Regularization is broader than a single penalty or weight-shrinking rule.

The original Transformer also used other techniques from this family (Vaswani et al., Attention Is All You Need, §5.4). The ice-cream example explains one form of the mechanism, not a recipe for every large language model (LLM).

The text and illustration were prepared with AI assistance. The cost comparison is our own simplified example.