Early Stopping

You run a bicycle repair shop and want to estimate repair costs from a short description of the fault. A model learns from old jobs whose actual costs you know. Over successive rounds of training, its average error on those jobs drops from 18 to 4 PLN. Will the last version give the best estimate for the next bike? Improvement on familiar jobs alone cannot tell you. The model may have adapted to quirks of past repairs.

So you set some jobs aside in a Validation Set. The model does not learn from them. They let you compare successive versions. At each round you check how many złoty, on average, the estimates differ from the actual costs. Early Stopping uses these checks to decide when to stop training. In the version below, you save the model with the lowest error so far on the held-out jobs and stop after a set number of checks without a new best. This rule appears in Deep Learning, §7.8, Algorithm 7.1.

The last round and the best saved version

Round	Error on training jobs	Error on held-out jobs
1	18 PLN	20 PLN
2	12 PLN	14 PLN
3	8 PLN	16 PLN
4	6 PLN	12 PLN
5	5 PLN	13 PLN
6	4 PLN	15 PLN

These are made-up results from one run. With a limit of two checks without improvement, you give the model another chance after round 3. In round 4 the error drops to 12 PLN, so you save that version and reset the counter. Rounds 5 and 6 fail to beat the record. You stop after round 6 and go back to the version saved in round 4.

Try a shorter limit

What happens if one check without improvement is enough to stop? Change the limit in the demo. It applies the rule to the numbers in the table.

With a limit of one check, training stops after round 3 and you keep the version from round 2. You miss the later improvement. Results can fluctuate in real training too. Before you stop the model, you cannot tell whether a better result is just around the corner. Prechelt shows one such run in §1.3, Figure 2.

Early Stopping is a form of Regularization, a change to training intended to help on new data. Limiting training time can reduce Overfitting, which here means adapting too closely to familiar jobs. Because you already used the held-out jobs to select a version, make the final assessment on a separate group of repairs called the Test Set.

The text and illustration were prepared with AI assistance. The table contains an original example.