Validation Set

You run an ice cream stand and want to predict daily sales from the temperature. You have three recorded days: at 20°C you sold 20 servings, at 22°C, 30, and at 24°C, 28. You compare three rules: always predict 24 servings, a straight line, and a curve passing exactly through all three results. The last rule makes no errors in your notebook. Should it plan tomorrow’s stock?

Checking those same days again cannot settle this. The flexible rule may also have fitted chance events, such as an unusually large tour group at 22°C. Before choosing a rule, you set other days aside to compare candidates. Validation Set is a separate portion of data used to select a model or its settings, rather than fit its learned numbers. This separation is described by Goodfellow, Bengio and Courville, Deep Learning, §5.3.

A notebook for learning, separate days for selection

We establish the rules and their numbers using only the training data. The line predicts twice the temperature minus 20. The curve is an arc fitted to the three notebook points. Mean absolute error here means: calculate how many servings each day’s prediction misses by, ignore the sign and average the differences.

On the days set aside, at 21°C, 23°C and 25°C, sales were 23, 25 and 31 servings respectively. The curve predicts 26.5, 30.5 and 22.5; its mean error is about 5.83 servings. The line predicts 22, 26 and 30: its error is 1 serving. This leads us to choose the line, although it lost 2 to 0 in the notebook. The Validation Set did not improve any rule; it helped choose among three. These data are invented, and fractional predictions mean expected sales, not splitting one serving.

After selection, a final assessment is still needed

The days used for selection have already influenced the decision. A third, disjoint portion of data, the test set, is therefore used only to assess the chosen solution. In this example, the line misses by an average of 1 serving on two previously unrevealed days. If you change the rule after seeing that result, those days also begin to serve selection; they lose their role as an independent test (§5.3).

This is an important limitation: after many attempts, selection can fit the Validation Set itself and produce an overly optimistic result. Three days set aside here are only a teaching aid, not a reliable assessment of real sales. No fixed split percentage guarantees a good measurement.

• Overfitting describes excessive fitting to training examples; separate data help reveal its effect.
• Fine-tuning is further training of an existing model; a Validation Set can help select settings for that training.
• Cross-entropy is another way of measuring error, often used for text prediction; the separation of data roles remains the same.

The text and illustration were prepared with AI assistance. The data, rules and demonstration are our own simplified example; the illustration is a metaphor for selection and keeping a separate sample.