Test Set

You run an ice cream stand and are choosing a rule for planning stock. You train several rules on recorded days, then compare their errors on separate days. A straight line wins: predicted sales equal twice the temperature minus 20. Its average error is one serving. Can you already say it will predict future days this well?

Not yet: those separate days helped you choose the winner. The result tells you how well it performed on the assessment used for selection. You need an evaluation that influenced neither learning nor that decision.

Test Set is a collection of examples set aside for the final evaluation of a fixed model. Neither its data nor its results are used to choose the model or its settings. This division of roles is described by Goodfellow, Bengio and Courville, Deep Learning, §5.3.

Freeze the choice first, then reveal the days

Before learning begins, you set aside two days that you will not use for any adjustments. After selecting the line, you check it on them:

Temperature	Actual sales	Line prediction	Absolute error
19°C	19 servings	18 servings	1 serving
26°C	31 servings	32 servings	1 serving

The average error is (1 + 1) / 2 = 1 serving. This evaluates the chosen rule on those two days. The test itself changed nothing in the rule. The data are invented, and two observations suffice here only to demonstrate the calculation.

What if, after seeing the result, you adjust the line to match these days too? You are starting to use them to develop the solution. Checking it on them again no longer gives an independent evaluation of the revised version. You need fresh, unused data. The textbook notes that a similar problem arises when solutions are tuned to the same public test over many years (§5.3).

What this result lets you say

The test should reflect the conditions in which the rule will be used. Evaluation on summer days in a city is insufficient for predicting winter sales by the sea. In the basic setting, we assume independent examples drawn from the same kind of conditions as the training data (§5.2). A copy of a notebook day does not become an independent example by moving to a different file.

A small collection gives an uncertain result: one unusual day can change the average substantially (§5.3.1). Test Set is a measurement on a particular sample, not a promise of behaviour in every situation.

• Validation Set is used to select the rule. Its demonstration lets you freeze the choice before revealing the final test with these same two days.
• Overfitting describes fitting the training data too closely; a separate test helps reveal the difference compared with new data.
• With Fine-tuning, meaning further training of an existing model, you need an evaluation unaffected by that training or the choice of settings.

The text and illustration were prepared with AI assistance. The sales example is an original simplification; the illustration is a metaphor for evaluating a fixed solution on separate material.