Fine-tuning

A trained model can process language, but you want it to classify company messages as “return”, “delivery”, or “payment”. General ability does not yet provide the way of working you expect. You prepare examples of messages with correct categories.

Fine-tuning is further training of an existing model or its adaptation parameters on new data. It changes the numbers controlling its behavior, so the adaptation can persist across later queries.

In the example, the model compares its predictions with specified categories, and training modifies the parameters. Pasting a few examples into a single prompt, the input text, is a different operation: it does not itself change weights.

You can train all weights or only small additions, such as LoRA. Fine-tuning does not guarantee improvement on every task; poor data can reinforce errors, and adapting to one domain can change other behavior. The starting point usually comes from Pre-training.

See also: Catastrophic Forgetting [Polski] — a substantial deterioration of previously learned abilities during further training.

Mechanism and details

In the original GPT, training on text was followed by training on labeled examples for specific tasks. The parameters of the model and the added output layer were optimized. This division is described in Improving Language Understanding by Generative Pre-Training, §3.2.

What actually changes

Full fine-tuning updates all parameters of the model being trained. It is also possible to train only selected parameters or small adaptation modules. For example, LoRA keeps the base weights frozen and trains matrices that describe their change; Hu et al., §2 and §4.1 provide the details. Freezing the base weights therefore does not rule out learning — you need to check whether other parameters are being optimized.

An original example: a dataset contains customer messages and the correct categories “return,” “delivery,” and “payment.” Training can adapt an existing model to this classification. Pasting a few such examples into an ordinary prompt changes its input but does not, by itself, update the parameters. The distinction between adaptation in context and fine-tuning is described by Brown et al., Language Models are Few-Shot Learners, §2.

Supervised fine-tuning uses examples of desired responses or labels. Without further qualification, the term “fine-tuning” does not specify a single objective function or the number of parameters being trained. The existing Order-consistency SFT is a particular way of training a scorer, rather than a general definition of this stage.

See also: Sequence Packing — arranging several text examples in one training input to reduce padding.