Checking those same days again cannot settle this. The flexible rule may also have fitted chance events, such as an unusually large tour group at 22°C. Before choosing a rule, you set other days aside to compare candidates. Validation Set is a separate portion of data used to select a model or its settings, rather than fit its learned numbers. This separation is described by Goodfellow, Bengio and Courville, Deep Learning, §5.3.

A notebook for learning, separate days for selection

We establish the rules and their numbers using only the training data. The line predicts twice the temperature minus 20. The curve is an arc fitted to the three notebook points. Mean absolute error here means: calculate how many servings each day’s prediction misses by, ignore the sign and average the differences.

On the days set aside, at 21°C, 23°C and 25°C, sales were 23, 25 and 31 servings respectively. The curve predicts 26.5, 30.5 and 22.5; its mean error is about 5.83 servings. The line predicts 22, 26 and 30: its error is 1 serving. This leads us to choose the line, although it lost 2 to 0 in the notebook. The Validation Set did not improve any rule; it helped choose among three. These data are invented, and fractional predictions mean expected sales, not splitting one serving.

After selection, a final assessment is still needed

The days used for selection have already influenced the decision. A third, disjoint portion of data, the test set, is therefore used only to assess the chosen solution. In this example, the line misses by an average of 1 serving on two previously unrevealed days. If you change the rule after seeing that result, those days also begin to serve selection; they lose their role as an independent test (§5.3).

This is an important limitation: after many attempts, selection can fit the Validation Set itself and produce an overly optimistic result. Three days set aside here are only a teaching aid, not a reliable assessment of real sales. No fixed split percentage guarantees a good measurement.

• Overfitting describes excessive fitting to training examples; separate data help reveal its effect.
• Fine-tuning is further training of an existing model; a Validation Set can help select settings for that training.
• Cross-entropy is another way of measuring error, often used for text prediction; the separation of data roles remains the same.

The text and illustration were prepared with AI assistance. The data, rules and demonstration are our own simplified example; the illustration is a metaphor for selection and keeping a separate sample.

The simplest rule is a straight line: "each extra degree means a few more scoops". In the example from this note, it is typically off by 27.5 scoops on the recorded days, because it does not see that sales drop in the worst heat. So you can give the rule more freedom and let the curve bend. The most flexible of the curves tried here passes exactly through all 10 points. Its error on the days from the notebook is 0.

It looks like a perfect rule. But the days in the notebook are already over, and the rule has to work tomorrow. On 400 new days from the same stand, the "perfect" curve is typically off by 236.5 scoops. A moderately flexible curve, which did worse on the notebook (10.9), is off by 16.8 on the new days.

Overfitting is a situation in which a model has a small error on the data used for learning and, compared with it, too large an error on new data of the same kind (Goodfellow, Bengio and Courville, Deep Learning, §5.2). The model has fitted itself to features of its own data set that do not repeat outside it. The opposite problem is called underfitting: the rule is too simple to describe well even the data it has seen. That is how the straight line in our example behaves.

Where the difference comes from

Every entry in the notebook is made of two things: a pattern (it is warmer, so more ice cream is sold) and chance (a tour group happened to arrive). The rule should capture the first and leave out the second. But the model does not know which is which. It sees only points.

How much chance the model absorbs depends on its flexibility, which the literature calls capacity: the ability to fit many different shapes. The curve in the example is described by 2 to 10 numbers that are chosen to match the data; how many there are is the degree of the curve plus one. A curve of degree 9 has 10 such numbers for 10 days, so it can hit every point, together with all the chance. The authors of the textbook describe it like this: a very flexible model can represent the right relationship, but also a great many others that fit the training data just as well, and there is little chance that it will pick exactly the one that holds up later (§5.2).

The typical course is as follows: as flexibility grows, the error on the training data falls, while the error on new data first falls and then starts to rise (§5.2, figure 5.3). A practical rule follows from this. A model is not judged on the data it learned from, and flexibility is not chosen by the error on that data, because that error will always point to the highest one. Part of the data is set aside: a validation set is used to choose the settings, and a separate test set is used for the final assessment (§5.3).

Try it yourself

Below is the same stand. All the numbers from this note can be reproduced: after clicking "Start over" you see 10 days, and experiments 1–5 set up the cases described.

It is worth doing two things that a static chart does not show. First, turn on "24 other notebooks" and change the degree: at a low degree the gray lines stay close together, at a high degree they spread out in all directions, even though each one comes from the same stand. Second, drag one point high up. The curve of degree 2 will move only a little, while the curve of degree 9 will reshape itself along its whole length to hit it.

What helps

• More data. With 80 days, the same curve of degree 9 is typically off by 14.8 scoops on the notebook and by 15.9 on new days. The difference almost disappears, because the curve can no longer hit every point (§5.2).
• Less flexibility. A curve of degree 3 on 10 days: 10.9 on the notebook, 16.8 on new days.
• Regularization, that is, a change to the way of learning that is meant to reduce the error on new data, not on the training data (§5.2.2). One example is a penalty for large coefficients, related to Weight decay: a curve of degree 9 with a penalty of λ = 0.01 is off by 5.8 on the notebook and by 21.2 on new days, instead of by 236.5.

None of these settings can be chosen by looking only at the error on the training data.

Limitations

• This is my own example with a single input quantity and simulated data. Chance alone gives a typical error of 15 scoops here, and no rule will go lower on new days.
• The "first better, then worse" shape is typical, not guaranteed. The authors note that in deep networks it is hard even to determine the model's actual flexibility, because it also depends on the learning algorithm (§5.2). So the number of parameters of an LLM does not tell you by itself whether the model is overfitted.
• A small error on new data applies to data from the same source. If the stand moves to the seaside, the notebook from the city promises nothing.

Related notes

• Weight decay limits the size of the weights and is one of the ways to reduce the difference between the error on training data and on new data.
• Grokking describes cases in which a model that looks overfitted starts to work well on new data after long further training.
• Fine-tuning on a small data set needs the same check: separate data that the model did not see while learning.
• Catastrophic Forgetting [Polski] is a different problem: there the model loses an old skill after new learning, while here it carries over poorly beyond its own training data from the start.

For the 50% variant, we collect the agent's successful and failed attempts on tasks with different human times. We fit a curve of the chance of success and then read off the time at which it crosses 50%. This is a prediction for the distribution of tasks of a given length; it does not mean that each of them has the same chance of being completed. METR: definition and methodology.

A teaching example: if the fitted curve reaches 50% at two hours, then two hours is the horizon. The agent may finish a successful task much faster than that. Nor does the result by itself say how successful the agent will be on a task from another domain.

Reading off the horizon requires a meaningful relationship between human time and difficulty for AI. When the curve is flat, a small change in the chance of success can move its crossing with the threshold a long way. The article on converting the METR horizon shows this problem with an example you can work through and discusses a new fit to the METR data. Nguyen and Fithian, §2–3.

When comparing horizons, keep the information about the task set, the agent's environment, the attempt budget and the chosen success threshold.

Suppose two agents work on the same three tasks. The first usually copes with A and B, the second with C as well. The success rate alone gives a ranking. IRT additionally tries to estimate how hard C is relative to the other tasks and what difference in ability explains the pattern of responses. This is a teaching example; a table this small is not enough for a reliable fit.

In the simple Rasch model, the chance of success rises as the agent's ability grows relative to the difficulty of the task. Both quantities are latent: we infer them from the results instead of equating difficulty straight away with the length of the instructions or with human time. Different variants of IRT add further parameters and assumptions.

In the analysis of METR horizons, the authors link latent difficulty to human working time through a learned curve. They add task-family effects and variation in the results of repeated runs. This makes it possible to check over which range time really separates tasks that are hard for AI from those that are easy. Nguyen and Fithian, §2.3 and §3.1.

Predicting responses well does not prove that the learned parameter covers every kind of ability an agent has. When interpreting it, check the model's assumptions and the results on tasks not used in the fit.

You could train the model on complaints with no restrictions at all. Then the weights shift wherever that helps the new task, including the ones that recognizing delivery and returns depended on. You could also penalize every change equally, but then the model clings to its old settings so tightly that it learns the new task poorly. What we need is an answer to the question: which weights may be moved, and which are better left alone?

Elastic Weight Consolidation (EWC) adds a penalty to training on the new task for weights moving away from their values after the old task, and the penalty is larger the more important a given weight was for the old task. The authors compare this to a spring attached to each weight: stiff for important weights, loose for the rest (Kirkpatrick et al., Overcoming catastrophic forgetting in neural networks, §2).

Two weights, one new task

Take just two weights. After training on delivery and returns, both have the value 1. The first is very important for the old departments (importance 10), the second almost irrelevant (importance 0.1). The new task requires only that the two weights add up to 4. Many settings satisfy this: (2, 2), but also (1, 3).

• Without protection, training moves both weights equally, to (2, 2). The new task is mastered, but the important weight has drifted by 1 and the loss on the old task rises from 0 to 5.05.
• A uniform penalty on both weights stops them halfway, at (1.67, 1.67). The old task suffers less, but the sum is 3.33 instead of 4.
• EWC shifts almost all of the movement onto the unimportant weight: (1.02, 2.80). The sum is 3.82, and the loss on the old task is only 0.16.

Loss is a number that says how wrong the model is; the smaller, the better. In the experiment below you can change the penalty strength λ and the kind of protection. The result is computed exactly for this two-weight example; it is not a simulation of a real network.

How we know which weight is important

In a real network, nobody supplies the importance in advance. It is estimated once the old task is finished, while its data are still available: you check how strongly a tiny change in a given weight changes the model's answers on the old examples. The paper uses the diagonal of the Fisher information matrix for this, that is, one importance number per weight. After that the data can be deleted; what remains are the old weight values and their importances.

During the new training, the sum of two terms is minimized (equation 3 in the paper):

$L_B$ is the loss on the new task, $\theta_i$ is the current value of weight number $i$, $\theta^*_{A,i}$ is its value after the old task, $F_i$ is its importance, and $\lambda$ says how much the old task counts relative to the new one. In our example $F = (10,\ 0.1)$ and $\lambda = 1$.

Limits of the method

The importance is an approximation computed at a single point, separately for each weight. The authors themselves show that their estimate can be overconfident about which weights are unimportant, and they call this the main weakness of the method (§2.2 and §3). In their experiments with Atari games, an agent with EWC learned many games one after another, but did not match separate networks trained for each game. The results concern digit recognition and games, not large language models. Our two weights and customer messages are our own simplification.

• Catastrophic Forgetting [Polski] is a substantial deterioration of earlier skills after new training; EWC tries to reduce it without access to the old data.
• Experience Replay solves the same problem differently: it stores some of the earlier examples instead of weight importances.
• Continual Learning sets the goal, that is, learning successive tasks while retaining the earlier ones; EWC is one of the methods.
• Fine-tuning is further training of a ready-made model; EWC changes its objective by adding a penalty for moving important weights.

The text and illustration were prepared with AI assistance. The two weights, their importances and the demonstration are our own simplified educational example; the illustration is a metaphor, not a diagram of the model's architecture.

Keep the first four? Then the memory knows only the beginning. Always hold the last four? Then it knows only the end. A fair draw of “four out of the whole” would require storing everything or knowing the number of messages in advance. The question is: with a fixed number of slots and a stream of unknown length, how do you give every message the same chance?

Reservoir Sampling is a way of maintaining a random sample of fixed size during a single pass over data whose count we do not know in advance. After each new message, the memory, called a reservoir here, contains a fair random sample of everything that has arrived so far. The simplest version, Algorithm R, is described by Vitter, Random Sampling with a Reservoir, §2, who attributes it to Alan Waterman.

The recipe in two steps

1. The first four messages go into memory without any draw.
2. Message number $t$ (the fifth, the sixth and every later one) gets into memory with chance $4/t$. If it gets in, it replaces one of the four stored messages, chosen at random.

So the fifth message gets in with chance 4/5, the hundredth with chance 4/100. New messages have a smaller and smaller chance of getting in, but the old ones have already been exposed to being thrown out many times, and these two effects balance exactly. Let us check this after the fifth message. It gets in with chance 4/5. The first message drops out only if the fifth got in (4/5) and landed on exactly its slot (1/4), that is, with chance 1/5. It stays with chance 4/5: the same as the fifth has. After $t$ messages, each of them is in memory with chance $4/t$.

An equal chance is not a guarantee

In the experiment, the stream has 20 messages. Eighteen are about delivery, and two, the ninth and the fifteenth, are about returns. Each run sends such a stream through 2000 times and counts how often a given position stayed in memory at the end. The draws are real, so the results differ by a fraction of a percent each time.

With Reservoir Sampling, every bar is at about 20%, that is, 4/20. At the same time, in most runs neither of the two returns messages stays in memory. The exact value is 3060/4845, about 63%: that is how many of all the foursomes chosen from 20 messages consist solely of the eighteen delivery messages. The method is fair to individual messages, but it does not make sure that every category has a representative.

Where this matters

In training with replay, Reservoir Sampling is sometimes the way of writing to the buffer: you do not need to know the length of the stream or the boundaries between tasks. Chaudhry et al., Continual Learning with Tiny Episodic Memories, §2, Algorithm 2 use it in exactly this way. In their experiments on image tasks it worked well, except with very small memories, where earlier classes disappeared from the buffer; strategies that ensure equal representation of classes did better then (§4.4). This is a result for image classification, not for LLMs.

The algorithm itself is not about machine learning. Vitter's 1985 paper describes sampling records from a magnetic tape of unknown length and proposes faster variants which, instead of drawing at every record, compute how many records to skip. The customer messages and the 20-position stream are our own example.

• Experience Replay needs a buffer of earlier examples; Reservoir Sampling is one way of deciding what goes into it.
• Continual Learning assumes learning from a stream of data with limited memory, that is, a situation in which not everything can be kept.
• Catastrophic Forgetting [Polski] can get worse when all the examples of an earlier category drop out of a small memory.

The text and illustration were prepared with AI assistance. The message stream and the demonstration are our own simplified educational example; the illustration is a metaphor, not a diagram of the algorithm.

Experience Replay (ER) means storing some earlier examples and reusing them for training. In a simple version, we mix examples from memory into a batch of new data. This memory is called a replay buffer: a container for practice data, not a record of everything the model knows. This joint training is described by Chaudhry et al., Continual Learning with Tiny Episodic Memories, §2, Algorithm 1.

What goes into the next exercise

Before losing the archive, we saved four messages along with their correct departments. We now have four new complaints. Without replay, the model trains on four new messages. With replay, it also practices on four old ones: it receives another signal that “Where is my parcel?” belongs to delivery. In this small example we use the entire buffer; larger buffers usually supply a sampled batch.

Counting memory slots is not enough, however. Four saved delivery questions contain no returns example at all. Replacing two of them with “How do I send my purchase back?” and “I want to return a shirt” leaves the number of replayed examples unchanged, but covers both earlier departments. The experiment below counts examples in a data batch; it does not predict model accuracy.

Replay can help only with available data

The model gets an opportunity to practice earlier answers, not a guarantee that it will retain every skill. A small random sample can entirely miss a rare category; researchers observed this problem with very small buffers (§4.4 of the same paper). Our messages are our own example, not a result of those experiments. The effect of training must be checked on separate old and new messages that were not used for training. The buffer also consumes memory and requires retaining selected data.

See also: Reservoir Sampling — a way of deciding which examples from a stream go into a fixed-size buffer while giving each one an equal chance.

In games, in Reinforcement Learning — learning actions from rewards — the stored record includes a situation, an action, a reward and the next situation. Sampling such records reduces dependence on events that occur immediately one after another (Mnih et al., Human-level control through deep reinforcement learning, Methods, “Training algorithm for deep Q-networks”). These are different data from a message with its correct department; Atari results should not be transferred directly to LLMs.

• Continual Learning sets the goal of further learning while retaining earlier abilities; ER is one way to pursue it.
• Catastrophic Forgetting [Polski] describes a substantial deterioration of earlier skills, which replay may reduce.
• Fine-tuning is further training of a model; replay changes the selection of its training data. Simply pasting old messages into a conversation is not this kind of model update.

The text and illustration were prepared with AI assistance. The messages and demonstration are our own simplified educational example; the illustration is a metaphor, not a diagram of the model's architecture.

You need to teach the model new things while preserving earlier abilities. This is the goal of Continual Learning: learning from data or tasks that arrive in sequence, while taking account of what the model has already learned. The name describes a problem and a family of approaches; it does not guarantee that forgetting will disappear. Kirkpatrick et al., Overcoming catastrophic forgetting in neural networks, §1 describe this need.

What changes in training and evaluation

Suppose the full archive of examples used for earlier training is no longer available. Before deleting it, we saved a small sample: 20 messages with their known correct departments. One possible method mixes this sample into the new complaint examples. Further training then also receives a signal to keep handling “Where is my parcel?” correctly. Reusing earlier examples is one of the strategies described by van de Ven and Tolias, Three scenarios for continual learning, §3.3.

See also: Experience Replay — storing a small sample of earlier examples and mixing it into new training.

The evaluation team keeps a separate test of old messages, which we do not use for training. After each update, we check both old and new categories. The numbers below are an invented example, not research results; each test contains 100 messages.

The last variant preserves the earlier ability better, although it recognizes complaints less well than the second one. The table shows what to compare; it does not promise this result from adding 20 examples.

Conditions change the difficulty

In our example, the model chooses among all the departments it has learned. A hint such as “this message belongs to the new category” would make the test easier. Whether such hints and task boundaries are available matters when comparing methods (van de Ven and Tolias, §2–2.3). A small sample may miss rare questions; a good average does not prove that every ability has been preserved.

• Catastrophic Forgetting [Polski] describes a substantial deterioration in old abilities; its demonstration shows how new training changes shared settings.
• Fine-tuning means training a model further on selected examples. After successive training rounds, we also check its earlier abilities.
• Knowledge Distillation can pass guidance from an older model to the one being updated; this is a different strategy from keeping the original messages (§3.3 of van de Ven and Tolias's paper).

The text and illustration were prepared with AI assistance; the messages, sample size and table results are our own educational example. The illustration shows the goal of preserving earlier abilities, not the model's structure.

Sequence Packing arranges several examples in one input sequence to reduce padding during training. In our example, all three fit into one eight-position row: seven slots with data, one empty. No new answers were created; their arrangement changed. The TRL 0.29.0 documentation, SFT Trainer, “Packing” describes this role.

Eight slots are our own simplification, rather than TRL's default row size. An ordinary batch of examples may be padded only to its longest text; grouping similar lengths also reduces wasted space. Attention Is All You Need, §5.1, already describes this grouping. Empty slot count alone does not determine speedup: that also depends on how computations are executed.

A shared row need not mean a shared conversation

You want “No” to remain an independent example, rather than a continuation of “Thank you”. The end marker itself is only a token. Ordinary blocking of future access — Causal Masking — still lets later text read earlier text in the same row.

To preserve independence, computation must know example boundaries and block access across them. Error evaluation must be controlled separately: do not teach prediction of the first token of “No” from the previous text's ending. Choosing assessed positions is Loss masking; omitting a position's penalty does not block reading it.

Specifically, in TRL 0.29.0 code: DataCollatorForLanguageModeling and BFD packing configuration, example lengths reconstruct boundaries and beginnings are excluded from error evaluation. This path requires a compatible attention implementation — the mechanism for using other text positions; the code warns of information leakage in unsupported configurations. This is a requirement of the specific solution, rather than a property of all packing.

Check two separate things

Enable packing, then disable access isolation. The space saving remains, but B1 can use the earlier example A. Squares represent end tokens; A1, B1, and so on are illustrative text pieces. The demonstration shows allowed connections without simulating model responses. The “full rows” variant reveals no savings when every example already fills eight slots.

Strategies differ in how they handle long texts. In TRL 0.29.0, the pack_dataset function, bfd fits whole examples into free slots but truncates the excess of an overlong example; wrapped can split text in the middle. Both retained data and boundaries must therefore be checked. Packing organizes training inputs, including Fine-tuning — further model adaptation using examples. It does not increase the maximum text length the model supports.

Reinforcement Learning (RL) learns a strategy for acting from experience and numerical assessments of actions' consequences. The goal is to increase expected total reward: the result a strategy produces on average across many attempts. A reward need not arrive immediately after a decision. The official Spinning Up, Part 1, “Key Concepts and Terminology”, “Reward and Return”, and “The RL Problem” describes the problem this way.

What the program learns from a game

In our game, the agent is the program choosing a route, and the environment is the game that executes the move and awards coins. The program receives an observation, the available information about the situation, such as its board position. Its policy is its rule for choosing actions; it can also specify probabilities for different moves.

After attempt A, the program records “2, then 0”. Attempt B gives “0, then 5”. The learning algorithm uses this experience to improve the policy, for example choosing B more often at this fork. This describes the goal, rather than one universal update recipe. The result 5 must be connected to the earlier choice of B. If the program always chooses A, it collects no information about B; learning therefore often requires trying less familiar actions too.

The invented game has fixed results and two turns. In a more difficult environment, the same move can have different consequences, so the expected result matters and one success does not prove a good strategy has been found. Simply comparing two known sums is not yet learning either.

In Behavioral cloning, the program learns to imitate actions from examples. In RL, assessments of consequences guide learning. In language models, an action can be generating a response, and a reward can assess its correctness or usefulness. RLHF [Polski] uses information from people for this purpose.

An important boundary: RL improves performance according to the chosen reward. If points reward bypassing rules rather than doing the task, the agent can learn Reward hacking [Polski]. More points need not mean the behavior we actually wanted.

Pick a route, then check the second turn

In this experiment you compare the objective of two strategies; you do not train an agent. Choose a route and reveal the later result. The slider changes the importance of the future reward, rather than the number of coins collected.

We calculate the return, the result for the entire attempt: first reward + γ × second reward. The symbol γ (gamma) weights the future reward. At γ = 1, both turns count fully: A gives 2, B gives 5. At γ = 0.2, we obtain A = 2, B = 1. This is a different evaluation objective for the same game. Boundary values γ = 0 and 1 are allowed here because the attempt ends after two turns; this is not an infinite reward sequence. Formalism source: Spinning Up, “Reward and Return”.

Loss masking determines which positions contribute to the loss, the numerical measure of error during learning. A token — a piece of text — may remain visible in the input even though the model is not directly assessed for predicting it.

When training only on the agent's responses, the user's text and the tool's result help it choose the next step, but receive no separate prediction penalty. The model therefore does not need to learn to generate every part of the entire record.

This differs from Causal Masking, which restricts information flow between positions. Having no loss term for a token need not mean having no effect on training: its computations may influence later positions that are assessed. “Don't Mask the Environment”, §2, examines the consequences of choosing which parts to assess.

See also: Sequence Packing shows why combining examples requires separate control over access to context and the positions assessed by the loss.

pass@k measures the chance that at least one of k attempts at a task passes a specified check. Results are summarized across tasks. The number k is part of the metric's meaning because it determines the attempt budget.

In a pool of four attempts with one success, the pass@1 estimate is 1/4. Of the six possible pairs, three contain a success, so the pass@2 estimate from this pool is 1/2. Do not confuse the number of attempts collected for estimation with the number allowed in the situation being measured.

A high pass@10 does not promise an equally good pass@1. Nor does it tell us whether the user can identify the correct attempt without a tester. Chen and colleagues, §3.1, describe pass@k estimation; the paper on agents uses this metric in a different context.

Continuous Batching allows the group of requests being served to change between successive model iterations. A completed response leaves the batch — a group processed together — and a new request can take its place.

The long response continues to grow while the short one is replaced by the next. A scheduler, a program that plans the work, selects the participants. There is no need to wait for the slowest member of the entire previous group.

Using slots this way can increase throughput, but does not guarantee a shorter wait for every user. Request lengths, memory limits, and scheduling rules matter. With similar lengths, the advantage over a fixed group may be small.

Mechanism source: Orca, §3, “Iteration-level scheduling”.

Mechanism and details

This is especially useful in Decode: one person needs a short response, another several hundred tokens. Merely gathering several requests before running the model does not ensure that an active batch's membership can change. A new request also needs Prefill; how this stage is handled alongside generation depends on the engine's implementation.

Does C have to wait for A?

An original experiment: two slots are available. A and B are ready at time 0 and need six and two rounds respectively. C becomes ready at time 1 and needs two rounds. Both variants perform the same computations, but in a fixed group, B's slot does not accept C until A finishes.

With Continuous Batching, C starts at time 2 and finishes at 4; with a fixed group, it starts at 6 and finishes at 8. Switch B to six rounds: in this case, early release of a slot disappears, and both strategies produce the same schedule.

Rounds here have an illustrative, equal duration. Prefill and transfers are outside the experiment; “ready” means ready for the displayed rounds. On a GPU, iteration time depends on its workload, so the number of occupied slots is not the device-utilization percentage.

The ability to join a request does not guarantee an immediate start: memory limits and scheduler policy apply (Orca, §4.2). PagedAttention helps manage memory but does not itself determine serving order.

I use AI-generated content as part of my daily learning process.

PagedAttention reads information from KV Cache stored in blocks that may occupy different locations in memory. A block table indicates where to find the next logical part of the conversation. The text remains in the correct order.

If a block holds descriptions of four tokens — pieces of text — a fifth requires a second block. The system does not have to reserve room for each conversation's maximum length in advance. It can also share unchanged blocks according to its memory management rules.

This is a way to address and organize data, rather than automatically shorten context or change the attention result. Tables and partially filled blocks still have a cost. The experiment shows how logical order is separated from physical location.

Mechanism source: Kwon et al., §4.1–4.3.

Mechanism and details

An original example: a block holds KV for four positions. A sequence of length five needs two blocks, or eight slots. The first block is full; the second contains one entry and three free slots. Logical blocks 0 and 1 can point to physical blocks 4 and 1. There is no need to move the existing prefix to append the next block during Decode.

Text order is not a memory address

The demonstration shows one sequence and blocks of four positions. Process successive positions, especially the transition from four to five. Then change the layout of the same data. We assume one scalar head: all keys are zero, the query is 1, and values at successive positions are 1, 2, 3… Attention weights are therefore equal, and the result is the mean of the values. For five positions, it is 3 in both layouts.

This illustrates addressing and allocation, not GPU runtime. Empty slots in the last block still occupy memory; PagedAttention neither removes nor compresses existing KV. FlashAttention focuses on organizing computation and transfers during attention. These mechanisms can complement one another. Block-based memory sharing is also useful in Prefix Caching.

I use AI-generated content as part of my daily learning process.

Prefix Caching reuses previously computed results for a common prefix of tokens — pieces of text — across requests. In suitable models these may be KV Cache data that let the model skip part of input processing.

The new question still needs processing, and its answer still needs generating. The cache is not a library of ready answers. The prefix and computational conditions must match: two documents with similar meanings need not share a cache.

Even a small change to an earlier part can prevent reuse of later blocks. The saving concerns repeated work, rather than every conversation; storing shared results also takes memory.

Mechanism source: vLLM 0.21.0, “Introduction” and “Limits”.

Mechanism and details

What matters is an identical token prefix in a compatible computational context, not similarity of meaning. Keys and values in later layers depend on the preceding context. The same final paragraph after a changed beginning does not automatically yield the same KV.

vLLM identifies a block by its tokens, the preceding block's identifier, and additional data, including the LoRA adapter ID and image input. In the version described, it stores full blocks for reuse. See the description of Automatic Prefix Caching. Compatibility of the model, positions, and other settings affecting KV is a condition for correct reuse; comparing texts alone is insufficient.

One changed token, different consequences

The cache in our example contains the sequence A B C D E F G H, divided into blocks of two tokens. Letters represent illustrative token IDs. A new request replaces exactly one position with X. Change its location and check how many whole blocks can be recovered.

Changing H to X leaves seven shared tokens, but only six in full blocks. Changing A to X gives zero hits, even though the next seven tokens match. All other conditions in the example stay fixed; the cache is available, and nothing has been evicted.

This caches computation, not a ready-made answer. Decode must still produce further tokens, and evicted blocks must be recalculated. PagedAttention explains organizing memory in blocks.

I use AI-generated content as part of my daily learning process.

Time to First Token (TTFT) is the time from a specified starting point in request handling to the first token, a piece of the response. In the client measurement described here, we start when the request is sent and stop at the first nonempty content.

If the question was sent at 0 ms and the first text arrived at 800 ms, TTFT is 800 ms. This can include network time, queuing, input preparation, and computation. Measuring only the model's time would use different boundaries.

A low TTFT does not guarantee a smooth continuation: a long pause may follow the first piece. Inter-token Latency [Polski] describes this. Comparisons must always state where the measurement starts and ends.

Mechanism source: NVIDIA NIM Benchmarking documentation, “Time to First Token”.

Mechanism and details

TTFT covers more than Prefill: the client may wait for the network, input preparation, the queue, computation, and result transmission. A measurement starting only inside the server has different boundaries. Such numbers should not be compared without checking where their timestamps come from.

The first character arrives quickly, the full response late

An original event trace: sending at 0 ms, server arrival at 30 ms, 10 ms of preparation, a 200 ms queue, 120 ms of Prefill, 5 ms to prepare the first result, and 35 ms to deliver it. Client TTFT is 400 ms; the time from request arrival to result preparation on the server is 335 ms.

After the first token, we receive four more, every 20 ms. The full response arrives at 480 ms. Increase their intervals to 100 ms: completion moves to 800 ms, but the first token still arrives at 400 ms. Then increase the queue and see which results change together.

The times are synthetic, the stages do not overlap, and each message contains one token. Real streaming can group tokens. TTFT describes neither the rhythm of Decode nor the time to receive the complete response; the documentation separates these metrics in “Inter-token Latency” and “End-to-End Request Latency”. Inter-token Latency [Polski] describes intervals and pauses after the first token. Prefix Caching can reduce prompt processing, while Continuous Batching affects when the scheduler admits a request for execution.

I use AI-generated content as part of my daily learning process.

Decode is the stage of generating a continuation after processing the initial input. In the usual loop, the model processes the most recently selected token — a piece of text — computes probabilities for the next one, and a selection rule appends one of the candidates.

The chosen piece becomes the input to the next step. KV Cache retains needed results from earlier computations, so the entire history does not have to be processed again.

Successive choices depend on each other, although several different responses can be served together. Here Decode means extending text with the model, rather than merely converting token numbers into characters. Prefill prepares the initial context.

Mechanism source: Shazeer, Fast Transformer Decoding, §2.4.

Mechanism and details

The first response token can be selected from the output of Prefill. If generation continues, this token becomes the input of the next model pass. Its new K and V are added to KV Cache, and the query uses the current and earlier positions. Selecting the token ID alone does not yet append its representation to the cache. This can be seen in the Hugging Face generation loop, “Cache storage implementation”.

A token selected but not yet processed

An original miniature model has three tokens: A, B, C. Their vectors are $(1,0)$, $(0,1)$, and $(-1,-1)$ respectively. We take $Q=K=V$ equal to these vectors; attention uses a scale of $1/\sqrt{2}$. From the result $h=(h_0,h_1)$, we create logits $[h_1,h_0,-h_0-h_1]$, and Greedy Decoding selects the largest.

The prompt A B gives approximately $h=(0{,}330;0{,}670)$, so the first answer is A. The cache still contains only the prompt's two positions. Only the next pass processes the selected A.

This is a fully specified demonstration model, without training, positions, multiple layers, or an end token. Its repetitive response has no linguistic meaning. Comparison with recomputing the entire prefix checks agreement of results, and the counter covers only positions for which K/V were calculated.

The cache removes repeated computation of earlier K/V, but attention still reads the context. With small batches, transferring weights or cache often becomes the bottleneck; this depends on the model, hardware, and sequence length. In Disaggregated Serving [Polski], Decode runs on separate resources after receiving the cache. Decode in this sense is not a tokenizer converting token identifiers back into text.

I use AI-generated content as part of my daily learning process.

FlashAttention organizes attention computation in blocks in the chip's small, fast memory. It does not write the full matrix of comparisons and weights to the large memory of the GPU, the card used for parallel computation.

Each chunk contributes part of the result, and the algorithm correctly updates the normalization and sum. You cannot simply compute separate chunk averages and average them: the chunks may have different weights.

The mathematical operation of exact attention is preserved, with the usual differences from computer arithmetic. For dense attention, the number of comparisons still grows quadratically. The gain comes from memory and execution organization, rather than removing dependencies between pairs.

Mechanism source: Dao et al., §2–3.2, Algorithm 1.

Mechanism and details

For dense attention, the number of operations still grows quadratically with sequence length. The gain comes from computation and memory organization. During training, some quantities are recomputed in the backward pass rather than stored. “Exact” means the operation itself is not approximated; a different order of floating-point operations can produce small numerical differences.

Averaging block results is not enough

Softmax normalizes weights across all allowed keys. Blocks can have very different sums of unnormalized weights. Each therefore needs the appropriate contribution to the result.

This can be done incrementally by maintaining the maximum score $m$, the sum $l=\sum_j e^{s_j-m}$, and the numerator $u=\sum_j e^{s_j-m}v_j$. The result is $u/l$. When a new block raises the maximum to $m'$, the previous $l$ and $u$ must be rescaled by $e^{m-m'}$ before adding new terms. The stable normalization mechanism is explained by Milakov and Gimelshein, §3, Algorithm 3.

An original example uses given scores $[0,0,\ln 3,\ln 3]$ and values $[0,0,1,1]$. Before normalization, the weights have proportions $[1,1,3,3]$, so the result is $6/8=0{,}75$. The two block results are 0 and 1. Their ordinary mean gives the incorrect 0.5.

The demonstration isolates aggregation for one query and scalar values. A full kernel calculates score blocks from Q and K on the GPU; here, given numbers let you verify the calculation by hand. Switching block size is not a benchmark.

Grouped-Query Attention limits the number of K/V heads, while FlashAttention changes how data is transferred and processed. These techniques can be combined. FlashAttention 2 is a later implementation development; the mechanism described here comes from the first paper.

I use AI-generated content as part of my daily learning process.

Prefill processes the known input, called the prompt, before further generation. The model creates numerical descriptions of tokens — pieces of text — and can retain some results in KV Cache for reuse.

Because the entire input already exists, many positions can be computed in parallel within a layer. Context access rules still apply: parallel work does not let an earlier position look at a prohibited future.

The final position's result allows the probabilities of the first new token to be computed. Choosing it is a separate step. Prefill does not cover all the time since sending the question: network time, queuing, and other stages also contribute. Decode describes further extension of the response.

Mechanism source: NVIDIA, Mastering LLM Techniques: Inference Optimization, section “Understanding LLM inference”.

Mechanism and details

A known prompt can be processed in parallel

All input tokens are already available, so many positions can be calculated simultaneously within a layer. Causal Masking still applies: the second position does not use the third. Parallel computation does not remove this dependency or the order of traversal through layers. The foundation is described by Vaswani et al., §3.1–3.2.

Prefill can also process the prompt in chunks [Polski], retaining the cache of earlier positions. vLLM documentation, “Chunked Prefill” describes using this division to interleave prompt processing with response generation for other requests.

Four positions, the same result

An original example isolates one attention head with dimension 1: all queries and keys are 1, and the values are $[2,4,8,10]$. Allowed comparison scores are equal, so each position returns the mean of its visible values: $[2,3,14/3,6]$. Compare processing all four together with chunks of two. In the second variant, the third position must still see the first two.

This calculates a single operation, without trained model weights or GPU measurements. In a full model, the cache is maintained separately for each layer. Changing the chunking may introduce small numerical differences in the implementation; it does not change the mathematical mask.

After the prompt comes Decode, where successive inputs depend on newly selected tokens. Prefill time alone does not cover the user's entire wait: the queue, input preparation, and delivery of the result also matter. See also: Time to First Token covers the client's wait for the first token, and Continuous Batching allows the group being served to change between iterations.

I use AI-generated content as part of my daily learning process.

Speculative Decoding produces a cheaper proposed continuation and then verifies it with the target model. In the classic algorithm, we accept the initial sequence of accepted tokens — pieces of text — until the first rejection, then compute a correction.

The large model can assess a known proposal at several positions in parallel. A correct sampling variant uses specific acceptance and correction probabilities to preserve the target distribution. Merely testing “does this sound sensible?” does not ensure that.

The speedup depends on the proposal's cost, the number of acceptances, and the hardware. If the small model often disagrees with the large one, extra work can cancel the gain. The demonstration below shows preservation of probabilities, rather than a guaranteed execution time.

Mechanism source: Leviathan et al., §2, Algorithm 1.

Mechanism and details

This accelerates Decode while preserving the target model's distribution under the algorithm's conditions. Checking whether a proposal “looks good” is not enough.

Why a correction is needed

Let $p$ be the target distribution and $q$ the draft distribution for the same prefix and a shared token set. Both include the chosen sampling rules, such as Temperature and Top-p Sampling. We accept a proposal $x\sim q$ with probability:

A sampled proposal has $q(x)>0$. After rejection, we sample from the distribution proportional to $\max(0,p-q)$. This step fills exactly the missing probability mass. If $p=q$, nothing is rejected, and no correction is needed. Leviathan et al., §2.3 and proof A.1.

An original example: for tokens A, B, C, let $p=[0{,}2;0{,}5;0{,}3]$ and $q=[0{,}6;0{,}3;0{,}1]$. The unconditional mass of accepted proposals is $[0{,}2;0{,}3;0{,}1]$. The remaining 0.4 must be split equally between B and C. Sampling the correction from ordinary $p$ would instead yield $[0{,}28;0{,}50;0{,}22]$.

The experiment shows the exact mass balance for one position, without simulating the whole loop or execution time. Preserving the distribution does not guarantee identical text for the same seed: the algorithms consume randomness differently, and computations have limited precision. This is discussed by Chen et al., §4.2 and §6.1.

Speedup depends on the acceptance rate, draft cost, and hardware. An overly expensive draft or many rejections can erase the savings even when distributional correctness is preserved.

I use AI-generated content as part of my daily learning process.

Beam Search retains several partial responses with the highest scores. At each step it extends them, compares the new sequences, and again keeps only a specified number. The size of this pool is called the beam width.

With width two, the system can keep beginnings A and B instead of immediately choosing A. At the next step it compares their continuations. The basic score combines the probabilities of successive tokens — pieces of text — although implementations may adjust the effect of length.

A larger pool costs more memory and computation, and can still discard the best path. A high probability for the entire sequence also does not guarantee that the response is true, diverse, or useful.

Mechanism and details

In Autoregressive Language Modeling, the basic sequence score is the sum of the log-probabilities of its tokens:

This is the logarithm of the probability product. A higher score is better; summation avoids multiplying very small numbers. The Hugging Face documentation, “Beam search” describes maintaining multiple sequences. Width 1 reduces this basic variant to Greedy Decoding.

Which prefix will you discard too early?

An original example starts with probabilities A = 0.45, B = 0.35, and C = 0.20. A beam of width 2 discards C. The best continuation of A later has probability 0.4, so the AA result is 0.18. Meanwhile, the omitted CA would achieve $0{,}20\cdot0{,}99=0{,}198$. Width 3 retains C and finds a better result. This is a complete, small tree of two tokens, without early termination or length normalization.

Beam Search remains an approximation: it does not expand a discarded prefix again. Width does not mean the number of tokens sampled from one distribution, as in Top-k Sampling.

For sequences of different lengths, the end token, stopping condition, and scoring function matter. For example, length_penalty in GenerationConfig divides the logarithmic score by the length raised to a specified power. This criterion should not be confused with pure probability. A wider beam does not guarantee better content either; the problem is discussed by Holtzman et al., §2.2–3.

I use AI-generated content as part of my daily learning process.

Greedy Decoding chooses the highest-scoring token, a piece of text, at every step and continues from that choice. It does not sample or return to skipped paths.

If A has probability 0.6 and B 0.4, it chooses A. But if the best next step after A has probability 0.51, the whole path has probability 0.306. A path through B with continuation 0.99 has probability 0.396. The local leader can therefore lose when evaluating two steps together.

The method is simple, but does not guarantee the most probable whole sequence or the best answer for a person. Beam Search keeps more paths at an additional cost.

Mechanism and details

The rule is presented in the Hugging Face documentation, “Greedy search”. Mathematically, we select $\arg\max_v p(v\mid x_{<t})$: among tokens $v$, we take the maximum given the existing sequence $x_{<t}$. A tie requires an established tie-breaking rule. The absence of sampling does not promise an identical result when the model, computation, or configuration changes.

A local winner can lose the whole sequence

This original example has exactly two steps. Initially, A has probability 0.6, and B has 0.4. After A, the best continuation has probability 0.51; after B, it has 0.99. Greedy selects A, then its best continuation: $0{,}6\cdot0{,}51=0{,}306$. However, the skipped path through B has $0{,}4\cdot0{,}99=0{,}396$. All distributions shown are invented and sum to 1; the sequence ends after two tokens.

Beam Search retains several partial sequences to defer such a choice. Top-k Sampling, by contrast, restricts the candidates for sampling the next token. With one unique maximum and $k=1$, token selection matches greedy.

The most probable text need not be true or useful. Holtzman et al., §2.2–3 distinguish probability maximization from the quality of open-ended continuations. This is separate from the search error shown above.

I use AI-generated content as part of my daily learning process.

Grouped-Query Attention (GQA) divides query heads into groups sharing key and value. A query is a numerical description used to read context, a key is used for matching, and a value conveys information. Each head still computes its own weights.

Eight query heads divided into two groups of four mean two key/value sets, rather than eight. One group gives MQA; one group per head corresponds to ordinary MHA.

Fewer sets reduce the cost of storing KV when dimensions match. This is a tradeoff in design and quality, rather than a switch guaranteeing a particular speed. Time also depends on hardware and the remaining computations.

Mechanism and details

For $H$ query heads and $G$ groups, we get $G$ K/V sets. When $G=1$, this is Multi-Query Attention. When $G=H$, we return to Multi-Head Attention. This family and the GQA-g notation are described by Ainslie et al., GQA, §2.2.

Calculate memory before guessing speed

An original calculation for KV Cache assumes batch size 1, 32 layers, 1024 stored tokens, a dimension of 128 for each key/value head, and 2 bytes per number. The K/V elements alone occupy:

The first factor of two denotes K and V. For eight query heads, the result is 128 MiB in MHA, 32 MiB for two KV heads, and 16 MiB in MQA. A MiB is $2^{20}$ bytes. We do not count model weights, buffers, memory alignment, or replication between devices.

Reducing the number of KV heads here does not shorten the context or remove query heads. It limits the number of separate K/V representations. The memory calculation does not predict quality or execution time; these also depend on training, hardware, and implementation.

When converting a checkpoint, the authors averaged K/V projections within groups and then continued Pre-training (§2.1–2.2). The results in §3 apply to specific T5 variants and tasks. They establish neither a universal number of groups nor a guarantee of preserving quality by simply switching the configuration.

I use AI-generated content as part of my daily learning process.

Multi-Query Attention (MQA) keeps multiple queries — descriptions of what the current positions need — but gives them a shared key and value set. Keys support matching, while values contain information to convey. All are lists of numbers.

Two heads can use the same key/value set yet assign different weights because they have different queries. Shared data do not force identical results.

The savings concern, among other things, KV Cache, the stored descriptions of earlier text. The change can affect quality, however, and does not imply a proportional speedup of the whole model. Grouped-Query Attention shares data within smaller groups.

Mechanism and details

In ordinary Multi-Head Attention, each head has its own query, key, and value projections. MQA removes the head dimension from K/V and their projections while retaining it in queries. This variant is defined by Noam Shazeer, Fast Transformer Decoding: One Write-Head is All You Need, §3.

The same memory, a different question

An original example has two keys: $k_1=(1,0)$ and $k_2=(0,1)$. Two query heads receive $q_1=(2,0)$ and $q_2=(0,2)$ respectively. Scaled Dot-Product Attention divides dot products by $\sqrt{2}$ and applies Softmax. The first head gives the keys weights of about 0.804 and 0.196; the second reverses those proportions. With shared $v_1=(1,0)$ and $v_2=(0,1)$, these are also the coordinates of two different results.

Where the savings come from

KV Cache stores earlier K/V. With eight query heads, MHA stores eight K/V sets, while MQA stores one. For the same sequence lengths, head dimensions, and precision, the cache tensors themselves are then eight times smaller. This is a size calculation, not an eightfold speedup of the whole model.

Shazeer's analysis in §2.4.1 and §3.1 links this change to memory transfer during successive generation steps. However, the architecture and its representational capacity change; this is not lossless removal of identical copies from an arbitrary trained MHA. Grouped-Query Attention retains several K/V sets as an intermediate variant.

I use AI-generated content as part of my daily learning process.

Rotary Position Embedding (RoPE) rotates pairs of numbers in queries and keys according to the position of a token, a piece of text. Queries and keys are vectors — lists of numbers compared in attention.

Imagine two arrows with specified lengths. Position determines their rotation angles. The comparison depends on the difference between angles, and therefore the difference between positions. A real model rotates many pairs at different frequencies.

This is an analogy for operations on numbers, rather than text moving in memory. RoPE does not prohibit access to the future or guarantee good performance beyond lengths seen in training. It differs from adding a positional signal in the original Positional Encoding.

Mechanism and details

Unlike vector addition in the original Positional Encoding, this changes the orientation of query and key. For a two-dimensional pair $u=(a,b)$, rotation by angle $\phi$ gives:

Rotation preserves vector length. Su et al., RoFormer, §3.1–3.2 derive the construction and combine many such pairs with different frequencies.

Shift both tokens and check the result

If the unrotated query and key stay fixed, then:

$m$ and $n$ are positions, and $\theta$ is the angle per unit of distance for the given pair. The result depends on the difference in positions. Shifting both positions together rotates the arrows but preserves their dot product.

An original example uses $q=k=(1,0)$ and an illustrative $\theta=30^\circ$. At distance 1, the dot product is about 0.866; at 6, it is −1; at 12, it is 1 again. This is one pair and a deliberately simple frequency, not a model's full recipe. The original paper uses $\theta_i=10000^{-2i/d}$ radians for pair index $i=0,\ldots,d/2-1$.

The displayed result is a component of Scaled Dot-Product Attention, before scaling and Softmax. It is not a probability. In this variant, we rotate Q and K; V does not require this rotation. RoPE does not replace Causal Masking.

A single pair's return to the same orientation also shows why one must not assume every score decreases monotonically with distance. The relative-position property alone does not demonstrate that the entire model works correctly on arbitrarily long text.

I use AI-generated content as part of my daily learning process.

KL Divergence compares two probability distributions, sets of probabilities for the same events. You must specify the reference distribution and the one approximating it. Each event's contribution depends on its reference probability and the ratio of the two probabilities.

For these numbers, comparing the first to the second gives about 0.368, and the reverse about 0.511, using natural logarithms. Direction matters: this is not a symmetric distance like the number of kilometers between cities.

In Knowledge Distillation, teacher and student predictions can be compared. A small KL indicates agreement, rather than the teacher's truthfulness. Assigning zero to an event with positive reference probability gives a mathematically infinite penalty.

Mechanism and details

$q_i$ and $p_i$ are probabilities of the same event $i$, such as a particular token. The formula and its relationship with Cross-entropy are given in the SciPy: entropy documentation. We have $D_{\mathrm{KL}}(q\|p)=H(q,p)-H(q)$: the entropy of the reference distribution is subtracted from Cross-entropy. When $q$ is fixed, this difference does not change the minimum with respect to the parameters of the model producing $p$.

The direction of comparison matters

An original example: for $q=[0{,}9;0{,}1]$ and $p=[0{,}5;0{,}5]$, the result is about 0.368. After swapping the distributions' roles, the result is about 0.511. We use the natural logarithm, so the unit is nats. KL Divergence is not a symmetric distance: the distribution on each side must always be specified.

If $q_i>0$ but $p_i=0$, the result is infinite. A term with $q_i=0$ is taken to be zero. These edge cases are defined by SciPy: rel_entr.

In Knowledge Distillation, the teacher's prediction can serve as the reference distribution and the student's as the approximation. A small value indicates agreement between these distributions on the evaluated data, not the truth of the teacher's response.

When implementing this, care is needed with argument order. PyTorch: KLDivLoss expects model log-probabilities as input and the reference distribution as target when log_target=False. For an “examples × classes” matrix, batchmean reduction sums terms over classes and averages over examples; mean averages all elements, producing a different scale.

Knowledge Distillation trains a student model using a teacher model's outputs or predictions. The student actually changes its parameters; simply sending a query to the teacher is not training.

If the teacher assigns A probability 0.60, B 0.35, and C 0.05, the label “A” omits the fact that B is a close alternative. The student can learn to reproduce the entire distribution. Another variant uses complete texts generated by the teacher.

The teacher need not be a single larger model, but classically serves as a source of behavior to imitate. Distillation can also transfer errors. KL Divergence helps measure agreement between predictions, while On-policy self-distillation is a more specific arrangement of data and teacher.

Mechanism and details

In Distilling the Knowledge in a Neural Network, §2, Hinton, Vinyals and Dean describe learning from soft probability distributions. Increased Temperature reveals more information about relationships between candidates. The same parameter is used on the teacher and student sides when comparing distributions. The authors combine this objective with learning from correct labels when available.

More information than a single answer

An original example: the teacher assigns illustrative tokens A, B, and C probabilities of 0.60, 0.35, and 0.05. The label “A” identifies only the winner. The full distribution also says that B was much more likely than C. The student can try to reproduce these proportions using Cross-entropy or appropriately directed KL Divergence. This is an additional training signal, not evidence that the teacher's labels are correct.

In text-generating models, comparing next-token distributions must be distinguished from training on entire generated sequences. Kim and Rush, Sequence-Level Knowledge Distillation, §2.2–3.2 describe both approaches for machine translation. In the sequence-level variant, the student trains on outputs found by the teacher's beam search. The text of such a response does not contain the full probabilities of all alternative tokens.

The result depends on the teacher, the data, and the student's capacity. It does not mean copying every capability or guaranteeing an improvement in every task. The separate note On-policy self-distillation describes a particular variant in TTPO; its conditions are not a general definition of Knowledge Distillation.

KV Cache stores keys and values from attention layers. These are lists of numbers used respectively to match fragments and convey their information. It does not store a ready answer or a summary of the conversation.

After processing A, B, and C, a new fragment D uses the stored descriptions of those positions and adds its own. In a model that blocks access to the future, new fragments do not change earlier descriptions, so they can be reused with compatible settings.

We save repeated computation but pay for memory to store the numbers. This cost grows with a long context. Prefill prepares the cache for a known input, and Decode extends it during the response. The required information must still be read and mixed.

Mechanism and details

In a model with Causal Masking, later tokens do not change the representations of earlier positions. This enables reuse of the cache for the same prefix and model settings. The information flow is described in Attention Is All You Need, §3.1 and §3.2.3, and cache updates in each layer in the Hugging Face documentation: How caching works.

What remains from the previous step

An original illustration: the model has processed tokens A, B, and C. When processing D, it uses the stored K and V for the three preceding positions and appends the pair for D. It compares the current position's query with the available keys; the result uses the corresponding values. The cache is neither a ready-made answer nor a textual conversation summary.

Storing results has a cost. A small original example for one layer: four positions, two KV heads, three numbers in each K and V vector, and four bytes per number:

The first factor of 2 means separate keys and values. The result includes only tensor elements, without allocation overhead. A full dynamic cache grows with the number of stored positions; limited-window variants can remove older positions.

An older article on KV-cache scaling develops the memory-cost topic. The article on moving caches between models [Polski] addresses a separate problem: matching tensor dimensions does not ensure matching meaning. Changing the model or prefix requires reassessing which part of the cache remains valid. See also: PagedAttention describes block-based memory organization, and Prefix Caching describes reusing KV for a shared prefix.

Top-k Sampling samples the next token, a piece of text, from the k highest-scoring candidates. It excludes the rest and renormalizes the retained probabilities to sum to one.

If A has probability 0.5, B 0.3, and the rest together 0.2, k=2 keeps A and B. After dividing by 0.8, their probabilities are 0.625 and 0.375. B can still win; the method does not give everyone equal chances.

A fixed k determines the number of candidates, rather than their total credibility. With a very flat distribution, even the leaders may have small probabilities. Top-p Sampling determines the set by cumulative probability, while Temperature adjusts the proportions.

Mechanism and details

A practical implementation can filter logits before Softmax: discarded positions are assigned negative infinity. Their probability after normalization is then zero. This mechanism is described in the Transformers: TopKLogitsWarper documentation.

Two candidates does not mean equal chances

An original example for $k=2$:

A and B had a combined probability of 0.80. We divide by this sum: $0{,}50/0{,}80=0{,}625$ and $0{,}30/0{,}80=0{,}375$. Sampling can still select B. With $k=1$ and a unique maximum, we always select the highest-scoring token.

A fixed $k$ controls the number of candidates, not their combined probability before the cutoff. Two retained positions can contain almost the entire distribution or only a small part of it. Top-p Sampling sets the boundary according to the sum of probabilities. Temperature changes the relative chances; positive scaling alone does not change the order of finite logits.

For ties, it is worth checking the implementation. The Transformers 5.17.0 code removes scores strictly below the score of the k-th candidate. It can therefore retain additional tokens with the same value at the boundary. The example in the table has no ties.

Top-p Sampling, or Nucleus Sampling, chooses the smallest leading group of candidates ordered by descending probability whose sum reaches threshold p. It then samples from that group after renormalizing the probabilities to one.

For probabilities 0.7; 0.15; 0.1; 0.05 and threshold 0.75, the first two remain: together they have 0.85. For 0.4; 0.3; 0.2; 0.1, three are needed. Each generation step can therefore admit a different number of tokens — pieces of text.

The threshold does not mean a minimum probability for each candidate or guarantee answer quality. Top-k Sampling fixes the number, while top-p responds to the distribution; neither filter assesses the truthfulness of the text itself.

Mechanism and details

The method and normalization are described by Holtzman et al., The Curious Case of Neural Text Degeneration, §3.1. The candidate set is recalculated at every step of Autoregressive Language Modeling.

The same threshold, a different number of tokens

An original example for $p=0{,}75$; the rows already contain sorted probabilities:

In the second row, A and B total only 0.70, so C must be included. After normalization, their chances are $4/9$, $3/9$, and $2/9$ respectively. D is excluded. The threshold does not mean retaining tokens each of which individually has a probability of at least 0.75.

Top-k Sampling would try to retain the same number of candidates in both cases. Top-p responds to the distribution of their chances. If we apply Temperature beforehand, the sum of successive probabilities changes, and the cutoff may fall in a different place.

The paper studied, among other things, open-ended text continuation by GPT-2 (§4.1). Its results do not establish one optimal threshold for every model and task. Nor is the parameter $p$ the probability that the generated response will be correct.