https://paweldubiel.com A public digital garden about AI, software engineering, books, and learning. en Wed, 07 Oct 2026 00:00:00 GMT https://paweldubiel.com/en/notes/42gq%E2%81%9D%20Sequence%20Packing https://paweldubiel.com/en/notes/42gq%E2%81%9D%20Sequence%20Packing Wed, 07 Oct 2026 00:00:00 GMT You are teaching a model to write short customer responses: “Thank you”, “No”, and “Yes”. After division into pieces of text, or tokens, the examples illustratively occupy 3, 2, and 2 positions, including each text's end https://paweldubiel.com/en/notes/42gk%E2%81%9D%20Reinforcement%20Learning https://paweldubiel.com/en/notes/42gk%E2%81%9D%20Reinforcement%20Learning Mon, 05 Oct 2026 00:00:00 GMT You want to teach a program to win a small game. At a fork, it can choose route A, collect two coins immediately, and finish the second turn without any new coins. Route B gives nothing on the first turn but leads to fiv https://paweldubiel.com/en/notes/42eo%E2%81%9D%20Przewidywanie%20wynik%C3%B3w%20polece%C5%84%20pomaga%20w%20p%C3%B3%C5%BAniejszym%20treningu%20agenta https://paweldubiel.com/en/notes/42eo%E2%81%9D%20Przewidywanie%20wynik%C3%B3w%20polece%C5%84%20pomaga%20w%20p%C3%B3%C5%BAniejszym%20treningu%20agenta Sun, 20 Sep 2026 00:00:00 GMT An agent working in a terminal tries to run a program. It sends a command, reads an error message and chooses its next action. In doing so, it must connect the output to the earlier command: recognize whether a file, mem https://paweldubiel.com/en/notes/42eo1%E2%81%9D%20Loss%20masking https://paweldubiel.com/en/notes/42eo1%E2%81%9D%20Loss%20masking Sun, 20 Sep 2026 00:00:00 GMT You are training an agent on records of conversations with tools. A record contains its commands and the tools' responses. You want to assess selected parts while still letting it read the rest as context for its next ac https://paweldubiel.com/en/notes/42eo2%E2%81%9D%20pass%40k https://paweldubiel.com/en/notes/42eo2%E2%81%9D%20pass%40k Sun, 20 Sep 2026 00:00:00 GMT The model produced ten programs, and one passed the tests. Does that mean a single response is likely to succeed? You need to distinguish “it worked on the first attempt” from “we found a success among many attempts”. pa https://paweldubiel.com/en/notes/42el%E2%81%9D%20Continuous%20Batching https://paweldubiel.com/en/notes/42el%E2%81%9D%20Continuous%20Batching Sun, 20 Sep 2026 00:00:00 GMT A server generates two responses at once. One finishes quickly; the other is long. If the next user must wait for the entire pair to finish, the freed slot is wasted for many steps. Continuous Batching allows the group o https://paweldubiel.com/en/notes/42ek%E2%81%9D%20PagedAttention https://paweldubiel.com/en/notes/42ek%E2%81%9D%20PagedAttention Sun, 20 Sep 2026 00:00:00 GMT A server handles conversations of different lengths. If it reserves a large contiguous memory region in advance for each entire future response, some space remains unused. You want to allocate smaller pieces as needed. P https://paweldubiel.com/en/notes/42em%E2%81%9D%20Prefix%20Caching https://paweldubiel.com/en/notes/42em%E2%81%9D%20Prefix%20Caching Sun, 20 Sep 2026 00:00:00 GMT You ask five questions about the same long document. Each input starts with the same instruction and document text; only the ending contains a new question. Computing the same beginning again would repeat work. Prefix Ca https://paweldubiel.com/en/notes/42en%E2%81%9D%20Time%20to%20First%20Token https://paweldubiel.com/en/notes/42en%E2%81%9D%20Time%20to%20First%20Token Sun, 20 Sep 2026 00:00:00 GMT You click “Send” and stare at an empty chat. For the user's comfort, it matters when the beginning of the response appears, even if writing the rest takes much longer. You need to measure that initial wait. Time to First https://paweldubiel.com/en/notes/42eg%E2%81%9D%20Decode https://paweldubiel.com/en/notes/42eg%E2%81%9D%20Decode Fri, 18 Sep 2026 00:00:00 GMT The first piece of the response has appeared. The next must account for what was just written: after “Today I drink tea” the model has a different context than after “Today I drink water”. This changing text must be exte https://paweldubiel.com/en/notes/42eh%E2%81%9D%20FlashAttention https://paweldubiel.com/en/notes/42eh%E2%81%9D%20FlashAttention Fri, 18 Sep 2026 00:00:00 GMT Attention compares many pairs of text fragments. Saving all intermediate results to a computing card's large memory can make data transfers expensive. You want to perform the same calculation while reducing movement and https://paweldubiel.com/en/notes/42ef%E2%81%9D%20Prefill https://paweldubiel.com/en/notes/42ef%E2%81%9D%20Prefill Fri, 18 Sep 2026 00:00:00 GMT You send a long question to a chat and wait for the first word. Before the model appends a response, it must process the content it has already received. This initial stage can require considerable work, especially for a https://paweldubiel.com/en/notes/42ei%E2%81%9D%20Speculative%20Decoding https://paweldubiel.com/en/notes/42ei%E2%81%9D%20Speculative%20Decoding Fri, 18 Sep 2026 00:00:00 GMT A large model generates text correctly but slowly. A smaller model can quickly propose several following pieces. You want to use its work while keeping the large model in control of the final result. Speculative Decoding https://paweldubiel.com/en/notes/42eb%E2%81%9D%20Beam%20Search https://paweldubiel.com/en/notes/42eb%E2%81%9D%20Beam%20Search Fri, 18 Sep 2026 00:00:00 GMT When choosing the first piece of a response, you may reject too early a path that would later turn out better. Checking all possible sequences is too expensive, however. You need a limited pool of alternatives. Beam Sear https://paweldubiel.com/en/notes/42ea%E2%81%9D%20Greedy%20Decoding https://paweldubiel.com/en/notes/42ea%E2%81%9D%20Greedy%20Decoding Fri, 18 Sep 2026 00:00:00 GMT The model could append several different pieces. The simplest choice is always to take the most likely one. This saves consideration of alternatives, but does the best first step lead to the best entire sequence? Greedy https://paweldubiel.com/en/notes/42ed%E2%81%9D%20Grouped-Query%20Attention https://paweldubiel.com/en/notes/42ed%E2%81%9D%20Grouped-Query%20Attention Fri, 18 Sep 2026 00:00:00 GMT The model reads earlier text in several ways in parallel. Each such computation is an attention head. You want to reduce the memory these heads occupy, but sharing a single data set completely may constrain the model too https://paweldubiel.com/en/notes/42ec%E2%81%9D%20Multi-Query%20Attention https://paweldubiel.com/en/notes/42ec%E2%81%9D%20Multi-Query%20Attention Fri, 18 Sep 2026 00:00:00 GMT The model connects the current piece of text to the earlier conversation through several parallel computations. These computations are called attention heads. If each head stores its own descriptions of the entire preced https://paweldubiel.com/en/notes/42ee%E2%81%9D%20Rotary%20Position%20Embedding https://paweldubiel.com/en/notes/42ee%E2%81%9D%20Rotary%20Position%20Embedding Fri, 18 Sep 2026 00:00:00 GMT The model compares pieces of text, but also needs to know how far apart they are. “Yesterday” beside a verb can play a different role from the same word in a distant quotation. How can position be included when comparing https://paweldubiel.com/en/notes/42dw%E2%81%9D%20KL%20Divergence https://paweldubiel.com/en/notes/42dw%E2%81%9D%20KL%20Divergence Wed, 16 Sep 2026 00:00:00 GMT Two models divide probabilities differently among the same answers. One assigns 90% to A and 10% to B; the other gives each 50% . You want to measure how well the second reproduces the first model's predictions. KL Diver https://paweldubiel.com/en/notes/42dx%E2%81%9D%20Knowledge%20Distillation https://paweldubiel.com/en/notes/42dx%E2%81%9D%20Knowledge%20Distillation Wed, 16 Sep 2026 00:00:00 GMT A large model performs a task well but is too expensive for everyday use. A smaller model could try to reproduce its behavior. It needs examples or assessments showing more than just a list of correct labels, however. Kn https://paweldubiel.com/en/notes/42dy%E2%81%9D%20KV%20Cache https://paweldubiel.com/en/notes/42dy%E2%81%9D%20KV%20Cache Wed, 16 Sep 2026 00:00:00 GMT The model appends a response piece by piece. It has already computed much information about earlier text, so computing it from scratch at every step would repeat work. You want to retain results that remain valid. KV Cac https://paweldubiel.com/en/notes/42du%E2%81%9D%20Top-k%20Sampling https://paweldubiel.com/en/notes/42du%E2%81%9D%20Top-k%20Sampling Wed, 16 Sep 2026 00:00:00 GMT When sampling the next piece, the model allows many extremely unlikely options. You want to keep some diversity but remove the long tail of candidates. You can admit only a specified number of leaders. Top k Sampling sam https://paweldubiel.com/en/notes/42dv%E2%81%9D%20Top-p%20Sampling https://paweldubiel.com/en/notes/42dv%E2%81%9D%20Top-p%20Sampling Wed, 16 Sep 2026 00:00:00 GMT Sometimes the model has one clear leader; sometimes it has many continuations with similar scores. Always admitting five candidates ignores this difference. You want to choose the group's size according to where the prob https://paweldubiel.com/en/notes/42do%E2%81%9D%20Cross-entropy https://paweldubiel.com/en/notes/42do%E2%81%9D%20Cross-entropy Tue, 15 Sep 2026 00:00:00 GMT Two models identify the same word as the best continuation, but the first gives it 80% probability and the second only 20% . Checking only the winner misses this difference. Training needs a measure of how strongly the m https://paweldubiel.com/en/notes/42ds%E2%81%9D%20LoRA https://paweldubiel.com/en/notes/42ds%E2%81%9D%20LoRA Tue, 15 Sep 2026 00:00:00 GMT You have a large model and want to adapt it to the style of your documents. Training all its weights — the numbers controlling its computations — requires a lot of memory. Can you train only a smaller correction to what https://paweldubiel.com/en/notes/42dq%E2%81%9D%20Masked%20Language%20Modeling https://paweldubiel.com/en/notes/42dq%E2%81%9D%20Masked%20Language%20Modeling Tue, 15 Sep 2026 00:00:00 GMT You want to teach the model to use the entire sentence. You hide a fragment in “Alice has a MASK ” or “Alice MASK a cat” and ask it to reconstruct the original. Information on both sides of the gap may be needed for the https://paweldubiel.com/en/notes/42dp%E2%81%9D%20Perplexity https://paweldubiel.com/en/notes/42dp%E2%81%9D%20Perplexity Tue, 15 Sep 2026 00:00:00 GMT You compare two models on the same text. You want to know which predicts its successive fragments better, rather than merely which guessed more individual winners. You need a number summarizing the probabilities assigned https://paweldubiel.com/en/notes/42dr%E2%81%9D%20Temperature https://paweldubiel.com/en/notes/42dr%E2%81%9D%20Temperature Tue, 15 Sep 2026 00:00:00 GMT The model usually chooses similar continuations, and you want to regulate sampling diversity. You do not change its knowledge or weights. You change only how strongly the best scoring candidates' lead affects their proba https://paweldubiel.com/en/notes/42dj%E2%81%9D%20Autoregressive%20Language%20Modeling https://paweldubiel.com/en/notes/42dj%E2%81%9D%20Autoregressive%20Language%20Modeling Mon, 14 Sep 2026 00:00:00 GMT You start a sentence: “Today I drink…”. After appending “tea”, possible continuations differ from those after “water”. The model must account for the text already chosen at every next step, rather than guess all position https://paweldubiel.com/en/notes/42dl%E2%81%9D%20Byte%20Pair%20Encoding https://paweldubiel.com/en/notes/42dl%E2%81%9D%20Byte%20Pair%20Encoding Mon, 14 Sep 2026 00:00:00 GMT A vocabulary containing every possible word would be enormous and would continually lack new names. A vocabulary of individual characters is small but turns text into a very long sequence. You need a compromise: frequent https://paweldubiel.com/en/notes/42dn%E2%81%9D%20Fine-tuning https://paweldubiel.com/en/notes/42dn%E2%81%9D%20Fine-tuning Mon, 14 Sep 2026 00:00:00 GMT A trained model can process language, but you want it to classify company messages as “return”, “delivery”, or “payment”. General ability does not yet provide the way of working you expect. You prepare examples of messag https://paweldubiel.com/en/notes/42dm%E2%81%9D%20Pre-training https://paweldubiel.com/en/notes/42dm%E2%81%9D%20Pre-training Mon, 14 Sep 2026 00:00:00 GMT You want the model to recognize relationships in language before adapting it to a specific job, such as classifying customer messages. Manually preparing answers for all the text would be impractical. The text itself, ho https://paweldubiel.com/en/notes/42dk%E2%81%9D%20Softmax https://paweldubiel.com/en/notes/42dk%E2%81%9D%20Softmax Mon, 14 Sep 2026 00:00:00 GMT The model scored three possible sentence continuations with the numbers 2 , 1 , and 0 . You want to sample a response according to their probabilities. Raw scores are not probabilities: they can be negative and do not su https://paweldubiel.com/en/notes/42dc%E2%81%9D%20Causal%20Masking https://paweldubiel.com/en/notes/42dc%E2%81%9D%20Causal%20Masking Mon, 14 Sep 2026 00:00:00 GMT You are teaching the model to append “cat” after “Alice has a”. The word “cat” already exists in the complete example. If the model could look at it while predicting, the task would appear easy but would not resemble cre https://paweldubiel.com/en/notes/42dd%E2%81%9D%20Cross-attention https://paweldubiel.com/en/notes/42dd%E2%81%9D%20Cross-attention Mon, 14 Sep 2026 00:00:00 GMT In translation, the model has two different texts: the original and the response still being created. When appending another translation word, it needs information from the original. Mixing only the response already writ https://paweldubiel.com/en/notes/42df%E2%81%9D%20Layer%20Normalization https://paweldubiel.com/en/notes/42df%E2%81%9D%20Layer%20Normalization Mon, 14 Sep 2026 00:00:00 GMT One fragment's description enters the next layer with numbers 2, 4 , another with 20, 40 . The signal magnitude can differ considerably. You want to aid computation by converting numbers within each description to a comp https://paweldubiel.com/en/notes/42de%E2%81%9D%20Position-wise%20Feed-Forward%20Network https://paweldubiel.com/en/notes/42de%E2%81%9D%20Position-wise%20Feed-Forward%20Network Mon, 14 Sep 2026 00:00:00 GMT The model gathered information from the rest of the sentence for a particular fragment. It now needs to process that description: combine features and compute a new representation. Reading other positions again is not th https://paweldubiel.com/en/notes/42db%E2%81%9D%20Token%20Embedding https://paweldubiel.com/en/notes/42db%E2%81%9D%20Token%20Embedding Mon, 14 Sep 2026 00:00:00 GMT The computer receives a number identifying a text fragment. The number 17 itself does not tell it what the fragment is or how it relates to others. It needs a useful numerical description that can be transformed during c https://paweldubiel.com/en/notes/42da%E2%81%9D%20B%C5%82%C4%99dy%20kwantyzacji%20cz%C4%99%C5%9Bciowo%20znosz%C4%85%20si%C4%99%20mi%C4%99dzy%20warstwami%20modelu https://paweldubiel.com/en/notes/42da%E2%81%9D%20B%C5%82%C4%99dy%20kwantyzacji%20cz%C4%99%C5%9Bciowo%20znosz%C4%85%20si%C4%99%20mi%C4%99dzy%20warstwami%20modelu Sun, 13 Sep 2026 00:00:00 GMT You want to run a language model on your own graphics card, but its weights take up too much memory. One solution is post training quantization: you replace numbers stored in the finished model with values from a smaller https://paweldubiel.com/en/notes/42da1%E2%81%9D%20Post-training%20quantization https://paweldubiel.com/en/notes/42da1%E2%81%9D%20Post-training%20quantization Sun, 13 Sep 2026 00:00:00 GMT You already have a trained model, but it occupies too much memory to run on your chosen computer. Much of that space stores the numbers controlling computation. Can they be stored less precisely to shrink the model? Post https://paweldubiel.com/en/notes/42da2%E2%81%9D%20Residual%20connection https://paweldubiel.com/en/notes/42da2%E2%81%9D%20Residual%20connection Sun, 13 Sep 2026 00:00:00 GMT The model processes a text description through many layers. If every layer had to rebuild all the information, forwarding it could easily become difficult. Retaining the input and adding only the computed change is helpf https://paweldubiel.com/en/notes/42da3%E2%81%9D%20LM%20head https://paweldubiel.com/en/notes/42da3%E2%81%9D%20LM%20head Sun, 13 Sep 2026 00:00:00 GMT The model has processed “Today I drink” and has an internal description of this content as a list of numbers. The user, however, is waiting for text. We need a transition from the description hidden in the model to score https://paweldubiel.com/en/notes/42cy%E2%81%9D%20Multi-Head%20Attention https://paweldubiel.com/en/notes/42cy%E2%81%9D%20Multi-Head%20Attention Sun, 13 Sep 2026 00:00:00 GMT One combination of information can mix different sentence relationships too strongly. In translation, relationships involving people, time, and grammatical form may all be useful. We want to let the model compute several https://paweldubiel.com/en/notes/42cz%E2%81%9D%20Positional%20Encoding https://paweldubiel.com/en/notes/42cz%E2%81%9D%20Positional%20Encoding Sun, 13 Sep 2026 00:00:00 GMT You read “BMW overtook Volvo” and “Volvo overtook BMW”. They contain exactly the same words, but changing their order reverses the cars' roles. If the model sees only word descriptions and compares them as an unordered s https://paweldubiel.com/en/notes/42cx%E2%81%9D%20Scaled%20Dot-Product%20Attention https://paweldubiel.com/en/notes/42cx%E2%81%9D%20Scaled%20Dot-Product%20Attention Sun, 13 Sep 2026 00:00:00 GMT The model needs to combine information from several sentence fragments. One is more useful, another less so. It needs a specific way to compute their contributions and then assemble the information into one result. Scale https://paweldubiel.com/en/notes/42cw%E2%81%9D%20Self-attention https://paweldubiel.com/en/notes/42cw%E2%81%9D%20Self-attention Sun, 13 Sep 2026 00:00:00 GMT In “Anna put the book down because it was heavy”, the word “it” alone explains little. It must connect to earlier fragments. A model processing text needs a way for one position's description to incorporate information f https://paweldubiel.com/en/notes/42cv%E2%81%9D%20Transformer https://paweldubiel.com/en/notes/42cv%E2%81%9D%20Transformer Sun, 13 Sep 2026 00:00:00 GMT You want to translate “The train will arrive tomorrow”. Translating each word separately is insufficient: the correct form depends on other words and their order. The model needs a way to combine information from text, e https://paweldubiel.com/en/notes/42be%E2%81%9D%20Ten%20sam%20ranking%20daje%20inne%20decyzje%20po%20zmianie%20kolejno%C5%9Bci https://paweldubiel.com/en/notes/42be%E2%81%9D%20Ten%20sam%20ranking%20daje%20inne%20decyzje%20po%20zmianie%20kolejno%C5%9Bci Sun, 30 Aug 2026 00:00:00 GMT A system answering questions from documents often operates in four steps. A search engine finds candidates, a scorer assigns each one a number indicating its usefulness, a threshold rejects low scores, and a reader model https://paweldubiel.com/en/notes/42be1%E2%81%9D%20Batched%20pointwise%20scoring https://paweldubiel.com/en/notes/42be1%E2%81%9D%20Batched%20pointwise%20scoring Sun, 30 Aug 2026 00:00:00 GMT You have a question and three retrieved documents. You want to score each one's usefulness, but three separate model calls repeat the same instruction and question. You can show all three together and ask for three score https://paweldubiel.com/en/notes/42be2%E2%81%9D%20nDCG%20at%2010 https://paweldubiel.com/en/notes/42be2%E2%81%9D%20nDCG%20at%2010 Sun, 30 Aug 2026 00:00:00 GMT A search engine returns ten results. Good documents may be on the list, but if the most useful is only tenth, the reader may never reach it. You need a measure accounting for both relevance and position. nDCG@10 measures