Byte Pair Encoding
A vocabulary containing every possible word would be enormous and would continually lack new names. A vocabulary of individual characters is small but turns text into a very long sequence. You need a compromise: frequent fragments are stored whole, while rarer ones are assembled from smaller pieces.
Byte Pair Encoding (BPE) builds text units by repeatedly merging frequent adjacent symbol pairs. These units become tokens, pieces of text processed by the model.
If k and o often appear together in the data, they can merge into ko. Then a frequent pair ko and t can form kot. This needs two merges because each combines two current units. The rules are learned beforehand and reused on new text.
Segmentation follows frequency, so it need not align with meaning or word structure. BPE defines the units; Token Embedding separately gives their identifiers numerical descriptions. The demonstration lets you step through successive merges.
Mechanism and details
Sennrich, Haddow and Birch adapted a compression algorithm for word segmentation in neural translation. Their variant starts from characters and an end-of-word marker. It counts pairs while accounting for word frequencies, merges the most frequent pair, and repeats the calculation. It does not merge symbols across word boundaries. The description and algorithm are in Neural Machine Translation of Rare Words with Subword Units, §3.2.
The rules are learned before the tokenizer is used
A simplified original example: if the pair k + o is most frequent, it creates the symbol ko. In the next step, ko + t may win, producing kot. Each operation merges two current symbols; merging this entire three-letter word requires two steps. I omit the end-of-word marker in the example.
When processing new text, the tokenizer applies previously learned rules. It does not rebuild the vocabulary for every prompt. Only the IDs of the resulting units are used to retrieve vectors through Token Embedding. Learning the text segmentation and learning the values of those vectors are separate operations.
BPE follows frequency, so token boundaries need not match morphemes or meaning. The variant in Sennrich's paper operates on characters; “Byte” in the name does not mean that every BPE tokenizer starts from raw bytes. When comparing models, the specific implementation must be checked.
An older article on the attention mechanism [Polski] provides broader context on embeddings and attention. Here, BPE is a separate entry about how vocabulary units are formed.