Tokenization and Byte-Pair Encoding: How Text Becomes Numbers

July 4, 2026 17 min read AI & Machine Learning

tokenizationbyte-pair-encodingbpellmnlptransformers

Contents

A large language model has never read a word in its life. Before a single matrix multiply happens, the string you typed is chopped into pieces, and each piece is looked up in a table and replaced by an integer. The model sees only those integers — a sequence of token IDs — and everything it “knows” about language is really a statistical relationship between numbers in that vocabulary. The layer that does the chopping is the tokenizer, and for almost every model you have heard of, the algorithm inside it is byte-pair encoding. It is the least glamorous part of the stack and one of the most consequential. Tokenization decides how long your prompt is, how much you pay, how well the model handles Arabic or Python or a phone number, and why a system that can write a sonnet cannot reliably count the letters in “strawberry.”

Byte-pair encoding was not invented for language models. Philip Gage published it in 1994 as a data-compression algorithm in The C Users Journal — a way to shrink files by repeatedly replacing the most common pair of adjacent bytes with a byte that did not appear in the data. It sat in relative obscurity for two decades. Then in 2016, Rico Sennrich, Barry Haddow, and Alexandra Birch borrowed the mechanic for a completely different purpose: not to compress text, but to carve an open, unbounded vocabulary of words into a fixed, manageable set of subword units for neural machine translation. That paper is the reason GPT, LLaMA, Claude, and nearly every modern model tokenize the way they do. This is how it works, why it wins, and where it quietly breaks.

The problem tokenization solves

You cannot feed raw characters to a transformer efficiently, and you cannot feed it whole words either. Both extremes fail, and understanding why makes the middle path obvious.

Feed it characters — one token per character — and every sequence becomes enormous. The word “tokenization” is twelve tokens instead of one or two. Your context window, measured in tokens, now holds a fraction of the actual text. Worse, the model has to learn to assemble meaning from tiny fragments across long spans, which wastes capacity on spelling before it can get to semantics.

Feed it words — one token per word — and the vocabulary explodes. English alone has hundreds of thousands of words, and that is before you count names, typos, hashtags, code identifiers, and every inflected form. Any word not in the vocabulary becomes a single <UNK> “unknown” token, and the model is blind to it. A fixed word vocabulary can never cover an open language. The moment someone types “antidisestablishmentarianismesque” or getUserByEmailAddress, you are stuck.

Subword tokenization threads the needle. Common words stay whole — “the,” “language,” “model” each become one token. Rare or novel words break into meaningful pieces — “tokenization” might become token + ization, and getUserByEmailAddress splits along its camel-case seams. Nothing is ever truly out-of-vocabulary, because in the worst case a word decomposes all the way down to individual bytes, and every byte is always in the table. You get a fixed vocabulary size, full coverage of any input, and sequence lengths that stay reasonable. BPE is the most popular way to build that subword vocabulary.

The BPE training algorithm

BPE has two phases that people constantly conflate: training the merges (done once, offline, on a big corpus) and applying them (done every time you tokenize a string). Start with training.

The algorithm is almost embarrassingly simple. Begin with a vocabulary of individual characters (or bytes). Then repeat one move — find the most frequent adjacent pair of symbols in the corpus and merge it into a new symbol — until you hit your target vocabulary size. Each merge you perform is recorded, in order, as a rule.

Here is the whole thing in Python, operating on a tiny toy corpus. Real implementations track word frequencies rather than re-scanning the corpus, but the logic is identical.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


from collections import Counter

# Toy corpus: word -> frequency. "_" marks a word boundary.
corpus = {"low_": 5, "lower_": 2, "newest_": 6, "widest_": 3}

# 1. Seed vocabulary with characters; split each word into symbols.
words = {tuple(w): freq for w, freq in corpus.items()}

def count_pairs(words):
    pairs = Counter()
    for symbols, freq in words.items():
        for a, b in zip(symbols, symbols[1:]):
            pairs[(a, b)] += freq
    return pairs

def merge(words, pair):
    a, b = pair
    merged = a + b
    out = {}
    for symbols, freq in words.items():
        new = []
        i = 0
        while i < len(symbols):
            if i < len(symbols) - 1 and symbols[i] == a and symbols[i+1] == b:
                new.append(merged)
                i += 2
            else:
                new.append(symbols[i])
                i += 1
        out[tuple(new)] = freq
    return out

merges = []
for step in range(10):
    pairs = count_pairs(words)
    if not pairs:
        break
    best = pairs.most_common(1)[0][0]   # most frequent adjacent pair
    words = merge(words, best)
    merges.append(best)
    print(f"merge {step}: {best[0]}+{best[1]} -> {best[0]+best[1]}")

Run it and the first few merges tell the story:

merge 0: e+s -> es
merge 1: es+t -> est
merge 2: est+_ -> est_
merge 3: l+o -> lo
merge 4: lo+w -> low

The algorithm discovered est_ — the English superlative suffix — purely from frequency, with no linguistic knowledge whatsoever. It found “low” as a unit because it appeared often. That is the entire magic: statistics over adjacency recovers morphology for free. The ordered list of merges is the trained tokenizer. Nothing else is learned; there are no weights, no gradients, no neural network in the tokenizer itself.

Applying the merges

Encoding a new string reverses nothing — it replays the merge list in order. Split the input into characters, then for each merge rule in the order it was learned, apply it everywhere it matches. Because the rules are ordered, e+s→es fires before es+t→est, so the tokenizer reconstructs exactly the segmentation the training corpus implied.

Input:  "newest"
Chars:  n e w e s t
Apply e+s   -> n e w es t
Apply es+t  -> n e w est
Apply ...   -> (further merges if they exist)
Result: n e w est   (or "new" + "est" if those merges were learned)

Decoding is trivial: concatenate the token strings back together. Because every merge is reversible and the base vocabulary covers every byte, the round trip is lossless — decode(encode(x)) == x for any input, including emoji, control characters, and text in scripts the tokenizer was never trained on. That guarantee is why byte-level BPE won.

Byte-level BPE and the Unicode problem

The 2016 Sennrich version operated on Unicode characters, which created an awkward edge case: what is the base vocabulary? Unicode has around 150,000 assigned code points. Seeding the vocabulary with all of them is wasteful, and any code point you leave out becomes a genuine unknown.

GPT-2 fixed this with byte-level BPE. Instead of Unicode characters, the base vocabulary is the 256 possible byte values. Any text in any language, any emoji, any binary-ish garbage, is first encoded as UTF-8 bytes, and BPE operates on those bytes. The base vocabulary is always exactly 256 symbols, coverage is total by construction, and there is no possible unknown token. This is what OpenAI’s tiktoken library implements and what powers the GPT-3, GPT-4, and GPT-4o family.

The trade-off is that non-ASCII text pays a byte tax. A single Chinese character is three UTF-8 bytes, so before any merges it costs three tokens; an emoji can be four. Merges learned from a mostly-English corpus recover common English words to one token but rarely learn multi-byte merges for underrepresented scripts. This is the mechanical root of the fairness problem discussed later.

The major algorithm families you will encounter in practice:

Algorithm	Merge criterion	Base unit	Used by	Notable trait
BPE (classic)	Highest pair frequency	Unicode char	GPT, RoBERTa, original NMT	Simple, deterministic
Byte-level BPE	Highest pair frequency	Raw byte (0–255)	GPT-2/3/4, LLaMA 3, tiktoken	No unknowns, ever
WordPiece	Highest likelihood gain	Unicode char	BERT, DistilBERT, Electra	Uses `##` continuation prefix
Unigram LM	Prob. under a unigram model	Char / byte	T5, mBART, ALBERT	Prunes from a large vocab down
SentencePiece	Wraps BPE or Unigram	Raw bytes, whitespace as `▁`	LLaMA 1/2, T5, many multilingual	Language-agnostic, no pre-tokenizing

WordPiece, used by BERT, is the closest cousin. It merges bottom-up like BPE, but instead of picking the most frequent pair it picks the pair that most increases the likelihood of the training data under a language model — roughly, the pair whose merged frequency is high relative to how often its parts appear separately. SentencePiece is not a competing algorithm so much as a wrapper: it runs BPE or a unigram model directly on raw text with whitespace treated as a normal symbol (the ▁ marker), which is why it handles languages without spaces, like Japanese, without a separate word-splitting step.

Vocabulary size: the central trade-off

Every tokenizer designer turns one knob before anything else: how many tokens are in the vocabulary. It is a direct trade between sequence length and embedding-table size, and both ends cost real money.

   small vocab (32K)                 large vocab (128K)
   -----------------                 ------------------
   more tokens per text              fewer tokens per text
   longer sequences                  shorter sequences
   smaller embedding matrix          larger embedding matrix
   worse rare-word handling          better multilingual coverage
        |                                     |
        |   attention cost ~ O(seq_len^2)     |
        +-------------------------------------+
              pick your poison

A bigger vocabulary means each token carries more text, so sequences are shorter — and since transformer attention scales with the square of sequence length, shorter sequences are cheaper to run and let more real content fit in a fixed context window. But a bigger vocabulary also means a bigger embedding matrix (vocabulary size times hidden dimension) and a bigger final softmax, which cost memory and compute on every forward pass. Push the vocabulary too large and most tokens are seen so rarely during training that their embeddings never learn anything useful.

The historical trend has been steadily upward as the multilingual payoff became clear:

Model	Tokenizer	Vocabulary size
Original GPT	BPE	40,478
GPT-2 / GPT-3	Byte-level BPE	50,257
LLaMA 1 / 2	SentencePiece BPE	32,000
LLaMA 3	tiktoken BPE	128,256
GPT-4o	Byte-level BPE (o200k)	~200,000

LLaMA 3’s jump from 32K to 128K tokens was not cosmetic. A larger vocabulary lets more non-English words and multi-byte sequences collapse into single tokens, which shortened sequences on multilingual and code data and improved efficiency measurably, at the cost of a much larger embedding table.

Why tokenization drives your bill and your context window

Two numbers you actually care about — how much a request costs and how much text fits — are both counted in tokens, not words or characters. The tokenizer is the exchange rate, and it is not one-to-one.

A useful rule of thumb for English is that one token is about four characters, or roughly 0.75 words. So 1,000 tokens is about 750 English words. A “128K context window” is not 128,000 words; it is 128,000 tokens, or roughly 96,000 words of English — and far fewer words if your text is code, or JSON, or a non-Latin script.

This matters concretely:

Pricing. API providers bill per token, input and output separately. The same paragraph translated into a language that tokenizes poorly can cost several times more to process, because it expands into more tokens.
Context budget. A retrieval-augmented pipeline that stuffs documents into the prompt is really stuffing tokens. Underestimate the tokenizer’s expansion on your data and you silently truncate. (See the token-budget discussion in RAG beyond toy demos.)
Latency. More tokens means more forward passes for generation and a bigger attention matrix for the prompt. Tokenization efficiency is throughput.

You can measure the exchange rate directly rather than guessing:

1
2
3
4
5
6
7


import tiktoken

enc = tiktoken.get_encoding("cl100k_base")   # GPT-4 / GPT-3.5-turbo

for text in ["Hello world", "def get_user(id):", "东京", "1234567890"]:
    ids = enc.encode(text)
    print(f"{len(ids):2d} tokens  {text!r}  ->  {ids}")

 2 tokens  'Hello world'  ->  [9906, 1917]
 6 tokens  'def get_user(id):'  ->  [755, 636, 3398, 3227, 1329]
 2 tokens  '东京'  ->  [tokens...]   # two Chinese chars, but 6 UTF-8 bytes underneath
10 tokens  '1234567890'  ->  [digit-by-digit]

Notice the last two lines. That is where the trouble starts.

Where tokenization breaks

BPE is frequency-driven and English-biased, and both properties produce failure modes that leak all the way up into model behavior. These are not bugs in the model; they are artifacts of how text was cut up before the model ever saw it.

Numbers. Most tokenizers split long digit strings inconsistently — “1234567890” might become several tokens, and “1000” and “1001” may tokenize completely differently. The model sees no clean place-value structure, which is a real contributor to why LLMs are shaky at arithmetic. Some newer tokenizers force single-digit tokenization specifically to give math a fighting chance.

Spelling and character counting. Because “strawberry” is one or two tokens, the model has no direct access to its individual letters. Asking “how many r’s are in strawberry” is asking the model to introspect the spelling of a token it only knows as an atomic ID. It genuinely cannot see the letters. This is the tokenizer’s fault, not a reasoning failure.

Code. Whitespace and indentation are semantically load-bearing in Python, but a naive tokenizer may merge runs of spaces unpredictably or split identifiers at awkward points. tiktoken’s cl100k_base added dedicated tokens for common whitespace runs precisely because code was tokenizing so wastefully.

Non-English text. This is the sharp one. A tokenizer trained mostly on English learns English words as single tokens but represents, say, Burmese or Amharic close to byte-by-byte. The same sentence can take three to five times as many tokens in an underrepresented language as in English. Users of those languages get less usable context and pay more per API call for identical meaning — a documented cross-lingual fairness problem baked into the exchange rate itself.

Glitch tokens. Training corpora contain oddities — a Reddit username that appeared thousands of times, a boilerplate string — that BPE dutifully promotes to single tokens. Because those tokens barely appear in the model’s training text (as opposed to the tokenizer’s), their embeddings are nearly random, and prompting the model with them produces bizarre, unstable output. The infamous SolidGoldMagikarp token was exactly this.

Building intuition: encode something yourself

The fastest way to internalize all of this is to install the library and watch the boundaries. Hugging Face’s tokenizers and OpenAI’s tiktoken both let you inspect the split directly.

1
2
3
4
5
6
7
8
9


from tokenizers import Tokenizer

tok = Tokenizer.from_pretrained("gpt2")
out = tok.encode("The tokenizer's failure modes are underappreciated.")

# Show the actual string pieces, not just IDs
print(out.tokens)
# ['The', 'Ġtoken', 'izer', "'s", 'Ġfailure', 'Ġmodes',
#  'Ġare', 'Ġunder', 'app', 'reci', 'ated', '.']

The Ġ marker is byte-level BPE’s representation of a leading space — a critical detail, because it means " token" (with a space) and “token” (without) are different tokens. Notice how “tokenizer” fractures into token + izer, how “underappreciated” shatters into four pieces because it is rare, and how the possessive 's becomes its own token. Every one of those boundaries was decided by frequency statistics over a 2019 web crawl, and every one of them shapes what the model downstream can and cannot do with your text.

The relationship to the rest of the model is clean: tokenization produces the integer IDs, an embedding table maps each ID to a vector, and only then does the network — trained by backpropagation — begin. When you later run quantization or fit a model onto a single GPU as in the local LLM deep dive, the token vocabulary and its embedding table are a fixed cost you inherit from this one offline decision.

Trade-offs, honestly

BPE won because it is simple, deterministic, and lossless, but it is not free of cost, and the alternatives it beat had real merits.

BPE is greedy and order-dependent. It merges the most frequent pair at each step without any lookahead, so the final segmentation is not provably optimal for compression or for downstream modeling. Research has shown BPE is measurably suboptimal for pretraining compared to unigram-model tokenization, yet BPE persists because it is faster and simpler.
The vocabulary is frozen at training time. Once merges are learned, adapting to a new domain (a fresh programming language, a medical corpus, a new slang) means either living with wasteful splits or retraining the tokenizer and the model. There is no cheap incremental update.
English bias is structural, not incidental. Because merges are frequency-driven and corpora are English-heavy, non-English users pay a token tax that no amount of model improvement fixes. Only a different or larger tokenizer helps, and that is a from-scratch decision.
Tokenization boundaries create silent behavioral cliffs. Arithmetic, spelling, and rhyming failures all trace back to how tokens cut the input, and they are invisible unless you go look at the token stream. Debugging a model without inspecting its tokenizer is debugging with one eye closed.
Unigram LM tokenization is arguably better but less popular. SentencePiece’s unigram mode produces probabilistically motivated, often cleaner segmentations, but BPE’s momentum, tooling, and mindshare keep it dominant. “Better on paper” lost to “already everywhere.”

Verdict

Tokenization is a 1994 compression hack doing a job it was never designed for, and doing it well enough that the entire industry standardized on it almost by accident. Byte-pair encoding is genuinely elegant: a single greedy rule — merge the most frequent adjacent pair — run over a corpus recovers morphology, keeps the vocabulary fixed and finite, and guarantees any input can be represented without unknowns. Byte-level BPE closed the last gap by making raw bytes the base, so the round trip is lossless for every string in every language. If you build or operate anything on top of a language model, this is the layer whose behavior you should be able to predict.

But it is also the layer where the most surprising failures are born. The model cannot spell because the tokenizer hid the letters. It fumbles arithmetic because the tokenizer scrambled place value. It charges a Burmese user five times more than an English user for the same sentence because the merges were learned from English text. None of these are model bugs; they are tokenizer artifacts that the model faithfully inherits. The practical lesson is blunt: when a model does something inexplicable with text, look at the token stream before you look at anything else. Half the time the answer is sitting right there in the split, decided years earlier by a frequency count over a web crawl no one remembers.