How BPE Tokenization Works

After reading this you will be able to predict roughly how many tokens a piece of text costs, explain why byte-pair encoding produces the tokens it does, and know why a model cannot count the letters in "strawberry".

What a tokenizer does and why it matters

A language model never sees your text as characters. Before any math happens, the text is chopped into tokens: short chunks of bytes, each mapped to an integer ID. The model only ever works with those integers. The tokenizer is the fixed lookup table that turns text into IDs and back.

Byte-pair encoding (BPE) is the most common way to build that table. The idea is short: start with single characters, then repeatedly glue together the most frequent adjacent pair into one new symbol. Do that a few thousand times and you get a vocabulary where common fragments like th, ing and the are single tokens, while rare strings stay in small pieces.

Here is the hook. Type strawberry into a typical tokenizer and it becomes something like str + aw + berry, three tokens, three opaque IDs. Ask the model how many r characters the word has and it has no reliable way to answer, because the individual letters were merged away before the model saw anything. The count is not a reasoning failure so much as a consequence of tokenization.

When to think about tokens, and when not to

Reach for a token count whenever money or context length is involved. API pricing is per million tokens, not per word or per character, so the token count is the number on your invoice. Context windows are measured in tokens too, so knowing that a document is 4,000 tokens rather than 3,000 words tells you whether it fits.

You can usually ignore tokenization for casual English prose, where the rule of thumb 4 \text{ chars} \approx 1 \text{ token} holds well enough. Start paying attention when your text is unusual: source code, JSON, non-English languages, long numbers, emoji, or repeated whitespace. Those cost far more tokens than their character count suggests, and the visualizer shows exactly where the cost lands.

The algorithm and its one rule

BPE training is a loop with a single decision rule. Given the current corpus split into symbols, count every adjacent pair, find the most frequent one, and merge it everywhere. Record that merge. Repeat until the vocabulary reaches the target size.

Here (a, b) is an adjacent pair of current symbols and \text{count}(a,b) is how many times that pair appears across the whole corpus. The winning pair (a,b)^* becomes a new symbol ab, and the vocabulary grows by one. The choice is greedy: it takes the best pair right now and never reconsiders.

Two consequences follow directly. First, early merges are the most common fragments in the language, because frequency drives every step. Second, the merge list is ordered. When you later tokenize new text, you apply the merges in the same order they were learned, so the vocabulary size acts like a dial: more merges means longer, fewer tokens.

A worked example you can reproduce

Take a corpus small enough to check by hand. Suppose after splitting into words and counting, you have these four word types with their frequencies:

The default corpus in the tool mixes English and code, so its first merges look like th, in, en, t and =-adjacent pairs from the code. Drag the vocabulary slider and watch the list rebuild in exactly this frequency order.

Reading the token count and the chars-per-token ratio

The most useful single number the tool reports is chars per token: total characters divided by total tokens. Higher is better, because it means each token carries more text and therefore costs less.

For ordinary English trained on a matching corpus, expect roughly 3.5 to 4.5. The sentence the quick brown fox is 19 characters. If a well-trained tokenizer turns it into 5 tokens ( the, quick, brown, fox plus a leading fragment), the ratio is 19 / 5 = 3.8. Now feed it dense JSON like {"id":42,"ok":true}. The same 19 characters might split into 11 tokens because braces, quotes and colons rarely merge, giving 19 / 11 \approx 1.7. Same length, more than double the token cost.

How vocabulary size changes the count

The comparison panel tokenizes the same input at vocabulary 300, 1000 and 2000. As the vocabulary grows, more merges are available, so common runs of characters collapse into single tokens and the count falls. The drop is fast at first and then flattens, because the early merges catch the highest-frequency pairs and each later merge helps less.

The chars-per-token ratio moves the opposite way. At 58 tokens the ratio is 200 / 58 \approx 3.4; at 34 tokens it is 200 / 34 \approx 5.9 for text that matches the corpus. Production tokenizers use vocabularies of 30,000 to 130,000 or more, far past the flat part of this curve, so their ratios for in-distribution English sit near 4.

Turning tokens into money

API providers bill per million tokens. Multiply your token count by the price and divide by a million.

Say a request sends 39 tokens at 2.50 dollars per million input tokens. The cost is 39 / 1{,}000{,}000 \times 2.50 = 0.0000975 dollars, about a hundredth of a cent. That looks negligible until you multiply by volume. At one million such requests per day the input alone is 39 \times 10^6 / 10^6 \times 2.50 = 97.50 dollars per day, roughly 2,925 dollars per month. If your text is dense JSON that doubles the token count, so does the bill.

Common mistakes

The errors below all come from treating tokens as if they were words or characters.

Counting words instead of tokens

Words and tokens diverge fast. A 750-word page of English is roughly 1,000 tokens, but the same page of code or JSON can be 2,000 or more. Budget in tokens.

Ignoring the leading space

the and the (with a leading space) are different tokens with different IDs. The tool marks the space with a visible ␣. This is why concatenating strings without spaces can change token counts unexpectedly.

Expecting the model to count characters

Once strawberry is three token IDs, the letters are gone. Character-level questions ask the model to recover information the tokenizer discarded.

Assuming all text costs the same per character

Text far from the training corpus falls back to single characters, so the same meaning in another language or format can cost three to five times more tokens.

Related tools

Once text is tokens, the model predicts the next one. The Next-Token Sampling Playground shows the actual next-token distribution of a small model and lets you reshape it with temperature, top-k and top-p, so you can see what happens after tokenization. When you are ready to turn token counts into a budget across models and context sizes, the LLM API Cost & Context Planner estimates spend per day and month, including RAG context and prompt caching.

Related tools

Next-Token Sampling Playground

See the actual next-token distribution of a small language model and reshape it live with temperature, top-k and top-p.

AI Learns to Drive (Neuroevolution)

Watch a population of neural-network cars evolve from crashing into corners to lapping a race track.

Backpropagation Step-by-Step

Walk through one forward and backward pass of a tiny neural network, number by number.

Decision Tree Builder

Grow a CART decision tree split by split and watch the partition map and tree diagram build together.

Diffusion Model Toy

Train a tiny denoising diffusion model and watch it turn pure noise back into a 2D shape.

Distribution Sketcher

Sketch the shape of a probability density — or paste raw data — and search 16 distributions for the ones that fit it best.