Tokenization
A Socratic walk-through of tokenization — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why can a model that writes flawless prose fail to count the letters in a single word?
Here is a failure that looks like a joke and is actually a clue. A system that can draft a contract clause, explain a proof, and refactor a function will, when asked how many times the letter r appears in strawberry, sometimes answer two.
The strange part is not that it gets a hard question wrong. It is that it gets an easy question wrong while getting hard ones right. Whenever difficulty and success run in opposite directions like that, the sensible move is to stop asking "why is it bad at this" and start asking "what is it actually looking at?" Because a task is only easy relative to what you can see.
Reasoning it through
REASONING #Start from a constraint. The model is a mathematical function over numbers; text has to be turned into numbers before it can enter. So somebody had to choose the unit.
Consider the options as a designer would. One number per character? Then the vocabulary is tiny and nothing is ever unrepresentable — but a page of text becomes a very long sequence, and the model must spend capacity rediscovering that t-h-e is a word. One number per word? Sequences get pleasantly short, but the vocabulary is unbounded: every name, typo, compound and inflection needs its own entry, and anything unseen becomes a blank.
So the field settled on the compromise between them: sub-word units. The dominant method, byte-pair encoding, is worth understanding because it is almost embarrassingly simple. Start with individual bytes. Count which adjacent pair occurs most often across the corpus, and merge it into a single new unit. Repeat, some tens of thousands of times, recording the merges in order. Frequent words end up as one unit; rare words end up as two or three pieces; anything at all, including emoji and languages the corpus barely contained, can still be spelled out from bytes. Typical vocabularies run from roughly fifty thousand to a couple of hundred thousand entries.
Now trace what that does to our question. The word strawberry is common enough that the tokenizer will have merged it into a small number of chunks — in several widely used vocabularies, something like str, aw, berry. What arrives at the model is not ten letters. It is two or three integers, each pointing at a learned vector.
So ask the question the way the model receives it: given the vectors for those chunks, how many r's are there? Nothing in that input is an r. The letters are not components the model can inspect; they are properties of a symbol it has only ever encountered whole. It is closer to being asked how many strokes are in a Chinese character you know the meaning and sound of but have never written.
Which raises the obvious objection: then how does it ever get spelling right? Because spelling information is recoverable indirectly. The corpus contains dictionaries, acrostics, spelling-out ("s-t-r-a-w-b-e-r-r-y"), word games, and code that manipulates strings, so the vectors do carry some sub-token information — learned as a fact about the token rather than read off it. That is why the failure is erratic rather than absolute: it is knowledge, and knowledge can be patchy. It is also why newer models often answer the strawberry question correctly. That is not because tokenization was solved; it is because the question became famous enough to be well represented in training and evaluation.
The same root explains a family of other oddities. Arithmetic is fragile when a tokenizer chops numbers into inconsistent chunks, so that 1,234 and 1,235 have unrelated internal structure — several recent tokenizers now deliberately split digits individually or in fixed groups to reduce this. Text in languages poorly represented in the merge table costs several times more tokens for the same meaning, which is a real cost and context penalty for those users. And a stray trailing space can change the tokenization of the following word, which is why prompts sometimes behave differently for reasons that look like superstition. There was even a period in 2023 when researchers found strings such as SolidGoldMagikarp — tokens present in the vocabulary but nearly absent from training text — that provoked bizarre behaviour, precisely because those entries had never been meaningfully learned.
The analogy
THE ANALOGY #Think of someone who learned to read entirely from whole-word flashcards, never phonetically. They read fluently, at speed, with full comprehension. Ask them what the third letter of a word is and they must picture the card and squint at it — possible, but a different and much shakier operation than the one their fluency is built from.
the flashcard reader can always look at the card again and simply read the letters off, whereas the model has no card to look at — the letters are absent from its input entirely, so its only route is memory of having been told.
Clarifying the model
THE MODEL #Three refinements. First, tokens are not syllables or morphemes. They are whatever statistics produced, so the pieces frequently cut across meaningful boundaries and the split of a word can look arbitrary. Nothing linguistic is guaranteed.
Second, this is not a defect of one product. It is a design trade — shorter sequences and a bounded vocabulary, paid for with an opaque relationship to characters — and every mainstream model makes some version of it. Character-level and byte-level models exist and do not have this weakness, but they pay in sequence length and compute.
Third, be careful not to over-explain with it. "Tokenization" has become a catch-all excuse for any model error, and it does not cover most of them. It genuinely explains character-counting, some arithmetic, rhyme and anagram trouble, and cross-language cost differences. It does not explain a fabricated citation or a reasoning slip. When a failure disappears the moment you ask the model to spell the word out first, that is good evidence tokenization was the cause — because spelling it out puts the letters into the context as tokens in their own right, where they can finally be counted.
A picture of it
THE PICTURE #How to readfollow the word down the left as it is chopped by the merge table into a handful of ids and then into vectors. At the diamond, everything depends on what the task needs: meaning travels the left branch and comes out fine, letters travel the right branch into the shaded hazard, where the model must reconstruct from memory something its input never contained. The dotted return edge is the practical fix — spelling the word out puts the letters back into the context as tokens of their own.
What became clearer
WHAT CLEARED #The model is not bad at counting. It is counting things it cannot see. Text reaches it as a few thousand-odd learned symbols, and the letters inside those symbols are, from its position, hearsay — facts it may have been told, not features it can inspect. Once you know that, the erratic pattern makes sense: fluent where the unit matches the task, shaky where the task reaches beneath the unit.
Where to go next
ONWARD #- Why languages outside the tokenizer's training distribution cost more tokens, and what that means for price and context limits.
- How character-level and byte-level models avoid this, and what they give up.
- Why digit tokenization choices measurably change arithmetic accuracy.
Key terms
TERMS #| Term | What it means |
|---|---|
| Token | the atomic unit of text a model reads and writes: usually a sub-word fragment drawn from a fixed vocabulary. |
| Byte-pair encoding (BPE) | the algorithm that builds that vocabulary by repeatedly merging the most frequent adjacent pair of units. |
| Vocabulary | the fixed set of tokens a model can represent, typically tens to hundreds of thousands of entries. |
| Embedding | the learned vector each token id is mapped to before the model does anything with it. |
Every term the collection defines is gathered in the glossary.