Character vs sub-word tokenisation

How fine to slice text into tokens — characters, whole words, or something between — and what each choice costs.

Character tokenisation treats every individual character as its own token. It needs no vocabulary decisions up front: any string is representable, and the vocabulary stays tiny (e.g. lowercase letters + space). The cost is sequence length — "transformer" becomes 11 tokens, so the model must learn how characters combine into meaning over many attention steps. A teaching setup often uses character tokenisation precisely because it keeps things transparent and keeps V small.

Word tokenisation sits at the opposite extreme: one token per whole word. A sentence becomes just a handful of tokens — short sequences, few forward passes, and each token already carries meaning. The price is a huge vocabulary (tens or hundreds of thousands of entries) and, worse, no composition: a word form the vocabulary never saw — a plural, a tense, a typo, a name — simply cannot be represented (it falls out of vocabulary). Sub-word tokenisation (BPE, WordPiece, SentencePiece) is the middle ground: common words stay whole, rarer ones split into known pieces ("bro" + "wn"), so sequences shrink without losing the ability to spell out anything.

The counter-intuitive part is where the cost goes. Smaller tokens (characters) make the vocabulary — and therefore the embedding table, output projection, and softmax (all sized by V) — smaller, not bigger; what they cost is sequence length, i.e. more forward-pass runs to read the same text. Larger tokens (words) flip it: few runs, but large V-sized matrices and lost coverage.

A subtlety the figure makes concrete: *most of a model's size is not the vocabulary. Model weights are the fixed transformer body — the attention and feed-forward matrices, scaling as roughly 12·d²·layers — plus the V·d embedding table. The body is where the gigabytes live, and the tokenizer never touches it; it only grows or shrinks the embedding slice on top. That is why real models are multi-gigabyte regardless of tokenizer (large d, many layers), and why our small reference moves only modestly across the triangle. Drag the point below: against that fixed reference it approximates vocabulary, tokens per page (runs), model size / RAM (shown split into the fixed body and the embedding slice), compute per page (≈ runs × 2·params), and coverage. Characters spend their budget on many runs; words spend it on a huge vocabulary* and lose coverage.

For a fixed reference model: vocabulary, tokens per page (runs), model size/RAM (split into the fixed transformer body and the V·d embedding the tokenizer moves), compute per page, and coverage. Most of the size is the body — tokenization only nudges the embedding slice.

For a concrete look at one point in that space, the chips below show a BPE-style segmentation of "the quick brown fox". Amber-outlined chips are multi-character sub-words; plain chips are single-character tokens. Notice "brown" splits into two pieces — neither "bro" nor "wn" is a standalone word, but both were frequent enough in training data to earn a slot.

Amber chips are multi-character sub-words; plain chips are single-char tokens. Same surface text, fewer chips than pure character tokenisation.

Related: Vocabulary · Greedy longest-match · Token ids