Vocabulary

The fixed set of symbols a model can ever read or write.

Vocabulary is the complete list of tokens the model recognises — its entire alphabet. Every string the model processes must be expressible as a sequence drawn exclusively from this list. A teaching vocabulary can be made deliberately small — for example the 26 lowercase letters a–z plus the space character, giving 27 tokens in total. That constraint keeps the math transparent without changing anything about how tokenisation works.

The vocabulary is fixed at training time and never changes at inference time. Its size, written as V, directly sets the width of the embedding table (V × dmodel) and the final logit layer (dmodel × V). Larger vocabularies — GPT-4 has about 100,000 — let the model handle subwords, numbers, punctuation, and code in every script, at the cost of larger weight matrices and a harder softmax over more candidates.

The chips below show each surface string paired with its integer id. Amber-outlined chips are multi-character tokens (sub-words); plain chips are single-character tokens. Every chip you see is an entry in the vocabulary; any string not in this list simply cannot be produced by the model.

Each chip is one vocabulary entry: the surface string on top, its integer id below. The whole set of chips shown here is the vocabulary.