Tokenization

Turning text into tokens into integer ids — everything downstream operates on ids, not text.

In this section

• Vocabulary — The fixed set of tokens a model knows, and why its size matters.
• Greedy longest-match — How the tokenizer segments text deterministically into known tokens.
• Characters vs sub-words — Single-character tokens vs word / BPE sub-word tokens, and the trade-offs of each.
• Token ids — Why every stage past the tokenizer works on integers, not text.