Token ids

How text tokens become the integers the network actually operates on.

Token ids are the bridge between language and arithmetic. After the tokeniser segments text into a sequence of surface strings, each string is looked up in the vocabulary and replaced with its position index — a plain integer. "the" might become 262; space becomes 220; "fox" becomes 21831. From this point onward the model never sees characters again. It only ever works with these integer indices.

The integers matter for two reasons. First, they are used as row selectors into the embedding table: to represent token 262 the network reads row 262 of a matrix of shape [V, dmodel], producing a vector of dmodel floating-point numbers. Second, at the output end the final linear layer produces one logit per vocabulary entry — a vector of length V — and the argmax (or sampler) returns the integer id of the predicted next token, which maps back to a surface string only at the very end. Everything in between — attention, feed-forward, residual additions — operates on vectors indexed by these ids.

The integer value itself carries no meaning beyond its position in the vocabulary. Token 262 ("the") is not numerically "close" to token 263 because they share a prefix; proximity in id space is arbitrary. The embedding layer's job is to replace that arbitrary index with a learned, geometrically meaningful vector. The chips below show each token's surface string on top and its integer id underneath — the transition from the text row to the numbers row is exactly what the tokeniser delivers to the network.