Embedding

Converting a token id into a learned vector — the first step of the forward pass.

Embedding is a lookup table. The model keeps a weight matrix of shape [27, 64] — one 64-dimensional row for each of the 27 tokens in the vocabulary (lowercase letters plus space). When the model processes a token it takes that token's id (an integer 0–26) and reads out the corresponding row: embed[id] → a vector of 64 numbers. No arithmetic happens here; the id is an index, and the row is the model's learned representation of that character.

Those 64 numbers are the starting values of the residual stream for that position. They encode everything the model currently "knows" about that token before any attention or feed-forward computation runs. The weights are learned by backpropagation: at first every row is random, but training nudges them so that tokens that tend to appear in similar contexts acquire similar directions in the 64-dimensional space. Space and a period might cluster together; the vowels might cluster; a capital-like pattern might emerge even with character-level tokens.

In the diagram, the embedding sits at the left end of the residual stream. The amber and teal stripes in the feature-rail below are the actual values in one row: positive dimensions glow amber, negative dimensions glow teal. At near-LOD each cell shows its raw number. All subsequent layers read from and write back to this same stream — the embedding vector is the origin of every influence the network will build up.

Related: The residual stream · The forward pass · Logits