llamers.

Understand

Standalone, bite-sized lessons for every concept on the playground. Start anywhere — each page stands on its own.

Orientation

  • Overview — What a decoder-only transformer is, and what it means to watch one think end to end.
  • The forward pass — The left-to-right journey of one token through the whole stack — the spine the animated pulse walks.
  • Autoregressive generation — How the model samples one token at a time and feeds it back in to grow the sequence.
  • Reading the diagram — The legend: rails vs ops vs weight chips vs controls, the teal/amber feature bars, and the pulse.

Input

  • The prompt — The text you type, and what pressing Generate sets in motion.
  • The input token — The single integer id fed into the network this step.

Tokenization

Turning text into tokens into integer ids — everything downstream operates on ids, not text.

Embedding & the residual stream

  • Embedding — Looking up a token id's learned vector — where text becomes numbers.

The residual stream

The running vector the whole network keeps refining as it flows right.

Normalization

  • RMSNorm — Rescaling the vector to unit root-mean-square before each sub-layer (Llama-style pre-norm).

Attention

Attention

The mechanism a token uses to read information from earlier tokens.

Feed-forward

The feed-forward network

The per-token MLP applied after attention — here the SwiGLU variant.

The block & the stack

  • The transformer block — Attention + feed-forward + two norms + two residual adds, as one repeatable unit.
  • Stacking layers — Repeating N identical blocks, and what depth buys the model.

Output

  • Final norm — The last RMSNorm before the vector is projected to the vocabulary.
  • Logits — The output (un-embedding) projection: one score per vocabulary token.
  • The output token — The id the sampler chose, emitted and fed back as the next step's input.
  • The generation loop — The feedback loop that regrows the sequence, and the max-length stop condition.

Sampling

Turning logits into a single chosen next token.

Memory & performance

The KV cache

Storing each position's keys and values so attention isn't recomputed every step.

Weights & parameters

  • Weights — The learned matrices — the numbers training changes; everything else is computed from them.
  • Weight layout — The [outDim, inDim] convention that makes y = W·x a plain matrix-vector product.
  • Parameters — Counting a model's parameters, and where they live across the stack.
  • Dense vs Mixture-of-Experts — Whether every weight runs for each token, or a router activates only a few experts.
  • Active vs effective parameters — Total vs active (compute) vs effective (memory), and why they diverge only at scale.

Training

Training

What changes when you flip the playground into training mode.

Beyond this model

The wider LLM landscape, beyond this small dense base model.

Mixture-of-Experts

Holding many feed-forward experts but running only a few per token — capacity decoupled from compute.

(More in progress — placeholders below.)