Cache occupancy

How the KV cache fills as generation proceeds — positions stored, memory used, and when it is full.

The KV cache is a fixed-size buffer indexed by sequence position. Each time a token is processed it claims one slot, writing its key and value tensors for every layer. Occupancy is the ratio of slots filled to the total context window: for a model with a 128-token context that has processed 11 tokens so far, occupancy is 11/128 — the stat badge shown in the diagram. When occupancy reaches 1.0 the cache is full and the model can no longer extend the sequence without eviction or truncation.

The memory cost is concrete and calculable: for each position, each layer stores a key vector and a value vector of size headdim × numheads (equal to dmodel). Across all layers and both tensors the cache holds 2 × numlayers × dmodel values per position. For a typical small model (dmodel = 256, num_layers = 4) that is 2 048 floats per position; at 4 bytes each, 128 positions costs about 1 MB. Larger models multiply accordingly.

In the diagram, positions from the original prompt (brighter border) are the first to fill the cache; generated tokens follow (normal border). The teal shading on each cell is the attention weight the current token places on that position — heavy on the prompt cat characters, lighter on spaces and recent generated tokens. The amber-highlighted rightmost cell is always the current token: it is writing to the cache right now and its attention query spans the entire filled lane.

Each cell is one cached position. The stat badge (top-left) shows positions used vs. the context window. Teal depth shows attention weight from the current token (amber); brighter borders mark prompt positions.