Cache correctness

The KV cache must be behaviourally transparent: greedy output is identical whether the cache is on or off.

Behavioural transparency is the formal requirement that the cache is a pure optimisation — it speeds up inference without changing what the model says. If a model generates "the cat sat" greedy without caching, it must generate exactly "the cat sat" greedy with caching. Any divergence is a bug, not a trade-off.

Why is this non-trivial? The cache is correct only if every stored key and value tensor is numerically identical to what a full recompute would produce for that position, and the attention computation for the current token reads exactly the correct t past positions — no fence-post errors in the position index, no stale entries from a previous generation, no off-by-one in the causal mask. Rotary position encodings (RoPE) add a subtlety: each key is encoded with the absolute position of the token that wrote it, and the query is encoded with the position of the current token; these offsets must agree with what a full run would compute. Getting this right requires careful bookkeeping of the write pointer as each new token fills its slot.

The project verifies this invariant with a dedicated test: a short sequence is decoded step-by-step with the cache enabled, then decoded again in a single full-sequence pass with the cache disabled, and the logits at every step are compared to floating-point equality. That test must stay green after any change to the attention layer, the position encoding, or the cache itself. It is the single check that separates "fast and correct" from "fast and subtly wrong."

The cache is transparent: every cell stores exactly the key/value a full recompute would produce. The test suite asserts token-for-token identical output with the cache on or off.