Train on your own text

The text you paste is the entire universe the model learns — it defines the vocabulary and every pattern the weights will encode.

Your corpus is the training data: the raw text pasted into the playground's corpus field. Everything the model can ever learn comes from this text and this text alone. There is no pre-training, no transfer, no prior knowledge. If a word never appears in the corpus the model will never produce it. If a phrase appears hundreds of times the model will learn to reproduce it reliably. The corpus is the complete specification of what "good" text means for this model.

Before training begins, the playground runs a tokenizer build step. It scans the corpus, counts every unique character (this runtime uses character-level tokenisation), and assigns each one an integer ID. The vocabulary size — the number of distinct tokens — is fixed at this point and determines the shape of the embedding matrix and the final projection layer. A corpus with 50 unique characters produces a vocabulary of 50; a corpus with 80 produces 80. Adding new text later that contains new characters requires rebuilding the tokenizer and reinitialising the weights, which is why "Build model" must run before "Train live."

Training then reads the corpus as a sequence of token IDs. Each step runs a full forward pass over every position at once — at each position the model predicts the next token — and averages the cross-entropy loss across all of those predictions. One backward pass then produces gradients for every weight, and the optimizer applies the update. (So each step trains on the whole corpus, not a single sampled position.) With a tiny corpus (a few hundred characters) the model will eventually memorise the text almost perfectly — the loss will fall very low and generation will reproduce the corpus nearly verbatim. With a larger, more varied corpus the model generalises to patterns rather than memorising, and generation produces novel text that follows the same style.

Choosing a corpus is therefore the main creative decision when training in the playground. A short nursery rhyme produces a model that rhymes in a very narrow way. A passage of code produces a model that generates code-shaped text. A mix of sentence types produces a model that has seen more variation and generalises better — at the cost of needing more steps for the loss to fall to the same depth.