Transformers power almost every modern language model, and the explanations are everywhere. Yet for most people the thing stays stubbornly abstract: boxes labelled "attention" and "feed-forward" with arrows between them, and no real sense of what is moving through those boxes.
The fastest way past that is to stop reading diagrams and start poking a real one. So below is a working transformer. It loads an actual GPT-2 model, the 124-million-parameter version OpenAI released, and runs it entirely in your browser. You type a prompt, and it shows you the genuine next-token probabilities the model computes, live. Nothing is sent to a server.
Load it, type something, and press run. Then come back and read the walkthrough underneath, which explains each stage you just watched.
This loads a real 124-million-parameter GPT-2 model and runs it entirely on your device. Nothing you type is sent anywhere. Press the button above to begin.
What just happened, stage by stage
When you pressed run, your text travelled through the same pipeline every transformer uses. Here is what each step was doing.
1. Tokenization: text becomes numbers
A model does not see letters. The first thing it does is chop your text into tokens using a fixed vocabulary of about 50,000 pieces. Common words are single tokens, rarer words get split into fragments, and the leading space is part of the token (that is why the tool shows a small dot for spaces). Each token is just an integer ID. Notice how "strawberry" might break into two or three pieces while "the" is one: that fragmentation is exactly why models sometimes miscount letters in a word.
2. Embedding: giving each token meaning and position
Each token ID is used to look up a vector of 768 numbers, its embedding. Think of it as the token's coordinates in a space where related meanings sit near each other. Because a bare set of vectors has no order, the model then adds a positional signal so it knows which token came first, second, third. From here on, everything is vectors.
3. Self-attention: how tokens gather context
This is the heart of the transformer, and the part worth slowing down for.
Every token produces three vectors: a query (what am I looking for?), a key (what do I offer?), and a value (what I will hand over). To build its next representation, a token compares its query against the key of every earlier token, and those comparisons become weights. A high weight means "pay attention to that token," and the result is a weighted blend of the other tokens' values.
This is how a model links "it" back to the noun it refers to, or connects "Paris" to "France" a few words later. GPT-2 stacks twelve of these attention blocks, each refining the context a little more. The panel in the tool shows a schematic of the last token attending to earlier ones. It is illustrative rather than the model's exact internal weights, because the browser build of GPT-2 does not expose those, but the idea it shows is the real one.
4. The feed-forward network: where knowledge lives
After each attention step, every token passes through a small two-layer network that expands it from 768 numbers to 3,072 and back. Attention moves information between tokens; this feed-forward layer processes each token on its own, and it is where a lot of the model's factual and stylistic "knowledge" is actually stored. Attention and feed-forward alternate, twelve times.
5. Unembedding and softmax: turning vectors back into words
After the final block, the vector for the last token is projected back onto the entire vocabulary. That produces one raw score, a logit, for every possible next token, all 50,000 of them. Softmax then squashes those scores into probabilities that sum to 100%. The bars you see in the tool are those real probabilities.
The knobs that make it feel creative or robotic
A model does not have to pick the single most likely token. How it chooses is controlled by three settings you dragged, and they matter enormously in practice.
- Temperature reshapes the distribution before choosing. Turn it down towards zero and the top token dominates, so the output becomes safe, deterministic and often repetitive. Turn it up and the probabilities flatten, so unlikely tokens get a real chance and the text becomes more varied, more surprising, and more prone to going off the rails.
- Top-k keeps only the k most likely tokens and throws the rest away before sampling. It is a blunt cap on how many options are on the table.
- Top-p (nucleus) is smarter: it keeps the smallest group of tokens whose probabilities add up to p. When the model is confident, that group is tiny; when it is unsure, the group grows. This trims the long tail of junk tokens without a fixed limit.
In the tool, candidates that fall outside your current top-k and top-p pool are struck through: they cannot be chosen no matter what. Watching that pool grow and shrink as you move the sliders is the clearest way to understand why the same model can sound crisp and factual or loose and inventive depending purely on these numbers.
Why this is worth understanding
Every large model today, from the ones writing code to the ones answering support tickets, is this same machine scaled up: more layers, more dimensions, far more training data, and refinements on top. The mechanism you just ran is the core. Once you have felt how tokenization fragments words, how attention pulls context together, and how temperature trades safety for creativity, a lot of otherwise mysterious model behaviour, from confident wrong answers to why prompts phrased differently give different results, starts to make sense.
If you want to go deeper on the ideas next to this, our explainers on what an AI agent's memory really is and the model context protocol build on the same foundations.
Sources and credit
- This interactive tool was inspired by the excellent Transformer Explainer from Georgia Tech's Polo Club of Data Science, which pioneered running GPT-2 live in the browser for teaching. Ours is an independent, lighter-weight build.
- The model is GPT-2 (124M), run in-browser via Transformers.js.
- Vaswani et al., "Attention Is All You Need" (2017), the paper that introduced the transformer.
Frequently asked questions
- How does a transformer predict the next word?
- It turns your text into tokens, converts each token into a vector, mixes in context through self-attention across many layers, and finally projects the last token's vector onto the whole vocabulary to produce a score for every possible next token. Those scores become probabilities through softmax, and one token is chosen. The process then repeats with the new token added.
- What is temperature in a language model?
- Temperature is a knob that reshapes the probability distribution before a token is picked. Low temperature makes the highest-probability token dominate, giving safe, repetitive output. High temperature flattens the distribution, letting less likely tokens win more often, which produces more varied and creative but riskier text. You can feel this directly with the temperature slider on this page.
- What is the difference between top-k and top-p sampling?
- Top-k keeps only the k most likely next tokens and samples from those. Top-p, also called nucleus sampling, keeps the smallest set of tokens whose probabilities add up to at least p, so the pool grows or shrinks depending on how confident the model is. They are often used together to cut off the long tail of unlikely, low-quality tokens.
- Is the GPT-2 model on this page really running in my browser?
- Yes. It loads a real 124-million-parameter GPT-2 model through Transformers.js and runs entirely on your device using your CPU. Nothing you type is sent to any server. The first load downloads the model once, then it is cached for later visits.
- What is self-attention in simple terms?
- Self-attention is how each token looks at the other tokens to gather context. Every token forms a query and compares it against the keys of earlier tokens to decide how much to borrow from each of their values. This is what lets a model connect a pronoun to the noun it refers to, or a verb to its subject, across a sentence.