Transformer Models Explained for Programmers

A transformer is a neural network built around one operation, attention, which lets every word in a sequence look at every other word and decide how much each one matters. That single idea is why the same architecture powers translation, chatbots, code assistants and image generators. This is how it works, described for someone who writes code rather than someone who does linear algebra.

Step 1: text becomes tokens#

A model cannot read letters. Text is split into tokens — common words, word fragments and punctuation — and each token has an integer ID from a fixed vocabulary of a few tens of thousands.

"Transformers are surprisingly simple."
-> ["Transform", "ers", " are", " surprisingly", " simple", "."]
-> [8291, 364, 527, 25839, 4382, 13]

Rare words become several tokens; common ones are a single token. This is why models are priced and limited in tokens rather than words.

Step 2: tokens become vectors#

Each token ID is looked up in a table to get an embedding: a list of a few hundred to a few thousand numbers. The table is learned during training, and it ends up placing similar tokens near each other. Add a second vector encoding the token’s position in the sequence, because attention itself has no idea about order.

import numpy as np

vocab_size, d_model, seq_len = 50_000, 512, 6
embedding = np.random.randn(vocab_size, d_model) * 0.02     # learned in reality
position  = np.random.randn(seq_len, d_model) * 0.02        # learned or fixed

ids = np.array([8291, 364, 527, 25839, 4382, 13])
x = embedding[ids] + position                               # shape (6, 512)

From here on, the model only ever manipulates that (sequence length, d_model) matrix.

Step 3: attention#

Attention answers one question for each token: which other tokens should I pay attention to, and by how much? It does this with three learned projections of every token’s vector:

• a query: what am I looking for?
• a key: what do I contain?
• a value: what will I contribute if attended to?

Each token’s query is compared with every token’s key. High similarity means high attention. The scores are turned into weights that sum to one, and the output for that token is the weighted sum of all the values.

def attention(x, Wq, Wk, Wv):
    Q = x @ Wq                                   # queries   (seq, d)
    K = x @ Wk                                   # keys      (seq, d)
    V = x @ Wv                                   # values    (seq, d)
    scores = Q @ K.T / np.sqrt(K.shape[1])       # every token vs every token (seq, seq)
    weights = np.exp(scores) / np.exp(scores).sum(axis=1, keepdims=True)   # softmax rows
    return weights @ V                           # weighted mix of values   (seq, d)

d = 512
Wq, Wk, Wv = (np.random.randn(d, d) * 0.02 for _ in range(3))
out = attention(x, Wq, Wk, Wv)

That is the entire mechanism. Everything else in a transformer is arranged around making this operation run many times, in parallel, at scale.

In the sentence “The cat sat on the mat because it was tired”, attention is what lets the token for “it” put most of its weight on “cat”. No rule says so; the projections learned that pattern from data.

Multiple heads#

One attention pass can capture one kind of relationship. Running several in parallel with different learned projections — “heads” — lets one head track grammar, another track which pronoun refers to what, another track position. Their outputs are concatenated and projected back to d_model. A typical layer has between 8 and 128 heads.

Step 4: the block, repeated#

A transformer block is:

1. Multi-head attention over the sequence.
2. Add the input back (a residual connection) and normalise.
3. A small feed-forward network applied to each token separately.
4. Add and normalise again.

The feed-forward part is where most of the parameters live and where facts appear to be stored; attention is where information moves between positions. Stack that block anywhere from 6 times (a small model) to over 100 times (a frontier model), and the output of the last block is a vector per token that encodes its meaning in context.

Step 5: predicting the next token#

For a language model, the final vector at the last position is multiplied by the embedding table in reverse to produce one score per vocabulary entry. Softmax turns those into probabilities; the model picks one (the most likely, or sampled with some randomness), appends it, and runs again. Generating a paragraph is that loop, one token at a time.

"The capital of France is" -> P(" Paris") = 0.91, P(" the") = 0.02, ...

Training is teaching the model to assign high probability to the token that actually came next, across trillions of tokens of text. Nothing else. Every apparent capability — answering, summarising, writing code — emerges from predicting the next token well enough.

Causal masking#

When generating, a token must not see the future. The attention scores for later positions are set to minus infinity before the softmax, so their weights become zero. That mask is the only difference between a model that reads whole documents (an encoder, used for classification and embeddings) and one that writes them (a decoder, used for generation).

Why it replaced what came before#

Recurrent networks processed text one token at a time, carrying a hidden state forward. Two problems: information from early in a long text faded by the end, and the sequential dependency meant no parallelism, so training was slow. Attention solves both. Every token reaches every other token in one step, and every position is computed at once, which is exactly what GPUs are good at. Scale followed, and capability followed scale.

Where to go next#

• Large language models explained — what training a transformer on text produces.
• What is fine-tuning? — adjusting a trained transformer.
• RAG explained — giving a transformer a real lookup.