How an LLM Actually Works

011. The whole thing in one sentence

An LLM is a function that takes a sequence of tokens and returns a probability for every possible next token. Generation is that function called in a loop: pick a token, append it, call again. Everything else in this part is detail inside that sentence.

As code, the entire runtime looks like this:

tokens = tokenize(prompt)
while not done:
    probs = model(tokens)       # one probability per vocabulary entry
    next_token = sample(probs)  # pick one
    tokens.append(next_token)
print(detokenize(tokens))

Three things are worth noticing before we open up model.

• The model is a pure function. Its weights are billions of numbers fixed at training time. They are identical for every user and every call, and nothing you type changes them.
• All state lives in the token list. The model has no session, no memory and no hidden scratchpad between calls. If it is not in tokens, the model does not know it.
• Chat is a formatting convention. A conversation is serialised into one long token sequence with special markers for system, user and assistant turns. The model simply continues that document.

The pipeline inside one call to model has five stages:

Text becomes token IDs, IDs become vectors, the vectors pass through a deep stack of identical blocks, and the final vector becomes a score for every token in the vocabulary. Sections 2 to 6 take these stages in order. Section 7 draws out what the design implies.

022. Tokenisation

A model never sees characters or words. It sees integers, each one an index into a fixed vocabulary of text fragments called tokens. The tokeniser is a separate, deterministic piece of code that runs before and after the neural network.

Why fragments

Characters would make sequences very long, and every step costs compute. Whole words would need an unbounded vocabulary and would fail on typos, new names and code. Sub-word fragments sit in between: common words are one token, rare words are several, and anything at all can be spelled out from smaller pieces.

Current models use vocabularies of roughly 30,000 to 250,000 tokens, and most recent ones use more than 100,000. The usual algorithm is byte-pair encoding (BPE). It starts from raw bytes and repeatedly merges the most frequent adjacent pair in a large text sample until the vocabulary is full. The result is a lookup table, learned once and then frozen.

Useful rules of thumb for English prose: one token is about four characters, or about three quarters of a word. A 1,000-word document is roughly 1,300 tokens. Code, numbers and other languages behave differently, which is where the consequences start.

What this explains

• Letter-level blind spots. Asked how many times a letter appears in a word, a model is reasoning about one or two opaque IDs, not a string of letters. It has to have learned the spelling of that token as a fact. Newer models mostly have, but the weakness is structural.
• Arithmetic quirks. Long numbers are split into chunks of digits, and the chunking does not align with place value. Models now handle this far better, and reach for a code tool when precision matters, but it is why raw mental arithmetic was an early embarrassment.
• Language cost. Tokenisers are trained mostly on English and code. The same sentence in Polish, Hindi or Japanese can take two to four times as many tokens. That means higher cost, slower output and less effective context for the same content.
• Whitespace and code. Indentation, brackets and common keywords get their own tokens, which is one reason code is handled efficiently. Minified or unusual formatting costs more than clean code.
• Tokens are the unit of everything commercial. Context limits, pricing, rate limits and output caps are all counted in tokens. Each vendor has its own tokeniser, so the same prompt has a different token count, and a different price, on each model.

One practical habit follows. When you estimate cost or check whether something fits in context, count tokens with the vendor’s tokeniser. Character counts and word counts will mislead you, especially for code and non-English text.

033. Embeddings

The first thing the network does is turn each token ID into a vector: a list of several thousand numbers. This is a plain table lookup. The table has one row per vocabulary entry, and its contents are learned during training like every other weight.

Meaning as geometry

Training pushes tokens that behave alike towards similar vectors. Python ends up near Ruby, Tuesday near Thursday. More usefully, consistent directions emerge: the step from walk to walked points roughly the same way as the step from jump to jumped. Nobody designs these directions. They appear because they make next-token prediction easier.

Two cautions keep this picture honest.

• Single dimensions mean nothing. There is no “formality” number at position 412. Concepts are directions that cut across many dimensions, and models pack in far more concepts than they have dimensions by letting them overlap slightly.
• The input embedding has no context. The token bank gets the same starting vector in “river bank” and “bank loan”. Sorting that out is the job of the layers above.

From static to contextual

Think of each position in the sequence as carrying a vector up through the stack. At the bottom it means “this token, in isolation”. Each layer edits it using information from other positions. By the top, the vector at a position no longer represents the token that went in. It represents everything the model has worked out that is relevant to predicting what comes next at that point.

That top vector is then compared against an output table, one row per vocabulary entry, to produce a score for every possible next token. So the model begins and ends with the same kind of operation: a mapping between tokens and points in space.

Why this matters later

The semantic search behind RAG, which part 8 of the practical AI module covers, uses the same idea with a different model. An embedding model is trained to produce one vector for a whole passage, so that passages with similar meaning land close together. If you have worked with vector similarity, you already have the right intuition: distance in the space stands in for similarity of meaning, imperfectly.

044. Attention

Attention is how positions in the sequence share information, and it is the only place in the architecture where they do. Every other operation works on one position at a time. If you understand attention, you understand how a model uses context.

The mechanism

A developer-friendly way in: attention is a fuzzy hash map lookup. A normal map takes a key, finds the one exact match and returns its value. Attention takes a query, scores it against every key, and returns a weighted blend of all the values.

At each position the model multiplies the current vector by three learned matrices to get three new vectors:

• a query: what this position is looking for
• a key: what this position offers to others
• a value: the information it hands over if selected

The position’s query is compared with the key of every earlier position using a dot product. The scores are normalised so they sum to 1. The output is the values of all those positions, blended by those weights, and it is added to the current position’s vector.

A worked example

Take the sentence “The server crashed because it ran out of memory”. When the model processes it, the vector at that position needs to find out what it refers to. One attention head might produce these weights (illustrative numbers):

The query from it matched the key from server most strongly, so most of what flows into the it position is the value from server. After this step, the vector at it carries “this refers to the server”. A later layer can then connect “ran out of memory” to a server, not to something else.

Three details that matter

Causal masking. A position may only attend to itself and earlier positions. This is what makes left-to-right generation possible. It also makes training efficient, because every position in a document is a prediction exercise at once.

Many heads. Each layer runs dozens of attention heads in parallel, each with its own query, key and value matrices. Different heads learn different relationships: the previous token, the subject of the verb, the matching open bracket, the earlier definition of this variable. One well-studied type finds an earlier occurrence of the current token and copies what came after it. That simple circuit is a building block for learning patterns from examples in the prompt.

Position. The blend operation is blind to order. Without help, “dog bites man” and “man bites dog” would look the same. Modern models fix this by rotating queries and keys by an angle that depends on position, so that the match score depends on how far apart two tokens are.

The cost

Each new token is compared against every token before it. Work per token grows linearly with context length, so total work for a sequence grows with the square of its length. A 100,000-token prompt implies on the order of ten billion pairwise scores for each head in each layer.

This single fact drives much of part 3: why long prompts are slow to start, why providers cache the keys and values of earlier tokens, why cached input is priced lower, and why context length is an engineering trade-off and not a free upgrade.

Attention(Q, K, V) = softmax( (Q K^T) / sqrt(d_k) + mask ) V

055. The transformer block and the stack

A transformer is one block design repeated many times. Each block has two sub-layers: attention, which moves information between positions, and a feed-forward network, which processes information at each position. Small models stack a few dozen of these blocks or fewer. The largest stack many dozens, in some cases more than a hundred.

The two “Add” nodes are the important part. Each sub-layer computes an adjustment and adds it to the vector it received. Nothing is overwritten.

The residual stream

Because every sub-layer adds to the same running vector, that vector acts as a shared workspace running the full height of the model. It is usually called the residual stream. Each layer reads what earlier layers wrote, contributes a small patch, and passes it on.

For a developer, the closest picture is a pipeline where every stage applies a diff to shared state. Early stages can leave information that a stage fifty layers later picks up. This design is also what makes very deep networks trainable at all.

The feed-forward network

The feed-forward sub-layer works on each position independently. It expands the vector to several times its width, applies a simple non-linear function, and projects it back down. In a standard dense model it holds around two thirds of all the parameters.

A useful reading is that it behaves like a very large learned key-value memory. Each of its internal units detects some pattern in the incoming vector and, when it fires, writes associated information back to the stream. Research that traces factual recall, such as completing “The Eiffel Tower is in” with “Paris”, finds much of the work happening in these layers.

So a rough division of labour is: attention decides what information goes where, and feed-forward layers decide what to do with it once it arrives. It is a simplification, but a serviceable one.

Depth is serial compute

Layers run in sequence, so the number of layers is the number of sequential processing steps the model gets for each token. That number is fixed. The model spends exactly the same compute producing the next token of “2 + 2 =” as the next token of a subtle architectural judgement.

Remember this for section 6. The only way a model can spend more computation on a hard problem is to generate more tokens.

Where the parameters are

When someone quotes a parameter count, almost all of it is the attention and feed-forward matrices, multiplied by the number of layers. One well-documented open model with 70 billion parameters has 80 layers, a vector width of 8,192 and 64 attention heads. The embedding tables are a rounding error by comparison.

Mixture of experts

Most of the largest models are now believed to be mixture-of-experts (MoE) designs. The change is confined to the feed-forward sub-layer. Instead of one large network, each layer holds many smaller ones, called experts, plus a small router. For each token, the router picks a few experts to run and ignores the rest.

This splits the parameter count into two numbers:

• Total parameters set how much the model can know and how much memory it needs to serve.
• Active parameters set how much compute each token costs, and therefore speed and price.

One published open model has 671 billion total parameters but activates about 37 billion per token, choosing 8 of 256 experts in each layer. It stores knowledge like a very large model and runs at roughly the cost of a mid-sized one. The trade is not free: an MoE model is generally weaker than a dense model of the same total size, and it needs all its experts held in memory.

For conversations, the takeaway is that “how big is it?” now has two answers, and they predict different things. Total size hints at capability. Active size hints at cost and latency.

066. From numbers to text

The network’s output is not a token. It is a score for every token in the vocabulary, and ordinary code outside the network decides which one to use. That separation explains randomness, reproducibility settings and structured output.

Logits and probabilities

The vector at the final position is compared against the output table to give one raw score per vocabulary entry. These scores are called logits. A softmax converts them into probabilities that sum to 1.

The shape of that distribution depends on the context (illustrative numbers):

A sharp distribution means the model is confident about the next token. It does not mean the token is true. Confidence here is about text likelihood, and part 3 shows how that gap produces hallucination.

Sampling controls

• Temperature divides the logits before softmax. Below 1 it sharpens the distribution towards the top choice. Above 1 it flattens it and lets unlikely tokens through. At 0 the sampler simply takes the highest-scoring token every time.
• Top-p keeps the smallest set of tokens whose probabilities add up to p, for example 0.95, and samples only from those. It cuts off the long tail of nonsense options while keeping real alternatives.
• Top-k keeps a fixed number of top tokens. It is cruder and less common now.

In practice, use low temperature for extraction, classification and code edits, where you want the same answer each run. Use the default or higher for drafting and brainstorming, where variety is the point. Some reasoning models fix these settings on the vendor side, so check before you build around them.

What the loop implies

Output is random by construction. The same prompt can give different answers because the sampler rolls dice. This is a feature of the design and not a defect. Any process you build on a model has to tolerate it, through validation, retries or evaluation over many runs.

There is no backspace. Once a token is emitted it becomes input, and the model treats its own earlier output as given. An early wrong turn tends to be followed by fluent text that justifies it. A model can correct itself, but only by appending “actually, that is wrong”. It cannot quietly revise.

Thinking out loud is real computation. Section 5 showed that compute per token is fixed. Writing intermediate steps buys more passes through the network, and puts partial results into the context where attention can use them. This is why asking for reasoning before the answer improves accuracy. Reasoning models are trained to do this at length before replying, which part 2 covers. Those thinking tokens are billed as output.

Input is cheap, output is slow. All prompt tokens can be processed in one parallel pass. Output tokens must be produced one at a time, each needing a full pass through every layer. That asymmetry is why output is priced several times higher than input and why long answers feel slow.

Stopping is learned. The model ends a reply by emitting a special end-of-turn token. If it hits your maximum token limit first, the output is simply cut off, mid-sentence or mid-JSON.

The sampler can be constrained. Because token choice is ordinary code, a system can mask out any token that would break a grammar or a JSON schema. That is how structured output modes guarantee valid syntax. They guarantee the shape, not that the content is right.

077. What falls out of this design

Six properties follow directly from sections 1 to 6. They account for most of the confusion you will meet when people reason about what these systems can do.

The model is stateless; products are stateful

Nothing persists inside the model between calls. A long chat works because the application re-sends the whole conversation every turn. Product features called “memory” work by storing notes elsewhere and inserting them into the prompt. When behaviour seems to carry over, look for what the application put in the context.

Recall is reconstruction, not lookup

There is no database inside the model. Facts exist only as statistical associations spread across billions of weights. Recalling one means regenerating it, in the same way the model generates everything else.

This is why a model can be right about the gist and wrong about the version number, the date or the citation. It is also why it cannot tell you where it learned something, and why it has no built-in signal for “I have no record of that”.

Two kinds of knowledge

Most practical AI engineering, which part 3 of the practical AI module covers, is the work of getting the right information into the context at the right time. When precision matters, put the source in the prompt and do not rely on recall.

In-context learning

A model can pick up a pattern from the prompt and apply it immediately, with no change to its weights. Show it three examples of your commit message format and the fourth follows suit. Give it your team’s conventions file and its code shifts to match.

This is the most useful property for day-to-day work, and it is temporary by nature. When the context ends, the learning is gone. Anything you want a model to “know” reliably has to be supplied every time.

It does not learn from your conversations

Typing into a model does not update it. Weights change only when the vendor runs a new training process. Whether a vendor may use your conversations as training data for a future model is a separate, contractual question. Business and API terms generally exclude it by default, and part 3 of the AI in the organisation module covers how to check.

Parameter count is capacity, not quality

More parameters give a model more room to store knowledge and skills. How well that room is used depends on the training data, the training recipe and the post-training, which part 2 covers. A recent small model regularly beats a large model from two years earlier. Treat a parameter count like a server’s RAM: relevant, but not a performance figure.

The whiteboard version

Text is split into tokens. Each token becomes a vector. The vectors climb a tall stack of identical layers, where attention moves information between positions and feed-forward layers process it. The top vector becomes a probability for every possible next token, one is sampled, and the loop repeats. The weights never change while you use it, and the prompt is the only thing it knows about you.