Every mechanism built so far, weighted sums, activation folds, attention’s queries and keys, describes one thing: how a model turns an input into an output. That description holds whether the model is being trained on its ten-trillionth token or answering the one question you just typed into a chat box. The forward pass, the sequence of matrix multiplications that produces a prediction, doesn’t know or care which situation it’s in.
And yet training a large model can occupy a warehouse of GPUs for months, while running that trained model to answer your question happens in a second or two on a fraction of that hardware. If the core computation is identical, where does that enormous difference come from?
It comes from the forward pass being only half the story in one of these jobs, and the other half changes everything about how much memory, computation, and structure the job needs.
One forward pass, borrowed by two different jobs
Strip away the labels “training” and “inference” for a moment and look at what a transformer actually does mechanically. Tokens get embedded into vectors. Those vectors pass through attention layers, where each token gathers context from its neighbors, and feed-forward layers, where each token’s vector gets folded by weights and an activation function. Eventually the last layer produces a set of scores, one per possible next token, and those scores get turned into a probability distribution. That entire sequence, embedding to final prediction, is called the forward pass, and it runs identically in both jobs. A training step runs a forward pass. Answering your question runs a forward pass. Same matrices, same attention mechanism, same folds.
Inference is the name for using an already-trained model to produce an output: you give it a prompt, it runs a forward pass, and it hands back a prediction. Nothing about the model’s weights changes during inference. The billions of numbers that make up the model are frozen, read-only, exactly as they were when training finished. Training is the process that set those weights in the first place, and it needs the forward pass too, but only as its first half. What training does with the forward pass’s output is where the two jobs stop resembling each other.
Inference runs the forward pass once per token and moves on. Training uses that same forward pass to gradually improve billions of weights, so training must carry extra machinery inference has no use for.
The other half of training
Training needs to know exactly how wrong it was, and in which direction
A forward pass by itself produces a prediction, and a prediction alone can’t improve a model. The prediction has to be compared against the correct answer, whatever the model was supposed to produce. That comparison has to be precise enough to report “wrong by this much, in this direction, and here’s exactly how each of the billions of weights contributed.”
The full mechanics of that comparison, called the loss function, and the machinery that traces the wrongness backward through every layer, called backpropagation and gradient descent, are dense enough to earn their own article next in this series. What matters here is narrower: that machinery is the extra thing training does that inference never touches, and it’s expensive in a very specific way.
To trace an error backward through a hundred-layer network, you need to know, layer by layer, exactly what each layer’s input looked like when it produced its output. That means every intermediate result computed during the forward pass, every attention weight, every post-activation vector at every layer, has to be kept around in memory rather than thrown away once it’s been used. Inference doesn’t have this requirement. Once a layer hands its output to the next layer, inference has no further use for that intermediate result and can discard it immediately. Training can’t discard anything until the backward pass has used it, which means training’s memory footprint is fundamentally larger than inference’s, before you even account for anything else.
The memory training carries
Weights are the smallest part of training’s memory bill
Training a model requires storing several extra copies of numbers the size of the model itself, for gradients and optimizer state, on top of the weights and activations, while inference only ever needs the weights.
Imagine the weights as a recipe you're gradually refining. Inference just cooks the recipe as written. Training has to also track, for every single ingredient amount in that recipe, how much to nudge it this round, and it has to remember a running history of how that ingredient has been nudged in previous rounds so it doesn't overcorrect or oscillate. That running history is not optional bookkeeping on the side, it's several full extra copies of the entire recipe, one for the current nudge and more for the momentum of past nudges.
For every weight in the model, training with a common optimizer like Adam keeps a gradient, the direction and size of that weight's nudge for the current step, plus two additional running averages Adam maintains per weight, one tracking the recent average gradient and one tracking the recent average of the squared gradient, used to adapt each weight's step size individually. That's three extra numbers per weight, each the same size as the weight itself, before you even count the activations kept for the backward pass.
So how much memory does training actually need, compared to inference?
A 7-billion-weight model doesn't need 7 billion numbers' worth of memory during training. It needs somewhere close to four to five times that, once weights, gradients, and optimizer state are all accounted for.
| Component | Present during inference? | Present during training? |
|---|---|---|
| Model weights | Yes, read-only | Yes |
| Layer activations | Discarded immediately after use | Kept for the entire backward pass |
| Gradients | Never computed | One per weight, every step |
| Optimizer state (e.g. Adam moments) | Never computed | Roughly two extra copies of the weights |
The actual parameters, needed by both training and inference, read-only during inference.
One number per weight, the direction and size of this step's proposed nudge, training only.
Adam's running averages of past gradients and their squares, training only, discarded once training ends.
Two different batch shapes
Training sees a whole sentence at once. Inference sees one token at a time.
There’s a second structural difference that has nothing to do with memory and everything to do with what data actually flows through the model at each step, and it comes from how language models are trained to predict text in the first place. A language model’s core task is next-token prediction: given everything so far, guess the next token. During training, the correct next tokens for an entire passage are already known in advance, since the training text itself is the answer key. That makes it possible to feed the model an entire sequence at once and have it predict every position’s next token in parallel, a technique called teacher forcing.
Teacher forcing lets training predict every position in a sequence simultaneously by feeding the model the real, correct tokens as input at every position rather than its own guesses.
Imagine practicing a speech by reading it off cue cards someone else wrote, one card ahead of where you are, rather than trying to remember your own last sentence. You get to practice predicting "and" comes after "bread" even if, five words earlier, you'd have gotten something wrong. Every position gets graded independently, all at once, using the correct history rather than whatever the model itself would have generated.
Concretely, if a training sentence is "the cat sat on the mat", the model is given the entire sentence as input and asked to predict, at every position simultaneously, whatever token comes next: predict "cat" after "the," predict "sat" after "the cat," predict "on" after "the cat sat," and so on, all in a single forward pass. This only works because a causal mask is applied inside attention, forcing each position's query to attend only to keys at or before its own position, never ahead.
What stops the model from cheating by reading the answer ahead in the input?
The causal mask. Without it, predicting "cat" would be trivial: the model could look ahead and copy the answer sitting right there in the input.
Inference has no such luxury, because the whole point of generation is that the future tokens don’t exist yet, they’re what you’re asking the model to produce. Inference has to run autoregressively: predict one token, append it to the sequence, run the forward pass again on the now slightly longer sequence, predict the next token, and repeat, one token at a time, until the response is finished. A hundred-token reply means running the forward pass roughly a hundred separate times, each one only slightly longer than the last.
Training (teacher forcing)
The entire sequence is known in advance. One forward pass, with a causal mask, predicts every position's next token in parallel.
Inference (autoregressive)
Only the prompt is known. The model predicts one token, appends it, and reruns the forward pass to predict the next, over and over.
Avoiding repeated work
The KV cache: inference’s answer to its own slowness
Autoregressive generation has an obvious inefficiency built into it. If token 50 requires attending back over tokens 1 through 49, and token 51 requires attending back over tokens 1 through 50, the query, key, and value computations for tokens 1 through 49 are being redone from scratch at every single step, even though those tokens’ keys and values never change once they’ve been computed. Recomputing the same keys and values dozens or hundreds of times per response would make generation dramatically slower than it needs to be.
The KV cache stores every previous token's key and value vectors the first time they're computed, so each new token only has to compute its own query, key, and value instead of recomputing the whole sequence's.
Picture the interview metaphor from the attention article, where every token holds up a key advertising what it offers. Without a cache, every new token walking into the room makes all the earlier tokens repeat their entire pitch from scratch. With a cache, each earlier token's pitch was written down the first time it was given, so a new token can just read the transcript instead of asking everyone to say it again.
At each generation step, the model computes the query, key, and value for only the newest token, then appends that token's key and value onto a growing cache rather than recomputing keys and values for the whole sequence. The new token's query attends over the full cache, old entries plus the one just added, exactly as before, but the cost of that step now scales with one new token's worth of computation instead of the entire sequence's.
A cache that grows with every generated token sounds like it just moves the cost elsewhere. Does it?
The cache isn't free: memory spent on it increases linearly with response length. What it buys is a trade of a much larger amount of repeated computation for a comparatively modest and predictable amount of extra memory.
The Nook of Wonder Theorems & Beautiful Patterns — counting the operations the cache saves
Say generating a response takes 100 steps, and at step n the model must attend over n previous tokens.
Without a cache: total key/value computations ≈ 1 + 2 + 3 + ... + 100 = 5,050
With a cache: total key/value computations = 1 per step × 100 steps = 100
That's roughly a 50x reduction in repeated key/value work for a 100-token response, and the gap widens with length, since the uncached cost grows quadratically while the cached cost grows linearly. It is the main reason a chat model can hold a long conversation without visibly slowing down as it goes.
Where the hardware actually strains
Compute-bound training, memory-bound inference
These structural differences add up to two jobs that stress a GPU in almost opposite ways. Training processes huge batches, often thousands of sequences at once, through teacher forcing’s single parallel forward pass. That means enormous matrix multiplications where the GPU’s arithmetic units are almost always busy, which is what people mean when they call training compute-bound: the bottleneck is how fast the chip can do math, and a well-tuned training run keeps those arithmetic units close to fully occupied.
A single user chatting with a model is a very different load. Generating one token at a time, for one conversation, means the matrix multiplications involved are comparatively tiny, the GPU spends most of its time waiting to fetch that model’s billions of weights from memory rather than actually crunching numbers. This is why single-request inference is usually called memory-bandwidth-bound: the bottleneck isn’t arithmetic speed, it’s how fast the weights can be streamed from memory to the chip’s compute units, over and over, once per generated token. Serving many users’ requests together, called batching, is the main lever providers use to push inference back toward compute-bound territory, since a batch of many users’ next-token predictions can share a single pass of loading the weights from memory.
| Training | Inference (single request) | |
|---|---|---|
| Typical batch | Thousands of sequences at once | One sequence, one token per step |
| Bottleneck | Arithmetic throughput (compute-bound) | Weight loading (memory-bandwidth-bound) |
| Duration | Days to months, continuous | Seconds, per request |
| Hardware utilization goal | Keep compute units saturated | Keep memory transfer saturated, usually via batching many users |
Two different kinds of randomness
Randomness serves opposite purposes in each job
Both jobs involve randomness, for reasons that point in different directions. Training relies on stochastic gradient descent, meaning each step sees a randomly sampled batch of data rather than the entire dataset at once.stochastic gradient descentgradient descent where each step uses one random sample of the data instead of all of it That keeps each step cheap, and it helps the model generalize better than a single perfectly accurate gradient over everything would. Many architectures also use dropout during training, randomly zeroing a fraction of neurons on each pass so the network can’t lean too hard on any single one.dropoutswitching a random share of neurons off each pass, as a regularizer Both of these are tools for producing a better final set of weights, and both are switched off completely once training ends.
Inference’s randomness serves a different goal: variety in output rather than quality of learning. Rather than always picking the single highest-scoring next token, which produces flat, repetitive text, generation samples from the probability distribution the forward pass produced. A temperature parameter flattens or sharpens that distribution, and top-k or top-p sampling narrows the candidate pool to the most plausible few first.top-kkeep only the k highest-scoring tokens before sampling
top-pkeep the smallest set of tokens whose scores sum to p None of it touches the weights. It changes only how a fixed probability distribution becomes one chosen word.
- Training's randomness (data shuffling, dropout) shapes which weights the model ends up with
- Inference's randomness (temperature, top-k, top-p) shapes which token gets picked from an already-fixed distribution
- Dropout and data shuffling are disabled entirely once a model is deployed for inference
- Turning inference's temperature to zero doesn't retrain anything: it always picks the highest-probability token, removing variety without touching a single weight
Where you've met this already
- Rehearsal versus performance: A rehearsal keeps stopping to adjust and take notes on what to fix. A performance just runs, using whatever was learned in rehearsal, without pausing to revise anything mid-show.
- Drafting versus reading a finished book: Drafting involves an editor’s notes, revisions, and multiple passes kept around for reference. A reader just experiences the finished, fixed text, start to finish.
- Cooking from a recipe you’re still perfecting: Perfecting a recipe means tracking every past attempt’s measurements and outcomes. Once it’s finalized, cooking it is just following the fixed steps.
- A calculator remembering recent entries: Keeping a running memory of recent button presses so it doesn’t have to re-enter earlier numbers is the same instinct behind caching keys and values.