The Tiga Explainers · Building Blocks · Part 13

GPU Kernels: Where the Speed of an AI Model Is Decided

Every layer of a model eventually becomes a small program running on thousands of GPU threads. How that program moves data matters more than how fast it calculates.

Good Omens Studio34 min readMachine Learning

Tiga, the White Tiger

Two jobs, one GPU. Job A adds 1 to each of one billion numbers. Job B multiplies two 4096 by 4096 matrices, which takes about 137 billion arithmetic operations. Both run on an NVIDIA H200, a chip rated at close to a quadrillion operations per second for matrix math. Which job finishes first?

Nearly everyone picks Job A, because one billion operations against 137 billion is a gap of more than a hundred times. The reasoning has a flaw, and by the end of section 3 you will be able to locate it with a calculator. Job B finishes in roughly 0.2 milliseconds. Job A needs about 0.8 milliseconds. The job with a hundred times more arithmetic wins by a factor of four.

Explaining this takes three ideas the series has not needed until now. The first is the kernel, the program a GPU actually executes. The second is the set of memories that program reads from and writes to. The third is a ratio called arithmetic intensity, which predicts whether a job is limited by arithmetic or by data movement. Each idea depends on the one before it, so we take them in that order. After that we look at two techniques for moving less data, then at FlashAttention, a kernel that made transformers much faster without changing any number they compute, and finally at a $100,000 competition announced on 22 October 2026 by Meta, Paradigm and Andromeda, whose entire task is writing faster kernels for Meta’s Muse Spark Open model.

So far this series has described a transformer as a sequence of operations on tensors: multiply, add, normalize, multiply again. That description leaves out where and how any operation physically happens. The answer decides how fast the model runs, and it is the subject of this article.

What a kernel is, and how a GPU runs one

In one sentence

A kernel is a function that a GPU runs thousands of times in parallel, once per thread, with each thread working on its own small piece of a tensor.

Intuition

Picture a hall with 10,000 teaching assistants and a stack of 10,000 exam papers. Every assistant receives the same one-page set of marking instructions and a different paper, and none of them talks to the others while marking. The instructions are the kernel. The hall finishes the whole stack in about the time one assistant needs for one paper, provided the instructions are short and no paper depends on another.

Technical

In CUDA, NVIDIA's programming model for its GPUs, a kernel is a function marked __global__. The program launches it with two numbers, the count of thread blocks and the count of threads per block, and the hardware creates that many threads, all running the same code. Each thread learns which element belongs to it by combining built-in variables (blockIdx, blockDim, threadIdx) into an index. This is the complete kernel for Job A:

__global__ void add_one(const half* x, half* y, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;   // my element
    if (i < n) y[i] = __hadd(x[i], __float2half(1.0f));
}
// launch: 256 threads per block, enough blocks to cover n elements
add_one<<(n + 255) / 256, 256>>(x, y, n);

For one billion elements the launch creates 3,906,250 blocks of 256 threads. Each thread performs one load, one addition and one store. Keep that ratio of arithmetic to memory traffic in mind, because section 3 turns it into a number.

The word "kernel" shares nothing with the operating system kernel. It names the core computation applied at each position of the data, the same sense used in convolution kernels.

Why does a GPU need hundreds of thousands of threads when a CPU gets by with a few dozen?

A CPU core is built to finish one thread as fast as possible, so it spends its transistors on large caches, branch prediction and out-of-order execution. A GPU gives up single-thread speed and spends the same silicon on arithmetic units. Reading from memory takes hundreds of clock cycles on both kinds of chip. The CPU hides that delay with caches. The GPU hides it with other threads: when one group of threads stalls waiting for data, the scheduler switches to another group that is ready, and the switch costs nothing because every resident thread keeps its registers on the chip. An H200 has 132 streaming multiprocessors (SMs, the chip's independent compute units), and each can hold up to 2,048 resident threads, roughly 270,000 in total. A kernel launched over a billion elements supplies far more threads than the hardware can run at once, and that surplus keeps the arithmetic units fed while other threads wait.

The hardware organizes those threads in four levels, and each level matches something physical.

  • Thread. One instance of the kernel, with its own registers.
  • Warp. A group of 32 consecutive threads that execute the same instruction at the same moment. If threads inside one warp take different sides of an if, the warp runs both sides one after the other with some threads switched off, so kernel authors try to keep the threads of a warp in agreement.
  • Thread block. Up to 1,024 threads that live on one SM for their whole lifetime. Threads in a block can share fast on-chip memory and wait for each other at a barrier.
  • Grid. All the blocks of one launch. Blocks cannot assume any order or any overlap in time, which lets the same kernel run on a GPU with 20 SMs or with 132 without modification.
grid of thread blocks one block, enlarged 8 warps · 32 threads each one warp highlighted runs on one SM
A launch is a grid of thread blocks. Each block stays on one SM, and each block divides into warps of 32 threads.

Every block needs data, and where that data sits determines how long the block waits. That is the subject of the next section.

The memory hierarchy: where the data lives

In one sentence

A GPU has several layers of memory in which the faster a layer is, the smaller it is and the fewer threads can see it.

Intuition

A restaurant kitchen has the same structure. The ingredient already in the chef's hand is available instantly. The counter at the chef's station takes a step to reach and is shared with the neighboring cooks. The walk-in storage room serves the whole kitchen but costs a minute for the round trip. The warehouse across town holds everything and costs an hour. Most of the planning in a good kitchen consists of limiting how many dishes need a trip to the warehouse.

Technical

The add_one kernel issues one read and one write per thread. Whether each costs a cycle or several hundred depends on which memory it touches, and an H200 has four kinds.

LevelCapacity on an H100/H200-class GPUVisible toApproximate latency
Registers256 KB per SM (64K 32-bit registers)one threadabout 1 cycle
Shared memoryup to 228 KB per SMthe threads of one blocktens of cycles
L2 cache50 MB per chipevery SMa couple hundred cycles
HBM (global memory)141 GB on H200, at 4.8 TB/severy SMseveral hundred cycles

Registers hold the operands the arithmetic units are working on right now, and each thread has its own. Shared memory is a scratchpad that the kernel author manages explicitly, unlike a cache, which decides for itself what to keep. HBM, short for High Bandwidth Memory, consists of stacked DRAM dies mounted on the same package as the GPU and connected through thousands of wires, which is how it reaches 4.8 terabytes per second. Every tensor lives in HBM by default: weights, activations and the KV cache. A kernel's inputs begin there and its outputs must end there, so every boundary between two kernels is a round trip to HBM.

If shared memory is so much faster, why not build the entire GPU memory out of it?

Because of cell size. A bit of SRAM, the technology behind registers, shared memory and caches, needs about six transistors, while a bit of DRAM, the technology behind HBM, needs one transistor and one capacitor. The whole chip therefore carries tens of megabytes of fast memory against 141 gigabytes of HBM, and that ratio has persisted across GPU generations. Kernels have to be planned around it.

The arithmetic units draw their operands from the top rows of the table, and the data they need starts in the bottom row. The next section puts a number on how far apart those two can be.

Arithmetic intensity: the ratio that predicts the winner

In one sentence

Arithmetic intensity is the number of operations a kernel performs per byte it moves to or from HBM, and comparing it with the GPU's own ratio of compute to bandwidth reveals which of the two will limit the kernel.

Intuition

Consider a factory with very fast machines and a loading dock of fixed width. If every part needs one quick cut, the trucks cannot deliver parts fast enough to keep the machines busy, and buying faster machines changes nothing. If every part needs two hundred operations, the dock sits idle most of the time and the machines are the limit. Which situation applies depends on the work done per part, and the machines alone do not settle it.

Technical

Write FLOPs for floating-point operations, where a multiply-add counts as two. A kernel's arithmetic intensity is I = FLOPs ÷ bytes moved to and from HBM. The GPU has an intensity of its own, its machine balance, which is the peak arithmetic rate divided by the HBM bandwidth. For an H200 running BF16 matrix math that is 989 trillion FLOP/s ÷ 4.8 trillion bytes/s, about 206 FLOP per byte.

A kernel needs at least FLOPs ÷ peak seconds to do its math and at least bytes ÷ bandwidth seconds to move its data, and it cannot finish before the larger of the two. When I is below 206, the data term is larger and the kernel is memory-bound. When I is above 206, the math term is larger and the kernel is compute-bound. Samuel Williams, Andrew Waterman and David Patterson drew this relationship as the roofline model in a 2009 paper. Attainable performance climbs along a slope set by memory bandwidth until it meets a flat ceiling set by peak arithmetic, and the intensity where slope meets ceiling is the machine balance. Real kernels land below the roof, and a well-tuned matrix multiplication typically reaches somewhere between 60 and 80 percent of the peak.

0.1 1 10 100 1,000 10,000 0.1 1 10 100 1,000 arithmetic intensity (FLOP/byte) attainable TFLOPS ridge · 206 memory-bound compute-bound add 1 · 0.25 attention · 63 matmul · 1,365
Everything left of the ridge waits on memory. Everything right of it waits on arithmetic.

Why can't a faster GPU rescue a memory-bound kernel?

Because the time of a memory-bound kernel depends only on bandwidth, and bandwidth has grown more slowly than arithmetic.

GPU Peak BF16 dense (TFLOPS) HBM bandwidth (TB/s) Machine balance (FLOP/byte)
A100 80 GB 312 2.04 about 153
H100 SXM 989 3.35 about 295
H200 989 4.8 about 206

From the A100 to the H200, peak matrix arithmetic grew about 3.2 times and HBM bandwidth about 2.4 times. A kernel limited by memory gains the smaller factor, a kernel limited by arithmetic gains the larger one, and so each new GPU generation lifts different kernels by different amounts. The H200 keeps the H100’s arithmetic and raises bandwidth by 43 percent, which helps memory-bound kernels and leaves compute-bound kernels exactly where they were.

The Nook of Wonder Theorems & Beautiful Patterns — resolving the opening riddle with a roofline

The riddle from the top of the article can be settled with nothing more than the machine balance from section 3. Both jobs are small enough to time by hand: each needs a number of floating-point operations and a number of bytes moved to HBM, and the slower of those two terms wins.

P = 989 × 1012 FLOP/s (BF16 dense tensor peak, H200)
W = 4.8 × 1012 bytes/s (HBM bandwidth)
FP16/BF16 = 2 bytes per element

Job A moves four bytes per element and does one operation per element, so its intensity is a quarter of a FLOP per byte, far left of the ridge. Its arithmetic is negligible and its time is the time to move the data.

Job A: add 1 to n = 109 elements
bytes = 109 × 2 (read) + 109 × 2 (write) = 4 × 109
FLOPs = 109
I = 109 / (4 × 109) = 0.25 FLOP/byte
tmem = 4 × 109 / 4.8 × 1012 = 0.83 ms
tmath = 109 / 989 × 1012 = 1.0 µs

Job B is the mirror image: each byte it moves feeds thousands of operations, so its intensity is far right of the ridge and its time is the time to do the arithmetic.

Job B: multiply two 4096 × 4096 matrices
FLOPs = 2 × 40963 = 1.37 × 1011
bytes = 3 × 40962 × 2 = 1.01 × 108 (read A, read B, write C, once each)
I = N / 3 = 1,365 FLOP/byte
tmath = 1.37 × 1011 / 989 × 1012 = 0.14 ms
tmem = 1.01 × 108 / 4.8 × 1012 = 21 µs
result: t ≈ 0.14 ms at peak, about 0.2 ms at 70% of peak

The two jobs sit on opposite sides of the ridge. That, not the operation count, is what decides the race.

Job B has 137 times more arithmetic and finishes four to six times sooner, because the data it needs is tiny relative to the work done on it. Job A reaches 0.25 × 4.8 TB/s = 1.2 TFLOPS, which is 0.12 percent of the tensor-core peak. The add runs on ordinary CUDA cores, whose peak is about 67 TFLOPS, so against that ceiling the figure is 1.8 percent, still negligible.

The intensity of a square matmul is N ÷ 3, so it grows with size. It crosses the H200’s balance of 206 at N ≈ 618. A 512 × 512 multiplication has I ≈ 171 and is memory-bound, which is why small matrix multiplications deliver far less than the chip’s headline number.

From measuring to fixing

Two ways to stop moving data: fusion and tiling

Section 3 ended with a diagnosis. Most kernels in a transformer sit left of the ridge, so the remedy is to move fewer bytes. Two techniques account for most of the gains, and they attack different waste. Fusion removes trips between kernels. Tiling removes repeated trips inside one kernel.

Fusion

In one sentence

Kernel fusion combines consecutive operations into a single kernel, so the intermediate tensors stay in registers or shared memory and never travel to HBM.

Intuition

A copy editor reviewing a manuscript can make three passes, one for spelling, one for punctuation and one for style. If the manuscript sits in an archive across town, each pass requires fetching it and returning it. Making all three corrections during one reading requires one fetch and one return, and the corrections themselves take no longer.

Technical

Take the step that follows nearly every linear layer in a transformer: multiply by a weight matrix, add a bias, apply an activation such as GELU. A framework that runs these as three separate operations launches three kernels. Let the output have 8192 × 8192 elements in FP16, which is 134 MB. The matmul kernel writes it once. The bias kernel reads 134 MB and writes 134 MB. The GELU kernel reads 134 MB and writes 134 MB. Counting only the traffic that follows the matmul's own inputs, the unfused version moves 5 × 134 = 671 MB. A fused kernel that applies the bias and the GELU while the result is still in registers writes the final 134 MB once. At 4.8 TB/s the difference is 140 µs against 28 µs.

The saving of 112 microseconds applies to one layer of one batch, and it repeats in every layer of the model. Each launch also carries a fixed overhead of a few microseconds, during which the GPU can sit idle, so three launches cost more than the traffic alone suggests. For small kernels, such as the single-token steps of text generation, launch overhead can exceed the kernel's own run time. Inference systems therefore record sequences of kernels with CUDA Graphs and replay them as one launch. Frameworks also fuse automatically in many cases, and PyTorch's torch.compile generates fused kernels for chains of elementwise operations.torch.compilePyTorch's compiler entry point, which traces a model and generates fused kernels, mostly written in Triton

If fusion saves this much, why doesn't a framework fuse everything automatically?

Three limits get in the way. Some operations need information from across a whole tensor before any output element is final, and softmax is the standard example, because each output depends on a sum over its entire row. Registers are finite, so a kernel that fuses too much runs out of them, spills values to slower memory and loses the benefit. And a fused kernel is compiled for specific shapes and data types, which forces compilers to choose between generality and speed. The first limit matters most for transformers, and section 5 is devoted to it.

Tiling

In one sentence

Tiling splits a large operation into blocks small enough to fit in shared memory, so each number fetched from HBM is used many times before it is discarded.

Intuition

A student writing a chapter does not walk to the library for each fact. They check out a stack of books on the chapter's topic, work through the stack at the desk, and return it once. The desk is small, so the stack has to be sized to fit, but any stack beats a separate trip per fact.

Technical

Matrix multiplication computes each output element C[i, j] from row i of A and column j of B. In a naive kernel every thread fetches its own row and column from HBM, so each element of A is fetched once per output column that uses it, N times in total. In a tiled kernel, one thread block owns a rectangle (a tile) of C, say 128 × 128 elements. It walks along the shared dimension K in slices. It copies a 128 × 32 slice of A and a 32 × 128 slice of B into shared memory once, every thread in the block multiplies from that copy, and the partial sums accumulate in registers. Each fetched number now feeds 128 outputs instead of one. GPUs of NVIDIA's Hopper generation add hardware for this pattern: the Tensor Memory Accelerator copies tile-shaped regions asynchronously while the tensor cores work on the previous tile, so copying and computing overlap.

How large should a tile be?

Larger tiles reuse each fetched number more often, but they consume shared memory and registers. A 128 × 128 tile with a 32-wide slice of K needs 16 KB of shared memory per pipeline stage, so four stages fit easily in 228 KB. Doubling the tile's side to 256 would quadruple the accumulator registers, leaving room for fewer resident blocks and less latency hiding. Edges matter too: a matrix 4,100 columns wide leaves a nearly empty final tile column, and the threads covering it do wasted work. No tile size wins for every shape and data type, which is why GPU libraries contain many variants of the same matmul and choose among them at run time, and why tuning kernels for a specific model's shapes can pay off.

The Nook of Wonder Theorems & Beautiful Patterns — how much reuse a tile buys

Section 3 gave the ceiling: a square matmul that moves each element once reaches N ÷ 3 FLOP per byte. Tiling is how a kernel approaches that ceiling, and how close it gets is set by the tile's shape. The arithmetic below derives the intensity of a tiled matmul from first principles.

Setting: C = A · B, with A (M × K) and B (K × N) in FP16 (2 bytes per element)

A naive kernel gives each output element its own thread, so every thread walks a full row of A and a full column of B. The reuse is one: each number fetched is used for a single output.

Naive: one thread per output element
loads per output = K (from A) + K (from B) = 2K elements = 4K bytes
FLOPs per output = 2K
I = 2K / 4K = 0.5 FLOP/byte

A tiled kernel instead gives a block a T × T square of outputs. It walks along K, loading one T-wide slice of A and one of B into shared memory, and every thread in the block multiplies from that copy. Now each fetched number feeds T outputs.

Tiled: one block per T × T tile of C
loads per tile = T rows of A + T columns of B = 2·T·K elements = 4·T·K bytes
FLOPs per tile = 2 · T2 · K
I = 2·T2·K / (4·T·K) = T / 2 FLOP/byte

The intensity doubles with the tile's side, which is why a 128 × 128 tile reaches 64 FLOP per byte, 128 times the naive kernel. A rectangular tile does a little better than a square one of the same area, because its two dimensions share the fetched slices differently.

T = 128 → I = 64 FLOP/byte (128 times the naive kernel)
Rectangular tile Tm × Tn: I = Tm·Tn / (Tm + Tn)
128 × 256 → I ≈ 85 FLOP/byte

The shared memory a tile needs is what bounds its side: a 128 × 128 tile with a 32-wide K-slice costs 16 KB per pipeline stage, which is how the hardware budget is spent.

Shared memory per pipeline stage, 128 × 128 tile, K-slice of 32:
(128 × 32 + 32 × 128) × 2 bytes = 16,384 bytes = 16 KB

The intensity above describes traffic between a block and the memory that feeds it. Production kernels lower the HBM share of the traffic further: neighboring blocks share tiles through the L2 cache, and on Hopper a cluster of blocks can fetch a tile once and multicast it to all of them.

C A rows B columns shared memory 16 KB per stage
The block loads one slice of A and one slice of B, uses every number in them 128 times, then slides along K to the next slice.

FlashAttention: fusing the operation that resists fusion

Attention is where the two techniques meet a real obstacle. For each attention head the model holds three tensors, Q, K and V, each of shape N × d, where N is the number of tokens and d, the head dimension, is typically 64 to 128. Attention computes S = Q Kᵀ, an N × N table in which entry (i, j) scores how strongly token i attends to token j. It applies softmax to each row of S to get P, and returns O = P V.

A straightforward implementation launches one kernel per step and stores S and P in HBM. Take N = 8,192 and d = 128. S has 67 million entries, which is 134 MB in FP16. Writing S, reading it back for the softmax, writing P and reading P for the final matmul moves four copies of an N × N table, 537 MB per head. The arithmetic for the head, 4N²d = 34 billion FLOP, takes about 35 µs at peak, while the traffic takes about 112 µs at 4.8 TB/s. The kernels are memory-bound by a factor of three, and the table is the cause.

The table also grows with the square of the context. Muse Spark Open advertises a context window of one million tokens. At N = 1,048,576, the S table for one head would hold 1.1 × 10¹² entries, which is 2.2 TB in FP16, more than 15 times the memory of an H200. No amount of tuning makes that table fit, so any kernel that handles such contexts must never build it.

Fusion is the natural cure, and softmax blocks it. Row i of P cannot be finalized until the maximum and the sum over all N scores of that row are known, and a kernel that holds one block of scores at a time has not seen them all.

In one sentence

Online softmax computes the exact softmax of a row one block at a time by carrying a running maximum and a running sum, both corrected whenever a larger score arrives.

Intuition

Imagine a scorekeeper who records every contestant's height as a fraction of the tallest contestant seen so far. When a taller contestant walks in, every earlier fraction is wrong by the same factor, and one multiplication repairs all of them. The scorekeeper never needs to look at the earlier contestants again.

Technical

The softmax of scores s_1 … s_N is exp(s_j − m) ÷ Σ exp(s_k − m), where m is the largest score. Subtracting m keeps the exponentials from overflowing and does not change the result. Process the row in blocks and keep three running quantities per query row: the maximum m seen so far, the sum l of exponentials so far measured relative to m, and an accumulator O holding the weighted sum of V rows so far on the same relative scale. When a new block arrives, set m_new to the larger of m and the block's maximum, multiply l and O by exp(m − m_new) to restate them relative to the new maximum, then add the new block's contributions. After the last block, divide O by l. The result equals standard softmax attention up to floating-point rounding.

The Nook of Wonder Theorems & Beautiful Patterns — why the running maximum keeps softmax exact

FlashAttention's claim is that the online recurrence computes the exact softmax, not an approximation. The claim rests on one algebraic identity, and the worked example below shows it holding on a row of three scores.

Update rule for one query row, one block of scores at a time
mnew = max(mold, max(block))
lnew = exp(mold − mnew) × lold + sum over block of exp(sj − mnew)
Onew = exp(mold − mnew) × Oold + sum over block of exp(sj − mnew) × vj
final output = O / l

Each new block can raise the running maximum. When it does, every earlier term was measured against the old maximum, so all of them are restated against the new one by a single multiplication. That is the step that keeps the sum exact.

Worked example: scores 2 and 5 arrive first, then 7 (tracking l only)
Block 1: m = 5
l = exp(2−5) + exp(5−5) = 0.0498 + 1 = 1.0498
Block 2: mnew = 7, scale = exp(5−7) = 0.1353
lnew = 0.1353 × 1.0498 + exp(7−7) = 0.1420 + 1 = 1.1420

Compute the same sum directly against the final maximum and the two routes agree to the last digit shown. In a real kernel the only differences come from floating-point rounding, because the additions happen in a different order.

Check, computed directly with m = 7
exp(2−7) + exp(5−7) + exp(7−7) = 0.0067 + 0.1353 + 1 = 1.1420

Both routes give the same sum, and the identity that makes them agree is exact algebra rather than a shortcut. That is the property FlashAttention’s kernel relies on: it never approximates the softmax, it only reorders the work.

FlashAttention, introduced by Tri Dao and colleagues in 2022, builds a kernel around this recurrence. Each thread block owns a tile of query rows. It loops over blocks of K and V, computes the scores for its tile into registers, updates m, l and O, and discards the scores. Only Q, K, V and O cross the HBM boundary, and the N × N table never exists anywhere. The memory footprint grows with N instead of N², which is what makes million-token contexts representable at all. K and V are re-read once per query tile, and how much of that re-reading reaches HBM rather than the L2 cache depends on tile sizes. The quadratic term is gone either way.

Is the result an approximation?

No. Many earlier methods sped up attention by dropping or compressing parts of the table, and they changed the output. FlashAttention computes the same function as the standard implementation and changes only the order of the work and where the intermediate values live. The kernel is tied closely to the hardware it was written for, and FlashAttention-3, published in 2024, rewrote it for Hopper GPUs to use asynchronous copies and FP8 arithmetic. A kernel tuned for one generation of chip leaves speed unused on the next.

MEMORY-BOUND0.25 FLOP/byte

Add 1 to a billion values. Reaches 0.12 percent of peak.

STILL MEMORY-BOUNDabout 63 FLOP/byte

Unfused attention, N = 8,192, d = 128. 34.4 GFLOP over 545 MB of traffic.

COMPUTE-BOUND1,365 FLOP/byte

4096 × 4096 matmul. Limited by the tensor cores.

Who writes kernels, and what a $100,000 prize shows

Kernels are written at several levels of the software stack, and each level trades effort for control. At the bottom sit vendor libraries. cuBLAS handles matrix multiplication, tuned by NVIDIA and called by most frameworks.cuBLASNVIDIA’s library of tuned matrix-multiplication kernels, called by most frameworks CUTLASS is the open-source set of templates for building matmul variants.CUTLASSNVIDIA’s open-source C++ templates for assembling matmul kernels from tile-sized parts

Above them, domain-specific languages let a researcher write a tile-level kernel in Python and leave register allocation and instruction scheduling to a compiler. Triton, from OpenAI, is the established example.TritonA Python-embedded language for writing GPU kernels at the level of tiles Helion is PyTorch’s newer kernel language.HelionPyTorch’s kernel language, with autotuning over many kernel configurations Hand-written CUDA remains the route when maximum control is needed, and FlashAttention is the best-known kernel written that way.

Meta integrated Helion into the Muse Spark Open release. Pasquale Pillitteri’s write-up of the release reports kernel speedups between 1.03 and 3.95 times on NVIDIA and AMD hardware, a figure that comes from that secondary report and not from Meta’s own tables. The range itself is informative. A well-tuned library kernel leaves little to find, and a kernel for an unusual shape can have a lot.

Serving a model gives kernels a second job. When a model generates text, each new token requires a pass over the active weights. Muse Spark Open is a mixture-of-experts model with 816 billion parameters in total and 42 billion active per token. Suppose its weights are stored in FP8, one byte each. A single sequence generating one token then reads about 42 GB and performs about 2 FLOP per byte read, one multiply and one add per weight. The balance of an H200 in FP8 is about 412 FLOP/byte (1,979 TFLOPS ÷ 4.8 TB/s), so decoding one sequence is memory-bound by a factor of about 200.

Doesn't batching fix that?

It helps, because B sequences generating tokens together share each weight read, which raises the intensity to roughly 2B. In a mixture-of-experts layer, though, the tokens of a batch are divided among experts. If each token chooses k of E experts, each expert’s weights are shared by only about B × k ÷ E tokens, so the intensity per expert is lower than the dense case at the same batch size. The kernels that group tokens by expert and multiply each group by that expert’s weights are where the memory-bound regime is hardest to escape.

That is the territory of the Muse Spark Open Kernel Challenge. According to Paradigm’s announcement, the challenge has two tracks that run in series and conclude on 19 November. The first consists of five kernel problems, and contestants try to write kernels faster than the SGLang baseline.SGLangAn open-source framework for serving language models that schedules requests and calls the kernels In the second, contestants modify SGLang directly, and the first track’s kernel work is a natural place to start. Entries run on H200 GPUs, the fastest submissions share $100,000, and first place also receives an H200 signed by the Meta team.

The prize pays for speed, and the sections above describe where speed on this hardware usually comes from: fewer trips to HBM for each useful operation.

Where you've met this already

Where you have seen it What is happening underneath
nvidia-smi reports GPU-Util at 100 percent while the model feels slow The figure measures the share of time in which at least one kernel was running, whatever fraction of the arithmetic units that kernel used. A memory-bound kernel can hold it at 100 percent with the arithmetic units mostly idle.
Hugging Face Transformers offers attn_implementation="flash_attention_2" The standard attention kernels are replaced by FlashAttention’s fused kernel, with the same outputs and no N × N table in HBM.
torch.compile speeds up a model without changing its math The compiler fuses chains of elementwise operations into single generated kernels and removes round trips to HBM.
Quantized checkpoints in FP8 or NVFP4 generate tokens faster Fewer bytes per weight means fewer bytes read per token, and decoding is memory-bound. NVIDIA’s serving recipe for Muse Spark Open uses NVFP4 weights with an FP8 KV cache.
A server delivers more tokens per second per GPU when it batches many requests Each weight read is shared by more tokens, which raises arithmetic intensity toward the ridge.
A long prompt costs more memory and time even though the weights are unchanged The KV cache grows with context length, and its bytes must be read for every generated token.