KV Cache: The Memory Bill Behind Every LLM Token

"Most of the energy in a GPU isn't spent on arithmetic. It's spent moving data around." - Bill Dally (Chief Scientist, NVIDIA)

Every token a language model generates has to look back at every token before it. Without a cache, that lookback is recomputed from scratch, every single step.

This week we're talking about the KV cache, the data structure that decides how much an LLM costs to run, and yes, this can get you a very comfortable job.

Here's the idea: the KV cache is the memory a transformer keeps during inference so it does not recompute attention keys and values for earlier tokens. At each decoding step the model computes the key and value vectors for the new token only, appends them to the cache, and attends over the whole cache. Per-token compute drops from quadratic to linear, and the price is GPU memory that grows with every token in every active request.

How much memory? Per token, per request: $$\text{bytes per token}=2\times L\times H_{kv}\times d_{head}\times {\text{bytes}}_{\text{dtype}}$$ The 2 is keys plus values, L is the number of layers, H_kv is the number of key-value heads, d_head is the head dimension. Llama-3-70B has 80 layers, 8 key-value heads (grouped-query attention), and 128-dimensional heads. In FP16 that is 2 x 80 x 8 x 128 x 2 = 327,680 bytes, about 320 KB per token. A single 128k-token request holds roughly 42 GB of cache before you count the 140 GB of weights. An older Llama-2-7B with full multi-head attention (32 KV heads) spends 512 KB per token, more than the 70B model, which is exactly why grouped-query attention exists.

This is where the money is. Hosted open-weight APIs charge somewhere between $0.07 and $0.90 per million tokens in 2026, and at those prices the margin is how many concurrent sequences you can fit on a GPU, which is a cache question. vLLM's PagedAttention (SOSP 2023) stopped pre-allocating cache for the maximum sequence length, cut memory waste from 60 to 80% down to under 4%, and raised throughput 2 to 4x. DeepSeek-V2's multi-head latent attention compresses keys and values into a low-rank latent, cutting the KV cache by 93.3% and raising maximum generation throughput 5.76x over their previous model. In 2026 the work moved down the stack: Huawei shipped a native vLLM backend for KV cache quantization, and a lossless entropy coder for the cache reported up to about 4x compression. Every one of those numbers is a serving bill, and the trillion-dollar infrastructure layer we covered in issue 1 is fighting over them.

How does the KV cache work?

During the prefill phase, the model runs the whole prompt in one pass and stores every layer's keys and values. During decode, each new token computes its query, key, and value once, appends the key and value to the cache, and computes attention against all cached keys. The cache is per layer, per request, and grows by one position per generated token. Decode is memory-bandwidth bound: the GPU spends most of its time reading the cache and weights, not multiplying, which is Dally's point in the quote above.

Is the KV cache used during training or inference?

Inference only. Training uses teacher forcing, so the full sequence is known up front and attention runs over all positions in a single parallel pass. There is nothing to cache. The KV cache exists because autoregressive generation produces one token at a time and would otherwise redo the prefix work at every step.

KV cache vs prefix cache vs paged attention

Now I'm going to explain what this means for you:

The Skill Employers EXPECT

Learn what the KV cache is and what it costs. Inference and platform teams assume you can:

• Explain why decode is memory-bound and prefill is compute-bound
• Calculate KV cache size for a given model, context length, and batch size from the config file
• Explain grouped-query and multi-query attention as KV cache reductions, not accuracy tricks
• Implement a KV cache in a toy transformer and prove the logits match the uncached version
• Describe PagedAttention and prefix caching in one sentence each

If you cannot say how many concurrent 32k-token requests fit on an 80 GB GPU, you cannot size a deployment.

The Skill That Separates You

Learn cache compression and measure what it costs in quality.

The 2026 serving stack quantizes, evicts, and compresses the cache. You should know how to:

• Quantize the cache to INT8 or FP8 per token and measure the perplexity change on a held-out set
• Explain eviction strategies (keep attention sinks, keep heavy hitters, drop the rest) and where they break
• Read the multi-head latent attention design and explain why the low-rank latent still supports RoPE
• Profile tokens per second vs batch size and find the point where the cache, not compute, is the ceiling
• Trade context length against concurrency with a number, not a guess

The Project To Learn These Skills This Week

Size a GPU fleet from first principles, then predict tokens per second. This is a graded Pro project on GRADuateML: LLM Serving Capacity: KV Cache Math and Decode Throughput. You get 16,000 benchmark runs across 48 model shapes and 5 GPU classes, a hidden test set, and instant grading on every milestone. The punchline: a linear regression on the raw columns scores an R-squared of 0.48, while a four-number roofline built from bytes read per decode step scores 0.98 before any machine learning. Not on Pro? The free challenge below is one decode step with a cache, by hand.

Requirements (each one is a graded milestone):

• Profile the runs: attention layouts (MHA, GQA, MQA), context lengths, and a throughput range of more than 7,000x from slowest to fastest.
• Compute KV cache bytes per token for every run from the model shape, and hit 327,680 for the Llama-3-70B shape.
• Work out how many concurrent requests fit in 92% of GPU memory. On one 80 GB GPU, a 70B-class GQA model with INT4 weights holds 3 requests at 32k context with an FP16 cache, 7 with FP8, and 0 with full multi-head attention.
• Prove in NumPy that decoding with a cache matches full recomputation exactly, with 16 key projections instead of 136.
• Fit a four-coefficient roofline model of decode step time with no machine learning: MAPE of 0.09 or lower.
• Beat it with a model built on memory-traffic features: MAPE of 0.062 or lower and R-squared of at least 0.97. Tuned boosting on the raw columns misses this bar.
• Serve the planner within 2 seconds, then write the capacity plan.

This project teaches a practical lesson for production ML:

Inference cost is a memory problem wearing a compute costume, and the KV cache is where the bill lives.

If you want to sharpen your machine learning skills even more, I also selected a free challenge problem for you this week. It is one decode step with a KV cache, by hand:

If you learned something from this newsletter, make sure to forward it to a friend.

GRADuateML

The full LLM serving capacity build, graded against hidden benchmark runs, is part of GRADuateML Pro.