Why evicting KV cache tokens can free almost no GPU memory on vLLM

NVIDIA researchers explain why token-level KV cache eviction often frees no memory on paged servers, and propose TriAttention as a fix.

A write-up from NVIDIA's Efficient AI lab argues that most KV cache compression methods that work in papers don't save memory in production serving. If you run a reasoning model or a local agent on one GPU, the takeaway is that evicted tokens are not the same as freed memory. NVIDIA's example: Qwen3-32B with 4-bit weights on a 24 GB GPU runs out of memory after roughly 24,000 generated tokens, short of the 32K-token traces reasoning models produce.

The post names two walls. Methods that choose tokens by observed attention scores, such as H2O and SnapKV, need scores that FlashAttention never exposes. And in a paged allocator a block is returned only when every slot in it is dead, so scattered survivors keep blocks pinned. vLLM's PagedAttention uses 16-token blocks by default, according to a DEV Community engine comparison. @_avichawla's thread sums it up: the cache "shrinks 90% logically and 0% physically."

NVIDIA's fix is TriAttention. It scores tokens from pre-RoPE query and key geometry, so it needs no attention scores, and about every 128 decoded tokens it packs survivors forward so trailing blocks empty out. NVIDIA reports that at a 3,072-token budget it matches full attention on AIME 2025 (40.8%), with 2.5× throughput and 10.7× less KV memory. These are NVIDIA's numbers. We found no independent reproduction, and it's unclear whether a mainline vLLM integration exists.

The levers available today are coarser. An FP8 KV cache halves per-token cache size in vLLM, though one L4 test hit backend start-up failures and advises checking accuracy on your own task. TensorRT-LLM evicts and offloads whole blocks by priority. Or lower the context limit first, as Octomind suggests, at the cost of cutting off long traces.

Sources