The rapid advancement of Large Language Models (LLMs) has shifted the primary challenge of artificial intelligence from model training to efficient production deployment. While techniques such as quantization, pruning, and distillation have successfully reduced the computational burden of model weights, the industry has identified a more persistent bottleneck in the inference pipeline: the management of Key-Value (KV) caches. As context windows expand from 8,000 to over 100,000 tokens, the memory consumed by these caches has become the limiting factor for concurrency, throughput, and latency in high-performance serving environments. Two seminal breakthroughs, PagedAttention and RadixAttention, have emerged as the foundational pillars of modern LLM serving, transforming how GPU memory is allocated and how redundant computations are bypassed.
The Technical Anatomy of the KV Cache Bottleneck
To understand why KV cache management is critical, one must examine the autoregressive nature of transformer-based models. Unlike traditional software that processes inputs in a single pass, LLMs generate text one token at a time. For every new token produced, the model must "attend" to every preceding token in the sequence. To avoid the prohibitive cost of recomputing the Key and Value vectors for every historical token at every step, serving engines store these vectors in a dedicated memory buffer known as the KV cache.
This cache effectively trades memory for speed, enabling practical autoregressive decoding. However, the memory footprint of the KV cache grows linearly with the sequence length and the number of concurrent requests. The memory required per token is calculated by the formula: $2 times textLayers times textKV Heads times textHead Dimension times textBytes per Value$. In a modern architecture like Llama-3 8B, which utilizes 32 layers, 8 KV heads, and 128-dimensional heads, a single token in FP16 precision requires approximately 128 KiB. While this seems negligible for a single token, a 100,000-token context window for a single user requires 12.8 GiB of GPU memory. In a multi-tenant environment where a single NVIDIA H100 GPU (80GB) might serve dozens of users, the KV cache quickly exhausts available High Bandwidth Memory (HBM), even before accounting for the multi-gigabyte weight of the model itself.
The Chronology of Inference Inefficiency
Prior to 2023, LLM serving engines relied on contiguous memory allocation. When a request was initiated, the system would allocate a single, continuous block of GPU memory to hold the KV cache. Because the engine could not predict the eventual length of the generated response, it typically reserved space equal to the model’s maximum context length (e.g., 4,096 or 8,192 tokens).
This approach led to two devastating forms of fragmentation. Internal fragmentation occurred because the majority of the reserved block remained empty until the very end of the generation process. External fragmentation occurred because the remaining free memory was often scattered in small, non-contiguous chunks that could not accommodate new requests. This "pre-allocation" strategy resulted in GPU utilization rates frequently hovering below 20%, as memory—not compute power—limited the number of batches the system could handle.
PagedAttention: The vLLM Revolution
In 2023, researchers at UC Berkeley introduced PagedAttention, a technique that drew direct inspiration from virtual memory and paging in operating systems. The core innovation was the decoupling of logical token sequences from physical memory locations. Instead of requiring a contiguous block of memory, PagedAttention divides the KV cache into fixed-size "pages" or blocks, typically containing 16 or 32 tokens.
Under this system, the serving engine maintains a "block table" that maps logical positions in a sequence to physical addresses in the GPU memory. As the model generates new tokens, the system allocates new physical blocks only when the current block is filled. This "on-demand" allocation eliminates internal fragmentation because memory is only used as tokens are actually produced. Furthermore, because blocks do not need to be contiguous, the system can utilize every available "page" of GPU memory, effectively solving the external fragmentation problem.
The impact of PagedAttention was immediate and profound. By allowing for much higher batch sizes (the number of requests processed in parallel), frameworks like vLLM demonstrated throughput increases of 2x to 4x compared to traditional systems like Hugging Face Text Generation Inference (TGI) at the time of its release.
RadixAttention and the Challenge of Prefix Reuse
While PagedAttention optimized where the KV cache was stored, it did not address what was being stored. In production environments, many requests are not unique. Thousands of users might interact with a chatbot using the same 2,000-token system prompt, or a Retrieval-Augmented Generation (RAG) system might prepend the same lengthy document to dozens of different queries.
In traditional systems, the "prefill" phase—where the model processes the input prompt to generate the initial KV cache—must be performed for every request. If 1,000 users send a query with the same 5,000-token prefix, the model recomputes that prefix 1,000 times, wasting massive amounts of compute and increasing the Time to First Token (TTFT).
RadixAttention, introduced by the SGLang team, addressed this by treating the KV cache as a persistent, searchable index. By utilizing a radix tree data structure, the system stores prompt prefixes as edges in a tree. When a new request arrives, the system performs a prefix match against the tree. If a match is found, the engine simply points to the existing KV blocks in memory and skips the prefill computation for that portion of the prompt.
Comparative Analysis: Radix Tree vs. Chain Hashing
The industry has seen two primary implementations of prefix caching: the Radix Tree (used in SGLang) and Chain Hashing (used in vLLM). While they achieve similar goals, their architectural approaches differ.
RadixAttention is particularly effective for deeply branching workloads, such as complex agentic workflows where a model might explore multiple reasoning paths from a single starting point. The tree structure naturally represents these branches. Conversely, vLLM’s chain hashing generates a unique hash for each KV block based on the tokens it contains and the hash of the preceding block. This creates a "content-addressed" cache where identical sequences of blocks can be identified and reused across any request.
Both methods have transformed the economics of long-context LLMs. For applications like coding assistants, where the user frequently appends small changes to a large file, prefix caching allows the system to reuse 99% of the previous computation, reducing TTFT from seconds to milliseconds.
Security Implications and Multi-Tenant Privacy
The sharing of KV blocks across different requests introduces a novel security vector: the prefix cache side-channel attack. In a multi-tenant cloud environment, an attacker could potentially infer whether a specific prompt was recently processed by another user by measuring the latency of their own requests. If a specific "secret" prefix results in an unusually low TTFT, it indicates a cache hit, confirming that the prefix exists in the system’s memory.
To mitigate this, developers have implemented "cache salting." By incorporating a tenant-specific identifier (a "salt") into the hashing or tree-matching logic, the system ensures that User A can only hit cache entries generated by User A. This preserves the performance benefits for individual users or organizations while maintaining a strict cryptographic boundary between tenants.
Implications for the Future of AI Infrastructure
The evolution of KV cache management marks a shift in AI research toward "system-aware" model design. We are now seeing the emergence of hierarchical KV caching, where the system manages a multi-tiered storage architecture. Active KV blocks are kept in GPU HBM (L1), recently used prefixes are moved to CPU RAM (L2), and long-term context is archived in NVMe storage or distributed caches (L3).
This hierarchy allows models to operate on "infinite" context windows by swapping pages in and out of GPU memory, much like how a modern computer uses swap space when RAM is exhausted. Furthermore, "cache-aware routing" is becoming standard in distributed clusters. Load balancers no longer just look at GPU load; they route requests to the specific server that already has the relevant KV cache in its memory, further maximizing the efficiency of prefix reuse.
Conclusion
The transition from contiguous memory allocation to paged and indexed KV management represents a fundamental maturing of LLM technology. PagedAttention solved the memory utilization crisis, while RadixAttention and prefix caching solved the redundant computation crisis. Together, these technologies have enabled the transition from experimental prototypes to scalable, multi-user applications capable of processing vast amounts of information in real-time. As context windows continue to grow, the ability to manage the "memory of the model" will remain just as vital as the architecture of the model itself. For the broader industry, these advancements mean lower costs, faster response times, and the ability to deploy increasingly sophisticated AI agents in production environments.
