The Physics of Inference – A Deep Dive into KV and Prompt Caching

Introduction

Viewed through an engineering lens, as the "Scaling Laws" face increasing scrutiny, I find myself agreeing with the growing consensus: Large Language Models (LLMs) are entering a "middle age" of calculated efficiency—a time for harvesting fruits rather than just planting forests.

In his Thanksgiving letter, Andrew Ng noted that while there may be bubbles in AI, they are certainly not in the application layer:

AI Application Layer: Underinvested. The potential here far exceeds common perception.
AI Inference Infrastructure: Still requires significant investment.
AI Training Infrastructure: I remain cautiously optimistic, though this is where a bubble might exist.

Context

As Generative AI transitions from experimental labs to large-scale commercial deployment, inference efficiency has become the critical variable determining economic viability. In the current landscape dominated by the Transformer architecture, the marginal cost of inference is constrained not by pure compute (FLOPs), but by the "Memory Wall."

As context windows expand from the early 4k tokens to 128k, 1M, and even 10M, managing the Key-Value (KV) Cache has emerged as the primary bottleneck for system throughput and latency.

This analysis spans from underlying physical principles to high-level application strategies. We begin by dissecting the mathematics of the KV Cache during decoding and its consumption of memory bandwidth. We then trace the architectural evolution from Multi-Head Attention (MHA) to Grouped-Query Attention (GQA), and finally to the Multi-Head Latent Attention (MLA) pioneered by DeepSeek. MLA, in particular, achieves extreme compression through the decoupling of low-rank matrix decomposition and Rotary Positional Embeddings (RoPE), laying the physical foundation for "disk-level caching."

On the system software front, we examine how vLLM’s PagedAttention borrows paging concepts from operating systems to solve fragmentation, and how SGLang’s RadixAttention utilizes Radix Trees for dynamic KV reuse. We also touch upon StreamingLLM, which exploits the "Attention Sink" phenomenon to bypass window limits for infinite streaming.

Finally, we survey the market implementation of Prompt Caching (Google, Anthropic, OpenAI, DeepSeek, Alibaba), contrasting the "High-Performance Memory" route against the "Architecture-Driven Low-Cost" route.

1. The Physical Bottleneck: Seeing Through the KV Cache

Before discussing optimization, we must understand—from first principles—why the KV Cache is the Achilles' heel of large model inference. It is not merely a question of capacity, but a conflict between Memory Bandwidth and Arithmetic Intensity.

1.1 The Autoregressive Nature of Transformer Decoding

Inference in Transformers occurs in two distinct phases:

Prefill Phase: The model processes all input tokens in parallel. Because this is highly parallelizable, it is usually Compute-bound. GPU utilization is high.
Decoding Phase: The model generates subsequent tokens one by one. This is an Autoregressive process; generating the $t$-th token depends on the internal state of the previous $t-1$ tokens.

In standard Self-Attention, the calculation is:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Here, $Q$ (Query) is the vector for the current step, while $K$ (Key) and $V$ (Value) hold information from all history tokens. To avoid recalculating the $K$ and $V$ projections for the entire history at every new step, the system stores these vectors in VRAM. This is the KV Cache.

1.2 The Math of VRAM Consumption

KV Cache size is a linear function of sequence length, multiplying with layers, heads, and dimensions. For a standard Transformer, it can be calculated as:

$$\text{Size}{KV} = 2 \times L{seq} \times B_{batch} \times N_{layers} \times H_{heads} \times D_{head} \times P_{prec}$$

Where:

$2$: Represents the two matrices, Key and Value.
$L_{seq}$: Current sequence length (context window).
$B_{batch}$: Batch size of concurrent requests.
$N_{layers}$: Number of layers.
$H_{heads}$: Number of attention heads.
$D_{head}$: Dimension per head.
$P_{prec}$: Precision (2 bytes for FP16).

Case Study: Llama-2 70B
Assuming FP16 precision, a sequence length of 4096, and a Batch Size of 1:

$N_{layers} = 80$
$H_{heads} = 64$
$D_{head} = 128$

The KV Cache for a single request is:
$$2 \times 4096 \times 1 \times 80 \times 64 \times 128 \times 2 \approx 10.7 \text{ GB}$$

If we extend the context to 100k tokens, this swells to 260 GB. This far exceeds the capacity of a single NVIDIA A100 (80GB) or H100. Consequently, memory capacity limits Batch Size, preventing the GPU cores from being fully utilized, driving up unit costs.

1.3 The Memory Wall

Beyond capacity, bandwidth is the silent killer. During decoding, for every token generated, the GPU must move the entire KV Cache from High Bandwidth Memory (HBM) to the on-chip SRAM for calculation.

Compute (FLOPs): Grows linearly.
Data Transfer (Bytes): Also grows linearly.

However, because the matrix multiplication degenerates into a vector operation (Query vector), the Arithmetic Intensity (FLOPs/Bytes ratio) is extremely low. Even with an H100's massive bandwidth (~3.35 TB/s), the GPU spends most of its time waiting for data. This is the definition of a Memory-bound scenario.

2. Architectural Evolution: From MHA to MLA

To shrink the KV Cache, architects have performed surgery on the heart of the Transformer.

2.1 Multi-Head Attention (MHA): The Expensive Baseline

In the original Attention Is All You Need, the model has $H$ Query Heads and $H$ Key/Value Heads.

Mechanism: Each Query Head has a unique KV pair. Maximum expressiveness.
Cost: Size is proportional to $H$. In the long-context era, this became unsustainable.

2.2 Multi-Query Attention (MQA): Radical Compression

Proposed by Noam Shazeer (2019).

Mechanism: All Query Heads share one Key Head and one Value Head.
Compression: $H : 1$. (e.g., 64x reduction).
Trade-off: Radical memory savings, but the model loses the ability to "attend" to different nuances simultaneously, often degrading perplexity. Used in PaLM and Falcon.

2.3 Grouped-Query Attention (GQA): The Golden Mean

Introduced with Llama-2, GQA became the standard for open-source models (Llama-3, Mistral, Qwen).

Mechanism: Query Heads are divided into $G$ groups. Each group shares a KV Head.
Example: Llama-2 70B uses 8 KV Heads for 64 Query Heads (8:1 compression).
Result: It sits on the Pareto Frontier—delivering performance near MHA with efficiency near MQA.

2.4 Multi-Head Latent Attention (MLA): DeepSeek's Revolution

DeepSeek-V2 (and V3) introduced MLA, which is not just a grouping strategy, but a fundamental reconstruction of storage.

2.4.1 Low-Rank Compression

Instead of storing the full $d_{model} \times L$ matrices, MLA assumes redundancy. It projects the input into a low-dimensional "Latent Vector" ($c_{KV}$) and stores only this compressed version. During computation, it projects this vector back up to the full dimension.
This reduces memory footprint from $O(H \times d_{head})$ to $O(d_{latent})$.

2.4.2 Decoupled RoPE

The challenge with compression is Rotary Positional Embeddings (RoPE). RoPE is geometrically sensitive; applying it to a compressed vector destroys position information.

DeepSeek's solution: Decoupling.

Content Head: Captures semantics, uses low-rank compression (No RoPE).
Position Head: A separate, tiny vector specifically carrying RoPE info.
Concatenation: They are joined only during the attention score calculation.

This allows the KV Cache to be 1/5th the size of GQA models. Crucially, it makes moving the cache to SSD/RAM feasible because the bandwidth requirement drops drastically.

Feature	MHA (Llama-1)	MQA (Falcon)	GQA (Llama-3)	MLA (DeepSeek-V3)
KV Heads	= Query Heads ($H$)	1	Groups ($G$)	Virtual/Dynamic
VRAM Usage	High (100%)	Very Low (~1-2%)	Medium (~12-25%)	Extreme (~5-10%)
Performance	Baseline	Lossy	Near Lossless	Lossless/Better
RoPE	Native	Native	Native	Decoupled

3. System-Level Management: OS Concepts Reborn

If architecture defines the "theoretical minimum," system software determines how we place that data on hardware.

3.1 PagedAttention (vLLM)

Before vLLM, memory was allocated statically based on "Max Sequence Length," leading to fragmentation and 60-80% waste.

3.1.1 The Principle

Inspired by Virtual Memory paging:

KV Block: Data is sliced into fixed blocks (e.g., 16 tokens).
Non-contiguous: Blocks can live anywhere in physical memory.
Block Table: Maps logical flow to physical blocks.

Impact:

Zero Waste: Internal fragmentation is limited to the last block.
Memory Sharing: Multiple requests sharing a System Prompt ("You are a helpful assistant...") point to the same physical blocks. Copy-on-Write is triggered only when they diverge. This is the foundation of Prompt Caching.

3.2 RadixAttention (SGLang)

vLLM handled allocation; SGLang handles discovery.

3.2.1 Radix Tree Structure

SGLang views the KV Cache not as a linear array, but as a Radix Tree.

Nodes: KV Cache states.
Edges: Token sequences.

3.2.2 Automatic Reuse

Scenario: User asks A, gets B. User asks C. The system sees the path A->B and resumes calculation from there.

LRU Eviction: When memory fills, leaves are pruned first.

3.3 StreamingLLM

For infinite streams (e.g., digital humans), simple sliding windows break the model.
MIT researchers discovered Attention Sinks: The first few tokens (usually 4) anchor the entire attention mechanism. StreamingLLM keeps these "sink tokens" permanently and slides the rest, allowing infinite length with stable perplexity.

4. Extreme Compression: Quantization

FP8: Supported by H100, halves memory usage with negligible loss.
INT4: Difficult due to "Outliers" in the Key/Value matrices. Techniques like SmoothQuant and KIVI migrate outliers to weights or keep them in high precision to make INT4 viable.

5. Market Landscape: The Battle of Caching

2025 marks the era of "Context Caching" as a standard product.

5.1 DeepSeek: The Price Butcher

Leveraging MLA, DeepSeek moves cache to Disk (SSD).

Price: $0.014 / million tokens (Hit). This is ~0.5% of OpenAI's price.
Storage: Free.
TTL: Hours to days. Ideal for long-tail knowledge retrieval.

5.2 Google Gemini: TPU Scale

Implicit: Automatic for Flash models.
Explicit: For Pro models. A "Lease" model—you pay a storage fee per hour. Only economical for high-frequency queries.

5.3 Anthropic Claude: High-Speed RAM Lease

Targeted at coding and high-interaction tasks.

TTL: 5 minutes.
Mechanism: Explicit breakpoints.
Economics: You pay a premium (1.25x) to write to cache. You must reuse it within 5 minutes to break even.

5.4 OpenAI & Alibaba

OpenAI: Conservative. 50% discount on hits. No write premium.
Alibaba (Qwen): Mixed mode. Strong support for long contexts (10M tokens).

Vendor	Mechanism	Storage Medium	TTL	Write Cost	Read Cost	Storage Fee
DeepSeek	Implicit	SSD/Disk	Long	1.0x	~0.05x	Free
Anthropic	Explicit	HBM	5 min	1.25x	0.10x	Included
Google	Hybrid	TPU HBM	1 hour+	1.0x	~0.25x	Hourly
OpenAI	Implicit	HBM	Dynamic	1.0x	0.50x	Free

6. Semantic Caching

Complementary to Prompt Caching (Server-side), Semantic Caching (Client-side) uses Embeddings (Vector DBs like Milvus) to match intent.

If a user asks "Price of apple?" and later "How much is an apple?", Semantic Cache returns the saved answer without hitting the LLM.
Tools: GPTCache.

7. Case Study: Prompt Cache in Agent Dev

Code Example

# Enabling Prompt Caching in Qwen
from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
  SystemMessage(
    content=[
      {
        "type": "text",
        "text": app_prompt_template.format(vars),
        "cache_control": {"type": "ephemera"}, # The explicit flag
      }
    ]
  ),
  HumanMessage(content=app_user_prompt_template.format(input_data))
])

# Enabling Prompt Caching in OpenAI
from openai import OpenAI

client = OpenAI()

completion = client.chat.completions.create(
    model="gpt-4o",
    # openai 支持 24h 保存 cache
    prompt_cache_retention: "24h"
    messages=[
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Tell me a joke."},
    ]
)

Summary

The history of Large Model inference is, essentially, a history of struggling against memory bandwidth.

Architecture & Hardware: DeepSeek's MLA proves that algorithmic innovation (low-rank compression) can unlock hardware potential (SSD storage), completely upending the business model.
Stateful APIs: The HTTP stateless protocol is no longer sufficient. LLMs are becoming "Stateful Operating Systems," and developers must manage "Context Lifecycle" just as they manage database connections.
The Cost Cliff: With prices hitting $0.014/M tokens, the bottleneck for RAG shifts from "how to retrieve less to save money" to "how much context can the model handle without hallucinating." Full Context is replacing sliced retrieval.
For developers, the strategy is clear: Use Anthropic/vLLM for high-frequency, low-latency tasks (coding assistants), and leverage DeepSeek's disk caching for massive knowledge analysis where cost is the primary constraint.

References

Optimizing Transformer Inference with Grouped Query Attention | Towards AI, accessed November 27, 2025, https://towardsai.net/p/machine-learning/optimizing-transformer-inference-with-grouped-query-attention
arXiv:2305.13245v3 [cs.CL] 23 Dec 2023, accessed November 27, 2025, https://arxiv.org/pdf/2305.13245
Attention Mechanisms in Transformers: Comparing MHA, MQA, and GQA | Yue Shui Blog, accessed November 27, 2025, https://syhya.github.io/posts/2025-01-16-group-query-attention/
Understanding Multi-Head Latent Attention, accessed November 27, 2025, https://planetbanatt.net/articles/mla.html
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model, accessed November 27, 2025, https://arxiv.org/html/2405.04434v2
DeepSeek API introduces Context Caching on Disk, cutting prices by an order of magnitude, accessed November 27, 2025, https://api-docs.deepseek.com/news/news0802
MLA: Redefining KV-Cache Through Low-Rank Projections and On-Demand Decompression - Hugging Face, accessed November 27, 2025, https://huggingface.co/blog/NormalUhr/mla-explanation
How PagedAttention resolves memory waste of LLM systems - Red Hat Developer, accessed November 27, 2025, https://developers.redhat.com/articles/2025/07/24/how-pagedattention-resolves-memory-waste-llm-systems
Introduction to vLLM and PagedAttention | Runpod Blog, accessed November 27, 2025, https://www.runpod.io/blog/introduction-to-vllm-and-pagedattention
vLLM and PagedAttention: A Comprehensive Overview | by Abonia Sojasingarayar | Medium, accessed November 27, 2025, https://medium.com/@abonia/vllm-and-pagedattention-a-comprehensive-overview-20046d8d0c61
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention - arXiv, accessed November 27, 2025, https://arxiv.org/html/2405.04437v1
When to Choose SGLang Over vLLM: Multi-Turn Conversations and KV Cache Reuse, accessed November 27, 2025, https://www.runpod.io/blog/sglang-vs-vllm-kv-cache
SGLang: Efficient Execution of Structured Language Model Programs - arXiv, accessed November 27, 2025, https://arxiv.org/pdf/2312.07104
Fast and Expressive LLM Inference with RadixAttention and SGLang | LMSYS Org, accessed November 27, 2025, https://lmsys.org/blog/2024-01-17-sglang/
Arxiv Dives - Efficient Streaming Language Models with Attention Sinks - Oxen.ai, accessed November 27, 2025, https://ghost.oxen.ai/arxiv-dives-efficient-streaming-language-models-with-attention-sinks/
Efficient Streaming Language Models with Attention Sinks - arXiv, accessed November 27, 2025, https://arxiv.org/html/2309.17453v4
[2309.17453] Efficient Streaming Language Models with Attention Sinks - arXiv, accessed November 27, 2025, https://arxiv.org/abs/2309.17453
Attention Sinks for LLM - Endless Generation - Analytics Vidhya, accessed November 27, 2025, https://www.analyticsvidhya.com/blog/2023/12/attention-sinks-for-llm/
FP8 E5M2 KV Cache - vLLM, accessed November 27, 2025, https://docs.vllm.ai/en/v0.6.3.post1/quantization/fp8_e5m2_kvcache.html
FP8 quantization with AMD Quark for vLLM — Tutorials for AI developers 8.0, accessed November 27, 2025, https://rocm.docs.amd.com/projects/ai-developer-hub/en/latest/notebooks/gpu_dev_optimize/fp8_quantization_quark_vllm.html
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization, accessed November 27, 2025, https://www.stat.berkeley.edu/~mmahoney/pubs/neurips-2024-kvquant.pdf
AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations - ACL Anthology, accessed November 27, 2025, https://aclanthology.org/2025.coling-main.158.pdf
FireQ: Fast INT4-FP8 Kernel and RoPE-aware Quantization for LLM Inference Acceleration, accessed November 27, 2025, https://arxiv.org/html/2505.20839v1
NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics, accessed November 27, 2025, https://arxiv.org/html/2505.16210v1
Gemini Developer API pricing, accessed November 27, 2025, https://ai.google.dev/gemini-api/docs/pricing
Context caching overview | Generative AI on Vertex AI - Google Cloud Documentation, accessed November 27, 2025, https://docs.cloud.google.com/vertex-ai/generative-ai/docs/context-cache/context-cache-overview
Context Caching In Google Gemini: Better Than RAG For Memory - Empathy First Media, accessed November 27, 2025, https://empathyfirstmedia.com/context-caching-google-gemini/
Prompt Caching Support in Spring AI with Anthropic Claude, accessed November 27, 2025, https://spring.io/blog/2025/10/27/spring-ai-anthropic-prompt-caching-blog
Prompt caching - Claude Docs, accessed November 27, 2025, https://platform.claude.com/docs/en/build-with-claude/prompt-caching
Prompt Caching is a Must! How I Went From Spending $720 to $72 Monthly on API Costs | by Du'An Lightfoot | Medium, accessed November 27, 2025, https://medium.com/@labeveryday/prompt-caching-is-a-must-how-i-went-from-spending-720-to-72-monthly-on-api-costs-3086f3635d63
Prompt caching - OpenAI API, accessed November 27, 2025, https://platform.openai.com/docs/guides/prompt-caching
Prompt Caching in the API - OpenAI, accessed November 27, 2025, https://openai.com/index/api-prompt-caching/
How does Prompt Caching work? - OpenAI Developer Community, accessed November 27, 2025, https://community.openai.com/t/how-does-prompt-caching-work/992307
Context Cache feature of Qwen models - Alibaba Cloud Model Studio, accessed November 27, 2025, https://www.alibabacloud.com/help/en/model-studio/context-cache
Qwen context window: token limits, memory policy, and 2025 rules - Data Studios, accessed November 27, 2025, https://www.datastudios.org/post/qwen-context-window-token-limits-memory-policy-and-2025-rules
Semantic Cache: How to Speed Up LLM and RAG Applications - Medium, accessed November 27, 2025, https://medium.com/@svosh2/semantic-cache-how-to-speed-up-llm-and-rag-applications-79e74ce34d1d
Semantic Cache: Accelerating AI with Lightning-Fast Data Retrieval - Qdrant, accessed November 27, 2025, https://qdrant.tech/articles/semantic-cache-ai-data-retrieval/
Semantic caching for faster, smarter LLM apps - Redis, accessed November 27, 2025, https://redis.io/blog/what-is-semantic-caching/
zilliztech/GPTCache: Semantic cache for LLMs. Fully integrated with LangChain and llama_index. - GitHub, accessed November 27, 2025, https://github.com/zilliztech/GPTCache
Reducing LLM Costs and Latency via Semantic Embedding Caching - arXiv, accessed November 27, 2025, https://arxiv.org/html/2411.05276v2
If your app process many similar queries, use Semantic Caching to reduce your cost and latency : r/LangChain - Reddit, accessed November 27, 2025, https://www.reddit.com/r/LangChain/comments/1f4rlx0/if_your_app_process_many_similar_queries_use/
图解vLLM Automatic Prefix Cache(RadixAttention), https://zhuanlan.zhihu.com/p/693556044
Gemini 3技术是跳蛙式超越 https://www.youtube.com/watch?v=EMQxQwoFSb4
Andre Ng issue 329 | deeplearning.ai batch https://www.deeplearning.ai/the-batch/issue-329/
The Architecture Behind vLLM: How PagedAttention Improves Memory Utilization https://medium.com/@mandeep0405/the-architecture-behind-vllm-how-pagedattention-improves-memory-utilization-2f9b25272110
Low Rank Decompositions of Matrices - YouTube https://www.youtube.com/watch?v=_FmolBCUo9M
The Inner Workings of DeepSeek-V3 · Chris McCormick https://mccormickml.com/2025/02/12/the-inner-workings-of-deep-seek-v3/
Decoupled RoPE with MHLA: Blending Rotary Positional Encoding and Latent Attention Like a Pro https://medium.com/@drpester001/decoupled-rope-with-mhla-blending-rotary-positional-encoding-and-latent-attention-like-a-pro-d13842134f4d
Efficient Streaming Language Models with Attention Sinks https://github.com/mit-han-lab/streaming-llm

推荐订阅源

Shadow Walker 松烟阁