使用 lmcache 能显著改善模型推理的 TTFT
微信公众号
·
2025-09-17
·
via 陈少文的网站
陈少文的网站 Posts 使用 lmcache 能显著改善模型推理的 TTFT Please enable Javascript to view the contents
1. LMCache 简介 TTFT 是指从请求发出到模型生成第一个 token 的时间。由于 Prefill 阶段需要把输入的上下文编码成 KV Cache，才能开始生成，在生成第一个 token 时需要大量的计算从而导致 TTFT 很高。
为了降低 TTFT，有一个思路就是将 Prefill 阶段计算出来的 KV Cache 缓存起来，下次遇到相同的上下文时，直接复用缓存的 KV Cache，就可以大幅降低 TTFT。
在模型推理的场景下，https://github.com/LMCache/LMCache 就是针对 KV Cache 缓存的一个开源项目，支持将 KV Cache 存储到内存、磁盘、Redis、GDS、Nixl 等多种存储后端。详情查看 https://docs.lmcache.ai/kv_cache/storage_backends/index.html 。
此外，lmcache 还提供了计算 KV Cache 大小的工具 https://lmcache.ai/kv_cache_calculator.html ，以 4k 中文估算，2k token 需要 106 MB 的 KV Cache，存储开销非常大。虽然 LMCache 有 LRU、FIFO、LFU、MRU 等缓存淘汰策略，但在生产环境中，通常还是需要配合大容量的存储后端，比如 Redis、3FS、大磁盘。
接下来我们通过一些 benchmark 来展示 LMCache 的效果。
2. 缓存到内存 1
2
3
4
5
6
7
8
9
10
nerdctl run -it \
\
\
= host \
memlock = -1 \
stack = 67108864 \
\
\
\
其他测试也都是基于这个镜像创建的环境，测试设备是 NVIDIA A100-SXM4-80GB。
1
unset $( env | awk -F= '/^LMCACHE_/ {print $1}' )
1
2
3
4
5
6
7
8
# Specify LMCache V1
export LMCACHE_USE_EXPERIMENTAL = True
# 256 Tokens per KV Chunk
export LMCACHE_CHUNK_SIZE = 256
# Enable CPU memory backend
export LMCACHE_LOCAL_CPU = True # default
# 50 GB of Pinned CPU memory
export LMCACHE_MAX_LOCAL_CPU_SIZE = 50 # default 5.0
1
2
3
4
5
6
7
export CUDA_VISIBLE_DEVICES = 7
/opt/venv/bin/vllm serve \
\
\
16384 \
\
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
1
2
3
4
5
6
7
/opt/venv/bin/vllm bench serve \
\
\
\
\
1024 \
16
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
============ Serving Benchmark Result ============
Successful requests: 1024
Request rate configured ( RPS) : 16.00
Benchmark duration ( s) : 72.91
Total input tokens: 225502
Total generated tokens: 202560
Request throughput ( req/s) : 14.04
Output token throughput ( tok/s) : 2778.23
Total Token throughput ( tok/s) : 5871.13
---------------Time to First Token----------------
Mean TTFT ( ms) : 62.06
Median TTFT ( ms) : 55.99
P99 TTFT ( ms) : 140.46
-----Time per Output Token ( excl. 1st token) ------
Mean TPOT ( ms) : 20.90
Median TPOT ( ms) : 20.73
P99 TPOT ( ms) : 36.28
---------------Inter-token Latency----------------
Mean ITL ( ms) : 20.39
Median ITL ( ms) : 15.81
P99 ITL ( ms) : 72.54
==================================================
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
============ Serving Benchmark Result ============
Successful requests: 1024
Request rate configured ( RPS) : 16.00
Benchmark duration ( s) : 72.35
Total input tokens: 225502
Total generated tokens: 202945
Request throughput ( req/s) : 14.15
Output token throughput ( tok/s) : 2805.13
Total Token throughput ( tok/s) : 5922.04
---------------Time to First Token----------------
Mean TTFT ( ms) : 32.65
Median TTFT ( ms) : 32.43
P99 TTFT ( ms) : 44.34
-----Time per Output Token ( excl. 1st token) ------
Mean TPOT ( ms) : 15.00
Median TPOT ( ms) : 15.07
P99 TPOT ( ms) : 16.15
---------------Inter-token Latency----------------
Mean ITL ( ms) : 14.99
Median ITL ( ms) : 14.72
P99 ITL ( ms) : 19.05
==================================================
1
( EngineCore_DP0 pid = 18318) [ 2025-09-18 05:07:16,918] LMCache INFO: Retrieved 776 out of total 776 out of total 776 tokens. size: 0.0414 gb, cost 2.0837 ms, throughput: 19.8891 GB/s; ( cache_engine.py:519:lmcache.v1.cache_engine)
可以看到建立 KV Cache 相关的日志信息。
指标 第一次测试 第二次测试 降低 TTFT 62.06ms 32.65ms 47% TPOT 20.90ms 15.00ms 28% ITL 20.39ms 14.99ms 26%
3. 缓存到磁盘 1
unset $( env | awk -F= '/^LMCACHE_/ {print $1}' )
1
2
3
4
5
6
7
export LMCACHE_CHUNK_SIZE = 256
export LMCACHE_LOCAL_DISK = "file:///data/models/lmcache/"
# 50GB of disk space
export LMCACHE_MAX_LOCAL_DISK_SIZE = 50
export LMCACHE_LOCAL_CPU = False
export LMCACHE_EXTRA_CONFIG = '{' use_odirect': True}'
export LMCACHE_USE_EXPERIMENTAL = True
1
2
3
4
5
6
7
export CUDA_VISIBLE_DEVICES = 7
/opt/venv/bin/vllm serve \
\
\
16384 \
\
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
1
2
3
4
5
6
7
/opt/venv/bin/vllm bench serve \
\
\
\
\
1024 \
16
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
============ Serving Benchmark Result ============
Successful requests: 1024
Request rate configured ( RPS) : 16.00
Benchmark duration ( s) : 72.92
Total input tokens: 225502
Total generated tokens: 202927
Request throughput ( req/s) : 14.04
Output token throughput ( tok/s) : 2783.03
Total Token throughput ( tok/s) : 5875.66
---------------Time to First Token----------------
Mean TTFT ( ms) : 63.79
Median TTFT ( ms) : 57.74
P99 TTFT ( ms) : 145.83
-----Time per Output Token ( excl. 1st token) ------
Mean TPOT ( ms) : 21.34
Median TPOT ( ms) : 21.15
P99 TPOT ( ms) : 37.31
---------------Inter-token Latency----------------
Mean ITL ( ms) : 20.78
Median ITL ( ms) : 15.88
P99 ITL ( ms) : 76.45
==================================================
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
============ Serving Benchmark Result ============
Successful requests: 1024
Request rate configured ( RPS) : 16.00
Benchmark duration ( s) : 72.49
Total input tokens: 225502
Total generated tokens: 201717
Request throughput ( req/s) : 14.13
Output token throughput ( tok/s) : 2782.66
Total Token throughput ( tok/s) : 5893.42
---------------Time to First Token----------------
Mean TTFT ( ms) : 39.40
Median TTFT ( ms) : 37.25
P99 TTFT ( ms) : 89.50
-----Time per Output Token ( excl. 1st token) ------
Mean TPOT ( ms) : 16.21
Median TPOT ( ms) : 15.99
P99 TPOT ( ms) : 20.42
---------------Inter-token Latency----------------
Mean ITL ( ms) : 16.17
Median ITL ( ms) : 14.85
P99 ITL ( ms) : 34.62
==================================================
1
2
( EngineCore_DP0 pid = 21129) [ 2025-09-18 05:23:16,568] LMCache INFO: Retrieved 12 out of total 12 out of total 12 tokens. size: 0.0006 gb, cost 0.6591 ms, throughput: 0.9724 GB/s; ( cache_engine.py:519:lmcache.v1.cache_engine)
( APIServer pid = 20851) INFO 09-18 05:23:22 [ loggers.py:123] Engine 000: Avg prompt throughput: 1700.5 tokens/s, Avg generation throughput: 1847.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
1
ls -alh /data/models/lmcache/
1
2
3
-rw-r--r-- 1 root root 14M Sep 18 05:20 vllm@-data-models-Qwen2.5-7B-Instruct@1@0@f2f002abf32763.pt
-rw-r--r-- 1 root root 14M Sep 18 05:20 vllm@-data-models-Qwen2.5-7B-Instruct@1@0@f838a2991593dd7.pt
-rw-r--r-- 1 root root 12M Sep 18 05:20 vllm@-data-models-Qwen2.5-7B-Instruct@1@0@fb7cf79a0adacc1.pt
指标 第一次测试 第二次测试 降低 TTFT 63.79ms 39.40ms 38% TPOT 21.34ms 16.21ms 24% ITL 20.78ms 16.17ms 22%
4. 缓存到 Redis 1
nerdctl run -d --name redis -p 6379:6379 redis:7
1
unset $( env | awk -F= '/^LMCACHE_/ {print $1}' )
1
2
3
4
5
6
7
8
9
10
11
12
13
# Specify LMCache V1
export LMCACHE_USE_EXPERIMENTAL = True
# 256 Tokens per KV Chunk
export LMCACHE_CHUNK_SIZE = 256
# Redis host
export LMCACHE_REMOTE_URL = "redis://x.x.x.x:6379"
# Redis Sentinel hosts (for high availability)
# export LMCACHE_REMOTE_URL="redis-sentinel://localhost:26379,localhost:26380,localhost:26381"
# LMCache Server host
# export LMCACHE_REMOTE_URL="lm://localhost:65432"
# How to serialize and deserialize KV cache on remote transmission
export LMCACHE_REMOTE_SERDE = "naive" # "naive" (default) or "cachegen"
1
2
3
4
5
6
7
export CUDA_VISIBLE_DEVICES = 7
/opt/venv/bin/vllm serve \
\
\
16384 \
\
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
1
2
3
4
5
6
7
/opt/venv/bin/vllm bench serve \
\
\
\
\
1024 \
16
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
============ Serving Benchmark Result ============
Successful requests: 1024
Request rate configured ( RPS) : 16.00
Benchmark duration ( s) : 72.90
Total input tokens: 225502
Total generated tokens: 202337
Request throughput ( req/s) : 14.05
Output token throughput ( tok/s) : 2775.41
Total Token throughput ( tok/s) : 5868.57
---------------Time to First Token----------------
Mean TTFT ( ms) : 67.79
Median TTFT ( ms) : 60.94
P99 TTFT ( ms) : 165.84
-----Time per Output Token ( excl. 1st token) ------
Mean TPOT ( ms) : 22.20
Median TPOT ( ms) : 21.75
P99 TPOT ( ms) : 40.42
---------------Inter-token Latency----------------
Mean ITL ( ms) : 21.43
Median ITL ( ms) : 15.96
P99 ITL ( ms) : 78.68
==================================================
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
============ Serving Benchmark Result ============
Successful requests: 1024
Request rate configured ( RPS) : 16.00
Benchmark duration ( s) : 72.91
Total input tokens: 225502
Total generated tokens: 202978
Request throughput ( req/s) : 14.04
Output token throughput ( tok/s) : 2783.98
Total Token throughput ( tok/s) : 5876.88
---------------Time to First Token----------------
Mean TTFT ( ms) : 50.34
Median TTFT ( ms) : 39.07
P99 TTFT ( ms) : 142.44
-----Time per Output Token ( excl. 1st token) ------
Mean TPOT ( ms) : 18.80
Median TPOT ( ms) : 17.32
P99 TPOT ( ms) : 35.65
---------------Inter-token Latency----------------
Mean ITL ( ms) : 18.17
Median ITL ( ms) : 15.13
P99 ITL ( ms) : 66.43
==================================================
1
( EngineCore_DP0 pid = 23013) [ 2025-09-18 05:29:58,971] LMCache INFO: Storing KV cache for 776 out of 776 tokens ( skip_leading_tokens = 0) for request cmpl-benchmark-serving1022-0 ( vllm_v1_adapter.py:988:lmcache.integration.vllm.vllm_v1_adapter)
1
nerdctl exec -it redis redis-cli KEYS "*"
1
2
3
4
5
6
4087) "vllm@/data/models/Qwen2.5-7B-Instruct@1@0@-7542097a982a0d29metadata"
4088) "vllm@/data/models/Qwen2.5-7B-Instruct@1@0@ab0b65969d69b56metadata"
4089) "vllm@/data/models/Qwen2.5-7B-Instruct@1@0@-49f1c05dccbcca9metadata"
4090) "vllm@/data/models/Qwen2.5-7B-Instruct@1@0@-2ede777a488b6923kv_bytes"
4091) "vllm@/data/models/Qwen2.5-7B-Instruct@1@0@-27a856291a779d38kv_bytes"
4092) "vllm@/data/models/Qwen2.5-7B-Instruct@1@0@7522166acc9c0267kv_bytes"
1
2
nerdctl exec -it redis redis-cli MEMORY USAGE "vllm@/data/models/Qwen2.5-7B-Instruct@1@0@-27a856291a779d38kv_bytes"
( integer) 524408
一个缓存块大约是 12 MB，与磁盘缓存块大小一致。
指标 第一次测试 第二次测试 降低 TTFT 67.79ms 50.34ms 25% TPOT 22.20ms 18.80ms 15% ITL 21.43ms 18.17ms 15%
5. 无 LMCache 对照 1
2
3
4
5
6
7
8
9
10
nerdctl run -it \
\
\
= host \
memlock = -1 \
stack = 67108864 \
\
\
\
1
2
3
4
5
6
7
8
9
10
11
export CUDA_VISIBLE_DEVICES = 7
python3 -m vllm.entrypoints.openai.api_server \
\
\
8000 \
\
4096 \
8192 \
128 \
\
1
2
3
4
5
6
7
vllm bench serve \
\
\
\
\
1024 \
16
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
============ Serving Benchmark Result ============
Successful requests: 1024
Request rate configured ( RPS) : 16.00
Benchmark duration ( s) : 73.42
Total input tokens: 225502
Total generated tokens: 203130
Request throughput ( req/s) : 13.95
Output token throughput ( tok/s) : 2766.85
Total Token throughput ( tok/s) : 5838.43
---------------Time to First Token----------------
Mean TTFT ( ms) : 61.55
Median TTFT ( ms) : 54.89
P99 TTFT ( ms) : 174.88
-----Time per Output Token ( excl. 1st token) ------
Mean TPOT ( ms) : 21.49
Median TPOT ( ms) : 21.27
P99 TPOT ( ms) : 36.38
---------------Inter-token Latency----------------
Mean ITL ( ms) : 21.00
Median ITL ( ms) : 16.70
P99 ITL ( ms) : 72.61
==================================================
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
============ Serving Benchmark Result ============
Successful requests: 1024
Request rate configured ( RPS) : 16.00
Benchmark duration ( s) : 73.84
Total input tokens: 225502
Total generated tokens: 203659
Request throughput ( req/s) : 13.87
Output token throughput ( tok/s) : 2758.13
Total Token throughput ( tok/s) : 5812.08
---------------Time to First Token----------------
Mean TTFT ( ms) : 59.70
Median TTFT ( ms) : 54.41
P99 TTFT ( ms) : 139.79
-----Time per Output Token ( excl. 1st token) ------
Mean TPOT ( ms) : 21.31
Median TPOT ( ms) : 21.08
P99 TPOT ( ms) : 36.62
---------------Inter-token Latency----------------
Mean ITL ( ms) : 20.78
Median ITL ( ms) : 16.63
P99 ITL ( ms) : 71.70
==================================================
指标 第一次测试 第二次测试 降低 TTFT 61.55ms 59.70ms 3% TPOT 21.49ms 21.31ms 1% ITL 21.00ms 20.78ms 1%
6. 总结 本篇主要是通过 benchmark 来展示 LMCache 的效果，并分别缓存到内存、磁盘、Redis 三种后端。
在 Qwen2.5-7B-Instruct 模型，使用 NVIDIA A100-SXM4-80GB 设备，16 个并发请求，测试结果如下：
缓存后端 TTFT 降低 TPOT 降低 ITL 降低 内存 47% 28% 26% 磁盘 38% 24% 22% Redis 25% 15% 15% 无 LMCache 3% 1% 1%
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。
