Running LLMs locally is mostly a memory bandwidth problem. For single-user inference the GPU or Apple Silicon almost always waits on memory. Every token needs the model's weights pulled across the bus, so if you know the bandwidth of the memory the weights live in, and how big the weights are, you can get a decent estimate of tokens/sec before spending anything.
The calculator below does exactly that. Pick your hardware tiers, the model you want to run, and the quantization. It tells you roughly how fast it'll go.
Primary memory GPU VRAM or Apple unified memory
System RAM falls back here if the model doesn't fit above
SSD last-resort streaming from disk
Model
Context & KV cache Llama 3 70B fp16 ~320 KB/token, 8B ~128, Mistral 7B GQA ~64
- tokens/sec
Memory footprint -
LLM inference on a single user is almost always memory-bandwidth bound. Every generated token requires the model's weights plus the full KV cache to be read from memory. So the speed ceiling is roughly bandwidth ÷ bytes read per token.
Weights bytes per token: full model size for a dense model, active parameter size for MoE (e.g. Mixtral 8x7B: 47B total, 13B active). Model size = params × bytes-per-param (int4 = 0.5, int8 = 1, fp16 = 2, etc.).
KV cache is read in full every token regardless of MoE. Total KV cache = context length × KV cache per token. Typical values for modern GQA models: Llama 3 70B fp16 ~320 KB/token, Llama 3 8B ~128 KB, Mistral 7B ~64 KB, Qwen 32B ~64 KB. Halve for fp8 quantization.
Allocation follows llama.cpp's default: the KV cache is pinned to primary memory, and weights fill whatever's left, spilling to system RAM then SSD. This means a 70B dense model whose weights spill to RAM will still have attention running at GPU speed, with only the FFN layers paying the RAM penalty. For MoE, only active experts are read per token, so weights in RAM hurt far less than they would for a dense model of the same total size. With Apple unified memory, there is no separate system RAM tier.
Ignored: prompt processing (prefill), compute-bound behaviour on very small models, and any software-side overhead. Real-world numbers are typically 60–90% of the theoretical ceiling. Treat the output as an optimistic upper bound.
A few things worth saying about the output:
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。