Cache-optimized LLM conversation history management.

Most LLM applications naively concatenate their system prompt, conversation history, and any dynamic context into a single message list — and rebuild it from scratch every turn. This works, but it leaves significant money and latency on the table by constantly invalidating the provider's prompt cache.

llmbuffer assembles your messages in the order that maximises cache reuse, manages the boundary between stable and changing content, and handles compaction when history grows too long — all without you having to think about it.

[Static System Prompt] → [Long-Lived History] → [Dynamic Context] → [Recent Messages]
       cached ✓                  cached ✓             not cached          not cached

The static system prompt and committed conversation history form a byte-stable prefix that is never mutated or re-ordered across turns. The frequently-changing parts — RAG results, timestamps, in-flight tool calls — live at the end where they can't invalidate the prefix.

Install

Optional extras for live benchmarking:

pip install "llmbuffer[anthropic]"    # Anthropic prompt caching
pip install "llmbuffer[openai]"       # OpenAI prefix caching

llmbuffer has zero required dependencies — just Python 3.9+.

Quickstart

Stateful (in-process)

from llmbuffer import PromptManager, AnthropicAdapter

manager = PromptManager(
    static_system_prompt="You are a senior software engineering assistant...",
    transition_mode="agent_cycle",   # auto-commit turns to the stable prefix
    adapter=AnthropicAdapter(),      # inject cache_control markers
    max_tokens=8_000,                # compact long-lived history beyond this
)

# Each turn:
manager.append({"role": "user", "content": user_message})
messages = manager.build_messages(dynamic_system_prompt=rag_context)
reply = anthropic_client.messages.create(messages=messages, ...)
manager.append({"role": "assistant", "content": reply})

Stateless (web app / serverless)

Pure functions over a JSON-serializable state dict — persist it anywhere between requests:

from llmbuffer import functional, new_state, dumps, loads

SYSTEM = "You are a senior software engineering assistant..."

# Load state from DB / session
state = loads(row.conversation_json) if row else new_state()

# Build messages, call LLM, store updated state
state = functional.append_message(state, {"role": "user", "content": text},
                                  transition_mode="manual")
messages = functional.build_messages(state, static_system_prompt=SYSTEM,
                                     dynamic_system_prompt=rag_context)
# ... call your LLM ...
state = functional.append_message(state, reply, transition_mode="manual")
state = functional.compact(state, max_tokens=8_000)   # explicit in the functional API
row.conversation_json = dumps(state)

Each function takes only the settings it uses — there's no config object to thread through. Compaction is an explicit compact() call in the functional API (the stateful PromptManager runs it automatically).

How it works

Message ordering

build_messages() always emits messages in this exact order:

Transition modes

Control when messages graduate from short-term into the stable long-lived history:

Transition hooks

Before messages move from short-term into the long-lived (cached) history, an optional transition_hook can rewrite them — useful for trimming verbose tool outputs or stripping content you don't want locked into the stable prefix forever.

def trim_tool_outputs(messages):
    """Keep only the last 20 lines of any tool output before it enters long-lived history."""
    result = []
    for msg in messages:
        if msg.get("role") == "tool":
            content = msg.get("content", "")
            lines = content.splitlines()
            if len(lines) > 20:
                kept = "\n".join(lines[-20:])
                msg = {**msg, "content": f"[…{len(lines) - 20} lines truncated]\n{kept}"}
        result.append(msg)
    return result

manager = PromptManager(
    transition_mode="agent_cycle",
    transition_hook=trim_tool_outputs,
)
# Functional API: pass the hook directly
# state = functional.append_message(state, msg, transition_mode="agent_cycle",
#                                   transition_hook=trim_tool_outputs)

The hook receives the list of short-term messages being committed and returns whatever should actually land in long-lived history. Drop messages entirely, summarise them, replace binary blobs with descriptions — the returned list is what gets cached.

Dynamic context: two channels

Context that changes during a conversation comes in two flavours, and they want different placement:

Volatile context is passed per-call and never stored — it sits after the cached prefix where it can't invalidate anything.

Durable context is appended as a keyed system message, riding the normal transition path so temporal ordering is preserved — a mid-turn update lands at the high-attention end of the list, not buried in the prefix:

manager = PromptManager(
    static_system_prompt=SYSTEM,
    initial_context=initial_world_state,    # seeds the stable prefix at creation
    context_key="world",
    max_tokens=8_000,
)
# later, when something changes:
manager.append_context("Update: the inventory now contains 3 keys.")

At compaction time, the initial context and all its deltas are consolidated: a ContextConsolidationHook receives every message for a key (initial block first, deltas in order) and returns the new, fully rewritten context, which is placed at the front of the compacted history — right after the static system prompt. The default hook concatenates losslessly; supply your own to apply diffs or summarise with an LLM:

def consolidate(key, messages):
    return rewrite_world_state(base=messages[0]["content"],
                               deltas=[m["content"] for m in messages[1:]])

manager = PromptManager(..., context_consolidation_hook=consolidate)

Keyed messages never pass through the lossy compaction hook — consolidation and lossy compaction are separate phases, and both always run once compaction triggers (the prefix is being rewritten anyway, so compact all the way down).

What lives outside the state: exactly two things are not carried in the serialized state — the static system prompt and the per-call volatile dynamic prompt. Everything else, including durable context and its deltas, round-trips through dumps()/loads(). In the stateless pattern: rehydrated state + your constant static prompt = the complete conversation.

Compaction

When the long-lived history exceeds max_tokens, a compaction hook reduces it to max_tokens // 2 (configurable). The default hook truncates oldest-first; supply your own to summarise instead:

def summarise(messages, target_tokens, adapter):
    summary = call_llm_to_summarise(messages)
    return [{"role": "system", "content": summary}]

manager = PromptManager(max_tokens=8_000, compaction_hook=summarise)
# Functional API: compaction is an explicit call
# state = functional.compact(state, max_tokens=8_000, compaction_hook=summarise)

Boundary metadata

Pass with_metadata=True to build_messages() to also get the predicted cacheable-prefix layout — useful for logging, debugging, or asserting prefix stability in your own tests:

messages, meta = manager.build_messages(dynamic_system_prompt=rag, with_metadata=True)
# meta == {"boundaries": [0, 12], "prefix_message_count": 13,
#          "prefix_tokens": 4203, "suffix_tokens": 310, "total_tokens": 4513}

These are predictions from prefix stability; actual cache hits are only reported in the provider's response usage metadata.

Request-budget compaction

compact() budgets the long-lived history in isolation. When your real constraint is the whole request (static system + history + dynamic context ≤ context window), use compact_for_request():

state = functional.compact_for_request(
    state,
    request_budget=128_000,            # whole-request token budget
    static_system_prompt=SYSTEM,       # measured (it's stable by contract)
    reserved_tokens=8_000,             # fixed headroom for dynamic + short-term content
)
# or: manager.compact_for_request(request_budget=128_000, reserved_tokens=8_000)

reserved_tokens is deliberately a declared constant, not a measurement of the current turn: if the budget tracked the fluctuating dynamic content, compaction could trigger on one turn and not the next — rewriting the long-lived prefix and invalidating the cache. Reserve your worst case and the derived budget stays deterministic.

Provider adapters

Subclass ProviderAdapter to add a new provider — override count_tokens() and/or apply_cache_markers().

Benchmark

The benchmark suite runs a multi-turn conversation through both llmbuffer and a naive approach, and reports cache hits from the provider's own usage metadata.

The naive approach puts the static and dynamic system prompts together at the start of every message list and drops the oldest messages when the context limit is hit — this is the default pattern in most LLM applications today.

Results (simulated, 15 turns, Anthropic pricing)

The simulated provider models provider prefix caching exactly: a turn is a cache hit when its message list shares a prefix with a previously-seen turn. Run --provider anthropic or --provider openai for live numbers.

Bottom line: llmbuffer's input costs are ~43% lower than the naive approach ($0.016 vs $0.028 for the same 15-turn conversation).

How to read these numbers:

• The cache hit ratio is the fraction of input tokens served from cache. It doesn't translate 1:1 into savings because cached tokens aren't free — on Anthropic pricing, cache reads cost 10% of the uncached rate. So savings vs. running with no caching at all = hit ratio × 90%: llmbuffer saves 76.7%, naive saves 59.5%.
• The llmbuffer-vs-naive comparison is the dollar figures: $0.016 / $0.028 ≈ 57%, i.e. ~43% cheaper. The gap comes from the miss turns: every time the dynamic context rotates (turns 4, 7, 10, 13) the naive approach takes a full cache miss — the changed system prompt invalidates the entire prefix. llmbuffer keeps the static system and long-lived history stable, so only the new suffix is uncached regardless of what the dynamic context does. The gap widens with longer conversations, larger system prompts, and more frequent dynamic-context changes — and again when compaction starts (naive trimming rewrites the prefix every turn once the limit is hit).

Run it yourself

# No API key needed:
uv run python -m llmbuffer.benchmark --provider simulated --compare --turns 15

# Live providers (needs API key):
uv run python -m llmbuffer.benchmark --provider anthropic --compare --turns 15
uv run python -m llmbuffer.benchmark --provider openai --compare --turns 15
uv run python -m llmbuffer.benchmark --provider gemini --compare --turns 15

# Ollama (local, needs server log access):
uv run python -m llmbuffer.benchmark --provider ollama \
    --ollama-log ~/.ollama/logs/server.log --compare

# JSON output:
uv run python -m llmbuffer.benchmark --provider anthropic --compare --format json

Development

# Clone and set up:
git clone https://github.com/scottpurdy/llmbuffer
cd llmbuffer
uv sync

# Run tests:
uv run pytest

# Run benchmark (simulated, no API key needed):
uv run python -m llmbuffer.benchmark --provider simulated --compare

The test suite includes explicit cache-stability tests asserting that the static system prompt and long-lived history are byte-identical across turns — verifying the cache prefix is never accidentally mutated.

License

MIT