How to measure prompt cache hit ratio in your Hermes Agent

If you run Hermes Agent for any real workload, the single biggest knob on your bill is the prompt cache. The Hermes developer notes say it plainly: cache-breaking forces dramatically higher costs. The fix is simple. Do not break the cache. The hard part is knowing whether your cache is actually working.

Hermes hides the model call behind a clean tool-and-skill loop, which is great for everything except observability. A cold prompt and a warm one look the same from the agent side. You only see the difference in the provider's invoice. This post shows how to attach a cache meter to your Hermes Agent and read the hit ratio in real time.

What you can measure

Anthropic, OpenAI, and Bedrock all return cache metadata in the response payload. The fields are named differently per vendor. If you wrap the model client before Hermes hands it a request, you can compute:

per-call cache hit ratio
cumulative ratio for the session
USD saved versus an uncached baseline
which prefix is performing well and which is not

Hermes does not expose this out of the box. You add it in the provider plugin.

The library

I built cachebench for exactly this problem. It is a small Python library that you wrap around the SDK call your provider plugin already uses. Every request and response passes through it, the cache metadata gets parsed, and you get a running tally.

pip install cachebench

It reads cache fields for Anthropic, OpenAI, and Bedrock today. About a hundred lines of vendor-neutral code, no provider lock-in.

Plugging it into Hermes

Hermes model-provider plugins use lazy discovery via providers.register_provider(ProviderProfile(...)). The cleanest place to add a meter is wherever your provider builds its client. You wrap the messages.create method (Anthropic) or chat.completions.create (OpenAI) with CacheTracker.wrap, and you are done.

from anthropic import Anthropic
from cachebench import CacheTracker, Provider

client = Anthropic()
tracker = CacheTracker(provider=Provider.ANTHROPIC)

# Replace the bare method with the tracked one once at startup.
client.messages.create = tracker.wrap(client.messages.create)

Hand the wrapped client to your provider profile and Hermes will use it for every model call. The wrapper is sync-and-async aware, so the same line works whether your profile uses the sync or async client.

Reading the meter

CacheTracker keeps the rolling counts in memory. The simplest report is tracker.aggregate():

print(tracker.aggregate())
# {'calls': 41, 'hit_ratio': 0.805, 'tokens_read_from_cache': 184320,
#  'tokens_written_to_cache': 14210, 'cost_usd': 0.94, 'cost_saved_usd': 1.21}

A good place to call this is the on_session_end hook. That keeps your turn-by-turn experience clean and gives you a single summary line when the agent finishes.

For a per-prefix breakdown:

for prefix_id, stats in tracker.by_prefix().items():
    print(prefix_id, f"{stats['hit_ratio']:.2f}", stats['calls'])

prefix_id is a stable hash of the cacheable portion of the request: system message, tools, and all prior turns. When the ratio drops you can see which prefix changed, then go fix it.

What kills your hit ratio

Three common patterns explain most of the bad runs I have seen.

A system message that includes the current time. One timestamp at the top of the prompt invalidates everything below it. Move time-sensitive context into a tool call instead.
A dynamic tool list. If your toolset changes between turns, the cached prefix is gone. Settle on a stable toolset for the session and re-register on demand.
Skill loading that drops or reorders skills mid-conversation. Hermes defers some of this for you, but custom plugins can still break it. Use --now only when you really mean it.

tracker.by_prefix() will surface the offending prefix the moment it appears.

Catching regressions

The richer use is automated. CacheTracker accepts a miss_alert_threshold and an on_miss_alert callback. Wire it to your logging once and any plugin or skill that drops the ratio below the floor produces a structured warning.

def alert(metrics):
    print(f"[CACHE-MISS] prefix={metrics.prefix_id} "
          f"ratio={metrics.hit_ratio:.2f} cost=${metrics.cost_usd(tracker.pricing):.4f}")

tracker = CacheTracker(
    provider=Provider.ANTHROPIC,
    miss_alert_threshold=0.6,
    on_miss_alert=alert,
)

You can also opt into miss-aware retry. When CachePolicy.miss_aware() is enabled and a call returns a cache-eligible prefix but a zero hit ratio (a silent provider miss), CacheTracker waits two seconds and retries once. Useful right after a system message change, when the cache is still propagating.

from cachebench import CachePolicy
tracker = CacheTracker(provider=Provider.ANTHROPIC, policy=CachePolicy.miss_aware())

What to do with the numbers

Once the meter is running, two small habits pay back fast.

Check tracker.aggregate() at session end. If the hit ratio is under fifty percent, something is invalidating the cache. Run tracker.by_prefix() and look for the prefix that ballooned recently.
Add a CI test that runs a short scripted conversation and asserts the hit ratio is above a floor. If a future plugin starts breaking the cache, the test catches it before your bill does.

Closing

You do not need a vendor dashboard to know how well your Hermes Agent is caching. About fifteen lines of wrapper code and a small Python library are enough. The biggest cost optimization for an agent that talks a lot is the one that already shipped, you just have to confirm it is working.

Repo: https://github.com/MukundaKatta/cachebench
PyPI: https://pypi.org/project/cachebench/
Hermes Agent: https://hermes-agent.nousresearch.com/

推荐订阅源

DEV Community