Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

Netflix Headroom: How to Cut AI Agent Costs 10x in Production [2026]

Netflix Headroom is a context optimization layer for LLM applications that sits between your application code and your model API, pruning, caching, and routing context to dramatically reduce token costs.

I watched a team's token bill jump from $400/month to $12,000/month in six weeks. They hadn't added more users. They'd added AI agents. A 10-step agent loop doesn't cost 10x a single call. It costs closer to 50x, because each step re-reads the entire conversation history, tool outputs, and system instructions. Netflix built Headroom to fix exactly this, and Tejas Chopra, Engineer at Netflix, presented the tool at the Linux Foundation's Open Source Summit North America 2025 in Denver. The result they're claiming: up to 10x cost reduction on production AI workloads without sacrificing output quality.

This isn't a research paper or a toy demo. It's a production system from a company running ML at planet scale. And the patterns inside Headroom are ones any engineering team can steal today.

Why AI Agent Costs Spiral Out of Control

Before I get into what Headroom does, let's talk about why you need it.

LLM inference costs in agentic workflows scale non-linearly. That's the part most teams don't internalize until the invoice lands. Here's the math: if your agent takes 10 steps and you're appending tool outputs and conversation history at each turn, the context window grows roughly like a triangle. Step 1 sends maybe 2,000 tokens. Step 5 sends 12,000. Step 10 sends 25,000+. You're paying for all that accumulated context on every single call.

The majority of your token spend isn't generation. It's input. Most teams I've talked to find that 70-85% of their LLM bill comes from the input/context side. Your agent is re-reading the same system prompt, the same tool definitions, and most of the same conversation history on every turn. Pure waste.

I've seen teams running agentic AI loops on GPT-4-class models hit $50-100 per complex task. At that price point, you can't ship the feature. You either eat the cost, downgrade to a weaker model and accept worse outputs, or you get clever about context management. Netflix chose option three.

As copyleftdev argues on Dev.to, the root cause is simple: teams feed agents raw, unfiltered data. Every API response, every database query result, every intermediate tool output goes straight into the context window without any pre-processing. It's the LLM equivalent of piping cat output straight to your model and hoping for the best.

What Is Netflix Headroom and How Does It Work?

Headroom is an open-source context optimization layer that Netflix built internally and presented at Open Source Summit NA 2025. The key architectural insight: it operates as middleware. A proxy layer between your application and the LLM API. Your code sends the full context it would normally send, and Headroom intercepts it, optimizes it, and forwards a leaner version to the model.

This is an architectural pattern, not a model-level change. You don't retrain anything. You don't need Netflix's infrastructure budget. You need to insert a layer that does three things:

1. Context pruning — Stripping irrelevant or redundant content from the context window before each LLM call
2. Prompt/KV caching — Reusing previously computed attention states for repeated prefixes
3. Tiered model routing — Sending simple subtasks to cheaper models and reserving frontier models for complex reasoning

Tejas Chopra presented this as a unified system rather than three separate optimizations, and that's the real insight. I've seen plenty of teams apply one of these techniques in isolation and call it a day. Headroom applies all three in a coordinated pipeline. The compounding effect is what gets you to 10x.

Here's the official talk from Open Source Summit NA 2025:

[YOUTUBE:UOWSHg18cL0|Headroom: A Context Optimization Layer for LLM Applications - Tejas Chopra, Netflix, Inc.]

Think of it like a CDN for your LLM calls. A CDN doesn't change the content your server produces. It makes delivering that content cheaper by caching, compressing, and routing intelligently. Headroom does the same thing for context windows.

Context Pruning: The Highest-Leverage Cost Reduction Lever

Context pruning is the single biggest bang-for-your-buck optimization in any agent framework pipeline. The idea is straightforward: before sending context to the model, analyze what's actually relevant to the current step and strip everything else.

In a typical 10-step agent workflow, by step 7 you're carrying forward tool outputs from step 2 that have zero relevance to the current decision. You're including the full text of API responses when the model only needed one field. You're sending the complete system prompt every time when 80% of it is static boilerplate that the model doesn't need to re-read.

Headroom's pruning works at multiple levels:

• Conversation-level pruning: Summarizing or dropping older conversation turns that are no longer relevant to the current task
• Tool output pruning: Extracting only the fields the model actually needs from structured API responses, instead of dumping the entire JSON blob
• Instruction pruning: Conditionally including system prompt sections based on what the current step requires

In my experience building agent systems, conversation-level pruning alone can cut context size by 40-60% on multi-step workflows. The trick is knowing what to cut. A naive approach — just truncate after N tokens — destroys performance. A smart approach uses relevance scoring, essentially asking "does this piece of context help the model answer the current question?" That's where Headroom's optimization logic lives.

The best context window is the smallest one that still produces the right answer. Everything else is wasted money.

This connects directly to the broader prompt engineering discipline. I've seen engineers obsess over prompt wording while completely ignoring prompt size. They're optimizing the wrong variable. A 2,000-token prompt that contains exactly the right context will outperform a 20,000-token prompt padded with irrelevant history. And cost 10x less.

Prompt Caching: Stop Paying for the Same Tokens Twice

Prompt caching is the optimization most teams know about but few implement well. The core idea: if the first 3,000 tokens of your prompt are identical across calls (system prompt + tool definitions + static instructions), you shouldn't pay full price for those tokens every time.

All three major providers now support some form of prompt caching. Anthropic's Claude, Google's Gemini, and OpenAI all offer mechanisms to reuse previously computed attention states for stable prefixes. The savings are real. Anthropic's documentation shows up to 90% cost reduction on cached input tokens and significantly lower latency.

But here's what most teams get wrong: prompt caching only works if you structure your prompts for cacheability. That means:

• Put stable content first. System prompt, tool definitions, and static instructions go at the top of your context window. Dynamic content (conversation history, current task) goes last.
• Minimize prefix variation. If you're randomly reordering tool definitions or injecting timestamps into your system prompt, you're busting the cache on every call.
• Batch similar requests. Agent steps that use the same tool set and system prompt should be routed through the same cached prefix.

Headroom handles this automatically. It restructures your context window to maximize cache hit rates, moving stable content to the prefix position and grouping dynamic content at the end. After shipping several agent-based features, I learned that this structural optimization — literally just reordering your prompt — can save 30-50% on input costs even without any content pruning.

If you're running LLM workloads at scale and sending the same system prompt on every call without cached prefixes, you're lighting money on fire. Full stop.

Tiered Model Routing: Use the Right Model for Each Step

This is the optimization that feels obvious in hindsight but almost nobody does well. Not every step in an agent workflow requires GPT-4-class reasoning. Some steps are classification tasks. Some are simple data extraction. Some are formatting. Sending all of these to your most expensive model is like taking a Ferrari to buy milk.

Tiered model routing means having a router that analyzes each agent step and directs it to the cheapest model capable of handling it:

Look at the cost column. A step that costs $0.03 on a frontier model might cost $0.001 on a mid-tier model and $0.0001 on a local LLM. If 60-70% of your agent steps can be handled by cheaper models — and in my experience, that's typical — you've just cut your total cost by 5-8x on routing alone.

I've seen this pattern work particularly well with agent orchestration frameworks that already decompose tasks into discrete steps. The router examines the step's requirements (does it need tool calling? long-context reasoning? or is it a simple yes/no decision?) and selects the appropriate tier. Headroom includes this routing logic as a core component.

The key insight from Netflix's approach: routing decisions should be data-driven, not hardcoded. You start by sending everything to the frontier model, log the results, then progressively shift simpler steps to cheaper models while monitoring quality. If quality stays above your threshold, keep shifting. It's A/B testing for model selection.

How Any Team Can Apply These Patterns Without Netflix's Budget

Here's the thing nobody's saying about Headroom: the individual techniques aren't novel. Context pruning, prompt caching, and model routing have all been discussed in the LLMOps community for over a year. What Netflix did is package them into a coherent, production-tested system. But you don't need their system to use their playbook.

Here's how I'd implement this incrementally, based on how I've actually rolled out similar optimizations:

Week 1: Instrument your context windows. Before you optimize anything, measure. Log the token count at each step of your agent loops. Calculate what percentage is system prompt, what's conversation history, what's tool outputs. I promise you'll be shocked at how much redundancy you're carrying. When I first did this on a project, I found that 62% of tokens at step 8 were from tool outputs the model never referenced again.

Week 2: Implement prompt caching. Lowest effort, highest impact. Restructure your prompts so stable content comes first, enable your provider's caching feature, and measure the savings. If you're on Claude, Gemini, or OpenAI, this is a configuration change, not an architecture change.

Week 3: Add basic context pruning. Start with the easy wins: truncate tool outputs to only the fields your model needs, summarize or drop conversation turns older than N steps, conditionally include system prompt sections. Even a crude implementation will cut 30-40% of your token spend.

Week 4: Prototype model routing. Identify 2-3 step types in your agent workflow that clearly don't need frontier-model reasoning. Route those to a cheaper model. Measure quality. Expand from there.

This four-week playbook can realistically get you a 5-8x cost reduction. The remaining push to 10x requires more sophisticated pruning (relevance scoring, semantic deduplication) and fine-tuned routing logic. That's where Headroom's codebase becomes genuinely useful as a reference architecture.

For teams already using frameworks like LangChain or CrewAI, many of these optimizations can be implemented as middleware layers or callbacks without restructuring your entire agent pipeline.

Why Netflix Open-Sourcing This Matters Right Now

The LLMOps tooling ecosystem is fragile. Case in point: TensorZero, an open-source inference optimization tool, was archived shortly after raising a $7.3M seed round. That's $7.3M that evaporated. When you build your cost optimization stack on tools from early-stage startups, you accept the risk that the tool disappears out from under you.

Netflix doesn't have that problem. They're not a VC-funded startup that might pivot or fold. They're a $250B+ company with one of the most respected engineering organizations in tech. When Netflix open-sources an internal tool, it comes with an implicit signal: this thing works at production scale, because Netflix was running it at production scale first.

Netflix Research has published extensively on their ML infrastructure. They run recommendation systems, content understanding models, and personalization engines across 260+ million subscribers. Their LLM cost sensitivity is real. They're not optimizing for fun. Their token bills are material to their bottom line.

This also tells you something about where the industry is heading. Netflix didn't build a feature flag or a quick hack. They built a proper middleware service for context optimization. That tells me production AI cost management is now a first-class engineering discipline. Not an afterthought you bolt on after launch. Part of the architecture from day one.

The Broader LLMOps Stack: What Else Matters for Cost

Headroom addresses the context side of the cost equation, but it's one lever among several. Having built production AI systems where cost was a hard constraint, here are the complementary techniques I've found actually move the needle:

Speculative decoding uses a small, fast draft model to generate candidate tokens, then verifies them with the larger model in a single forward pass. 2-3x speedups on generation-heavy workloads with zero quality loss, because the final output is mathematically identical to what the large model would have produced.

RAG over stuffing. Stop cramming your entire knowledge base into the context window. Use vector embeddings and semantic search to retrieve only the relevant chunks. This is context pruning applied to external knowledge, and it's one of the most commonly missed optimizations I see.

Structured outputs over free-form generation. When you need the model to return data in a specific format, use function calling or structured output modes. These constrain the generation space, reducing output tokens and eliminating the follow-up parsing call you'd otherwise need (which often means another LLM call, compounding costs).

Batch processing where latency allows. OpenAI's batch API offers 50% cost reduction for non-real-time workloads. If your agent has steps that don't need sub-second responses — background analysis, document processing, bulk classification — batch them. I've shipped features where 40% of agent steps could run async, and batching those alone cut the bill meaningfully.

The teams I've seen run AI most cost-effectively treat their LLM calls the way good engineers treat database queries: every call should be justified, every context window should be as lean as possible, and you should always be asking "do I actually need the frontier model for this?"

What's Next for LLM Cost Optimization

Headroom represents the current state of the art, but this space is moving fast. Here's where I think this goes:

Context windows will keep growing, but filling them is still a bad idea. Just because Gemini offers a 2M token window doesn't mean sending 2M tokens is smart. Larger contexts mean higher costs and slower inference. The teams that win will use large windows selectively and small windows by default. Having run benchmarks on this, I can tell you: a well-pruned 8K context almost always beats a lazy 128K context, both on cost and on output quality.

Model routing will become automatic. Right now, routing decisions require manual configuration or custom logic. Within a year, I expect routing layers that automatically learn which steps need frontier models based on quality feedback loops. Headroom's architecture is well-positioned for this.

The "context optimization layer" will become as standard as a CDN. Just as nobody serves web traffic without a CDN today, nobody will run production AI agents without a context optimization layer by 2027. It'll be considered negligent. The same way running production databases without connection pooling is considered negligent today.

The local inference movement — running local LLM models on your own hardware — is the logical extreme of this cost optimization trend. If you can run the cheap tiers of your model routing stack on local hardware (an M4 Max Mac, a consumer GPU), you eliminate API costs entirely for those tiers. I've written about running local agentic AI on Mac specifically because this combination of local small models + cloud frontier models is the most cost-efficient architecture I've found.

The teams that figure out context optimization now will have a structural cost advantage for years. Everyone else will keep watching their token bills climb and wondering why their AI features can't turn a profit. Netflix just handed you the playbook. Use it.

FAQ

What is Netflix Headroom?

Netflix Headroom is an open-source context optimization layer for LLM applications. It sits between your application and the LLM API as middleware, intercepting and optimizing context windows before they're sent to the model. It uses context pruning, prompt caching, and tiered model routing to reduce inference costs by up to 10x.

How does context pruning reduce LLM costs?

Context pruning removes irrelevant or redundant content from the context window before each LLM call. In multi-step agent workflows, context grows with every turn as tool outputs and conversation history accumulate. Pruning strips out information the model doesn't need for the current step, which directly reduces the number of input tokens you pay for.

Can small teams use Netflix's LLM cost optimization patterns?

Absolutely. The individual techniques — prompt caching, context pruning, and model routing — don't require Netflix-scale infrastructure. You can implement prompt caching by restructuring your prompts and enabling your provider's caching feature. Context pruning can start with simple rules like truncating tool outputs. Model routing can begin by manually directing simple tasks to cheaper models.

What is tiered model routing for AI agents?

Tiered model routing directs each step of an agent workflow to the cheapest model capable of handling it. Complex reasoning goes to frontier models like GPT-4.1, while simple tasks like classification or formatting go to cheaper models like GPT-4o Mini or local models. Since most agent steps don't require frontier-class reasoning, this can reduce costs by 5-8x.

Why is prompt caching important for production AI?

Prompt caching reuses previously computed attention states for repeated prefixes in your LLM calls. Since most agent calls share the same system prompt and tool definitions, caching can reduce input token costs by up to 90% for those stable portions. All major providers — Anthropic, Google, and OpenAI — now support some form of prompt caching.

How does Headroom compare to TensorZero?

TensorZero was an open-source LLMOps tool that was archived after raising a $7.3M seed round, highlighting the instability of startup-built LLMOps tooling. Headroom comes from Netflix, a $250B+ company with no risk of pivoting away from the project. As a production-proven tool from a major engineering organization, it offers a more stable foundation for building cost optimization infrastructure.

---

Originally published on kunalganglani.com