In the previous parts of this series, we built a Unified Model Interface, added Tools, and created Autonomous Agents. But as your AI applications move from “cool demo” to “production tool,” you’ll hit a wall: The Context Window.
LLMs have limits. If you feed an agent thousands of lines of logs, it becomes slow, expensive, and eventually, it crashes because it can’t “remember” any more. This is where Middleware saves the day.
What is Middleware in LangChain?
Think of Middleware as a protective layer that sits between your User and your LLM “Brain.” It intercepts messages before the model sees them and after the model responds.
Middleware allows you to:
- Summarize: Compress long histories so you never hit token limits.
- Protect: Detect and redact PII (Personally Identifiable Information).
- Optimize: Add retries for failed tool calls or fall back to cheaper models.
- Guardrail: Prevent infinite loops or excessive costs.
1. Solving the “Memory Problem” with Summarization
The most common use case for middleware is preventing “Context Overflow.” If a user has a long debugging session with your agent, the history grows exponentially.
Instead of just deleting old messages (losing context), we use SummarizationMiddleware. This tool watches the conversation and, when a threshold is met, uses a second "Summarizer LLM" to compress the old history into a concise paragraph while keeping the most recent messages untouched.
from langchain.agents.middleware import SummarizationMiddleware# Configure the compression logic
log_compressor = SummarizationMiddleware(
model="ollama:phi4-mini", # A fast, cheap model for summarizing
trigger=("messages", 10), # Kick in after 10 messages
keep=("messages", 3), # Always keep the last 3 messages raw
)
2. Implementing the “Log Compressor”
Let’s see how this works in a real SRE (Site Reliability Engineering) scenario. Imagine an agent analyzing a massive stack trace.
from langchain.agents import create_agent# Pass the middleware during agent creation
agent = create_agent(
model=main_model,
tools=[analyze_stack_trace],
middleware=[log_compressor] # The secret sauce
)
By adding this one line, your agent becomes infinitely more durable. Even if the conversation goes on for 100 turns, the middleware will continuously “crunch” the oldest parts of the chat, providing the model with a “The story so far…” summary instead of 100 individual messages.
3. Why This Is Better Than “Windowing”
Many developers use a simple “Window” approach, where they only send the last 5 messages to the LLM. The problem? The model “forgets” why the conversation started.
Summarization Middleware is smarter:
- Context Preservation: The model still knows the intent of the first message because it’s in the summary.
- Token Efficiency: You only pay for a few hundred tokens of summary instead of thousands of tokens of raw chat history.
- Developer Experience: It happens automatically “under the hood.” You just invoke the agent as usual.
Conclusion: Engineering for Longevity
Building an AI agent that works once is easy. Building an agent that can handle a 2-hour debugging session without getting confused or crashing is hard.
Middleware transforms your agents from fragile scripts into production-ready software. By delegating tasks like summarization, PII detection, and error handling to the middleware layer, you allow your “Main Model” to focus on what it does best: solving the user’s problem.
💬 What do you think?
Drop your thoughts, questions, or suggestions in the comments below!
Check out my YouTube channel for more exciting content! [YouTube Channel Link — Harsha Selvi]
Disclaimer: This text has been rephrased using AI tools, and some parts are derived from various sources to provide a comprehensive overview.
#AI #LangChain #SoftwareEngineering #Python #LLM #AIEngineering #MachineLearning