Originally published on Towards AI.
Over the past year, memory has become one of the most overused — and misunderstood — concepts in AI agent design.
But before I start, I want to add a few words, most of us building AI agents today didn’t start as “AI engineers”. We come from backend engineering, data engineering, or data science.
That background shapes how we think about systems: scalability, reliability, clear lifecycles, and predictable failure modes.
And when we bring LLMs and agents into production, we still care about the same things:
This article is written from that mindset, not “what sounds impressive in demos”, but what leads to a reasonable trade-off between AI capabilities, backend architecture, and long-term system health.
You hear phrases like long-term memory, short-term memory, context engineering, persistent agents, and stateful conversations everywhere.
But if you look closely at most real implementations, many teams either:
This article aims to cut through the hype and explain, in practical terms, how memory for AI agents actually works, which approaches exist today, and what trade-offs they come with.
If this piece gives you something practical you can take into your own system:
👏 leave 50 claps (yes, you can!) — Medium’s algorithm favors this, increasing visibility to others who then discover the article.
🔔 Follow me on Medium and LinkedIn for more deep dives into agentic systems, LLM architecture, and production-grade AI engineering.
Long-term memory is anything that persists across sessions, restarts, and disconnections (includes the agent’s past behaviors and thoughts that need to be retained and recalled over an extended period of time; this often leverages an external vector store accessible through fast and scalable retrieval to provide relevant information for the agent as needed).
Typical characteristics:
Common forms of LTM:
Think of long-term memory as durable knowledge, not working context.
Short-term memory (often called working memory or execution state, includes context information about the agent’s current situations; this is typically realized by in-context learning which means it is short and finite due to context window constraints) is:
In practice, what we call “short-term memory” in agents usually combines:
Short-term memory exists to reduce overhead and improve reasoning continuity, not to replace persistence.
The most widespread approach today is actually stateless.
For every user request:
history = db.load_last_messages(user_id, limit=20)
prompt = build_prompt(history, user_message)
response = llm(prompt)
This approach does not use short-term memory at all.
Each request is fully independent.
A more advanced approach introduces explicit short-term memory.
This is the model used by frameworks like LangGraph.
Conceptually:
class ChatState(TypedDict):
user_id: str
messages: list[dict]
SocketIO one of the most common and well-known framework for building chat based applications.
On connect
On each message
On disconnect
RAM usage grows with:
Requires:
Socket-based systems have edge cases:
This approach can be production-ready, but only if memory management is treated as a first-class concern.
Many implementations add context variables (for example, ContextVar in Python) to avoid passing state through every function.
This is useful — but limited.
Context variables:
They are an access pattern, not a memory strategy.
state through dozens of function callsstate = get_current_state()
state["messages"].append(new_message)
Context variables are a convenience layer, not a memory system.
A newer and increasingly popular approach is Memory as a Tool.
Before dismissing this approach as “too complex”, I would strongly recommend trying it at least once.
Even if you don’t end up using Memory as a Tool in production, it forces you to think differently about agent design:
In practice, this approach is one of the best ways to truly understand:
Many engineers are familiar with prompts and APIs, but far fewer have hands-on experience with ReAct-style loops or explicit reasoning-driven tool calls. Trying this pattern — even in a small prototype — helps close that gap.
I’ll link a few articles below that go deeper into ReAct, reasoning models, and tool-based agents for those who want to explore this further.
medium.com
medium.com
Instead of automatically injecting memory, you expose memory retrieval as a tool:
Tool: retrieve_chat_history(user_id, chat_id, offset=0, limit=10)
The agent decides:
This pattern assumes:
Without a ReAct-style agent:
This is not a drop-in replacement for traditional memory injection.
With modern multi-step reasoning models, this approach is becoming increasingly viable — and often superior for large-scale systems.
!!! Many people might immediately start talking about security risks and so on, and they will be absolutely right, but let’s figure it out.
Memory used as a tool does introduce security considerations, but not fundamentally different from any other tool.
Prompt injection can attempt to trigger a memory read or write, but it cannot access anything beyond what the memory tool explicitly allows. The real security boundary is not the prompt, but the tool implementation.
This is why memory must:
In practice, prompt injection cannot recall another user’s memory unless there is already a bug in isolation or authorization logic. The larger risk is memory poisoning (writing bad memory), which should be mitigated by validation and write policies inside the tool wrapper, but that’s an absolutely different topic, my folks. If you are interested in this topic, I can also write my observations, experience, and thoughts on this matter.
There is no single “correct” memory strategy.
Use hybrid memory, since it integrates both short-term memory and long-term memory to improve an agent’s ability for long-range reasoning and accumulation of experiences.
In practice, hybrid architectures work best:
Memory is not magic.
Most problems attributed to “lack of memory” are actually:
Before adding complexity, ask:
If you answer those questions honestly, the right architecture usually becomes obvious.
And that’s a wrap! If you’ve read this far, it probably means you found this article useful or insightful. If that’s the case, consider leaving a few claps or sharing it with your team, please. Thanks for reading! 🚀
Published via Towards AI
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。