​

getty

Despite $30 billion to $40 billion spent on enterprise generative AI, only 5% of custom solutions have reached production with sustained value, according to MIT Media Lab's Project NANDA report, "The GenAI Divide." McKinsey found that 88% of organizations have deployed AI, yet only 1% of leaders describe their companies as "AI mature."​ That is not a rounding error. It is a structural failure—and in my experience building AI-native products, the root cause is almost always the same: The systems have no memory.

Every foundation model available today is stateless. It forgets everything the moment a session ends. Users must re-explain their role, their project context, their preferences, their constraints from scratch every time they interact. The AI does not compound. It resets. McKinsey's research shows knowledge workers already spend approximately 19% of their workweek searching for and regathering information. Layering stateless AI on top of that fragmented workflow does not solve the problem—it doubles it.

The MIT report's own conclusion reinforces this: "Most GenAI systems do not retain feedback, adapt to context or improve over time." BCG has quantified the downstream effect. Seventy-four percent of companies cannot move AI beyond proof of concept.

I call this the Memory-Value Gap: the distance between what an AI system could deliver with accumulated context and what it actually delivers starting cold each session. Closing the Memory-Value Gap is the highest-leverage investment in AI product strategy today. Here is how product and engineering leaders can start.

For product leaders, the shift requires treating memory as a design primitive rather than a feature to add later. When every competitor has access to the same models, the differentiator becomes what the system remembers about your organization. In my experience, this creates three compounding advantages.

• First, user retention improves structurally. Tribe AI's research on context-aware memory architectures found that systems retaining user preferences across sessions achieve 40% to 70% higher retention rates.

• Second, onboarding compresses. Brandon Hall Group research shows new enterprise hires take six to 12 months to reach full productivity. Early adopters of organizational memory report 25% to 40% reductions in that timeline because new team members get instant access to the reasoning behind past decisions—not just the documents.

• Third, the knowledge base deepens with every interaction, creating a defensible asset that competitors cannot replicate by switching to a better model.

The practical framework I use with product teams breaks memory into four layers, each requiring different architectural decisions. Conversation memory is the active context window—ephemeral, cleared on session end. Session memory spans a single task and needs explicit life cycle management so it does not pollute future sessions. User memory stores individual preferences and carries the heaviest compliance burden. Organizational memory captures institutional knowledge such as decision rationale, failed approaches, domain terminology. This is the hardest to build because it lives in scattered tools such as Slack, Jira and meeting recordings and not in any single database.

Most teams only implement the first layer. The organizations in that 5% achieving real ROI have invested in all four.

For engineering leaders, the most common mistake I see is treating a single vector database as a complete memory solution. Embedding every conversation turn and running cosine similarity retrieval at inference time breaks down predictably: Recall precision drops as the corpus grows, token costs spike when retrieval consumes large portions of the context window, and latency compounds with each retrieval hop.

The correct approach uses heterogeneous storage, or structured databases for deterministic facts like user preferences and policies, vector or graph stores for fuzzy semantic knowledge like past decisions and institutional context. Mixing everything into one embedding space forces your highest-value information to compete with noise for retrieval slots.

Two recent developments illustrate where production-grade memory architecture is heading. Anthropic's Managed Agents platform, launched in April 2026, virtualizes agent components into stable abstractions that outlast any particular model. When models improve, the memory layer does not need to be rebuilt. Their engineering team noted that agent harnesses "encode assumptions about what the model can't do" and those assumptions go stale as models get better. Separately, Anthropic's Memory Tool API enables agents to persist knowledge across sessions through a developer-controlled file system, with internal evaluations showing an 84% reduction in token usage on extended workflows.

These are not features exclusive to one vendor. The pattern: Separating memory from model logic, designing memory scopes as isolation boundaries, running consolidation offline and persisting what did not work alongside what did, applies regardless of which platform you choose.

The enterprise AI market stands at roughly $115 billion in 2026 and is projected to reach $273 billion by 2031. The gap between the 5% succeeding and the 95% stalling will not close with better models. It will close when product and engineering teams treat memory as the most carefully architected component in their stack—not an afterthought. The models will keep improving. That is table stakes. Memory is what compounds.​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?