I attended 10 technology conferences in Q2/2026 to learn how solution providers are accelerating AI adoption for their customers. I wrote about how AI agents are shaping the future of work and a series on what CIOs need to know. 

How AI agents tap into the organization’s intelligence was a common theme at these conferences. Some called it a knowledge graph, others a context layer, and a few a semantic layer. Several solution providers announced branded products for these capabilities, including Atlassian’s Teamwork Graph, SAP Knowledge Graph, and Snowflake semantic views. Others referred to the capabilities, such as using Appian’s Data Fabric as a context layer and Domo’s semantic layer.

Intelligent AI agents: A goal with many components

I have to confess that I found the jargon confusing – which of course meant it was time to research, solicit answers from experts, and simplify.

Irfan Khan, president and chief product officer of SAP Data & Analytics at SAP, told me, “Semantic layers, context, and knowledge graphs each play a distinct role in helping AI reason effectively across the enterprise.”

Here’s how Khan distinguishes between them.

• Context determines which information is relevant for a specific task or decision at a specific moment.
• A semantic layer gives data consistent business meaning.
• The knowledge graph then connects those meanings, relationships, APIs, metadata, and business entities into a structured network. It helps AI reason across the enterprise rather than simply retrieve data, producing responses and actions that are relevant, reliable, and grounded in business context.

The graphic depicts the architecture as I understand it. I’ve already covered AI orchestration in one article and am working on a research paper on AI orchestration platforms. I introduced FinAI, a cousin of FinOps as a discipline to avoid AI cost debt, and wrote about developing smarter AI agents with data fabrics. My article on AI governance and strategy paved the way for the AI Strategy and Governance Workshops I lead at StarCIO.

In today’s article, here’s how I unpack and de-jargonize what goes on in the brains of AI agents.

Start with high-quality, trusted data sources

Would you build a house on top of a shoddy foundation? To build the AI agent’s brain, define what AI-ready data is in your organization and build data products around reusable data assets.

“High-quality, well-governed data paired with clear security, privacy, and accountability frameworks is the foundation AI agents rely on to operate in the right business context,” says Cássio Christianini, head of AI go-to-market Americas at SAP. “These aren’t just risk management measures; they’re what enable organizations to scale AI with confidence.” 

CIOs should be paranoid about data risks, including misclassified data, leaked IP, and unobservable data pipelines. Chief data officers should develop their checklist around data management, governance, security, and privacy to rubber-stamp which sources are AI-ready, who can use them, and how they can be used.

Knowledge graphs are the connective tissue

Knowledge graphs are a form of graph database that encode relationships and their meanings. The confusing part is that knowledge graphs can appear as data sources, as a foundation layer connecting data sources, and as a platform for implementing the context layer. Some examples:

• As data sources, knowledge graphs can store complex taxonomies, categorizations, and other ontologies. For example, an HR platform’s underlying understanding of job titles, skills, and competencies may be stored as a knowledge graph.
• In the context of the diagram I presented, knowledge graphs are a data management layer that exposes relationships between data stored across various data sources. These knowledge graphs may store metadata only in their nodes and relationships, and link to the original data for complete information. Customer Data Platforms are one example.
• A knowledge graph in the context layer can be used to optimize how information is presented to AI agents. For example, the organizational knowledge graph may store everything related to customers and prospects, while the context layer may contain only information that marketing and sales AI agents can use.

“A context layer connects those relationships and is often the difference between simple information retrieval and making trustworthy decisions,” says Tony Grout, chief product and technology officer at M-Files. “A knowledge graph is one implementation of that context layer, mapping relationships between business entities and information in a structure that AI can navigate. Relationship context is an essential capability because it allows AI to reason over information that reflects how the organization actually works, not just what individual documents say.”

It’s important to understand that knowledge graphs are often implemented with graph databases. Matvey Ayre, founding engineer at Tiger Data, says that graph databases answer a different question than memory and semantic layers do. “A graph database’s job is pathfinding and link analysis: how two things connect, the shortest path, who’s in a fraud ring, and they’re excellent at that,” Ayre says.

What to look for when defining a semantic layer

From an architectural perspective, semantic layers can be independent layers as I’ve depicted them, or foundational components of knowledge graphs or the context layer. The architecture is less important – what semantic layers do and where organizations need them is what requires discovery and solutioning.

Some view semantic layers as extensions of a data catalog’s data definitions. Your knowledge graph may be designed using ambiguous or conflicting terminology from data sources. The semantic layer can provide sanctioned business terminology for presenting information in the context layer.

Another view on the semantic layer versus knowledge graph is, “Your AI agents need a deterministic semantic layer to translate business intent into accurate SQL. Knowledge graphs store relationships; semantic layers resolve them into business metrics.”

Barr Moses, co-founder and CEO of Monte Carlo, says that a semantic layer is only as trustworthy as the definitions behind it. “If two teams define ‘active customer’ differently, every agent built on top inherits that disagreement as a silent error. Look for a layer that’s version-controlled, observable, and tied to the underlying data’s lineage, not a glossary sitting next to the data,” says Moses.

The context layer: Optimized, governed, trusted information for AI agents

With raw knowledge and the organization’s semantics captured, we can now answer the key question: How is information presented to AI agents?

“The semantic layer provides a shared business vocabulary so every consumer, whether a dashboard or AI agent, interprets data consistently, says Animesh Kumar, co-founder and CEO at The Modern Data Company. “Once meaning is established, the next challenge is understanding what is relevant, and this is where a context layer comes in. It assembles runtime context from both structured and unstructured systems to provide the information needed for decision-making.”

Context layers require at least three key attributes:

• Optimized so that AI agents can retrieve relevant information quickly, easily, and inexpensively.
• Governed, so that there’s transparency, guardrails, observability, and monitoring on what information is available to which agents and how they can use it.
• Trusted, because of the investments made in making data sources AI-ready, developing knowledge graphs, and creating a semantic layer where needed.

 “A context layer turns a sea of metrics, logs, traces, and events into signal, and what agents need is a derived layer sitting above raw telemetry to enable fast, precise retrieval,” says Francois Martel, field CTO at NeuBird AI. “The context layer lets an agent traverse those tiers on demand instead of scanning everything, and two properties make it real: every investigation writes back what it learns, and it is exposed as tools that an agent calls.”

My diagram shows memory at the same level as the context layer. Memory is key to AI agents learning, processing feedback, and dismissing what’s less relevant. It deserves its own deep dive, and I will cover questions about the AI agents’ memory in separate articles.

Governance and security considerations

I’ve presented these layers from an architecture perspective, not an implementation one. Governance needs to be defined from day-0 and added incrementally as the organization engineers its “AI brain.”

Yes, I think an agile approach is needed to drive a focus on incremental outcomes, and DevOps is needed to bring feedback from operational disciplines.

“Governance has to start one level earlier than most teams assume, with the knowledge asset itself, supported by access and privacy controls such as role- and field-level permissions, encryption, masking of sensitive data, and keeping data within the enterprise boundary rather than extracting it for external training,” says Sanjay Koppikar, chief product officer and co-founder at EvoluteIQ. “At runtime, agents need least-privilege access, zero-trust identity, and hardened guardrails with centralized policy and monitoring to show not only what an agent did, but why.”

David Torgerson, VP of technology and security at Lucid Software, says the most critical security consideration isn’t just restricting access but rather ensuring clean documentation and visible workflows. “When developing context and semantic layers, you must treat them as a distinct technology layer with its own governance, controls, and cross-functional accountability. If you build these knowledge foundations on top of undocumented legacy processes, AI will simply amplify those hidden vulnerabilities at machine speed. Safely developing these assets requires starting with tightly scoped use cases and grounding your security controls in how modern collaboration actually happens,” Torgerson says.

Developing an easy but scalable approach to context

Smaller organizations may not need all the layers, especially if data sources are relatively clean, have simple relationships, and are developed using business-friendly terms. Even larger organizations can start with an MVA – a minimally viable architecture developed for specific agentic use cases.

“Start with one consequential decision and ask what a person or an agent would genuinely need to know to make it with confidence,” says Dan Onions, global SVP of data and AI at Quantexa. “Build that context once as a governed, reusable data product, refresh it continually, and reuse it everywhere a decision gets made. Scalability comes from reuse, not from scope.”

Well said! But not easily implemented. Start with AI agent use cases with high business value, establish the data’s AI readiness, iterate on AI governance, and then slowly engineer the brain’s knowledge, semantic, and context layers. Establish FinAI as you deploy more AI agents into production.