The AGI moment? Databricks’ new releases zero in on support and deployment of AI agents

Major enterprise platform companies are racing to build tools for a new class of users: artificial intelligence agents.

One example of this was apparent today with the latest releases from Databricks Inc. The company unveiled a new architecture – Lake Transactional/Analytical Processing – that enables AI agents to access operational and analytics workloads on a primary copy of data that resides in a data lake.

By placing this data in the same open format, Databricks believes that agents will have the capability to observe and reason across a multitude of production databases within an enterprise and take action accordingly. It’s an important milestone in the realization of artificial general intelligence or AGI, the ability of AI to match or exceed human capabilities, according to Databricks co-founder and Chief Executive Ali Ghodsi.

“We believe that AGI is already here,” Ghodsi (pictured) said during his keynote remarks at the Data + AI Summit in San Francisco today. “AI does not have an intelligence problem right now. It’s plenty smart. The problem is that AGI is not completely permeating our organizations. The question is: ‘How do we enable this at work?’”

Clearing a path for autonomous employees

Ghodsi’s belief in AGI’s arrival is being constrained by a set of factors encompassing context, cost and control. Databricks wants to build a platform that can remove these obstacles while serving a new class of autonomous employees that can launch numerous versions of an application at the same time while creating and discarding entire software environments in minutes.

Databricks’ launch of its real-time Lakehouse is powered by Reyden, a new compute engine designed to deliver millisecond query latency for tens of thousands of concurrent users and agents. The name originated from “Reynold’s Dream Engine,” a nod to the role of Databricks’ co-founder Reynold Xin in the new release’s creation.

Xin appeared during the keynote session to demonstrate Reyden’s features, which included consistent low-latency response times when thousands of AI agents hit the same query concurrently.

“None of the other existing systems can do that,” Xin told the conference gathering. “It is probably the single largest introduction we have done since the launch of Lakehouse.”

Enhancements for Genie AI

Databricks also addressed the challenge of agentic context with the introduction of a set of enhancements for its Genie AI platform. Today’s launch of Genie One is designed to help business teams automate work that is grounded in real business data.

In a session with the media following his keynote, Ghodsi described the differentiators that Databricks has built into its new agentic offering. “Genie One computes whereas other agents recite,” Ghodsi explained. “That’s a unique advantage that it has.”

Databricks is now powering its suite of AI co-workers with Genie Ontology, a live context layer that continuously learns from internal and external business data. Ghodsi likened Genie Ontology to page rank algorithms used by Google Search to identify the most relevant information.

“We’re doing the same thing for the enterprise,” Ghodsi said. “Genie Ontology searches behind the scenes. We think this is the missing puzzle piece for agents.”

The company focused a number of its new products on ways to reduce the cost for enterprises to implement artificial intelligence. A key element in this approach is Unity AI Gateway, Databricks’ governance solution for delivering security controls, cost management and agent monitoring for enterprise AI.

Cost has become a significant issue as AI token usage has skyrocketed in many organizations, Ghodsi noted. “Every organization is super worried about costs going through the roof,” he said. “It’s the No. 1 question that we get asked.”

Another area that worries organizations in the rapidly expanding AI era is security, and Databricks has continued to develop new enterprise offerings in this area. In March, the company announced Lakewatch, an agentic security information and event management or SIEM solution to deploy defensive security agents and automate threat detection. This was followed by the news today that Databricks will acquire Panther Labs Inc., an AI security operations center platform whose clients include leading model provider Anthropic PBC.

Channeling signals into productivity

Today’s announcements from Databricks, and those from its competitor Snowflake Inc. earlier this month, highlight the emergence of agent-based systems of engagement such as Genie where enterprise work will actually get accomplished. Yet, as SiliconANGLE’s research analysts have recently noted, the back end to drive this new model will require systems of intelligence, where agents learn from enterprise data and channels signals into real organizational productivity.

This embodies the architecture that Databricks is seeking to build, an enterprise platform that can effectively link the place where work gets done with the place where enterprise intelligence is built. It’s wrapped into Databricks’ vision of AGI, a challenge to equip AI agents with key enterprise data that has not been an easy one to resolve.

“Getting this context into AI is harder than one can imagine,” Ghodsi admitted. “This is the problem we’ve been focused on at Databricks.”

While Databricks is clearly focused on building the agentic infrastructure of the future, its belief that the AGI moment is upon us may be a matter of interpretation. In an appearance at the summit today, OpenAI Group PBC president and co-founder Greg Brockman reminded attendees that developers remain very much a part of the picture and the opportunities are significant.

“It’s almost like AGI is a feeling, not a defined thing,” Brockman told the gathering. “It’s never been a better time to be a builder. Creativity is being unleashed. Anything you want to do, you can build it.”

Photos: Robert Hof/SiliconANGLE