Databricks declares the end of pipelines with a unified platform for operational and analytical data

Databricks Inc. is using its Data + AI Summit today in San Francisco to unveil a new data architecture designed to eliminate one of enterprise computing’s oldest bottlenecks: the separation between transactional databases and analytical systems.

The company is also introducing a real-time analytics engine that it says removes the need for separate serving infrastructure while delivering millisecond response times.

The new architecture, called Lake Transactional/Analytical Processing, unifies operational and analytical workloads on a single copy of data stored in a data lake. Databricks said the approach enables applications, analytics systems and artificial intelligence agents to access the same data without the change data capture pipelines, extract/transform/load processes and replicated databases that have traditionally connected operational and analytical environments.

The company said conventional architectures are ill-suited to the emerging world of AI agents, which continuously read, analyze and act upon data in near-real time.

“You’ve got more code being written than ever before, which means you’ve got lots more applications,” said Shanku Niyogi, Databricks’ vice president of product management. “Those applications are powered agents that need to both reason and act on data more quickly than humans can manage. So the data stack becomes the bottleneck.”

Enterprises have long maintained separate systems for transaction and analytical processing. Operational applications typically write data to transactional databases, while analytical systems consume copies of that data through ETL and change data capture pipelines, which monitor databases for modifications and propagate them to downstream destinations.

Databricks argues that this architecture introduces latency, complexity and governance challenges that become more pronounced as AI-driven applications proliferate.

Niyogi said many organizations are struggling to manage the growing number of pipelines required to synchronize operational and analytical systems.

“We’ve been joking that CDC is ‘continuous data corruption,’” he said. “Every time something changes, you’ve got a new pipeline.” He cited a large banking customer that now maintains “hundreds of thousands of Postgres databases, each with CDC pipelines bringing data back to the lake.”

Built on Lakebase

LTAP builds upon Databricks’ Lakebase database platform, introduced last year. Lakebase separates database computing from storage. LTAP writes transactional data directly into open columnar formats such as Delta Lake and Apache Iceberg while maintaining PostgreSQL compatibility for applications.

Niyogi said the architecture allows transactional applications to continue operating with native PostgreSQL performance while making data instantly available for analytics and machine learning workloads.

“You’re getting Postgres performance and Postgres semantics,” he said, “but underneath the covers, when we write storage out to the lake, we’re instantly writing that to columnar formats, which means any analytics engine now has access to all of your operational data. There are no pipelines and no latency.”

Columnar storage is a database architecture that stores data sequentially by column instead of by row to speed up analytical queries.

Databricks said LTAP relies on open formats and it plans to open-source technology that enables PostgreSQL data to be stored in the Apache Parquet format while preserving compatibility.

New analytics engine

The company today is also introducing Lakehouse//RT, an analytics engine that it said brings real-time query performance directly to lakehouse environments. Traditionally, organizations seeking speedy access to analytical data have had to deploy specialized serving systems, caches or real-time databases alongside their data lakes.

Lakehouse//RT is powered by a new execution engine called Reyden that Databricks claims can deliver response times as low as 10 milliseconds for smaller workloads and under 100 milliseconds for larger workloads while supporting tens of thousands of concurrent users and agents. The company said customers have reported up to 16 times better performance than existing real-time serving architectures.

Niyogi described the product as a major evolution of the lakehouse concept. “With Lakehouse RT, we can actually serve data now directly out of the warehouse to tens of thousands of concurrent users with very low latency,” he said.

The company sees both announcements as foundational technologies for AI-driven enterprises, where agents will increasingly execute business processes and make operational decisions.

“Agents need the best data,” Niyogi said. “If they’re getting stale or wrong data, they act poorly.” Traditional architectures featuring separate transactional systems, analytical systems and serving layers “are just not a platform that you can put millions of agents on,” he said.

LTAP is available through as an upgrade for Lakebase customers, while Lakehouse//RT is entering beta test. Databricks said existing Lakehouse customers can adopt Lakehouse//RT as a drop-in replacement for current warehouse deployments and will receive access through their existing subscriptions, with promotional pricing planned during the first year.

Image: Pixabay