Unity Catalog was designed for the open lakehouse. Previously, data teams were stuck in silos, often forced to duplicate data across platforms just to use the tools they wanted. Every new platform or tool meant copying datasets, rebuilding access policies from scratch, and keeping everything in sync. The result was increased costs from redundant storage, policies that drifted out of sync, and fragmented data access and discovery.
When we open sourced Unity Catalog and launched Open APIs, we broke down the silos that previously kept customers locked-in. Enterprises could finally keep one copy of data, use any compute engine, and govern everything from one place. The UC ecosystem has thrived since. Today, thousands of customers use Unity Catalog to govern and access Delta Lake and Apache Iceberg tables, with dozens of integrations in the growing Unity Catalog ecosystem — from Apache Spark and Trino to DuckDB and Confluent Tableflow.
UC managed tables are where openness meets performance. These advanced tables use Predictive Optimization and Liquid Clustering to automatically tune data layouts, run compaction and vacuuming, and keep statistics fresh — delivering up to 20× faster queries and 50% lower storage costs, while staying fully accessible through open APIs.
Now in Beta, external engines, such as Apache Spark, Flink, and DuckDB, can create and write to UC managed Delta tables with centralized governance and automatic optimizations.
With the Beta, external engines can:
Because every operation flows through UC managed tables built on catalog commits, you get serialized commits that prevent log corruption and complete auditability of every read and write. Predictive Optimization continues to run seamlessly, even on tables accessed by external engines. Catalog commits also lay the groundwork for features like multi-statement, multi-table transactions that require a centralized commit coordinator.
The thriving UC ecosystem is continuing to grow as engines expand support for external access to managed tables. Delta Kernel — the open source Java and Rust library for reading, writing, and committing to Delta tables — abstracts the low-level protocol details so connector developers can focus on UC integration, not Delta implementation. Apache Spark, Delta Flink, and DuckDB have all leveraged Delta Kernel to support external writes to UC managed tables and integrate with catalog-managed commits, and the ecosystem continues to grow. By handling the low-level protocol complexity, Delta Kernel makes it straightforward for any engine to integrate with Unity Catalog which contributes to a growing ecosystem of connectors.
For an external engine to access data in UC, it needs a secure way to authenticate and get scoped access to cloud storage without requiring broad, static permissions or credentials tied to a specific account. Unity Catalog handles this through credential vending, which is now generally available (GA): UC issues short-lived, scoped credentials to external engines on demand, with access policies enforced centrally.
Thousands of customers have used UC Open APIs and two additions make it production-ready at enterprise scale. External engines can now authenticate to UC using machine-to-machine (M2M) OAuth, meeting enterprise security requirements without relying on personalized access tokens (PATs), which are per-user, long-lived, and hard to rotate. And credentials are refreshed automatically by engines via the UC credential vending APIs, so pipelines that run for hours complete reliably without tokens expiring mid-job.
With credential vending, enterprises can read, write, and create managed and external tables in Unity Catalog from any compatible engine or tool. These credentials are short-lived, scoped to the requested resource, and governed by UC privileges. This means your platform team retains full control over which principals can access data externally and what they can do with it.
With Unity Catalog’s Open APIs, we've empowered our teams to use their preferred tools while maintaining governance and data consistency. We can leverage the benefits of managed tables within a truly interoperable data and AI platform that works across multiple compute engines.— Sudipta Das, Director of Enterprise Data Operations at PepsiCo
Credential vending extends not only to tables but also unstructured data. Volume credential vending is now in Public Preview, so external clients can request temporary, scoped credentials to access images, PDFs, and videos stored in volumes with Unity Catalog governance. The same access control model, audit trail, and scoped credentials apply whether you're querying a table or processing a raw video file externally.
We're continuing to invest in making external access more capable. Credential vending today governs coarse-grained access controls for external engines. We've also developed functionality to enforce attribute-based access controls (ABAC) for external reads, which makes governance fine-grained. This makes it possible to enforce row and column level ABAC policies when UC managed tables are read rom external engines.
To get started with credential vending, see our documentation. To use the Beta of external access to managed Delta tables:
Data and AI Summit 2026 is almost here! Join us June 15-18, 2026 at the Moscone Center in San Francisco, California to learn how leading organizations are using Unity Catalog to govern data and AI across engines. Register today to get a first look at what’s coming next for open, unified governance.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。