by Jonathan Brito, Gene Pang and Harsh Motwani

For years, ingesting semi-structured data like JSON, XML, or CSV meant a difficult tradeoff. Data teams could build ETL pipelines to schematize data for fast queries and sacrifice flexibility, or store data as strings to stay flexible and pay the cost in slow query performance. To solve this tradeoff, we collaborated with the Delta and Spark communities to introduce the Variant data type and brought it to the Parquet and Iceberg communities to unify the lakehouse as a single, open standard for semi-structured data.

We are excited to announce that Variant is now Generally Available in Databricks. This launch includes Variant Shredding, also Generally Available, a performance optimization that uses Predictive Optimization to automatically improve query performance on Variant data. With Variant, teams can flexibly ingest semi-structured data without compromising downstream query performance.

Flexible ingestion at scale

Over 5K teams are writing Variant using Databricks. These teams most commonly use Variant to ingest events from streaming sources like Kinesis or Event Hub, JSON payloads from APIs, and schemaless data from databases like PostgreSQL and MongoDB.

Variant is particularly useful to handle schema changes from ingestion sources. For example, an upstream application may change its API types. This leaves downstream teams scrambling to update relevant pipelines, perform backfills of existing data, and handle a cutover. Even worse, most enterprises have separate data platforms and application teams, making these schema changes unpredictable. With Variant, users can ingest all of their semi-structured data flexibly into their tables.

Variant removes the upfront cost of working with semi-structured data. Building pipelines to schematize data takes time, requiring data engineers to justify their time investment. Variant flips the paradigm – teams can effortlessly land data first, and then figure out its utility for the rest of the business.

Faster, smarter queries with Predictive Optimization

Databricks users execute 500M+ Variant queries per month across 160+ TB of Variant data. Databricks makes reading Variant as fast as reading schematized data on managed tables. Using Shredding, Variant stores common fields as columns in the underlying Parquet files. Predictive Optimization trains on a user's unique workload and query patterns, and using machine learning, identifies the most critical shredded fields and collects statistics on them to improve file skipping. As a result, Databricks scans only the files and columns required for a query, avoiding unnecessary work and boosting performance.

Variant shredding delivers nearly 4x faster reads than unshredded Variant — and 30x faster reads than storing JSON as a string:

With Variant, Databricks is unlocking lightning fast performance at scale:

Using Variant in Databricks

With Databricks, you can use Variant across your entire data stack.

Our users commonly use two tools to ingest semi-structured data as Variant:

1. Auto Loader, a source to incrementally process semi-structured files from object storage

2. Zerobus, a fully managed ingestion service to write directly into tables without using a message bus

Both ingestion approaches write data into either Delta or Iceberg, allowing any client to interoperate with data in the lakehouse. To simplify setup, use Genie Code in the Lakeflow Pipelines Editor to easily generate Auto Loader ingestion pipelines using natural language.

Teams can then directly consume Variant data in Lakehouse. Since data is intelligently shredded during the ingestion process, dashboards and reporting can query the data directly as fast as structured data.

In the near future, we plan to expand Variant support even further to include Liquid Clustering by Variant fields, expanded SQL functions, and further feature integrations.

Get started with Variant today

With Variant, you no longer have to make a tradeoff between flexibility and performance when using semi-structured data. Databricks uses Predictive Optimization, which tracks workload and query patterns, to automatically write Variant data for the best performance across products.

Getting started with Variant is easy – try it here.