Migration as a strategic evolution

BigQuery is often the standard for starting fast, but for many enterprises, scale eventually turns that simplicity into a management challenge. When your workloads reach a point where varying on-demand costs and slot reservations necessitate a trade-off between performance and your budget, together with the increased complexity of managing data governance, it’s time to rethink the architecture.

By consolidating ETL, Storage, BI, and multi model AI into a single, open and simplified Lakehouse architecture with a unified governance layer, organizations eliminate proprietary silos and gain predictable performance at any scale. This transition allows teams to move toward an open format that simplifies operations, streamlines compliance from data to AI, and unlocks new AI-driven use cases.

A successful migration requires more than copying tables. It demands a phased strategy: moving data out from a proprietary storage, transforming logic thoughtfully, and validating results with automated tooling to capture ROI. This guide outlines a pragmatic framework across Process, Technology, and People for navigating that transition with minimal disruption and measurable business impact.

Process: The migration journey and governance evolution

The Process pillar defines how you migrate. Success depends on choosing the right entry point and managing the transition period effectively.

Assess

The migration journey runs in sequence: assess the estate, choose an entry point, migrate, validate in dual operation, then decommission. Assessment comes before everything else: you cannot pick the right entry point without knowing what you run today. Profile the BigQuery estate first — datasets, query history and slot consumption — to find which workloads drive cost, which dashboards are actually used, and which tables are never queried at all. Lakebridge, the open-source Databricks Labs migration toolkit, includes a BigQuery profiler that automates this discovery, so migration waves are planned around what is actually used rather than what merely exists.

Choose Strategy

• BI-first: Prioritize the dashboards decision-makers see every day. This is the path for an analytics leader whose dashboards are slow or capped by concurrency limits, and whose analysts want AI features. Rebuild the most-used dashboards on Databricks, reading BigQuery data in place at first, and switch teams over one use case at a time, proving parity side by side. The payoff is visible from day one: faster dashboards and natural-language Q&A with Genie.
• ETL-first: Prioritize the backend to solve spiraling costs or performance bottlenecks. By offloading heavy processing to the Photon/Spark engine, you stabilize the "engine" and create a clean foundation for future AI. It suits a data platform owner watching costs climb and pipeline windows slip: move the backend first and let the results speak. The trade-off is visibility — business users see little until pipelines land — so publish cost and runtime wins with each wave. When in doubt, the assessment decides: dashboard complaints point BI-first; pipeline cost points ETL-first.

Whichever entry point you choose, migrate in waves, not in one big bang. A big-bang cutover concentrates all the risk in a single moment — and if anything breaks, so does trust in the whole program. Waves keep the blast radius small: rank workloads on two axes — value to the organization (how visible it is to leadership, how directly it touches revenue, how urgent the compliance deadline behind it is, how many people rely on it day to day) and migration complexity — and start where value is high and complexity is low. Each wave then delivers a visible business win, reconciles before the next starts, and leaves the team's playbook faster for the one after it. Reserve big bang for the rare small, low-risk estate where running two platforms costs more than it protects.

Execute

1. Lift and shift: Migrate existing SQL logic "as-is" to Databricks SQL. This ensures rapid decommissioning of legacy costs and immediate performance gains.
2. Modernize: Once stable, refactor high-value pipelines into Lakeflow Spark Declarative Pipeline for automated orchestration, built-in data quality, and a unified framework for both batch and streaming.

Most organizations navigate a Dual Operation phase using Lakehouse Federation to "shadow" workloads for validation. The key to ROI is setting clear "success criteria" (e.g., 99.9% parity) to trigger the immediate decommissioning of legacy pipelines, eliminating the cost of running two platforms.

Dual operation works in both directions, and the right bridge depends on your entry point. BI-first teams use Lakehouse Federation so Databricks can read BigQuery while dashboards move first. ETL-first teams flip it: move ingestion and transformation to Databricks, land curated tables in open formats, and let BigQuery keep serving existing dashboards and applications from that same single copy — no dual-write pipelines, no export jobs, no second copy to reconcile. Migrating in waves keeps this dual-operation window short and narrow: only the workloads in the current wave carry the cost of running on two platforms at once, so the bill for dual operation stays proportional to what's actually in flight, not the whole estate. Treat BigQuery-as-serving-layer as a transitional state rather than a destination: the external tables are read-only on BigQuery’s side and carry limitations such as manual schema refresh after schema changes, so plan the cutover of the serving layer onto Databricks SQL as the closing step of the journey.

Govern as an engine

Unity Catalog transforms governance from a hurdle into a strategic engine. It offers a seamless 3-tier mapping (Project -> Catalog, Dataset -> Schema, Table -> Table) that replicates BigQuery permissions while adding automatic, end-to-end lineage and treating AI models as first-class citizens.

The mapping goes deeper than the object hierarchy. Unity Catalog gives you the same fine-grained protection natively: row filters restrict access row by row, and column masks and tags handle column-level security — so protection travels with the data instead of being rebuilt from scratch. And one sequencing rule from the field: migrate permissions before data, so each wave's cutover changes where a table lives, never who can see it.

Technology: building the open foundation

The technical pillar focuses on moving from a closed, proprietary storage model to an open, high-performance architecture.

Modernizing from BigQuery to Databricks involves three workstreams: Data Migration, Logic Migration, and Validation.

Data migration

Choose the path by volume and freshness. Bulk history moves fastest through BigQuery’s export to Parquet on Google Cloud Storage, usually the economical route at scale, and the export doubles as a frozen point-in-time copy that simplifies validation. Continuously updated tables are read through the Storage API connector or redirected at the source, and small, frequently changing datasets can stay queryable through federation until their wave arrives. Everything lands in open formats, ready for the medallion layers.

Logic migration

Never hand-convert an estate. Three tiers cover it: rule-based transpilation (e.g. Lakebridge) for the bulk of routine SQL, LLM-assisted conversion for dialect quirks, and engineers reserved for the genuinely complex tail. Orchestration follows the same pattern: scheduled queries and Composer DAGs map to Lakeflow Jobs.

Validation

Budget for validation as seriously as for the migration itself — in the field it often consumes comparable effort. Run it at three levels: completeness (row counts), consistency (schemas and types), and accuracy (aggregate reconciliation plus row-level hashing), automated with Lakebridge reconcile, which supports BigQuery as a native source. One field lesson: small built-in function differences between the two SQL dialects can break hash comparisons — investigate mismatches before assuming data loss. Parity here is what triggers decommissioning in the Process journey.

Open table formats

The open foundation pays off before the migration even finishes. Because Delta Lake and Apache Iceberg are open formats, the tables you land on Google Cloud Storage are readable by more than Databricks: BigQuery reads Delta through BigLake external tables and Iceberg through Iceberg external tables, and catalog federation between Unity Catalog and BigQuery (currently in preview) lets both platforms govern and query the same tables without copying them. Storage is decoupled from the engine — one copy of data, many engines. That interoperability is not a side benefit; it is the capability behind the low-risk transition patterns described in the Process section.

People: Organizing the Modern Data + AI team

The third pillar provides the ultimate ROI of a migration: a more capable, unified workforce.

Ending the handoff culture. In legacy stacks, the gap between SQL Analysts and Data Scientists creates silos. In Databricks, shared Notebooks allow the entire "squad" to live in the same workspace, reducing communication overhead.

Building the new skillset. Genie acts as the bridge for SQL-heavy teams. Analysts can use natural language to generate Python or Spark code, turning traditional analysts into versatile data practitioners without a steep learning curve. Pair that day-to-day AI assist with structured training, Databricks Academy courses and role-based certifications, so the new skillset sticks organization-wide instead of depending on a few self-taught early adopters.

Engineering rigor. Teams move from "writing queries" to "building data products" by adopting software engineering best practices like Unity Catalog, Git integration and CI/CD.

Lessons from the field

Six patterns repeat across successful BigQuery migrations based on our migration project experience:

• Profile before you plan. In most estates, a significant share of BigQuery tables are rarely or never queried. Profiling query history and slot usage first means you migrate the workloads that matter and retire the rest.
• Define parity criteria before dual-run starts. Agree the success threshold (for example, 99.9% reconciliation across row counts, aggregates and hashes) up front — without it, the shadow period has no exit condition.
• Don’t hand-convert SQL. Rule-based transpilation plus AI-assisted conversion handles the bulk of the estate; reserve engineers for the genuinely complex remainder.
• Map governance one-to-one first, modernize second. The project → catalog, dataset → schema, table → table mapping into Unity Catalog preserves existing permissions — including row- and column-level policies — on day one; richer tagging, attribute-based controls and lineage-driven governance can evolve after cutover.
• Treat BI-first as a change-management project. The technology is often the easier half; the analysts whose dashboards move need enablement, champions and a feedback loop.
• Decommission aggressively. Every week both platforms run, ROI erodes. Celebrate switch-offs, not just go-lives.

Conclusion

Migrating to Databricks isn't one leap — it's a sequence of decisions you can actually plan for: which entry point fits your team, how to sequence waves so each one banks a win before the next starts, and which bridge keeps BigQuery and Databricks running side by side until the last dashboard cuts over. Get that sequencing right, and Process, Technology, and People reinforce each other: data migration, logic transformation, and validation retire the estate on one side, while analysts querying in natural language with Genie and engineers shipping governed data products build on it on the other.

The result isn't just an open lakehouse replacing a proprietary warehouse — it's a migration your organization can trust wave over wave, and a team that's ready to build on what it lands on.

Ready to plan your migration? Explore the Databricks migration hub , assess your estate with the open-source Lakebridge toolkit ,or contact your Databricks account team for a migration assessment.