惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

罗磊的独立博客
美团技术团队
Apple Machine Learning Research
Apple Machine Learning Research
Hugging Face - Blog
Hugging Face - Blog
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
月光博客
月光博客
WordPress大学
WordPress大学
The Cloudflare Blog
阮一峰的网络日志
阮一峰的网络日志
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
博客园_首页
博客园 - Franky
博客园 - 司徒正美
酷 壳 – CoolShell
酷 壳 – CoolShell
爱范儿
爱范儿
Jina AI
Jina AI
Last Week in AI
Last Week in AI
雷峰网
雷峰网
IT之家
IT之家
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
博客园 - 聂微东
小众软件
小众软件
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
V
V2EX

Databricks

Using AI_Functions in Your Data Warehouse: Top Use Cases How Scottish Water Made Its Capital Investment Data Conversational With Databricks Genie What are AI Hallucinations? Smart Routing in Unity AI Gateway: Match frontier quality with 30%+ lower cost per task Databricks Network Configuration delivery to Tens of Millions of Serverless VMs How Amtrak is building the data backbone for its largest transformation in over 50 years How a major freight railroad scaled pipeline creation with Genie Code The Future of Data Analytics: Why AI is rewriting the Analyst’s Job Description Taking AUTO CDC to the next level: Solving the hardest real-world use cases Open-sourcing Metals v2: Databricks’ Java and Scala language server for multi‑million line codebases Modern Risk Demands a Real-Time Foundation: The CRO’s Mandate Electric joins Databricks to bring WASM Postgres to AI agent sandboxes How to ground Genie Agents in both structured data and documents without losing governance Innocent until combined: Blocking the lethal trifecta with Omnigent Contextual Policies Introducing FILE type: a native column type for multimodal data Managing AI Coding Costs at Scale What is an AI Assistant? What are Agentic Workflows? What is Tool Calling? Kimi K3 from Moonshot AI is now available on Databricks through Unity AI Gateway Introducing OfficeQA Pro V2: A New Benchmark for Enterprise Grounded-Reasoning BigQuery to Databricks: A Strategic Framework for Modern Migration Unity AI Gateway is Generally Available Granular Usage Attribution for dbt Pipelines with Query Tags - Cloned Databricks joins the Open Secure AI Alliance to advance AI safety and security The New Monday Morning Report: How Generative AI can deliver the insights your executives need. Ingest semi-structured data faster and more efficiently with Variant - Now Generally Available Databricks Completes Acquisition of Panther: Accelerating the Security Lakehouse Era Foundations for an AI-forward healthcare organization Agentic media buying cannot scale without the right foundation. See how buyers and sellers get there on Databricks.
Backstage with Lakebase, part 3
Cameron Casher, Shanil Anushka Fernando, Kevin Hartman · 2026-08-01 · via Databricks

In the first part of this series, running Backstage on Databricks Lakebase gave us one-second database branching. In part two, meanwhile, Unity Catalog absorbed that operational database into the enterprise governance plane.

But here’s the payoff that actually changes the org chart. 

In a normal stack, answering ‘who owns the infrastructure running up our cloud spend, and what did it cost?’ crosses two boundaries. The ownership graph lives in Backstage (owned by platform engineering), while the cost data lives in a data warehouse (governed by the data team). Answering the question requires an ETL pipeline, a Jira ticket or a Slack thread.

Separated compute makes sharing viable

The reason a FinOps analyst can run massive analytical queries against the exact same underlying storage without impacting the live portal is that Lakebase isolates compute per workload.

Backstage gets its own isolated, autoscaling compute envelope. During normal portal use, catalog queries ran at 55–65 ms end-to-end, and searches hit two to four milliseconds. Because your web application and your analytical workloads aren't contending for the same compute cluster, they can finally safely share the same data substrate.

The workaround: Lakehouse federation auth

To join the live Postgres data to our analytical billing data, we use Databricks Lakehouse Federation. However, Lakehouse Federation’s Postgres connector currently only supports static user/password credentials. Because Lakebase authenticates app identities via OAuth JWTs, the federation engine needs a parallel auth path.

The workaround is creating a native Postgres role with SCRAM-SHA-256 auth, wired to federation separately from the OAuth identity the app uses:

You’re now managing two auth paths for the same database.

The FinOps join
With the foreign catalog live, a FinOps analyst can write a single query that pulls the Backstage resource name directly from the operational Postgres table, and joins it to Lakebase's own billing rows

With the foreign catalog live, a FinOps analyst can write a single query that pulls the Backstage resource name directly from the operational Postgres table, and joins it to Lakebase's own billing rows in system.billing.usage:

Real result:

The left side of that row comes directly from inside the live Backstage Postgres catalog; the right side comes from a Unity Catalog system billing table. Those two things have historically never been in the same SQL engine, and now they join with zero data movement.

Why not just use ETL?

A skeptic might ask why we don't just use a Python script to sync an RDS instance to a Delta table once an hour.

The answer is branching. When a developer creates an ephemeral, 1-second database clone to test a PR, you would have to dynamically provision new ETL pipelines just to get cost visibility into that temporary test environment. With Lakebase, the moment the branch is created, its billing and ownership data are instantly queryable. (In this POC, the dropped test branch was automatically and independently attributed 0.0107 DBU).


Operationalizing convergence

This three-part series started with a 1-second database branch, moved through unified governance and landed here — a single SQL query that joins operational ownership data to cloud billing data with zero pipelines between them. That's the proof that convergence works technically. The question practitioners will ask next is: what does it take to operationalize this?

Two things stood out from this POC that are worth calling out for teams planning to follow this path.

The federation auth gap

The Lakehouse Federation workaround we described – a native Postgres role with static credentials wired separately from the OAuth identity the app uses – is the right approach today. Every team that wants to join their Lakebase operational data with analytical tables in Unity Catalog will need to set up this parallel auth path. Federation probably shouldn't run as your application user anyway, so the separation has a security upside, but password rotation is on you. For teams adopting this pattern, the steps can be packaged into a repeatable script – generate a secure password, create the role with read-only grants, wire the connection, create the foreign catalog. One-time setup, minutes once you know the pattern. Natively supporting OAuth JWTs in Federation would eliminate this workaround entirely.

Branch cost visibility for dev teams

The FinOps join answers the platform question: what does this infrastructure cost and who owns it? But the same billing data tells a second story that matters to engineering managers: what does the development process itself cost?

In the branching workflow from part one, every pull request creates an ephemeral CI branch and every developer has their own feature branch. These show up as independent line items in system.billing.usage, broken down by branch_id and endpoint_id. An engineering manager can see exactly how much compute their team's dev/test branching consumed in a sprint versus production – and make informed decisions about branch lifecycle policies.

The key is that ephemeral branches should be treated as ephemeral in the billing data too. CI branches created with a short TTL auto-expire if cleanup fails for any reason – a direct push to main, a workflow error, a missed event. Without lifecycle controls, orphaned branches can accumulate quietly, each one with an active compute endpoint billing against the project. The test branch cost 0.0107 DBU. That's trivial. Thirty orphaned branches running for a month are not.

The point isn't that branching is expensive – it's a cost vs productivity gain. When a team eliminates two days of environment wait time per sprint and stops maintaining 20-30% of their codebase in mock objects, the 0.0107 DBU per branch isn't a line item to manage – it's the cheapest productivity investment the team has ever made. And unlike most productivity investments, this one is measurable: the infrastructure tells you exactly what it cost, per branch, per developer, per sprint. That's a conversation most engineering teams have never been able to have with their database.

What comes next

Before we wrap, there's one more point to the FinOps story that should be called out. Lakebase endpoints scale to zero. When a branch isn't being queried, its compute suspends and the bill stops. The 0.0107 DBU figure is the cost of a branch that ran, not the cost of a branch that exists; a fleet of ephemeral branches sitting between test runs contributes nothing.

Across this series, we proved the infrastructure works – real app, real benchmarks, real governance, real cost data. From our side, Databricks and Thoughtworks are working together to take this from POC to practice: real development teams, real sprints, real velocity measurements. The constraint that kept operational and analytical data in separate worlds for thirty years is dissolving.

There's a Monday morning takeaway for every piece of this series. Branch your next migration on a real schema. Rewrite one mock-heavy suite against a branch. Join your billing data to your ownership graph.The teams that move first will define what comes next.