Data and AI work has always had a maintenance problem. Data pipelines break all the time due to not only code issues but also data problems such as upstream schema changes or late-arriving data. ML models drift, and degrading models keep serving confident, wrong answers long before anything throws an error. The burden of keeping data and AI assets running in production is falling on data teams, and it's only growing. The rise of LLMs and agentic tools has made it faster than ever to build pipelines and ship models. As a result, data teams report spending most of their time fighting fires rather than building.
To help data teams with this operational burden, we’ve built Genie ZeroOps: an autonomous background agent that monitors your data and AI assets (such as pipelines, jobs, tables and ML models) and takes action before or when things go wrong. Because it runs inside Databricks, it has secure and easy access to:
Here's the process it runs for every failure:
Why do you need a purpose-built agent for data and AI operations? Can’t you use the same coding agent that helps you build software and get the same results? The answer is – “no, not really”.
Coding agents were built for software engineering, but data engineering and AI are fundamentally different:
When something breaks, you need to: detect it, assess root cause, remediate with a fix, and verify it works without side effects.
Examine each step, and you’ll find coding agents typically fall short. For detection, they can lack context, such as telemetry or choke on extremely large context, like Apache Spark™ logs. For assessment, finding the root cause and its impact, they often lack access to lineage data. They also don’t have a purpose-built harness for data and AI work, which makes the process more costly and time-consuming. Coding agents can write code for remediation, but they often lack the context to do it right and can’t fix issues that are data-related. But the step that is most challenging for coding agents is verification.
Verification requires testing code fixes against real production data in an isolated environment. You can't give an external agent access to production data, and even if you did, running code against it risks side effects that can have devastating consequences.
For an agent to safely handle the verify step, it needs to be part of the data platform itself. Genie ZeroOps is part of the Databricks Platform, and that’s what makes it succeed where coding agents fail.
Machine learning workloads in particular showcase the benefits of a purpose-built agent for operations work.
Production ML introduces some additional challenges to data engineering. A model can have no pipeline errors and still be producing bad predictions, which means keeping pipelines running isn't enough, you need to watch whether the model's outputs are still trustworthy.
When they aren't, Genie ZeroOps diagnoses the cause, builds a corrected candidate, and validates it before it touches live traffic. For a pipeline fix, it validates against a shallow clone of a table. For a model, it trains a candidate on corrected features and evaluates it against the same eval suite and criteria the production model was held to -- not a generic benchmark. It surfaces the candidate only if it's measurably better, and lets you ramp it on live traffic before it takes over.
What makes those fixes trustworthy is context. Genie ZeroOps for ML is built on the same foundation as Genie Code, Genie Ontology and native integration with the Databricks ML stack (Feature Store, MLflow, model serving, notebooks). It knows which features your model uses, how your team evaluates it, and what 'good' means for your business, so it reasons the way your senior ML engineers would.
You configure which assets Genie ZeroOps monitors and what it's authorized to do. Everything runs under Unity Catalog governance, so it can only access data your own credentials allow. Issues surface in an inbox-style UI, prioritized by severity, each with a root cause analysis and a proposed fix. Nothing gets applied to production without your approval.
The sandbox is the technical trust layer. Shallow cloning means the fix is tested with real data but production is never touched. Scoped permissions and network isolation mean the sandboxed environment can't reach outside its boundaries. What was tested is exactly what gets applied.
This is the value of Genie ZeroOps - it lets you scale your operations safely. It does the heavy lifting while you stay in control.
Genie ZeroOps is entering private preview in the coming weeks, starting with support for jobs, pipelines, tables and ML workloads. Apps, and Lakebase databases are on the roadmap.
Talk to your Databricks account team to request early access. In the meantime, explore other members of the Genie family like Genie One and Genie Code.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。