Originally published on Towards AI.
There’s a painful gap between how most data talks are given and how data actually flows inside real companies. Conference slides show clean, linear pipelines. Production reality is messier: three different teams calling the same Kafka topic with different schemas, a dbt model nobody owns that silently joined the wrong dimension table for six months, and an “AI-powered” feature that turns out to be a single GPT-4 API call with no retry logic, no monitoring, and no idea what happens when the context window fills up.
So let me try to sketch out what a genuinely modern data platform looks like, one that handles AI workloads, enforces governance without requiring a full-time compliance team, and can actually be deployed by a team of four rather than forty.
The traditional warehouse-centric stack, think Redshift or BigQuery at the centre with some Airflow DAGs feeding it, was designed around a pretty simple contract. Structured data comes in, SQL queries go out, BI dashboards update overnight. That was fine.
AI changes the contract pretty fundamentally. Your LLM pipeline might need:
None of these fit cleanly into the old paradigm. The warehouse assumes structure. The data lake has structure but no transactions. Neither was built to serve a vector database at 10ms latency.
The Lakehouse architecture, Delta Lake, Apache Iceberg, or Apache Hudi sitting on object storage, is the honest answer to this. It gives you the flexibility of a lake with enough transactional integrity to trust your data. But it’s only part of the answer.
You’ve probably seen the medallion diagram before. Bronze is raw, Silver is cleaned, Gold is business-ready. Simple enough. But there are a few things people tend to gloss over.
The fourth tier, what I call the Platinum or AI-native layer, is the piece most teams miss. Once your data is clean and business-ready in Gold, there’s still work left to make it AI-ready. Embeddings need to be computed and stored somewhere fast. Fine-tuning datasets need to be curated, versioned, and tracked. Feature vectors for real-time ML models need to be pre-materialised so your serving endpoint doesn’t have to compute them at query time.
One thing I’d push back on here: a lot of teams treat this fourth tier as optional, something to add later. In practice, if you don’t design for it from the start, retrofitting it into an existing Iceberg table structure is genuinely painful. You’ll end up with embedding tables that aren’t linked to their source rows in the lineage graph, which makes auditing almost impossible.
For years, data engineers debated Lambda architecture vs. Kappa architecture. Lambda said: run two pipelines, one for real-time and one for batch, then merge the outputs. Kappa said: just use streaming for everything.
Both camps were partly right. Kappa’s “streaming for everything” ideal runs into the reality that batch processing is still cheaper and more reliable for large historical datasets. But Lambda’s “dual pipeline” approach creates synchronisation nightmares. The honest answer in 2026 is something like this:
The key insight in this architecture is that Iceberg acts as the unification point. Flink can write micro-batches to an Iceberg table every 30 seconds. Spark can overwrite entire partitions of that same table in a nightly batch run. Both writes are ACID-safe. Downstream consumers, whether a BI tool or an LLM-powered agent, always see a consistent snapshot.
The latency SLOs are also worth calling out explicitly. Your real-time path has to meet latency requirements measured in seconds. Your batch path is measured in hours. Those two pipelines need different monitoring, different alerting, and often different teams owning them. I’ve seen organisations collapse those two SLOs into one on-call rotation and it ends badly.
Here’s where I want to spend some real time, because governance is the thing that usually gets designed last and regretted first.
Most teams treat data governance like a regulatory tax. You do the minimum required to pass the audit, you slap a data catalogue on top of whatever already exists, and you call it done. That works fine until your LLM training pipeline accidentally ingests PII that was supposed to be masked in Silver. Or until a churn model’s predictions are challenged in a lawsuit and you can’t reconstruct which version of which feature was used at inference time.
A few things here worth calling out explicitly.
Column-level lineage is not optional when you’re doing AI. Table-level lineage, “this table was derived from those three tables,” is useful. But it doesn’t tell you which specific columns fed a model. If a regulator asks why your credit-scoring model produced a particular output, you need to trace back from the model’s input features to their source columns in the raw data, including any transformations that happened along the way. OpenLineage is the open standard here, and both Airflow and dbt can emit lineage events that populate tools like Marquez or DataHub.
Data contracts are the real unlock. The idea is simple enough: before a producer team changes a table’s schema or SLA, they have to negotiate that change with all downstream consumers. In practice this means defining a YAML contract (Avro schemas work well here) that specifies the column names, types, nullability guarantees, and expected freshness SLA. Break the contract, break the build. Tools like Soda Core or custom Great Expectations suites can enforce this at ingestion time.
PII classification used to be manual. Now it doesn’t have to be. An LLM tagging pipeline, yes, using an LLM to govern data that might be fed into an LLM, can scan new tables, identify columns that look like names, emails, phone numbers, or health identifiers, and automatically apply the relevant governance tags. These tags then propagate downstream to trigger column-level masking policies in Unity Catalog or whatever metastore you’re using.
Now we get to the part people find exciting, but often underestimate. Putting an LLM on top of your data platform is straightforward. Putting one there that’s reliable, governed, and cost-efficient is considerably harder.
Three specific patterns here are worth unpacking.
Text-to-SQL is deceptively tricky. The demos are always impressive, you type “what was our revenue by country last quarter” and out comes beautiful SQL. But in production, SQL generation quality degrades sharply when your schema is large, when column names are ambiguous (is status the order status or the payment status?), or when the query requires joining more than three or four tables. Tools like Vanna.AI address this with a feedback loop, storing validated query-SQL pairs as few-shot examples. DSPY from Stanford takes a more systematic approach of optimising the prompting chain using a small labelled dataset. Neither is perfect, but both are considerably more reliable than a raw "here is my schema, generate SQL" prompt.
RAG pipelines need governance too. When your RAG system retrieves chunks from an internal knowledge base, it’s effectively bypassing all the row-level security you’ve built into your lakehouse. A user who shouldn’t be able to see customer PII can, in theory, ask a question that causes the RAG system to retrieve and surface that information. Solving this properly requires filtering your vector search results by the same access policies you apply to your structured data, which means your embedding pipeline needs to store metadata alongside each vector indicating which roles are permitted to see it.
The feature store is still underutilised. Most teams I talk to have at least heard of feature stores but haven’t built one properly. The value proposition is this: you compute a feature, say, a user’s 30-day purchase frequency, once, store it, and serve it to both online model endpoints (at millisecond latency via Redis) and offline training pipelines (from the lakehouse). Without a feature store, that calculation gets duplicated, often subtly differently, between your training pipeline and your serving pipeline. That inconsistency is one of the most common sources of model performance degradation in production.
Data observability has become its own category. Tools like Monte Carlo, Acceldata, and Metaplane essentially run health checks on your pipelines and alert you when something breaks silently. And “breaking silently” is the killer. A pipeline that errors out is annoying. A pipeline that runs successfully but produces subtly wrong numbers for three weeks is catastrophic.
The minimum viable observability setup for a production platform:
The AI-specific layer adds model monitoring. Prediction distributions should look roughly similar over time. Feature distributions fed to the model should match what the model saw during training. Embedding similarity in a RAG system should stay within an expected range. When these drift, your model’s accuracy is likely degrading even if your business metrics haven’t caught up yet.
Let me close with something concrete. Suppose you’re building this for a mid-size e-commerce company, around two million orders per year, a team of eight data engineers and three ML engineers.
Your Kafka cluster handles clickstream and CDC from Postgres. Flink jobs enrich and normalise events in real time, writing to Iceberg tables on S3 every 90 seconds. A nightly Airflow DAG triggers dbt runs that build the Silver and Gold layers. A Feature Store job runs every hour, recomputing the top 200 features for your recommendation model and writing them to Redis for sub-10ms serving.
Your LLM setup is a LangChain agent with three tools: a Text-to-SQL tool over the Gold layer (restricted to read-only, with row-level security enforced by Databricks Unity Catalog), a RAG tool over your product documentation (with metadata-filtered vector search), and a custom tool that hits your feature store API.
Governance is Unity Catalog for access control and column-level masking. DataHub for the catalogue and lineage. OpenLineage emitted by both Airflow and dbt. Data contracts enforced by Soda Cloud on every Silver table.
Observability is Monte Carlo for data quality alerting and Grafana with custom dashboards over your pipeline execution metadata.
Is it perfect? No. You’ll hit schema drift you didn’t anticipate. You’ll have a Flink job that starts lagging when Kafka partition counts grow unevenly. Your Text-to-SQL tool will occasionally generate a query that joins the wrong dimension and nobody will notice for a week.
But it’s recoverable. Because you have lineage to trace the problem back to its source, contracts to catch schema changes, observability to alert you when distributions shift, and a governance layer that means the LLM can’t leak data it shouldn’t see.
That’s the real goal: not a perfect platform, but one that fails detectably and fixes cheaply.
The teams that build great data platforms aren’t the ones with the fanciest tooling. They’re the ones that were disciplined about governance from day one, who treated observability as a first-class concern rather than a retrospective, and who resisted the temptation to reach for an LLM before the underlying data quality problems were solved.
The LLM is only as good as the data it sees. The pipeline is only as trustworthy as the contracts that define it. And the whole thing is only as useful as the trust that business stakeholders have in it.
Build that trust deliberately, layer by layer, Bronze to Platinum.
Published via Towards AI
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。
