惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Apple Machine Learning Research
Apple Machine Learning Research
J
Java Code Geeks
博客园 - 聂微东
Microsoft Azure Blog
Microsoft Azure Blog
量子位
T
Tailwind CSS Blog
Vercel News
Vercel News
I
InfoQ
Stack Overflow Blog
Stack Overflow Blog
U
Unit 42
Engineering at Meta
Engineering at Meta
L
LangChain Blog
大猫的无限游戏
大猫的无限游戏
D
Docker
博客园_首页
P
Proofpoint News Feed
月光博客
月光博客
T
The Blog of Author Tim Ferriss
MyScale Blog
MyScale Blog
酷 壳 – CoolShell
酷 壳 – CoolShell
Martin Fowler
Martin Fowler
腾讯CDC
N
Netflix TechBlog - Medium
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant
Data Warehouses, Data Marts, Data Lakes, and Lakehouses -...
Anthony Gich · 2026-05-03 · via DEV Community

If you’ve been around data engineering long enough, you’ve probably heard these terms thrown around in meetings:

  • “Just dump it in the data lake”
  • “We’ll expose it through the warehouse”
  • “That goes into the mart”
  • “We’re moving to a lakehouse architecture”

And honestly… it can sound like four different ways of saying the same thing.

They’re not.

Each one solves a slightly different problem in the data ecosystem. Once you understand the “why” behind each, the architecture suddenly feels a lot less like buzzwords and more like a clean system design.

Let’s break it down in a practical, engineer-first way.


1. The Big Picture (Why all these systems exist)

In most companies, data doesn’t come from one place — it flows in from everywhere:

  • User clicks from web/mobile apps
  • Payments and transactions
  • Logs from servers
  • Third-party APIs (Stripe, Shopify, etc.)
  • IoT or streaming data (Kafka, sensors, etc.)

Now here’s the problem:

Raw data is messy. Business users don’t want messy.

So we build systems that progressively refine data from:

Raw → Clean → Structured → Business-ready

That’s where these four concepts come in:

  • Data Lake → store everything raw
  • Data Warehouse → structured analytics-ready data
  • Data Mart → department-specific slices of warehouse data
  • Lakehouse → hybrid of lake + warehouse

2. Data Lake — “Store everything first, figure it out later”

A data lake is basically a massive storage system where you dump raw data in its original format.

Think of it like:

A giant warehouse where you throw every box in as-is, without opening it.
Or even better: a farm storage system where everything is stored right after harvest, unprocessed and mixed together.

Characteristics:

  • Stores structured, semi-structured, and unstructured data
  • Cheap storage (usually object storage like S3)
  • Schema is applied when reading, not writing (schema-on-read)

Example tools:

  • Amazon S3
  • Azure Data Lake Storage
  • Google Cloud Storage

Example:

You might store:

/events/clicks/2026/05/01.json
/logs/api/2026/05/01.log
/payments/stripe/2026/05/01.parquet

Enter fullscreen mode Exit fullscreen mode

No transformations. No enforcement. Just storage.

Here’s the catch:

If you’re not careful, a data lake becomes a data swamp — lots of data, zero usability.


3. Data Warehouse — “Clean, structured, and business-ready”

A data warehouse is where data goes after it has been cleaned, transformed, and modeled for analytics.

Think of it like:

A well-organized supermarket where everything is cleaned, packaged, labeled, and placed on the right shelves.
You don’t pick raw potatoes from the soil — you get them washed, sorted, and priced.

Characteristics:

  • Structured data only
  • Schema-on-write (you define structure before loading)
  • Optimized for analytics queries (OLAP systems)
  • Highly curated and trustworthy

Example tools:

  • Amazon Redshift
  • Snowflake
  • Google BigQuery

Typical workflow:

  1. Extract data from sources
  2. Transform (clean, join, aggregate)
  3. Load into warehouse tables

Example SQL model:

CREATE TABLE sales_fact (
    order_id INT,
    customer_id INT,
    product_id INT,
    amount DECIMAL(10,2),
    order_date DATE
);

Enter fullscreen mode Exit fullscreen mode

Now business analysts can run queries like:

SELECT product_id, SUM(amount)
FROM sales_fact
GROUP BY product_id;

Enter fullscreen mode Exit fullscreen mode


4. Data Marts — “Department-specific mini warehouses”

A data mart is a subset of a data warehouse focused on a specific business domain.

Think of it like:

A grocery store or specialty shop — like a bakery, butcher, or vegetable shop.
It doesn’t sell everything. It only sells what its customers actually need.

Characteristics:

  • Smaller scope than a warehouse
  • Built for a specific team (finance, marketing, sales)
  • Faster queries for targeted use cases

Example:

A marketing data mart might include:

  • Campaign performance
  • Customer acquisition metrics
  • Ad spend data

Example structure:

CREATE TABLE marketing_campaign_performance AS
SELECT
    campaign_id,
    SUM(clicks) AS total_clicks,
    SUM(impressions) AS total_impressions
FROM ad_events
GROUP BY campaign_id;

Enter fullscreen mode Exit fullscreen mode

Why it exists:

Instead of everyone querying a massive warehouse, teams get pre-optimized datasets.


5. Data Lakehouse — “Best of both worlds”

Now this is where things get interesting.

A lakehouse combines:

  • The flexibility of a data lake
  • The structure and performance of a data warehouse

Think of it like:

A modern retail system where the warehouse and supermarket are combined into one smart facility.
Raw goods arrive, but they are immediately tracked, organized, and made queryable without losing flexibility.

Characteristics:

  • Uses low-cost storage (like a lake)
  • Adds structure, ACID transactions, and governance
  • Supports both analytics and ML workloads

Example tools:

  • Apache Spark + Delta Lake
  • Apache Iceberg
  • Apache Hudi

Why it matters:

In traditional setups:

  • Data lakes = flexible but messy
  • Warehouses = clean but expensive and rigid

Lakehouses try to remove that tradeoff.


6. How They Work Together (Real Architecture Flow)

A modern data pipeline often looks like this:

[ Data Sources ]
      ↓
   DATA LAKE (raw storage)
      ↓
ETL / ELT pipelines (Airflow, Spark)
      ↓
DATA WAREHOUSE (modeled data)
      ↓
DATA MARTS (team-specific views)
      ↓
Dashboards / BI tools

Enter fullscreen mode Exit fullscreen mode

Or in a lakehouse setup:

[ Data Sources ]
      ↓
DATA LAKEHOUSE (single system)
      ↓
BI + ML + Analytics directly

Enter fullscreen mode Exit fullscreen mode


7. Practical Example (Mini Pipeline)

Let’s say we’re processing e-commerce data.

Step 1: Raw data in S3 (Data Lake)

{
  "order_id": 101,
  "user_id": 55,
  "amount": 250,
  "timestamp": "2026-05-01T10:00:00Z"
}

Enter fullscreen mode Exit fullscreen mode

Step 2: Spark transformation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("etl").getOrCreate()

df = spark.read.json("s3://datalake/raw/orders/")

clean_df = df.dropna() \
             .withColumnRenamed("amount", "order_amount")

clean_df.write.mode("overwrite").parquet("s3://warehouse/sales_fact/")

Enter fullscreen mode Exit fullscreen mode

Step 3: Load into warehouse (Redshift example)

COPY sales_fact
FROM 's3://warehouse/sales_fact/'
IAM_ROLE 'arn:aws:iam::123456:role/RedshiftRole'
FORMAT AS PARQUET;

Enter fullscreen mode Exit fullscreen mode

Step 4: Create a data mart

CREATE TABLE sales_summary AS
SELECT
    DATE(order_date) AS date,
    SUM(order_amount) AS revenue
FROM sales_fact
GROUP BY DATE(order_date);

Enter fullscreen mode Exit fullscreen mode


8. Common Pitfalls (Where most teams mess up)

1. Turning the data lake into a swamp

Dumping everything without metadata or structure leads to chaos.

2. Over-modeling too early

Trying to build perfect schemas upfront slows everything down.

3. Duplicating logic across marts

You end up with inconsistent metrics like “Revenue_v1”, “Revenue_final”, “Revenue_real_final”.

4. No governance layer

Without access control and cataloging, nobody trusts the data.


9. Best Practices (From real-world systems)

1. Use layered architecture

  • Raw (lake)
  • Cleaned (staging)
  • Modeled (warehouse)
  • Aggregated (marts)

2. Standardize transformations

Use tools like:

  • dbt
  • Apache Airflow
  • Spark jobs with clear ownership

3. Define a single source of truth

One metric definition per business KPI. No duplicates.

4. Treat data like software

Version it, test it, document it.

5. Monitor everything

  • Pipeline failures
  • Data freshness
  • Schema changes

10. Conclusion — The mental model that matters

If you strip away the jargon, it’s really simple:

  • Data Lake → store everything
  • Data Warehouse → clean and organize it
  • Data Mart → tailor it for teams
  • Lakehouse → unify storage and analytics

The real skill in data engineering isn’t memorizing definitions.

It’s knowing:

When to keep data raw, when to structure it, and when to specialize it.

Once that clicks, designing data systems becomes a lot more intuitive — and honestly, more fun to build.