惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

V
Visual Studio Blog
MongoDB | Blog
MongoDB | Blog
Engineering at Meta
Engineering at Meta
云风的 BLOG
云风的 BLOG
Microsoft Azure Blog
Microsoft Azure Blog
B
Blog RSS Feed
T
The Exploit Database - CXSecurity.com
P
Privacy & Cybersecurity Law Blog
Know Your Adversary
Know Your Adversary
月光博客
月光博客
I
InfoQ
阮一峰的网络日志
阮一峰的网络日志
NISL@THU
NISL@THU
爱范儿
爱范儿
S
Securelist
博客园 - 叶小钗
C
CERT Recently Published Vulnerability Notes
Recorded Future
Recorded Future
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
aimingoo的专栏
aimingoo的专栏
D
DataBreaches.Net
G
GRAHAM CLULEY
P
Proofpoint News Feed
A
About on SuperTechFans
Google DeepMind News
Google DeepMind News
C
Cyber Attacks, Cyber Crime and Cyber Security
Threat Intelligence Blog | Flashpoint
Threat Intelligence Blog | Flashpoint
T
Tor Project blog
Stack Overflow Blog
Stack Overflow Blog
T
Threat Research - Cisco Blogs
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
T
Tailwind CSS Blog
有赞技术团队
有赞技术团队
Hugging Face - Blog
Hugging Face - Blog
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
Recent Announcements
Recent Announcements
P
Proofpoint News Feed
The GitHub Blog
The GitHub Blog
The Cloudflare Blog
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
Last Week in AI
Last Week in AI
Y
Y Combinator Blog
Jina AI
Jina AI
大猫的无限游戏
大猫的无限游戏
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
罗磊的独立博客
博客园 - 【当耐特】
H
Help Net Security
F
Fortinet All Blogs
T
The Blog of Author Tim Ferriss

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant Common SOC 2 Failures (Real World) Stop Vibe-Checking Your AI App: A Practical Guide to Evals How to Use SonarQube and SonarScanner Locally to Level Up Your Code Quality Your Next To-Do App Is Dead — I Replaced Mine with an OpenClaw AI Sign a Nostr event in 60 lines of Python using coincurve — no nostr-sdk, no nbxplorer, no rust toolchain ITGC Audit Explained Like You’re in Big 4 Patch Tuesday abril 2026: Microsoft parcha 163 vulnerabilidades y un zero-day en SharePoint Stop scraping everything: a better way to track competitor price changes Listing on MCPize + the Official MCP Registry while routing payments OUTSIDE the marketplace — how I kept 100% of my x402 revenue Building an AI-Powered Risk Intelligence System Using Serverless Architecture Why We Ripped Function Overloading Out of Our AI Toolchain Testing AI-Generated Code: How to Actually Know If It Works SaaS Churn Is Killing Your Business. Here Is What to Do About It (Without a Support Team) The Speed of AI Is No Longer Linear - And Self-Improving Models Are Why How to Implement RBAC for MCP Tools: A Practical Guide for Engineering Teams From Standard Quote to Persuasive Proposal: AI Automation for Arborists I built a CLI that scaffolds complete multi-tenant SaaS apps Axios CVE-2025–62718: The Silent SSRF Bug That Could Be Hiding in Your Node.js App Right Now The dashboard that ended our friendship Data Pipelines Explained Simply (and How to Build Them with Python) The Hidden Cost of AI Systems Nobody Talks About. undefined vs undeclared, and how typeof behaves Switching from file-based jobs to NATS/Kafka in Rust without changing code io_uring Adventures: Rust Servers That Love Syscalls Why Agentic AI is Killing the Traditional Database The POUR principles of web accessibility for developers and designers Quantum Neural Network 3D — A Deep Dive into Interactive WebGL Visualization How To Install Caveman In Codex On macOS And Windows Automation Pipeline Reliability: Why Your Workflow Breaks When Nobody Is Watching I Built an 'Open World' AI Coding Agent — It Works From ANY Folder From Freelancing to Product: A Tech Service Company's SaaS Transformation China's AI Giants: Adding Tencent Hunyuan & ByteDance Doubao to AI University (74 Providers) On the Vibe Coders and Their Lies clerk: Auto-Summarize Your Claude Code Sessions AI Weekly — 2026/04/10–04/17 | The Model Lockdown Is Here, but the Toolchain Is the Real Battleground AI 週報 — 2026/04/10–2026/04/17 模型封鎖潮來了,但工具鏈才是真戰場 Maybe this is how Open-Source apps are born... 🚀 Fine-Tune LLMs with LoRA and QLoRA: 2026 Guide tRPC v11 + Next.js App Router: End-to-End Type Safety Without the Boilerplate ShadCN UI in 2026: Why I Stopped Installing Component Libraries and Started Owning My Components SaaS Billing in React Server Components: Stripe + Supabase Without a Single `useEffect` Join our DEV Weekend Challenge — $1,000 in Prizes Across TEN winners! Submissions Due April 20 at 6:59 AM UTC. Implementing FSRS Spaced Repetition in Flutter + Supabase — Adding Memory Science to an AI Learning App "I Texted My Localhost From the Train — Claude Code Fixed the Bug Before I Got Home" I Built a Sales Prep AI and It Went Deeper Than Expected Design to Code #2: One JSON, Eleven Outputs Solving the 100M-Row Problem: A Summary Table Pattern for High-Volume Push Notification Logs Flutter Web With Wasm: What Actually Changes For Developers I Built 50 Royalty-Free Soundtracks for My Side Project in a Weekend Using AI Music Generation The Vibe Coding Security Checklist: 7 Things to Check Before You Ship Stop Letting Googlebot Guess Fix Your React App's SEO Right Desconstruindo o Streaming do LinkedIn: Como Criar um Engine de Extração de Vídeo de Alta Performance com HLS e FFmpeg (EDA Part-1) EDA (Exploratory Data Analysis) Explained With Real Life — Why Looking at Your Data Is the Most Important Step in Machine Learning Brand Relationship Management at Scale: Our 4-Touch Outreach System for 200+ Brands Why String.fromEnvironment() Might Return an Empty String in Dart JGuardrails 1.0.0 — Hardening Java LLM Apps Against Jailbreaks, Toxicity, and Prompt Injection Plan and Schedule a Full Week of Threads Content From One Claude Conversation Coding Cat Oran Ep3, Five Tables Changed Everything Updated: BFF Pattern I'm done watching freelancers get buried by 200 proposals. So I'm building the alternative. This is my first post BFS Algorithm in Java Step by Step Tutorial with Examples Tracking LLM Pricing Monthly: An Open Dataset for 22 AI Models How We Measure Content ROI on a Comparison Site: Revenue Attribution Without Perfect Data Introducing Nova AI Ops: The AI-Native Operating System for SRE Teams I built a free desktop video downloader for Windows — Grabbit How Talkie OCR Helps Vision-Impaired & Dyslexic Users Read the World Around Them VRCFaceTracking安装和iPhone面捕配置教程,有bug Even CrowdStrike Can't See Your Agents The Automation Gold Rush: What n8n Workflows and Claude Are Opening Up for Developers Right Now
BigQuery, Snowflake, Redshift, Databricks, Fabric: where each one silently inflates your bill
Aditya Soman · 2026-05-18 · via DEV Community

TL;DR

  • Cloud data warehouses trap you with hidden fees: the Scan Tax (charging per terabyte scanned), the Idle Tax (60-second minimums for inactive compute), and the Complexity Tax (opaque billing units).
  • The major incumbents, BigQuery, Snowflake, Redshift, Databricks, and Fabric, force you into punishing trade-offs between bankrupting your budget on exploratory queries, eating costs for idle time, or suffering through agonizing resume latencies.
  • MotherDuck provides a modern cloud data warehouse alternative designed to eliminate these taxes with a strict 1-second billing minimum, true scale-to-zero architecture, and flat compute pricing for workloads ranging from gigabytes to petabytes with Managed DuckLake (in preview).

My worst on-call wakeup wasn't a database melting down at 3 AM. It was an email from finance.

Someone had run a query in a BI tool, and it generated a $50,000 Google BigQuery bill overnight. It was a simple, innocent-looking query, the exact kind a junior analyst writes to explore a new dataset. But that single query triggered a full table scan on a massive, unpartitioned table, and the meter just spun and spun.

Back when we were managing our own on-prem Teradata and Oracle clusters, the pain was upfront. You paid for the hardware, the power, the cooling, and the army of DBAs needed to keep it all running. We moved to the cloud to escape that management tax, only to find a whole new set of hidden ones.

The major cloud data warehouses aren't just selling you compute and storage. They are built on pricing models with hidden "taxes" that punish you for growing, for experimenting, and sometimes, even for being idle. Choosing a data warehouse today is like picking a commercial electricity plan. Some plans look incredibly cheap on paper but have massive "peak demand" charges that bankrupt you the moment you actually need the power.

After years of signing the checks and getting burned, I've decoded the pricing models of the big five: BigQuery, Snowflake, Redshift, Databricks, and Fabric. Here is exactly where the bodies are buried.

The actual storage of your data is largely a solved, commoditized problem. Across the major vendors, storage costs are cheap and highly predictable, often hovering around $23.00 per terabyte per month on-demand for Snowflake, or dropping to $0.01 per gigabyte per month for long-term storage in BigQuery. When CTOs complain about their data warehouse bills, they aren't complaining about S3 buckets. The real financial battleground is compute, concurrency, and architecture. That's where vendors make their margins.

The three hidden taxes designed to drain your cloud budget

Almost all surprise cloud costs stem from three specific pricing mechanics.

The Scan Tax punishes you for asking questions of your data. The Idle Tax punishes you for not running queries 24/7. The Complexity Tax (and its ugly cousin, Egress Fees) punishes you for not having a Ph.D. in vendor-specific billing models.

Vendor Pricing Unit Billing Minimum The Hidden Penalty Ideal Workload
Google BigQuery Pay-per-TB Scanned Per query Scan Tax: Unpredictable costs for ad-hoc exploration. Sporadic, well-defined queries on partitioned data.
Snowflake Per-second Credits 60 seconds Idle Tax: Pays for unused time on short queries. High-throughput BI and ETL with consistent, predictable usage patterns.
AWS Redshift Provisioned / Serverless RPUs 60 seconds / Hourly Idle & Complexity Tax: High operational overhead. Predictable, high-volume workloads with dedicated ops.
Databricks Databricks Units (DBUs) Opaque / Variable Complexity & Egress Tax: Obscured true cost. All-in-one data science and large-scale Spark ETL.
Microsoft Fabric Capacity Units (CUs) Opaque Complexity Tax: Obscured resource consumption. Enterprises fully committed to the Microsoft/Power BI ecosystem.
MotherDuck Compute-time only 1 second Predictable time-based billing; no scan or idle penalties. Modern cloud data warehouse for interactive BI to large-scale batch processing.

The scan tax: paying a penalty to analyze your own data

Google BigQuery, AWS Athena, and Azure Synapse Serverless rely heavily on a pay-per-TB-scanned model. The pitch is seductive, especially for startups: "You only pay for what you query."

At around $5.00 to $6.25 per terabyte processed, it sounds like a bargain, until a single poorly written query costs you thousands of dollars. It's the equivalent of going to a massive public library where you aren't charged for the book you read, but rather a fee for every single book you had to move out of the way to find it.

This model is exactly where my $50,000 bill came from. The query was devastatingly simple:

SELECT user_id, COUNT(event_id) FROM events_log GROUP BY 1;

The problem? It lacked a WHERE clause on a partitioned date column. It triggered a full scan of a petabyte-scale table.

(For the curious: serverless scan-based engines allocate compute slots to brute-force read every underlying file block from cold storage into memory if the query planner cannot prune files via a partition key. You are paying for the massive physical I/O overhead of that distributed read, regardless of how small the final result set is.)

For truly sporadic, well-defined weekly reports, this model can be cost-effective. For interactive, exploratory BI where query patterns are unpredictable by nature, you are flying blind. It forces engineering teams to become "cost police," constantly reviewing queries and enforcing strict partitioning schemes just to avoid financial catastrophe.

The idle tax: paying for compute you aren't even using

I learned about the Idle Tax the hard way while building a customer-facing analytics dashboard. We initially set our provisioned data warehouse to run 24/7, but when the first bill arrived, my jaw dropped. To save money, we aggressively configured the cluster to auto-suspend after one minute of inactivity. The cost went down, but the support tickets flooded in. Our users were suffering through 10-second "resume" latencies every time they loaded a dashboard after a few minutes of quiet. We were stuck thrashing between burning budget and ruining the user experience.

Snowflake and Amazon Redshift are the clearest examples of the idle tax in practice. Their pitch is "decoupled compute and storage," giving you production-grade scalability. You're paying for "virtual warehouses" or "RPUs" (billed per RPU-hour) that carry a hard billing minimum, often 60 seconds.

Imagine you run 30 short, 5-second queries in an hour to power a customer-facing BI dashboard. You are not billed for 150 seconds of compute. You are billed for 30 queries * 60 seconds = 1800 seconds. You just paid for 1,650 seconds of pure idle time.

It's like a taxi meter that charges you for a full mile even if you only drive one block.

This model is especially punishing for customer-facing embedded analytics or ad-hoc BI, where queries are spiky and short-lived. You are left with a terrible architectural choice: either over-provision a warehouse and eat the idle tax, or set it to auto-suspend aggressively and make your users suffer through long resume latencies.

For massive, 24/7 ETL workloads, a provisioned model can be highly efficient. The problem lies in applying it to intermittent workloads.

The complexity tax and egress fees: when you need a PhD to understand your bill

I once spent an entire week auditing our cloud bill only to discover that a junior engineer had accidentally scheduled a massive, daily production ETL job using Databricks' "All-Purpose" compute instead of the purpose-built "Jobs Compute." That single checkbox mistake silently tripled the cost of the pipeline for months. The pricing model was so opaque that nobody caught it.

This is the reality of platforms like Databricks and Microsoft Fabric. The pitch is a "unified analytics platform." The reality is a labyrinth of proprietary billing units like DBUs (Databricks Units) or CUs (Capacity Units) that are nearly impossible to map back to actual hardware consumption.

What exactly is a DBU? The answer depends on the VM type, the cloud region, and whether it's for an automated job or an interactive notebook. It's like trying to buy a car and being quoted a price per spark-plug ignition. It is a direct tax on not being a platform expert.

Alongside opaque compute units, these providers often extract massive, hidden network egress charges when you try to move data out of their ecosystem, penalizing integrations and compounding the complexity tax. vendor docs pricing page

Redshift carries its own complexity tax, requiring full-time database experts to manage Workload Management (WLM) queues and cluster resizing just to keep costs in check. This operational overhead isn't just theoretical. MotherDuck's Mega instance at $12.00/hr is 2.2x faster and 70% cheaper than a comparable 4-node Redshift ra3.16xlarge cluster, without requiring a dedicated team to manage it.

An alternative: predictable pricing with a 1-second minimum and zero idle tax

After getting burned by all three taxes, I started looking for a warehouse built on a fairer, more transparent philosophy. That's when I found MotherDuck. What I was really looking for was simple: a billing model I could explain to a finance team without a spreadsheet, and a cold-start fast enough that I'd never have to choose between saving money and not embarrassing myself in front of users. MotherDuck was the first warehouse where both of those were true at the same time.

True scale-to-zero with a 1-second minimum

MotherDuck has a strict 1-second minimum charge. If a query runs for 500ms, you are billed for 1 second. If it runs for 5 seconds, you are billed for 5 seconds. End-user compute (called "Ducklings") spins up in about 100ms, so there is no painful trade-off between saving money and delivering fast performance. The 60-second minimum waste simply does not exist.

Flat compute pricing, not a scan tax

You pay a flat, hourly rate for the compute you use (e.g., ~$0.60/hr for the Pulse instance), not a penalty based on how many terabytes a query happens to touch. You can run that full table scan without fear of a five-figure bill. The cost is predictable because it is based on execution time, a metric engineers can actually reason about and optimize.

Simple, SQL-first, no DBU math

MotherDuck bills in standard compute units that map directly to vCPU and RAM. The pricing is public, flat, and easy to understand. You don't have to deal with the JVM overhead of Spark or the convoluted cluster configurations of Databricks and Redshift. Connecting via the Python SDK takes seconds, without configuring complex IAM roles or service accounts.

Petabyte-scale without the penalty

The assumption that scale-to-zero warehouses can't handle petabyte workloads is now outdated. MotherDuck now supports petabyte-scale workloads through Managed DuckLake (in preview), giving you the same cost-predictability and ease of use whether you are querying a few gigabytes of local CSVs or petabytes of cloud data.

Code tells the story: from expensive to predictable

To control costs on a scan-based engine, you have to rewrite the query, add a WHERE clause, and pray your tables are perfectly partitioned:

-- Still risky if a user forgets the WHERE clause
SELECT user_id, COUNT(event_id)
FROM events_log
WHERE event_date = '2026-05-18' -- Must partition and filter by this!
GROUP BY 1;

Enter fullscreen mode Exit fullscreen mode

The MotherDuck equivalent: just run the query. Cost is per second of execution, not per TB scanned.

-- cost is per second of execution, not per TB scanned
SELECT user_id, COUNT(event_id)
FROM events_log
GROUP BY 1;

Enter fullscreen mode Exit fullscreen mode

Putting it to the test: matching the right model to your workload

The right architecture depends entirely on your use case.

For startups to enterprise scale: You need to avoid the idle and complexity taxes at all costs. MotherDuck is designed to grow with you. With Managed DuckLake (in preview), you can scale from gigabytes to petabytes with the same simple, scale-to-zero model. It is a highly cost-effective alternative to heavy platforms like Azure Synapse.

For customer-facing embedded analytics: You need low latency, high concurrency, and strict cost controls. ClickHouse is a strong baseline here due to its raw query speed and incredible 10x storage compression, but managing ClickHouse clusters introduces significant operational overhead. MotherDuck gives you the columnar performance benefits without the management burden. Its hypertenancy model isolates compute per user, preventing "noisy neighbors." (Each isolated user query runs inside its own secure, lightweight environment, completely decoupling one user's compute spikes from another's. You get the security and predictable performance of a single-tenant architecture with the cost efficiency of a multi-tenant one.) That predictable per-user cost model lets you offer more competitive and profitable pricing for your own SaaS product.

For ad-hoc BI and interactive dashboards: You are running lots of short, spiky queries. The 60-second minimum from Snowflake will destroy your budget. The 1-second minimum from MotherDuck saves you from paying for compute you never actually used.

Conclusion

Unpredictable data warehouse bills are a feature, not a bug, of the incumbents' pricing models. They were designed in a different era, and their business models rely heavily on the waste generated by the Scan Tax, the Idle Tax, and the Complexity Tax.

The choice of a data warehouse is an architectural decision with deep financial consequences. Choose a partner whose business model supports yours, not one that profits from your idle time or accidental table scans.

After years of fighting surprise bills, I found that a simpler, more transparent model wasn't just cheaper. It gave my team back the time we were burning on query audits and cost reviews, time we could spend building instead. That's the real cost of a bad pricing model: not just the dollar amount on the invoice, but everything your engineers stopped doing to manage it.


Frequently Asked Questions

Which serverless warehouse minimizes idle costs?

The 60-second billing minimum is what kills budgets for intermittent workloads. Every short query, a 3-second dashboard refresh, a 7-second ad-hoc lookup, gets rounded up to a full minute. Multiply that across dozens of concurrent users and you are paying for compute that never ran. A 1-second minimum with 100ms cold-start eliminates that rounding error entirely.

Which architecture provides a better price-performance ratio for spiky, intermittent query patterns?

Provisioned systems are designed for sustained, predictable throughput. When your workload is spiky, you are paying peak rates during quiet periods and scrambling during bursts. A serverless engine that bills strictly for execution time matches your actual usage curve, so your bill tracks your activity rather than your worst-case capacity estimate.

How do I avoid the complexity tax and egress fees in cloud data warehouse pricing?

The complexity tax compounds quietly. You end up needing a FinOps specialist just to interpret the bill, let alone optimize it. The cleaner path is a platform that prices in units you can reason about without a certification: vCPU time and RAM, billed at a public flat rate. Egress fees are a separate trap. If moving data out of the platform costs money, your integration architecture is constrained by your billing model, which is backwards.

What are cost-effective alternatives to BigQuery for a small data team?

The scan model is fine when you control the query patterns. Small teams rarely do. Analysts explore, iterate, and occasionally forget partition filters. A compute-time model removes that risk entirely: a runaway query costs you the seconds it ran, not the terabytes it touched. That distinction matters enormously when you don't have a dedicated data engineering team reviewing every query before it hits production.

How do I get predictable pricing to replace my unpredictable Snowflake costs?

Snowflake's 60-second minimum is invisible until you do the math. If your dashboard fires 40 short queries per session and users open it 500 times a day, you are paying for hours of compute that lasted seconds. Switching to per-second billing converts that hidden multiplier into a straightforward calculation: how long did the query actually run? That's the number on your bill.

Are there cost-effective alternatives to Azure Synapse that don't require massive price jumps when scaling?

Synapse's pricing tiers create awkward inflection points where crossing a usage threshold forces you into a much higher cost bracket. A flat compute model with no tier boundaries scales linearly: double the workload, roughly double the cost. Managed DuckLake extends that same model to petabyte-scale, so growth doesn't suddenly trigger a renegotiation with your vendor.

Can a scale-to-zero serverless warehouse handle petabyte-scale data?

Scale-to-zero and petabyte-scale were mutually exclusive until recently. The assumption was that handling large data volumes required persistent, warm infrastructure. Managed DuckLake separates compute from storage cleanly enough that you can query petabytes without keeping compute running between queries. You pay for the seconds your query runs, regardless of how much data it touches.

Which data warehouse offers the most predictable pricing for embedded analytics and customer-facing dashboards?

Predictability in embedded analytics requires two things: fast cold-starts (so you aren't paying for warm standby) and per-user compute isolation (so one customer's heavy query doesn't inflate everyone else's bill). ClickHouse wins on raw speed but demands operational investment most product teams can't justify. Hypertenancy solves the isolation problem architecturally, and a 100ms spin-up means you aren't paying to keep compute warm between user sessions.