Prologue - What is Large-scale Processing?

Published: March 18, 2026

3 AM, October 2025. A single DNS configuration error on an AWS server brought Snapchat, Roblox, and McDonald's to a standstill. 3,500 companies across 60 countries were stopped cold by one small crack.

Systems are far more fragile than we think. Large-scale processing isn't a trend about boosting server specs. It's the engineering discipline that keeps services alive at the edge of their limits.

So where does "large-scale" actually begin? 10,000 users? A million? That's the wrong question. Large-scale isn't a number. It's the moment a system hits the ceiling of its available resources. That's why what's a normal Tuesday for Amazon can be a catastrophe for a growing startup.

This series is about how to detect that ceiling, understand why systems break, and build things that hold.

The Signals Before a System Breaks

Systems don't collapse without warning. There are always signs. The Google SRE team calls them the Four Golden Signals.

ref. Google SRE: Monitoring Distributed Systems

Latency: How long does it take to handle a request? A gap between successful and failed response times is often the first sign something's wrong.
Traffic: How much demand is hitting the system right now? Think RPS — requests per second.
Errors: How many requests are failing? Explicit 500s, silent wrong responses — both count.
Saturation: How "full" is the system? This is the most direct signal of large-scale stress. When latency starts climbing, saturation is usually already on its way up.

If any one of these looks off, the system is already approaching its limit.

So What Exactly is "Large"?

The Golden Signals tell you the state of a system. But to actually fix things, you need to understand the nature of the load. The same word — "large-scale" — means something completely different depending on what's overwhelming the system.

Traffic (Too many requests)
How many requests per unit time? How many connections can the system hold?

TPS / QPS: Transactions or queries per second. The real measure of system throughput.
Concurrency: Simultaneous active connections. The deciding factor during flash sales or ticketing rushes.

Volume (Too much data)
How large is the data, and how fast does it need to move?

Throughput: Data transferred per second (MB/s). The usual bottleneck in video streaming or large file uploads.

Complexity (Too hard to process)
How much computation does a single request require? How many systems does it touch?

Logic Latency: The more complex the logic, the slower the response — and the faster saturation spikes.

Real outages usually involve all three at once. But if you can't separate the causes, you can't fix them.

Where Business Thinking Meets Engineering

Picture a factory floor. One slow machine holds up the entire line. It doesn't matter how fast everything else runs.

Eliyahu M. Goldratt formalized this as the Theory of Constraints (TOC): *"The throughput of any system is determined by its weakest link — the Constraints."

ref. Lean Enterprise Institute: TOC

Servers work the same way. The point where Saturation hits 100% first — that's the Bottleneck. Large-scale engineering is about finding which component saturates first as traffic grows, then eliminating that constraint with the right strategy.

When You Hit a Wall

Once you've found the bottleneck, you need to increase capacity. There are two ways to do it.

ref. GeeksforGeeks: Vertical and Horizontal Scaling

Vertical Scaling (Scale-up): Upgrade the single node — more CPU, more RAM. Fast to implement, but there's a ceiling. And it's expensive.
Horizontal Scaling (Scale-out): Add more nodes and distribute the load. More complex, but theoretically limitless.

  [Single Server]         [Multiple Servers]

  ┌─────────────┐         ┌───┐ ┌───┐ ┌───┐
  │   CPU ↑↑↑   │         │ S │ │ S │ │ S │
  │   RAM ↑↑↑   │    →    │ 1 │ │ 2 │ │ 3 │
  │   SSD ↑↑↑   │         └───┘ └───┘ └───┘
  └─────────────┘           Load Balancer

      Scale-up                Scale-out
    (Has limits)        (Infinitely expandable)

Scale-up buys simplicity at the cost of a ceiling.

Scale-out removes the ceiling at the cost of complexity.

Neither is the right answer. There's only the right trade-off for the constraint you're solving.

What's Next

At the end of the day, large-scale processing is Strategic Bottleneck Management — controlling Latency and Errors by managing Saturation.

ref. AWS Well-Architected Framework

Next up: a single HTTP request makes its way to a server by passing through 7 layers — the OSI model. We'll trace that journey and see exactly where large-scale traffic creates bottlenecks at each layer, and what engineers have done about it.

推荐订阅源