title: "Welcome to the Distributed Systems World — The Challenges Nobody Warned You About"
published: false
description: A friendly tour of the six big headaches that come with distributed systems, with small .NET examples.
tags: distributedsystems, dotnet, architecture, beginners
series: Fundamentals of Distributed Systems
cover_image:

You wrote your first ASP.NET MVC app. Controller, service, a SQL database — beautiful. It runs on your laptop, it runs on one server, life is good.

Then somebody says the word "microservices" in a meeting and suddenly there's talk of replication and CAP theorems and consensus algorithms named after Greek philosophers. Cool, cool, cool. Everyone is nodding. You are also nodding.

This series is for that moment. We're going to walk through the world of distributed systems together — slowly, with pictures, with small bits of .NET code, and without pretending any of it is obvious. By the end you'll actually know what those words mean, and more importantly, you'll know why they exist.

This first article is the lay of the land: what a distributed system even is, and the six big challenges that show up the moment you have more than one machine.

Let's go.

So… what is a distributed system?

The textbook definition is annoying, so here's the human one:

A distributed system is a bunch of computers that work together so that, to the user, they look like one computer.

That's it. That's the whole idea.

Think about what happens when you order food on an app. You tap a button. Two hundred milliseconds later you see "Order placed."

In that 200 ms, this happened behind the scenes:

• A load balancer picked one of dozens of identical web servers to handle your tap
• Your request hit a payments service that might literally be in another country
• The order got written to a database that's split across many machines
• A message landed in a queue telling the restaurant's screen to start beeping
• A cache somewhere updated so the next person sees the right available stock

A whole little conversation between machines. You saw one button. They had a committee meeting.

That conversation — coordinating many independent machines to behave like one trustworthy system — is the entire field of distributed systems. And it is genuinely hard.

Why don't we just… not?

Fair question. One machine is way simpler. So why does anyone leave?

Three reasons:

1. Scale. One machine has a ceiling. Best CPU, max RAM, fastest disk — there's still a top. The moment your traffic punches through it, your only move is more machines.

2. Availability. Single machines die. Hardware fails, kernels panic, somebody trips on a power cable. If your whole app lives on one box, your whole app dies with it. With many, the survivors keep serving.

3. Geography. Light is fast but not infinite. A user in Tokyo hitting a server in Virginia waits roughly 150 ms just for the round trip — before the server does any work. To feel "instant" worldwide, you need machines close to people.

So we go distributed. And in exchange we inherit every problem in the rest of this article. Distribution is not free. You're trading simplicity for capability.

Now, the challenges.

Challenge 1: The network is a liar

Inside one process, when you call a method, it works. The CPU doesn't "lose" your function call. Methods don't sometimes arrive twice. Life is sane.

The network is not sane.

The moment your code talks to another machine, your message can be:

• Lost — never arrives, no error, no nothing
• Delayed — arrives five seconds after you gave up waiting
• Duplicated — the same request lands twice (and your "charge the credit card" endpoint runs twice 🙃)
• Out of order — request #2 arrives before request #1

There's a famous thought experiment called the Two Generals Problem: two generals on opposite hills need to attack at the same time. They send messengers through enemy territory. The brutal twist: there is mathematically no way for both to be 100% sure the other one got the message. Not "hard." Actually impossible.

That's the network. Every cross-machine call lives with that uncertainty.

Here's what this looks like in practice. The bad version we've all written:

// Pretending the network is reliable. It isn't.
var result = await _httpClient.GetAsync("/api/orders/42");
var order = await result.Content.ReadFromJsonAsync<Order>();

The version that respects reality:

try
{
    using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(3));
    var result = await _httpClient.GetAsync("/api/orders/42", cts.Token);
    result.EnsureSuccessStatusCode();
    return await result.Content.ReadFromJsonAsync<Order>();
}
catch (TaskCanceledException)   { /* timed out — now what? retry? fail? */ }
catch (HttpRequestException)    { /* the connection died — same question */ }

Notice the comments. The catch blocks aren't really about logging — they're about decisions. Do I retry? How many times? If I retry a "charge card" call, am I charging twice?

We'll spend a whole article on this later — timeouts, retries with backoff, idempotency keys, circuit breakers. The toolbox for living with a lying network.

For now, the takeaway is just: the network will betray you. Code defensively.

Challenge 2: Many copies, one truth

Okay, so now you've got many machines holding your data. That's great — if one dies, the others still have it. But it creates a new problem nobody warned you about.

If I write X = 5 to machine A, and you read X from machine B before A has told B about the change, what do you see?

The old value? The new one? An error? Pick one and you have to live with the consequences.

The standard solution is a pattern called leader-follower replication, and underneath it sits a family of algorithms called consensus — Paxos, Raft, Zab. Different names, same vibe:

1. One node is elected the leader. Everybody agrees on who it is.
2. All writes go through the leader. No exceptions.
3. The leader writes the change to its own log, then ships it to all followers.
4. The leader waits until the majority of followers say "yep, got it."
5. Then it commits and tells the client "done."

That "majority" rule is the magic. It's called a quorum, and it guarantees that two conflicting decisions can never both win — even when some machines crash, even when the network is being weird.

The cool part? You're already using this. You just don't think about it:

• SQL Server Always On availability groups → leader-follower
• PostgreSQL streaming replication → leader-follower
• MongoDB replica sets → leader-follower
• Kafka ISR (in-sync replicas) → leader-follower-ish

Same picture, different sticker.

Challenge 3: How do you grow?

Traffic doubles. Then doubles again. You have two moves:

Scale up (vertical) — make the machine bigger. More CPU, more RAM, faster disk. Your code doesn't change at all. Easy.

The downside? There's a biggest machine on Earth, and you can hit it. Also, no matter how huge it is, it's still one machine. When it dies, everything goes with it.

Scale out (horizontal) — add more machines and split the work across them. Growth is basically unlimited. One dies? The others shrug and keep going.

The downside? Now your code has to handle coordination. State, sessions, locks, ordering — all the things that were free on one machine now need real engineering.

For an ASP.NET MVC app, scaling out usually means three things:

Web tier: run many copies of your app behind a load balancer. Easy if your controllers are stateless. The classic gotcha is in-memory session — works perfectly with one server, breaks horribly with two. The fix:

// Program.cs — sessions that survive across many instances
builder.Services.AddStackExchangeRedisCache(o =>
{
    o.Configuration = builder.Configuration.GetConnectionString("Redis");
});
builder.Services.AddSession();
// Now any web server can serve any user, any time.

Service tier: break the monolith into smaller services that scale independently. Maybe payments needs 10× the instances and search needs 2×. You can't do that when they're all one app.

Data tier: read replicas for read-heavy workloads, and partitioning for write-heavy ones — which is the next challenge.

Challenge 4: When the load isn't fair

So your data outgrew one machine. You split it across many — this is called partitioning (or sharding, same thing). A router decides which machine handles which request, usually by hashing some key like userId.

Beautiful. In theory.

In practice, here's what happens. You shard your users across three partitions. Then one influencer with 50 million followers does a livestream. Every request involving that user — their profile, their feed, their comments, their everything — slams partition #2. Partition #1 is bored. Partition #3 is napping. Partition #2 is on fire.

This is the hot partition problem, and it's one of the most common ways distributed systems fail in production. Not because the total load is too high. Because the load is uneven.

How do you fix it? A few moves:

• Pick a high-cardinality key. userId is great. country is terrible (good luck when one country has 90% of your traffic).
• Hash the key. Don't shard by userId directly — shard by hash(userId). The hash spreads things out evenly.
• Split the hot keys. For known whales, write their data across sub-partitions (userId#1, userId#2, …). This is called write-sharding.
• Put a cache in front. Hot reads never even hit the database.

We'll dedicate a whole article to this — consistent hashing, range vs. hash sharding, the nightmare joy of rebalancing data while the system is still serving traffic. It's a deep topic.

Challenge 5: Testing this stuff is genuinely brutal

A bug in a regular app is a bug. A bug in a distributed system is a bug that:

• Only shows up under load
• Only on machine #17
• Only after it's been running for 11 days
• Only when the network adds 200 ms of latency
• At 3 AM
• During a holiday

This is why people obsess over the testing pyramid:

• Many unit tests at the bottom. Fast, pure logic, no I/O. xUnit or NUnit. They run in milliseconds. They catch the dumb stuff before it leaves your laptop.
• Some integration tests in the middle. Your code talking to a real database, a real cache, a real queue. Slower, more useful.
• A few end-to-end tests at the top. The whole system, real browser, real network. Playwright is the go-to. Slow, expensive, but they catch the things nothing else can.

The ratio matters. 70/20/10 is a healthy default. Flip it (lots of E2E, few units) and your build takes 45 minutes and everyone gives up on running tests.

A really clean trick in .NET is TestContainers — it spins up actual SQL Server, Redis, Kafka, whatever, in Docker, just for your test run:

var db = new MsSqlBuilder()
    .WithImage("mcr.microsoft.com/mssql/server:2022-latest")
    .Build();

await db.StartAsync();
// Run integration tests against a *real* SQL Server.
// When tests finish, the container disappears.

No more "works on my machine because I have SQL Server installed but Jenkins doesn't."

And beyond normal tests, distributed systems need chaos testing — deliberately killing nodes, dropping packets, freezing clocks. Netflix made this famous with Chaos Monkey. We'll cover it in the resilience article.

Challenge 6: Shipping is half the job

Here's the thing nobody tells you when you start: writing the code is maybe half the work. The other half is keeping it running 24/7 across all those machines.

You need three things working together:

A CI/CD pipeline. Every commit gets built, tested, packaged into a container, and deployed automatically. GitHub Actions, Azure DevOps, GitLab CI — pick one. The rule is: humans are not allowed to SSH into production and "just fix it." That way lies madness.

Observability — the three pillars. When something breaks in production at 2 AM, you need to know what broke and why, fast:

• Logs — what happened. Use Serilog with structured JSON output, not Console.WriteLine.
• Metrics — numbers over time. CPU, requests per second, error rate, p95 latency. Prometheus, Datadog, Application Insights.
• Traces — one user request followed across every service it touched. OpenTelemetry is the standard.

The OpenTelemetry setup is honestly absurd how easy it is:

// Program.cs — distributed tracing in three lines
builder.Services.AddOpenTelemetry()
    .WithTracing(t => t
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSqlClientInstrumentation()
        .AddOtlpExporter());

Now every request through your MVC app produces a trace that shows every outgoing HTTP call and SQL query, with timings. When someone complains "the page is slow," you stop guessing and start looking.

Alerting and on-call. Metrics are useless if nobody's watching. You set thresholds, route alerts to Slack / PagerDuty / Opsgenie, and someone is on call to respond. This is honestly more of a culture problem than a tech problem, but we'll get into it.

Quick recap

Six challenges. None of them go away. All of them are solvable.

The honest summary: distributed systems aren't magic, they're just a long list of trade-offs. Once you see the trade-offs clearly, the technology choices start making sense.

What's next

Next article we dig into communication between services — synchronous (REST, gRPC) vs. asynchronous (queues and events). When to use which, and we'll build a small ASP.NET MVC app that uses both.

After that: data and storage, the CAP theorem (without the confusing diagrams), resilience patterns, and observability in depth.

If something in this article felt fuzzy — good. That means we have something to dig into. Drop a comment with the challenge you want explored first and I'll bump it up the queue.

See you in part 2. 👋

Follow the series for more — every article = simple language, lots of diagrams, small .NET examples.