Introduction

Modern web applications are no longer just simple static websites. Years ago, many websites were mostly:

• static pages
• blogs
• content websites
• simple request-response applications

But modern software systems are very different. Today’s applications do things like:

• video uploads
• realtime notifications
• analytics processing
• AI inference
• payment processing
• image optimization
• email delivery
• realtime chat
• activity tracking
• stream processing

Modern backend systems are now closer to continuously running software systems rather than just: “serve HTML and return response.”
And this creates a very important architectural problem.

The Problem With Long Running Tasks

Suppose a user uploads a video. Your backend now needs to:

• save the file
• generate thumbnails
• compress the video
• notify followers
• update analytics
• scan for moderation

Some of these tasks may take:

• several seconds
• minutes
• sometimes even longer

Now imagine if the user had to wait for ALL of this before receiving a response.

Client
   |
   v
Backend
   |
   +--> Compress Video
   +--> Generate Thumbnail
   +--> Send Notifications
   +--> Update Analytics
   |
   v
Response Returned

This creates a terrible user experience.

The application becomes:

• slow
• blocked
• harder to scale

Why Async Processing Exists

This is exactly why asynchronous systems exist.

Instead of making users wait:

Client -> Backend -> Everything Happens Here

modern systems usually do this:

Client -> Backend -> Queue -> Worker

The backend quickly stores a job into a queue.

Then background workers process the heavy tasks separately.

Now:

• API becomes fast
• users get immediate response
• heavy processing happens in background

This is one of the most important concepts in backend engineering.

Real World Examples

This architecture exists almost everywhere.

Sending Emails

User Signup
    |
    v
Queue Email Job
    |
    v
Email Worker Sends Email

Image Processing

User Uploads Image
    |
    v
Queue Resize Job
    |
    v
Image Worker Processes File

Payment Notifications

Order Created
    |
    v
Queue Notification Job
    |
    v
Notification Worker

What Is a Queue?

A queue is simply

A middle layer between producers and workers.

Instead of directly doing heavy work:

Backend -> Heavy Task

we place the task into a queue:

Backend -> Queue -> Worker

This gives us:

• asynchronous processing
• better performance
• better scalability
• loose coupling

Messaging Systems

To implement queues and async systems, we use messaging systems.

All of them solve similar problems differently.

Distributed Systems

As Systems Grow, Architecture Evolves. Initially, a single application may work perfectly fine.

This is called a:

Monolith architecture.

Monolith Architecture

+----------------------+
|      Monolith        |
|----------------------|
| Auth                 |
| Orders               |
| Payments             |
| Notifications        |
+----------------------+

Everything lives inside one application. This is actually completely normal. Most successful applications start this way.

But as systems grow:

• traffic increases
• teams grow
• deployments become difficult
• scaling becomes harder
• failures affect entire application

Eventually systems evolve into:

Microservices architecture.

Microservices

Instead of one giant application, each service becomes independent.

+---------+
| Auth    |
+---------+

+---------+
| Orders  |
+---------+

+------------+
| Payments   |
+------------+

+----------------+
| Notifications  |
+----------------+

Benefits:

• isolated deployments
• independent scaling
• smaller codebases
• better team ownership

But now another important problem appears.

How Do Services Communicate?

Suppose:

• Order service creates order
• Payment service charges customer
• Notification service sends email

How should they communicate?

Most beginners first think:

Order Service ---> HTTP ---> Payment Service

This works initially.

But distributed systems become difficult quickly.

Problems With Direct HTTP Communication

Imagine:

Order Service ---> Payment Service

What if:

• payment service is down?
• network becomes slow?
• retries create duplicate requests?
• traffic spikes suddenly?

Now systems become tightly coupled. One service failure can affect everything. And remember the problem we discussed earlier:

long-running tasks should not block users

That same problem exists here too.

Suppose:

• sending email becomes slow
• payment provider becomes delayed
• analytics system becomes overloaded

Should users wait? Of course not.

So even microservices need:

• asynchronous communication
• buffering
• scalable messaging systems

Event-Driven Communication

Instead of directly calling services, now Order service simply publishes an event:

Order Service ---> Message Broker ---> Payment Service

order_created

It does NOT care:

• who consumes it
• how many consumers exist
• whether consumers are temporarily offline

This creates:

• loose coupling
• better scalability
• better fault tolerance

This Is Where Kafka Comes In

Apache Kafka became extremely popular because it solves these problems at very large scale.

Kafka is heavily used in:

• analytics systems
• payment pipelines
• notification systems
• activity tracking
• realtime monitoring
• distributed systems
• stream processing

Companies use Kafka because it handles:

• huge traffic
• distributed systems
• realtime event streaming
• scalable consumers
• durable event storage

Kafka Is Not Just a Queue

This is important. Kafka can absolutely work like a queue. But Kafka is much more than that.

Kafka is fundamentally:

A distributed event streaming platform.

Meaning:

• events can be stored
• replayed later
• consumed by multiple services
• processed at massive scale

This becomes extremely powerful in modern architectures.

Kafka in One Simple Sentence

Kafka is basically:

Producer ---> Kafka ---> Consumer

Producer sends events. Consumers receive events.

Kafka stores events safely in between.

Basic Kafka Architecture

+------------+
| Producer   |
+------------+
       |
       v
+----------------+
| Kafka Topic    |
+----------------+
       |
       v
+------------+
| Consumer   |
+------------+

Important Kafka Terms

So now we will implement a very simple

Which Go Package Are We Using?

We will use:

IBM/sarama

Why?

Because:

• beginner friendly
• stable
• widely used
• pure Go
• simple learning curve

Later in the series we may also explore:

• franz-go
• async producers
• advanced performance tuning

But Sarama is perfect for learning fundamentals gradually.

Kafka Setup (Modern KRaft Mode)

Older Kafka versions required Zookeeper.

Modern Kafka supports:

KRaft mode

which removes Zookeeper completely.

We will use the modern setup.

Install Kafka (Mac)

Using Homebrew:

brew install kafka

Start Kafka:

kafka-server-start /opt/homebrew/etc/kafka/kraft/server.properties

Install Kafka (Linux)

Install Java:

sudo apt update
sudo apt install default-jdk -y

Download Kafka:

wget https://downloads.apache.org/kafka/3.9.1/kafka_2.13-3.9.1.tgz

Extract:

tar -xzf kafka_2.13-3.9.1.tgz
cd kafka_2.13-3.9.1

Start Kafka:

bin/kafka-server-start.sh config/kraft/server.properties

Create Kafka Topic

Open another terminal:

kafka-topics \
  --create \
  --topic orders \
  --bootstrap-server localhost:9092

Verify:

kafka-topics \
  --list \
  --bootstrap-server localhost:9092

You should see:

orders

Create Go Project

mkdir go-kafka-tutorial
cd go-kafka-tutorial

Initialize Go module:

go mod init github.com/<your-github-username>/go-kafka-tutorial

Install Sarama:

go get github.com/IBM/sarama@latest

Project Structure

go-kafka-tutorial/
├── producer/
│   └── main.go
├── consumer/
│   └── main.go
├── go.mod
└── go.sum

Writing Our First Producer

Create:

producer/main.go

Code:

package main

import (
    "fmt"
    "log"

    "github.com/IBM/sarama"
)

func main() {
    config := sarama.NewConfig()

    config.Producer.Return.Successes = true

    producer, err := sarama.NewSyncProducer(
        []string{"localhost:9092"},
        config,
    )

    if err != nil {
        log.Fatal(err)
    }

    defer producer.Close()

    message := &sarama.ProducerMessage{
        Topic: "orders",
        Value: sarama.StringEncoder("new order created"),
    }

    partition, offset, err := producer.SendMessage(message)

    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf(
        "message sent to partition %d at offset %d\n",
        partition,
        offset,
    )
}

Writing Our First Consumer

Create:

consumer/main.go

Code:

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/IBM/sarama"
)

type Consumer struct{}

func (Consumer) Setup(sarama.ConsumerGroupSession) error {
    return nil
}

func (Consumer) Cleanup(sarama.ConsumerGroupSession) error {
    return nil
}

func (Consumer) ConsumeClaim(
    session sarama.ConsumerGroupSession,
    claim sarama.ConsumerGroupClaim,
) error {

    for message := range claim.Messages() {
        fmt.Printf(
            "received message: %s\n",
            string(message.Value),
        )

        session.MarkMessage(message, "")
    }

    return nil
}

func main() {
    config := sarama.NewConfig()

    group, err := sarama.NewConsumerGroup(
        []string{"localhost:9092"},
        "order-group",
        config,
    )

    if err != nil {
        log.Fatal(err)
    }

    defer group.Close()

    consumer := Consumer{}

    for {
        err := group.Consume(
            context.Background(),
            []string{"orders"},
            consumer,
        )

        if err != nil {
            log.Fatal(err)
        }
    }
}

Running the Application

Start consumer first:

go run consumer/main.go

Now in another terminal:

go run producer/main.go

Consumer output:

received message: new order created

You just built your first Kafka publisher/subscriber system using Go.

Conclusion

In this part we learned:

why async systems exist
long running task problems
queues and background workers
distributed systems basics
monolith vs microservices
service communication problems
messaging systems
Kafka fundamentals
creating producer and consumer using Go

Most importantly:

You now understand:

WHY Kafka exists.

That foundation matters much more than memorizing APIs.