If you’ve ever run docker run hello-world, you’ve witnessed magic. A container spins up in milliseconds, isolated from your host, with its own filesystem, network, and process tree. But what actually happens when you hit Enter?

As DevOps engineers, we often treat Docker as a black box. We write YAML, push images, and hope for the best. But when things break—when a container won’t start, when disk space vanishes, or when networking fails—knowing the internals isn’t just "nice to have." It’s survival.

In this deep dive, we’ll peel back the layers of Docker. We’ll explore how Linux kernel primitives like OverlayFS and Namespaces power containers, decode the mysterious {} syntax in docker inspect, and learn how to troubleshoot like a pro.

1. The Image: It’s Not a File, It’s a Stack

Most people think a Docker image is a single binary. It’s not. An image is a stack of read-only layers.

How It Works

Every instruction in your Dockerfile (RUN, COPY, ADD) creates a new layer. These layers are stored in a content-addressable storage system (usually under /var/lib/docker/overlay2/).

• Union File System: Docker uses a Union File System (specifically OverlayFS) to merge these layers into a single view.
• Content Addressability: Each layer is identified by a SHA256 hash. If two images share the same base (e.g., ubuntu:22.04), they share the same disk space. No duplication.

The JSON Truth

You can see this structure using docker image inspect. The output is a JSON object containing:

• RootFS: A list of layer hashes.
• History: The build steps that created each layer.

docker image inspect nginx:latest | jq '.[0].RootFS.Layers'

Key Insight: Images are immutable. You never change an image; you only create new ones with additional layers on top.

2. The Container: Just a Process, But Isolated

A running container is simply a process on your host Linux kernel. But it’s a special process, wrapped in isolation primitives.

The Linux Kernel Primitives

Docker doesn’t invent virtualization; it leverages existing Linux features:

1. Namespaces: Provide isolation.
   • PID: The container sees only its own processes.
   • NET: Its own network stack (IPs, ports).
   • MNT: Its own filesystem view.
   • UTS: Its own hostname.
2. Control Groups (cgroups): Limit resource usage (CPU, Memory, I/O). This prevents one container from starving the host.
3. OverlayFS Mount: The writable layer (the "container layer") is mounted over the read-only image layers.

Inspecting the Reality

When you run docker inspect <container_id>, you’re querying the Docker Daemon’s internal state database. The JSON output reveals the truth:

{
  "State": {
    "Status": "running",
    "Pid": 12345,
    "ExitCode": 0
  },
  "HostConfig": {
    "Memory": 536870912,
    "CpuShares": 1024
  }
}

Key Insight: docker run is essentially a wrapper around runc (the OCI runtime), which sets up namespaces, cgroups, and mounts, then executes your entrypoint.

3. Networking: Virtual Wires and Bridges

How does a container talk to the world? Through Linux Network Namespaces and Virtual Ethernet (veth) pairs.

The Default Bridge

By default, Docker creates a Linux bridge (docker0) on the host.

1. Each container gets a virtual interface (eth0) inside its network namespace.
2. This eth0 is connected via a veth pair to the docker0 bridge.
3. NAT (iptables): Outbound traffic is masqueraded so it appears to come from the host IP.

Inspecting Networks

docker network inspect bridge

This returns JSON showing which containers are attached, their IPs, and MAC addresses.

Key Insight: Containers on the same bridge can communicate via IP. For name resolution, Docker runs an embedded DNS server at 127.0.0.11.

4. The Power of {{}}: Mastering docker inspect --format

One of Docker’s most powerful but underused features is the --format (or -f) flag. It uses Go’s text/template engine to parse JSON output directly in the CLI.

Why Use It?

docker inspect returns massive JSON blobs. Often, you just need one field. Parsing JSON with jq is great, but --format is built-in and faster for simple queries.

Syntax Rules

• {{ }}: Delimiters for template expressions.
• .: Represents the current object.
• {{.Field}}: Accesses a key.
• {{.Parent.Child}}: Traverses nested objects.
• {{index .Array 0}}: Accesses array elements (Go templates don’t support [0]).

Real-World Examples

Get Container IP:

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' my-container

List Environment Variables:

docker inspect -f '{{range .Config.Env}}{{.}}{{"\n"}}{{end}}' my-container

Check if Running:

docker inspect -f '{{if .State.Running}}✅ Up{{else}}❌ Down{{end}}' my-container

Format Mounts Like JSON:

docker inspect -f '{{range .Mounts}}
{
  "Type": "{{.Type}}",
  "Source": "{{.Source}}",
  "Destination": "{{.Destination}}"
}
{{end}}' my-container

Pro Tip: Always wrap your template in single quotes ('{{...}}') to prevent the shell from interpreting special characters.

5. Storage Deep Dive: OverlayFS & Copy-on-Write

Where do files live? How does Docker save space? The answer is OverlayFS.

The Layer Cake

Imagine your image has 3 layers:

1. Base OS (ubuntu)
2. App Dependencies (nginx)
3. Your Code (app)

When you start a container, Docker adds a 4th layer: The Writable Layer.

Copy-on-Write (CoW)

• Read: If a container reads a file from the image, it reads directly from the read-only layer. Fast.
• Write: If the container modifies a file, Docker performs a copy-up:
  1. Copies the file from the read-only layer to the writable layer.
  2. Modifies the copy in the writable layer.
  3. The original remains untouched.

⚠️ Performance Warning: CoW copies the entire file, not just changed blocks. Modifying a 1GB log file triggers a 1GB copy. This is why you should use Volumes for databases and heavy I/O.

Directory Structure

/var/lib/docker/overlay2/
├── <layer-hash>/
│   ├── diff/       # Files unique to this layer
│   ├── lower       # Link to parent layers
│   ├── merged/     # The unified view (mounted into container)
│   └── work/       # Internal workspace

6. Troubleshooting Like a Senior Engineer

When things go wrong, don’t guess. Inspect.

Step 1: Check Status

docker ps -a

Look at the STATUS and EXIT CODE.

• 0: Clean exit.
• 137: OOM Killed (Out of Memory).
• 125: Docker daemon error.

Step 2: Read Logs

Logs are stored in JSON files under /var/lib/docker/containers/<id>/<id>-json.log.

docker logs --tail 100 -f my-container

Step 3: Exec Into the Container

If the container is running but behaving strangely:

docker exec -it my-container /bin/sh

Now you can check files, network connectivity (curl, ping), and environment variables inside the namespace.

Step 4: Check Resources

docker stats

Real-time CPU, memory, and I/O usage. If a container is using 100% CPU, you’ll see it here.

Step 5: Disk Space

docker system df

Identify unused images, containers, and volumes. Clean up safely with:

docker system prune --dry-run  # Always dry-run first!

Conclusion

Docker is not magic. It’s a clever orchestration of Linux kernel features:

• OverlayFS for efficient, layered filesystems.
• Namespaces for isolation.
• cgroups for resource control.
• Go Templates for powerful CLI inspection.

Understanding these internals transforms you from a Docker user into a Docker engineer. You stop guessing why a container failed and start knowing exactly where to look.

Next time you run docker run, remember: you’re not just starting an app. You’re creating a namespace, mounting a union filesystem, and isolating a process—all in milliseconds.

📚 Further Reading

• Docker Storage Drivers
• Linux Kernel OverlayFS Documentation
• Go Text Template Package

Happy Containerizing! 🐳