Every engineer has a war story about a deployment gone wrong.

Mine came before I built this project. A Friday afternoon release, a ten-minute window where the app was completely unreachable, and an inbox full of user complaints by the time it recovered. The fix was already deployed. The damage was already done.

I built this project to make sure that never happens again.

What you are about to read is a complete, honest walkthrough of how I built a blue-green deployment pipeline on AWS EKS from scratch, on Ubuntu, with nothing but a terminal and determination. I will show you the actual commands, the real challenges I hit, and the exact solutions that fixed them. By the end you will have a working mental model and a repository you can fork today.

The pipeline averages 29 seconds end to end. The traffic switch itself takes under one second. Rollback takes under five seconds. And I have the screenshots to prove every single claim.

What Blue-Green Deployment Actually Means

Before we touch a single terminal command, let us get the concept locked in because it is simpler than most articles make it sound.

You run two identical production environments at all times. One is called blue. One is called green. At any given moment, one is live (serving all real user traffic) and the other is idle (running, healthy, but not serving anyone).

When you want to release a new version:

1. Deploy the new version to the idle environment
2. Verify it is healthy
3. Flip a switch that moves all traffic from live to idle

That switch is instantaneous. Users experience nothing. If the new version has a critical bug, you flip the switch back. Rollback is not a redeploy. It is the same switch command in reverse.

On Kubernetes, that switch is a single field change in a Service manifest. That is genuinely all it is.

kubectl patch service bluegreen-service \
  -p '{"spec":{"selector":{"app":"bluegreen-app","version":"green"}}}'

One command. One second. Zero downtime.

The Stack

Before we dive into the build, here is what the complete system uses:

The application itself is intentionally minimal. A Node.js Express server with two endpoints: /health and /. The health endpoint is what makes everything else possible. The main page displays a giant color and version number so the traffic switch is visually obvious in a browser.

The Architecture

Here is how every component connects:

Developer
    |
    | git push to main
    v
GitHub Actions (29 seconds average)
    |
    +-- Configure AWS credentials
    +-- Log in to Amazon ECR
    +-- Connect kubectl to EKS
    +-- Detect idle environment
    +-- Build image and push to ECR
    +-- Deploy to idle environment
    +-- Health check idle pods
    +-- Switch traffic (the selector patch)
    |
    v
Amazon EKS Cluster
    |
    +-- NGINX Ingress (public AWS ELB URL)
            |
            v
    Kubernetes Service (selector: version=blue OR version=green)
            |
            +-- Blue Deployment  (2 pods, v1.0.0)
            +-- Green Deployment (2 pods, v2.0.0)

The public URL never changes. The NGINX Ingress forwards to the Service. The Service routes to whichever environment the selector points at. Everything else is just automation around that one field.

The Application

const express = require("express");
const app = express();

const PORT        = process.env.PORT        || 3000;
const APP_VERSION = process.env.APP_VERSION || "v1.0.0";
const APP_COLOR   = (process.env.APP_COLOR  || "blue").toLowerCase();
const COLOR_HEX   = APP_COLOR === "green" ? "#2C9E5E" : "#1C6EA4";

app.get("/health", (req, res) => {
  res.status(200).json({
    status:  "healthy",
    color:   APP_COLOR,
    version: APP_VERSION,
  });
});

app.get("/", (req, res) => {
  // Returns full-page HTML showing which environment served this request
  // The giant color and version number make the switch visually undeniable
});

app.listen(PORT, () =>
  console.log(`color=${APP_COLOR} version=${APP_VERSION} port=${PORT}`)
);

APP_COLOR and APP_VERSION are injected by Kubernetes from the Deployment manifest. When blue is live, every response carries "color":"blue". The moment traffic switches, responses immediately return "color":"green". This is what you watch during the live demo.

Why the Health Endpoint Matters So Much

Kubernetes readiness probes call /health on a schedule. A pod is only added to the Service routing pool after its readiness probe passes. This means Kubernetes will never route traffic to a pod that has not already proven it can respond correctly.

But that is not enough on its own. The CI/CD pipeline also calls /health manually from inside an idle pod before touching the Service selector. Two independent verification layers before any user traffic moves.

The Dockerfile

# Stage 1: install dependencies only
FROM node:20-alpine AS deps
WORKDIR /app
COPY package*.json ./
RUN npm install --omit=dev

# Stage 2: runtime image
FROM node:20-alpine AS runtime
WORKDIR /app

# Non-root user for security
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN chown -R appuser:appgroup /app
USER appuser

EXPOSE 3000
ENV PORT=3000 APP_COLOR=blue APP_VERSION=v1.0.0 NODE_ENV=production

CMD ["node", "server.js"]

Three decisions worth explaining.

First, the multi-stage build. The final image contains only what is needed to run the app. Build tools, npm cache, and intermediate artifacts from the deps stage are discarded. Smaller image, faster pulls, smaller attack surface.

Second, the non-root user. Running containers as root is a well-documented security risk. Creating a dedicated user takes four lines and meaningfully limits blast radius if the application is ever compromised.

Third, no HEALTHCHECK instruction. We rely on Kubernetes probes instead because they integrate directly with the pod lifecycle and the rollout gating logic.

The Kubernetes Manifests

Four files. Each has a specific, non-overlapping responsibility.

deployment-blue.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bluegreen-blue
  labels:
    app: bluegreen-app
    version: blue
spec:
  replicas: 2
  selector:
    matchLabels:
      app: bluegreen-app
      version: blue
  template:
    metadata:
      labels:
        app: bluegreen-app
        version: blue
    spec:
      containers:
        - name: bluegreen-app
          image: 677276115158.dkr.ecr.us-east-1.amazonaws.com/bluegreen-app:blue
          env:
            - name: APP_COLOR
              value: "blue"
            - name: APP_VERSION
              value: "v1.0.0"
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "250m"
              memory: "256Mi"
          readinessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 15

The version: blue label is everything. The Service selector matches on it. Resource limits are non-negotiable in production. Without them a misbehaving pod can consume all node resources and starve other workloads.

The green Deployment is identical except for version: green, APP_COLOR=green, and APP_VERSION=v2.0.0.

service.yaml (The Traffic Switch)

apiVersion: v1
kind: Service
metadata:
  name: bluegreen-service
spec:
  type: ClusterIP
  selector:
    app: bluegreen-app
    version: blue   # THE SWITCH. Change to "green" to move all traffic.
  ports:
    - port: 80
      targetPort: 3000

This is the entire mechanism. Change version: blue to version: green and every new connection goes to green. In-flight connections on blue drain naturally. The pipeline does this with:

kubectl patch service bluegreen-service \
  -p '{"spec":{"selector":{"app":"bluegreen-app","version":"green"}}}'

The CI/CD Pipeline

This is where everything comes together. The full deploy.yml workflow runs on every push to main.

The Detect Idle Step (The Heart of the Pipeline)

LIVE=$(kubectl get service bluegreen-service \
  -o jsonpath='{.spec.selector.version}' 2>/dev/null || echo "blue")

if [ "$LIVE" = "blue" ]; then IDLE="green"; else IDLE="blue"; fi

echo "live=$LIVE" >> $GITHUB_OUTPUT
echo "idle=$IDLE" >> $GITHUB_OUTPUT

Rather than hard-coding which environment to deploy to, the pipeline reads the current cluster state and computes the answer. Run it a hundred times and it always deploys to the correct idle environment. The || echo "blue" fallback handles first-time runs where the Service does not exist yet.

The Health Check Gate

POD=$(kubectl get pods -l version=${IDLE} \
  -o jsonpath='{.items[0].metadata.name}')

kubectl exec "$POD" -- wget -qO- http://localhost:3000/health | grep '"status":"healthy"'

Running the health check with kubectl exec means the request goes directly to the pod's localhost, bypassing the Service and Ingress entirely. If grep does not find "status":"healthy", it exits non-zero, the pipeline step fails, and the traffic switch step never runs. Users see nothing.

The Traffic Switch

kubectl patch service bluegreen-service \
  -p "{\"spec\":{\"selector\":{\"app\":\"bluegreen-app\",\"version\":\"${IDLE}\"}}}"

echo "Traffic is now live on: ${IDLE}"

That single patch command is the entire release. Everything before it was verification. Everything after it is cleanup.

The Challenges I Actually Hit (And How I Fixed Them)

Challenge 1: AWS gives you a hostname, not an IP address

Every tutorial I found used jsonpath='{.status.loadBalancer.ingress[0].ip}' to get the Ingress address. On AWS, that returns empty. AWS ELB gives you a hostname like:

ae91af9007367451cbf51b4cfb8fb340-1660787795.us-east-1.elb.amazonaws.com

I spent time staring at curl: (6) Could not resolve host: health before realising the INGRESS_IP variable was empty and curl was trying to resolve health as a hostname.

Fix: Use .hostname instead of .ip:

export INGRESS_HOST=$(kubectl get ingress bluegreen-ingress \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

curl http://$INGRESS_HOST/health

Store it as INGRESS_HOST not INGRESS_IP so the name never misleads you.

Challenge 2: Pods stuck in ImagePullBackOff

After deploying the manifests, all four pods hit ImagePullBackOff. The cluster could not pull the images from ECR because the node IAM role had no permission to access the private registry.

This is the step most tutorials either skip or bury in a footnote.

Fix: Find the node instance role and attach the ECR read policy:

ROLE_NAME=$(aws iam list-roles \
  --query "Roles[?contains(RoleName,'NodeInstanceRole')].RoleName" \
  --output text)

aws iam attach-role-policy \
  --role-name  "$ROLE_NAME" \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

No credentials hard-coded anywhere. The node role handles authentication transparently.

Challenge 3: The workflow files were never actually in the repo

I pushed the project to GitHub, went to the Actions tab, and saw the "Get started with GitHub Actions" page. The workflow files existed on my machine but the .github/workflows/ folder was empty because I had created the files in a downloaded ZIP earlier and never actually written them on the Ubuntu machine.

Fix: Create the files directly on Ubuntu using heredocs and explicitly add the hidden folder:

cat > .github/workflows/deploy.yml << 'EOF'
# ... workflow content
EOF

git add .github/
git commit -m "Add GitHub Actions CI/CD workflows"
git push

Always verify with ls -la .github/workflows/ before pushing. Hidden folders are easy to miss.

Challenge 4: The curl loop did not capture the switch moment

During the first demo run, I started the curl loop after the switch had already happened. Terminal 1 only showed green responses. No evidence of the transition.

Fix: Start the monitoring loop first, then trigger the switch. The loop must already be running before the patch command fires. With sleep 1 between requests, the transition window is narrow. Setting sleep to 0.5 or removing it entirely makes the switch moment easier to capture.

Proving Zero Downtime Live

This is the most important part of the project and the most satisfying moment.

Open two terminals.

Terminal 1:

while true; do
  curl -s http://$INGRESS_HOST/health
  echo
  sleep 1
done

Terminal 2:

kubectl patch service bluegreen-service \
  -p '{"spec":{"selector":{"app":"bluegreen-app","version":"green"}}}'

Watch Terminal 1. You will see:

{"status":"healthy","color":"blue","version":"v1.0.0"}
{"status":"healthy","color":"blue","version":"v1.0.0"}
{"status":"healthy","color":"green","version":"v2.0.0"}
{"status":"healthy","color":"green","version":"v2.0.0"}

The color changes. The status never wavers. Not a single failed request. That line where blue becomes green is your zero-downtime proof.

Open the same URL in a browser before and after. Full blue page becomes full green page. Same URL. Same cluster. Different version. Zero downtime.

The GitHub Actions Pipeline in Action

After pushing the workflow files, every commit to main triggers the full pipeline automatically. Here is what a successful run looks like:

Set up job              1s
Checkout                1s
Configure AWS credentials   0s
Log in to Amazon ECR    1s
Connect kubectl to EKS  4s
Detect idle environment 3s
Build and push image to ECR  10s
Deploy to idle environment   3s
Health check idle pods  2s
Switch traffic          1s
Complete job            0s

Total: 29 seconds

Every step green. Every step automated. No manual intervention. No SSH into servers. No hand-run scripts.

The rollback workflow is equally clean. Go to Actions, select Rollback, choose the target environment, click Run workflow. Traffic switches back in under 5 seconds.

Key Takeaways

The mechanism is simpler than you think. The entire blue-green strategy is one label selector field in a Kubernetes Service. Everything else is automation around that field.

Dynamic detection beats hard-coding. Reading the live cluster state to find the idle environment makes the pipeline self-correcting and safe to run repeatedly without manual tracking of which environment is currently live.

Two verification layers beat one. Kubernetes readiness probes are necessary but not sufficient. An explicit health check from inside the idle pod, before the selector patch, adds a second independent gate that catches edge cases probes miss.

AWS is not Azure. If you have done this on Azure before, expect a few AWS-specific surprises: hostnames instead of IPs for load balancers, IAM role policies instead of registry attachment flags, and eksctl instead of az aks create. The concepts are identical. The commands differ.

Screenshots are not optional. If you are building this for a portfolio or a capstone, capture every stage while the cluster is live. Once you tear it down, the URL is gone and the proof is gone with it.

The Repository

The full source code is at:

github.com/gbadedata/zero-downtime-bluegreen-eks

It includes all Kubernetes manifests, the complete GitHub Actions workflows, the Dockerfile, the application code, and ten screenshots from the live deployment. Fork it, adapt it, and build on it.

What I Would Add Next

Prometheus and Grafana for real-time metrics during the switch. Right now the only signal is the curl loop. A proper monitoring stack would give error rate, latency, and request volume dashboards that make the switch moment quantifiable.

Canary releases as a step between all-or-nothing blue-green and fully automated rollout. Send 5% of traffic to green, monitor for 10 minutes, then ramp to 100%. Achievable with NGINX Ingress weight annotations.

Terraform for infrastructure so the EKS cluster, ECR repository, IAM roles, and VPC are all version-controlled and reproducible with a single terraform apply.

Automated rollback triggered by a post-switch error rate spike. If the error rate on the new environment exceeds a threshold within two minutes of the switch, automatically patch the selector back. No human required.