20 Oct, 2024

This post describes my lessons learned after 10 years running production environments in sizes ranging from "just getting started" to a "Series F" company with 4-9 SLAs.

Like any good developer, I'm impatient and expect tasks to be fast, especially if I need to do them often, like deploying. Nothing is worse than hitting "deploy" and having a 10-15 minute delay before I can validate my changes. It's a pain felt by the whole team, multiple times, every day. It breaks their flow. And engineers are expensive, so this lost time and concentration costs a lot of money too.

Let's start with the typical setup for my company, analyze what takes the most time, and see how we can strip these layers away to make something that's faster and simpler.

CI/CD has been a Best Practice™ for a long time now, but it's slow. Even with special concern for caching, it's just much slower than running go build locally, and it's a lot more complex.¹ For a big application at my company, compiling locally takes ~10 seconds, whereas building the binary in Github workers takes several minutes.

Gradual rollouts are slow too, taking 5-10 minutes as k8s rolls out updates to pods across the fleet. This is an especially important practice when your deploys take so long, since reverting any change will be just as painful. You need to be sure it's going to work the first time.

But if you could deploy in 10 seconds, do you need to gradually increase traffic for every change, or could you just redeploy?² I would just redeploy. The YOLO of devops.

I know what I want. Let me deploy quickly and undo changes quickly. So how do we do it?

Let's go back in time and use the basics: bash, rsync, and a service manager. At a high level, my typical deploy script looks like this:

#!/usr/bin/env bash
set -ex

TMP_DIR=/tmp/app

rm -r "$TMP_DIR"
mkdir "$TMP_DIR"

go build -o "${TMP_DIR}/app"
cp app.service env.ini "${TMP_DIR}/"

rsync -chazP "$TMP_DIR" server1:

ssh server1 '
    sudo systemctl stop app && \
    sudo cp -R app/* /home/_app/ &&
    sudo mv app /home/_app/app && \
    sudo chown -R _app:_app /home/_app/* && \
    sudo chmod 400 /home/_app/env.ini && 
    sudo systemctl start app'

rm -r "$TMP_DIR"

This example script³ builds our binary, moves it to our dedicated daemon's home folder, adjusts permissions and ownership, and restarts the service.

"But wait!", I hear you say. "You're only deploying to one server. This means we have no redundancy, and restarting the service results in a brief outage." We certainly have no redundancy in this scenario -- I'll write an entire post espousing the benefits of vertical scaling over horizontal and why redundancy isn't all it's cracked up to be until massive scale. It's too much to fit into this post. But to your second point, restarting a service doesn't mean you need to have an outage -- simply a delay! Your reverse proxy should keep the request open until the server comes back online. In Caddy, you do this via try_duration.

So it really is that simple: a small bash script, building locally, rsync'ing the changes, and restarting the service. It's just the bare essentials of a deployment. That's how I deploy in 10 seconds.

In my next two posts, I'll:

• Show you how I provision a Debian server using Ansible
• Compare trade-offs of horizontal and vertical scaling

Subscribe on RSS or email to follow along.

#devops