CI/CD for Side Projects: 3 Pragmatic Design Choices

Side projects are both a learning ground and a platform for testing new ideas for many of us. I, too, have been working on my own side products for years. Unfortunately, the CI/CD processes for these projects often remain at the "manually SSH in, git pull, systemctl restart" level. This situation has led to significant time loss and errors, especially when I had to fix something at 3:00 AM.

Bringing the massive CI/CD pipelines we use in corporate projects to side projects is often overkill. It doesn't make sense in terms of cost, time, and complexity. Therefore, for my side projects, I've made three main pragmatic CI/CD design choices to be fast, reliable, and easy to maintain. In this article, I will explain these choices and their reasons based on my own experiences.

1. Minimalist Git-Triggered Deployment: The Simplicity of Push-to-Deploy

Manual deployments, even for small changes, accumulate over time and become a significant burden. At one point, for the backend of one of my side products, I had to connect to the server and run commands one by one for every update. This was both tedious and increased the risk of typing incorrect commands when I was tired. On one occasion, I accidentally confused the production database with the test database and feared several hours of data loss.

To solve this problem, I chose the simplest and fastest method: automatic deployment triggered by git push. This system is generally ideal for single-server, simple web applications or API services. What I essentially do is run a script using the post-receive hook of the Git repository. This script updates and restarts the application when new code arrives.

💡 Simplicity Always Wins

In side projects, choosing the simplest approach that gets the job done, rather than complex solutions, increases the long-term sustainability of the project. Unnecessary engineering is an enemy of motivation.

Here's what a typical post-receive hook looks like:

#!/bin/bash
# post-receive hook on your bare Git repository

# The bare repository is usually at /var/repo/myproject.git
# The working directory for your application is at /var/www/myproject

# Git hook variables
read oldrev newrev refname

# Ensure we're pushing to the main branch
if [ "$refname" = "refs/heads/main" ]; then
  echo "Push received for main branch. Deploying..."
  cd /var/www/myproject || exit

  # Pull the latest changes
  unset GIT_DIR # Important for non-bare repo operations
  git pull origin main

  # Install dependencies (if any)
  /usr/bin/pip install -r requirements.txt

  # Run database migrations (if any)
  /usr/bin/python manage.py migrate

  # Restart the application service
  /usr/bin/systemctl restart myproject-backend.service

  echo "Deployment complete for main branch."
else
  echo "Push received for $refname. No deployment triggered."
fi

This script runs on every push to the main branch. It navigates to the application directory, pulls the latest changes, installs dependencies, runs database migrations, and restarts the systemd service. This way, I simply push the code with git push origin main, and the server handles the rest. This method provided great relief in my side projects after the manual deployment nightmare I experienced in an ERP company.

The main advantage of this approach is speed and simplicity. The disadvantage is its reliance on a single server and that rollback is limited to manually reverting to an old commit and pushing again. However, for side projects, this trade-off is usually acceptable.

2. Environment Consistency with Containerization: The Docker Compose Advantage

"It worked on my machine!" is one of the oldest and most frequently heard complaints in the software world. I've often encountered this problem even in side projects. I remember spending hours on setup issues, especially due to different library versions, operating system differences, or nuances in database installation. While developing the backend for one of my side products, a query that worked locally failed on the test server due to PostgreSQL version differences, which took me 2 days to resolve.

To fundamentally solve this problem, I use Docker and Docker Compose. Putting all application components (backend service, database, cache services like Redis) into containers guarantees environment consistency. No matter where it runs, it operates with the same dependencies and configuration.

Here is a simple docker-compose.yml example:

version: '3.8'

services:
  web:
    build: .
    ports:
      - "8000:8000"
    volumes:
      - .:/app
    environment:
      DATABASE_URL: postgres://user:password@db:5432/mydatabase
      REDIS_URL: redis://redis:6379/0
    depends_on:
      - db
      - redis
    # Add resource limits for side projects to prevent runaway processes
    deploy:
      resources:
        limits:
          memory: 512M
        reservations:
          memory: 256M
    restart: always

  db:
    image: postgres:14-alpine
    environment:
      POSTGRES_DB: mydatabase
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    volumes:
      - pgdata:/var/lib/postgresql/data
    deploy:
      resources:
        limits:
          memory: 1G
        reservations:
          memory: 512M
    restart: always

  redis:
    image: redis:7-alpine
    deploy:
      resources:
        limits:
          memory: 256M
        reservations:
          memory: 128M
    restart: always

volumes:
  pgdata:

This configuration defines the web service (my application code), the db service (PostgreSQL), and the redis service. Each runs in its isolated environment, and dependencies are determined by depends_on. Especially the deploy.resources.limits and reservations parts are vital for someone like me who runs many side products on a VPS. Last month, a memory leak in the backend of one of my side products caused it to be OOM-killed after exceeding the cgroup memory.high limit. These limits prevent other services from being affected.

Docker Compose also provides great convenience in the development environment. With docker compose up, the entire environment starts up in seconds. In production, I can deploy with a simple docker compose up -d. This approach has been a breath of fresh air for me after the monolithic dependency hell I experienced in a production ERP system.

3. Simple Testing and Linting Steps: Fast Feedback Loop

In our side projects, we often don't have time to write comprehensive test suites. However, completely foregoing tests also carries significant risks. In my Android spam application, I once realized that certain numbers were being blocked incorrectly due to a regex error. This bug slipped into production, and I didn't notice it for 3 days until user complaints started coming in. Manual tests are cumbersome and can be forgotten.

Therefore, I include at least basic linting and a few critical unit test steps in my CI/CD pipeline. These steps improve code quality and help me catch the most obvious errors before they reach production. Typically, in Python projects, I check code style with flake8 or black and run tests for critical functionalities with pytest.

ℹ️ Minimum Coverage, Maximum Benefit

Instead of aiming for 100% test coverage in side projects, writing tests that cover the most critical functionalities and run quickly provides maximum benefit with minimum effort. This is a golden rule, especially for those working with limited time and resources.

Here's an example of a simple test and linting workflow on GitHub Actions:

name: CI for My Side Project

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.10'
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - name: Run Flake8 Linter
        run: |
          pip install flake8
          flake8 . --max-line-length=120 --exclude .git,__pycache__,venv
      - name: Run Pytest
        run: |
          pip install pytest
          pytest

This workflow is automatically triggered on every push or pull request to the main branch. It first installs dependencies, then checks code style with flake8, and finally runs tests with pytest. If linting or tests fail, the deployment process is halted, providing me with instant feedback. This way, I prevent small but critical errors, as mentioned above, from slipping into production.

These steps became indispensable for me after an incident I encountered in a client project: at 03:14 AM one night, a WAL rotation alarm went off, and the reason was a piece of code deployed without testing, which wrote massive amounts of temporary data to the database. Automated tests could have caught such issues much earlier.

4. Deployment Strategies and Rollback Mechanisms: Safe Exit Routes

In side projects, we generally deploy with a "fire and forget" approach. But things don't always go smoothly. A system crash after a faulty deployment and the difficulty of rolling back can be demotivating. In a client project, after a deploy on April 28th when the disk was 100% full, we saw that the application failed to start. Since there was no rollback mechanism, we had to spend 4 hours manually fixing it. Such situations demonstrate why a simple "undo" strategy is important even in side projects.

Full-fledged Blue-Green or Canary deployments can be complex for side projects. However, a simple rsync-based Blue-Green approach or symbolic link switching offers relatively safer deployment and quick rollback capabilities. My preference is to use two separate directories holding different versions of the application and a symbolic link pointing to the active one.

Here is a simple deploy script and ln -s usage:

#!/bin/bash
# Simple Blue-Green style deploy script

APP_DIR="/var/www/myproject"
CURRENT_RELEASE_DIR="$APP_DIR/current"
RELEASE_DIR_A="$APP_DIR/releases/release_a"
RELEASE_DIR_B="$APP_DIR/releases/release_b"
GIT_REPO="/var/repo/myproject.git"

# Determine which release directory is currently active
if [ -L "$CURRENT_RELEASE_DIR" ] && [ "$(readlink "$CURRENT_RELEASE_DIR")" == "$RELEASE_DIR_A" ]; then
  TARGET_RELEASE_DIR="$RELEASE_DIR_B"
  OLD_RELEASE_DIR="$RELEASE_DIR_A"
else
  TARGET_RELEASE_DIR="$RELEASE_DIR_A"
  OLD_RELEASE_DIR="$RELEASE_DIR_B"
fi

echo "Deploying to $TARGET_RELEASE_DIR"

# Clean target directory and clone/pull latest code
rm -rf "$TARGET_RELEASE_DIR"/*
mkdir -p "$TARGET_RELEASE_DIR"
git clone "$GIT_REPO" "$TARGET_RELEASE_DIR" # Or git pull if directory exists and is a repo

cd "$TARGET_RELEASE_DIR" || exit

# Install dependencies, run migrations, build frontend etc.
/usr/bin/pip install -r requirements.txt
/usr/bin/python manage.py migrate --noinput
npm install && npm run build # If you have a frontend

# Test if the new version is working (e.g., health check)
# A simple curl to a health endpoint could work here
# curl -f http://localhost:8001/health || { echo "Health check failed!"; exit 1; }

# Switch the symlink
ln -snf "$TARGET_RELEASE_DIR" "$CURRENT_RELEASE_DIR"
echo "Switched to new release: $(readlink "$CURRENT_RELEASE_DIR")"

# Restart application service (using systemd for example)
/usr/bin/systemctl restart myproject-backend.service

echo "Deployment finished. Old release is in $OLD_RELEASE_DIR for rollback."

This script deploys by switching between two separate directories named release_a and release_b. The current symbolic link always points to the active version. When a new version is ready, it's first deployed to TARGET_RELEASE_DIR, tested, and then the current link is changed to point to the new version. If a problem occurs, I can quickly rollback by manually reverting the current link to the old OLD_RELEASE_DIR.

This method ensures atomic updates of the application and allows me to deploy with near-zero downtime. I use this structure for the backend of my own side product, and in case of an error, I can revert to the old version within minutes.

5. Observability and Monitoring: What's Happening, How Do I Know?

Not knowing the status of the system after deployment is like flying blind. In side projects, we typically don't have the time or resources to set up comprehensive monitoring solutions like Prometheus/Grafana. However, this doesn't mean we monitor nothing at all. Late detection of errors not only negatively impacts user experience but can sometimes lead to serious data issues. Last month, I experienced a performance drop in the financial calculators of my side product. While examining logs with journalctl, I noticed a particular query was taking longer than expected.

My pragmatic observability approach for side projects is based on using Linux's built-in tools and simple logging. I run my applications as systemd services and direct their logs to journald.

Here is a simple systemd unit file:

[Unit]
Description=My Side Project Backend
After=network.target postgresql.service redis.service

[Service]
User=myuser
WorkingDirectory=/var/www/myproject/current
ExecStart=/usr/bin/python /var/www/myproject/current/app.py
Restart=always
StandardOutput=journal
StandardError=journal
# Add memory limits for the service
MemoryHigh=256M
MemoryMax=512M

[Install]
WantedBy=multi-user.target

This systemd unit sends my application's logs directly to journald (StandardOutput=journal). This allows me to follow live logs with the journalctl -u myproject-backend.service -f command. Additionally, by using cgroup limits like MemoryHigh and MemoryMax, I prevent the application from consuming more memory than expected. If the application exceeds these limits, it's automatically restarted by systemd, and a relevant warning is logged to journald.

I use similarly simple methods to monitor the performance of services like PostgreSQL. For example, querying the pg_stat_activity table to find long-running queries or using the log_min_duration_statement setting to log queries longer than a certain duration helps me detect issues like PostgreSQL WAL bloat early. For Redis, I check if my OOM eviction policy choices are working correctly using CONFIG GET maxmemory-policy and INFO memory commands.

These simple tools, while not replacing the dashboards and alarms offered by comprehensive systems like Prometheus, allow me to quickly understand what's happening when a problem occurs. For me, this is sufficient for side projects.

6. CI/CD from a Security Perspective: Closing Vulnerabilities

Security in side projects is often an afterthought. The mindset of "who would bother with my small side project?" is common. However, this is a major misconception. On a backend I built for my own site, the service stopped due to excessive requests to an API endpoint. I solved this problem with fail2ban rules and rate limiting in Nginx, but this experience showed me how important basic security steps are even in side projects.

Integrating basic security checks into my CI/CD process helps me reduce such risks. Instead of a full-fledged security audit, I focus on a few easily implementable and high-impact steps.

⚠️ Security Cannot Be Neglected

Even side projects can be vulnerable to attacks. Including basic security measures in the CI/CD process provides a significant risk reduction with minimal effort.

Here are some basic security steps I've added to my CI/CD:

Dependency Scanning: I check for known vulnerabilities in the libraries I use. The pip-audit tool is great for this in Python projects.
```
# Dependency scanning step in CI/CD pipeline
pip install pip-audit
pip-audit -r requirements.txt
```
This command scans all dependencies in the requirements.txt file against known CVEs and warns if any are found. In a production ERP, we realized a critical vulnerability in an old library too late, and it cost us dearly. Since then, I don't skip this step.
Basic Security Configurations: At the application level, I follow best practices for rate limiting and JWT/OAuth2 patterns. If I use Nginx, I limit requests to my API with simple limit_req directives.
```
# Example of rate limiting in Nginx
limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s;

server {
    # ...
    location /api/ {
        limit_req zone=mylimit burst=20 nodelay;
        # ...
    }
}
```
This configuration limits requests to 10 per second from a specific IP and allows a "burst" of up to 20 requests. This is my first line of defense against simple DDoS attacks or excessive usage.
Kernel Module Blacklisting: Rarely, some kernel modules can lead to security vulnerabilities or consume unnecessary resources. On my own VPS, I blacklist non-critical kernel modules like algif_aead to create an additional layer against potential vulnerabilities like CVE-2026-31431.
```
# /etc/modprobe.d/blacklist.conf
blacklist algif_aead
```
This is a small but effective step I take to improve overall system security.

These steps are pragmatic security measures that I integrate into my side projects' CI/CD process and have saved me from many headaches. While they may not be as comprehensive as what a full-fledged security team would do, they are the minimum precautions that should be taken instead of just saying "it'll be fine."

Conclusion

Side projects, especially when we work like a one-person army, often proceed with a "get it done, no matter how" mentality. However, my 20 years of field experience have shown that these pragmatic CI/CD approaches save me time and reduce stress in the long run. Because I have personally experienced the risks of manual deployments, the headaches caused by environment inconsistencies, and the problems arising from insecure code, I apply these three fundamental design choices (minimalist git-triggered deployment, environment consistency with containerization, and simple testing/linting steps) to every new side project.

These steps can be thought of as mini-versions of the complex CI/CD processes in large corporate systems. Each aims to provide maximum benefit with minimum effort, making the project more sustainable and less problematic. In my experience, thanks to these approaches, my side projects develop faster and run more stably. In my next article, I will delve deeper into the simple monitoring and alerting mechanisms I use in my side projects.

推荐订阅源

DEV Community