惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

P
Proofpoint News Feed
博客园_首页
WordPress大学
WordPress大学
大猫的无限游戏
大猫的无限游戏
有赞技术团队
有赞技术团队
阮一峰的网络日志
阮一峰的网络日志
Hugging Face - Blog
Hugging Face - Blog
博客园 - 【当耐特】
酷 壳 – CoolShell
酷 壳 – CoolShell
Y
Y Combinator Blog
Vercel News
Vercel News
The GitHub Blog
The GitHub Blog
T
The Blog of Author Tim Ferriss
云风的 BLOG
云风的 BLOG
博客园 - 司徒正美
Engineering at Meta
Engineering at Meta
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
J
Java Code Geeks
Stack Overflow Blog
Stack Overflow Blog
N
Netflix TechBlog - Medium
Martin Fowler
Martin Fowler
宝玉的分享
宝玉的分享
G
Google Developers Blog
Last Week in AI
Last Week in AI

The Practical Developer

The Libuv Thread Pool Trap: Why Node.js Async APIs Stall Under Load Postgres Covering Indexes with INCLUDE: Eliminate Heap Fetches on Read-Heavy Workloads Postgres DISTINCT ON: The Fastest Way to Get the Latest Row Per Group Postgres Transaction Isolation: The Anomalies Your App Actually Faces in Production Linux TCP Tuning for Node.js Microservices: The Kernel Settings That Stop Silent Connection Drops Under Load Postgres HOT Updates and Fillfactor: Why Not All Writes Are Created Equal Database Connection Pool Leaks: Finding the Promise That Never Returns Its Seat Linux OOM Killer in Production: Why Your Node.js Containers Die Without a Stack Trace Postgres Materialized Views: Refresh Strategies That Do Not Lock Your Dashboards API Dependency Health Checks: Why /health Is Not Enough Authorization with Zanzibar Tuples: How Google Manages Permissions and How To Build the Same Check in Node.js Postgres Advisory Locks: The 20-Character Primitive That Replaces Redis for Coordination Dead Letter Queues: The Message Queue Pattern That Saves You at 2 a.m. File Descriptor Exhaustion: The Kernel Limit That Silently Drops Node.js Connections Graceful Degradation: The Pattern That Turns Total Outages into Partial Success PostgreSQL Full-Text Search: Dropping Elasticsearch for 90% of Use Cases S3 Presigned Multipart Uploads: Stop Your API Server from Being a File Upload Bottleneck MessagePack vs JSON: The Binary Serialization Switch That Cut Our Internal RPC Overhead by 40% DNS Caching in Node.js: The Silent Cause of Production Latency Spikes Reliable Cron Jobs: The Pattern That Stops Double Runs, Missed Executions, And The 2 AM Page GraphQL Query Complexity: Stop the OOM Query Before It Reaches Your Resolver Node.js Event Loop Lag: The Hidden Metric Behind Random Latency Spikes API Request Validation with Zod: The Schema That Catches Bad Input Before It Corrupts Your Database Load Shedding in Node.js: How to Reject Traffic Before You Drown Request Hedging: Cut Tail Latency In Half Without Overprovisioning Git Bisect: The Automated Binary Search That Finds Breaking Commits in Minutes Node.js Garbage Collection Tuning: Stop Letting V8 Pause Your Event Loop Node.js Server Timeouts: The Settings That Stop Slow Clients from Holding Sockets Hostage Postgres BRIN Indexes: The Time-Series Secret That Shrinks Indexes by 99% Event Sourcing with PostgreSQL: The Pragmatic 80% Solution
GitHub Actions In A Monorepo: Caching, Path Filters, And ...
The Practica · 2024-05-10 · via The Practical Developer

The team has a monorepo with three apps: web, api, mobile. A CI pipeline runs everything on every PR: install, lint, typecheck, test, build, deploy preview. Total time: 25 minutes. A PR that touches only api/ waits the same 25 minutes as one that touches everything.

This is the default monorepo CI pattern, and it’s wrong. PRs should run only the jobs affected by their changes. Caching should be aggressive. Long-running jobs should run in parallel where possible. Secrets should be scoped so a compromised PR cannot exfiltrate prod credentials.

This post is the working setup: path filters, cache configuration, reusable workflows, and the secret-boundary pattern that prevents lateral compromise. Total CI time on a typical single-app PR drops to ~4 minutes.

Path filters: only run what changed

Use path filters to skip jobs entirely:

# .github/workflows/api.yml
name: api
on:
  pull_request:
    paths:
      - 'api/**'
      - 'shared/**'
      - '.github/workflows/api.yml'

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npm test --workspace=api

A PR that touches only web/ doesn’t trigger this workflow at all. The CI dashboard shows what was actually relevant.

For more nuanced filtering, dorny/paths-filter computes a per-job filter:

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      api: ${{ steps.filter.outputs.api }}
      web: ${{ steps.filter.outputs.web }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            api: 'api/**'
            web: 'web/**'

  api-test:
    needs: changes
    if: needs.changes.outputs.api == 'true'
    runs-on: ubuntu-latest
    steps: ...

  web-test:
    needs: changes
    if: needs.changes.outputs.web == 'true'
    runs-on: ubuntu-latest
    steps: ...

One workflow, multiple conditional jobs. Cleaner than separate workflows for some teams.

Caching: the highest-leverage CI improvement

actions/setup-node@v4 with cache: 'npm' (or pnpm, yarn) caches dependencies automatically. For other caches, the explicit pattern:

- name: Cache build artifacts
  uses: actions/cache@v4
  with:
    path: |
      .next/cache
      node_modules/.cache
    key: ${{ runner.os }}-build-${{ hashFiles('**/package-lock.json') }}-${{ github.sha }}
    restore-keys: |
      ${{ runner.os }}-build-${{ hashFiles('**/package-lock.json') }}-
      ${{ runner.os }}-build-

key is the exact match. restore-keys are progressively more lenient fallbacks; first match wins. Result: even on a fresh PR, you get most of yesterday’s cache.

For Docker layer caching, use docker/build-push-action with a registry-based cache:

- uses: docker/build-push-action@v5
  with:
    context: ./api
    push: true
    tags: ghcr.io/.../api:${{ github.sha }}
    cache-from: type=registry,ref=ghcr.io/.../api:buildcache
    cache-to: type=registry,ref=ghcr.io/.../api:buildcache,mode=max

For monorepo TypeScript projects, the tsc --build incremental cache plus a workflow cache turns a 90s typecheck into 5 seconds.

Matrix builds: parallel where possible

Tests across multiple Node versions, multiple OSes, multiple databases:

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        node: ['18', '20', '22']
        os: [ubuntu-latest, macos-latest]
    runs-on: ${{ matrix.os }}
    steps: ...

fail-fast: false continues other matrix jobs after one fails. Useful for “we want to know which Node version broke it.”

For tests that take a long time, shard across runners:

jobs:
  test:
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    runs-on: ubuntu-latest
    steps:
      - run: npm test -- --shard=${{ matrix.shard }}/4

Vitest, Jest, and Playwright all support sharding. A 12-minute test suite becomes 3 minutes on 4-way parallel.

Reusable workflows

Don’t copy-paste the same setup across multiple workflows. Factor out:

# .github/workflows/_node-setup.yml
name: node-setup
on:
  workflow_call:
    inputs:
      node-version: { required: true, type: string }

jobs:
  setup:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ inputs.node-version }}
          cache: 'npm'
      - run: npm ci
# .github/workflows/api.yml
jobs:
  setup:
    uses: ./.github/workflows/_node-setup.yml
    with: { node-version: '20' }

  test:
    needs: setup
    runs-on: ubuntu-latest
    steps:
      - run: npm test --workspace=api

Reusable workflows make CI changes one-place-to-edit. Major maintenance win.

A pull request from a fork can run workflows. Without care, those workflows can read your secrets and exfiltrate them. The default behavior is reasonable:

  • pull_request events from forks have no access to repository secrets.
  • pull_request_target events do have access, and run on the target branch’s code, not the fork’s. Use it carefully.

Rule: any workflow that needs production secrets should run on push to main, not on pull_request. Preview environments use scoped credentials, not prod ones:

on:
  push:
    branches: [main]    # only on main; forks can't push to main

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production       # GitHub Environments adds approval gates
    steps:
      - uses: actions/checkout@v4
      - run: ./deploy.sh
        env:
          AWS_KEY: ${{ secrets.PROD_AWS_KEY }}

The environment: production line gates this on a manual approval (configurable in repo settings). Deploys cannot proceed without an approver.

For preview environments on PRs, use a scoped IAM role (or separate AWS account) that can only manage preview resources:

- run: ./deploy-preview.sh
  env:
    AWS_KEY: ${{ secrets.PREVIEW_AWS_KEY }}  # different role, smaller scope

A compromised PR can drop preview environments; it can’t touch production.

Concurrency: cancel old runs

When a PR gets a new push, cancel the old workflow run. Saves money and surfaces the latest result.

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Same group → previous run is cancelled. Makes the dashboard cleaner and CI cheaper.

Self-hosted runners: when worth it

GitHub-hosted runners are billable per minute. For a heavy CI workload, self-hosted runners on EKS (via actions-runner-controller) can be 5-10x cheaper.

The trade-off: you operate the runners. They need scaling, security patching, monitoring. For small teams, GitHub-hosted is fine. For 10+ engineers running CI all day, self-hosted often pays back.

Caveat: never self-host runners that handle PRs from forks. The fork’s code runs on your hardware; security implications are real.

Pre-merge vs post-merge

Default GitHub setup runs CI before merge. For very large monorepos, “merge queue” (in beta as of 2024) runs CI after a merge candidate is selected, sequentially, so two PRs that pass individually but break together are caught.

For most teams under 50 engineers, pre-merge CI plus required status checks is enough. Merge queues are for very high-velocity repos where the rate of “passes individually, breaks combined” is significant.

Observability for CI itself

A few queries worth running periodically:

  • Median PR time-to-merge from CI start. Should be under 10 minutes.
  • % of CI failures that are flaky (pass on retry). Should be under 5%.
  • Most-expensive jobs by total minutes consumed.

The third one is where you find optimization opportunities. Often a single test suite or build step dominates the bill; speeding it up by 30% is the win.

The takeaway

A monorepo CI that runs everything on every PR is paying for nothing. Path filters, aggressive caching, matrix sharding, reusable workflows, and proper secret boundaries take a 25-minute pipeline to 4 minutes. The work is one or two days of focused tuning. The payoff is faster PR cycles and lower bills, every day, for years.

Don’t write CI configs once and forget them. Re-tune quarterly as the codebase grows and patterns emerge. The compounding wins are real.


A note from Yojji

The kind of CI-engineering discipline that takes a slow, expensive pipeline and turns it into a fast, cheap one (path filters, caching, matrix sharding, secret boundaries) is the kind of long-haul DevOps work Yojji’s teams put into the products they ship for clients.

Yojji is an international custom software development company founded in 2016, with teams across Europe, the US, and the UK. They specialize in the JavaScript ecosystem (React, Node.js, TypeScript), cloud platforms (AWS, Azure, GCP), and full-cycle product engineering, including the CI/CD work that decides whether your team ships fast or waits in the build queue.