Introduction: Build Times and the Importance of Cache

Slow build times in your CI/CD pipelines can be a serious issue, reducing developer productivity and putting your projects at risk. Starting every build from scratch can take hours, especially for large projects. This situation increases developer waiting time due to repetitive and time-consuming downloads, compilations, and tests, while also driving up the costs of your CI/CD infrastructure. This is precisely where build cache management comes into play.

Cache stores the results of previously executed operations, allowing us to reuse these results when the same or similar operations are requested again. In the context of CI/CD, build cache helps significantly speed up the next build by storing dependencies, compiled code, or test results. This not only saves time but also reduces costs. In my own projects, I've seen build times reduced by up to 70% with the right build cache strategies. In this post, we will delve into 3 practical build cache management approaches you can use to shorten build times in your CI/CD pipelines.

1. Caching Dependencies

One of the most time-consuming parts of CI/CD pipelines is downloading all the external libraries and packages your project needs. Especially for large projects or those with frequently updated dependencies, these download operations can lead to significant time loss. One of the most fundamental and effective uses of build cache is to cache these dependencies.

Many package managers (npm, yarn, pip, Maven, Gradle, etc.) have their own local cache mechanisms. In a CI/CD environment, however, these local caches need to be made persistent. For example, caching the node_modules folder in a Node.js project or the .venv or ~/.cache/pip directory in a Python project prevents these files from being re-downloaded in the next build. The most important point to consider when implementing this approach is ensuring cache consistency. If dependency files like package.json or requirements.txt change, the old cache must be invalidated, and new dependencies must be downloaded.

# Example: Caching dependencies with npm (GitLab CI/CD)

<figure>
  <Image src={cover} alt="An abstract graphic illustrating build cache management in a CI/CD pipeline" />
</figure>

cache:
  key: ${CI_COMMIT_REF_SLUG}-npm-${CI_PROJECT_DIR}
  paths:
    - node_modules/
  policy: pull-push

build_job:
  script:
    - npm ci # npm ci installs based on package-lock.json and is more suitable for caching
    - npm run build

In the GitLab CI/CD example above, the node_modules folder is cached. The key field creates a unique cache key based on the branch and project directory, allowing separate caches to be used for different branches. policy: pull-push downloads the cache at the start of the job and uploads the updated cache back to the server at the end of the job. This method reduces the dependency download time to almost zero after the first build. However, it's important not to forget that the cache needs to be refreshed with every change in the dependency file.

ℹ️ Cache Key Strategies

Designing the keys used for dependency caches is critical. Using a key that includes the hash of files like package-lock.json or yarn.lock ensures that the correct cache is used even with minor changes in dependencies. This prevents unnecessary downloads while guaranteeing that dependency changes are correctly reflected.

2. Caching Compiled Code

Especially when working with statically typed languages (Java, C#, Go, Rust) or languages with complex compilation processes (C++, TypeScript), the compilation phase of code can also take significant time. Caching these compilation results can also substantially shorten build times. Many compilers and build tools have the ability to store intermediate build artifacts.

For instance, in a Java project using Gradle, the compiled classes and source files can be stored thanks to Gradle's internal caching mechanism. Similarly, compilations done with tsc's --build or --incremental modes in TypeScript projects also benefit from caching. Making these compiled code caches persistent in a CI/CD environment is managed similarly to caching dependencies.

# Example: Using Gradle build cache in CI/CD (Jenkins)
# Jenkinsfile snippet

pipeline {
    agent any
    options {
        // Preserve workspace to make build cache persistent
        keepWorkspaces()
    }
    stages {
        stage('Build') {
            steps {
                sh './gradlew build' // Gradle automatically uses build cache
            }
        }
    }
}

In CI/CD tools like Jenkins, the keepWorkspaces() option prevents the workspace from being deleted after the job completes, allowing Gradle's build cache to be carried over to the next job. However, caution is needed here as well. If a change in the source code causes the compiled code to become invalid, this old cache should not be used. The incremental build features provided by build tools help ensure this consistency.

⚠️ Risks of Compiler Cache

When using compiler caches, especially with complex projects or when switching between different compiler versions, unexpected issues can arise. Sometimes, old compilation outputs can become incompatible with the current source code, leading to runtime errors. Therefore, when compiler caches are used, it's important to always have a full clean build option available and to use it periodically.

3. Incremental Testing and Caching Test Results

In software development processes, tests are the cornerstone of quality. However, running all tests on every build in a project with thousands of tests can also take a significant amount of time. To solve this problem, incremental testing and caching test results approaches come into play.

Incremental testing is based on the principle of running only the tests associated with the changed code. Many test frameworks (e.g., Jest, Pytest) support this capability. By analyzing the changed files, it runs only the tests that touch those files. This is much faster than running the entire test suite. Caching test results means skipping those tests by reusing the results of the previous test run, if no changes have been made to the tests.

# Example: Incremental testing with Jest (Nx Monorepo)
# package.json snippet

"scripts": {
  "test:e2e": "jest apps/my-app/e2e/ --ci --changedSince=main",
  "test:unit": "jest --changedSince=main",
  "test:all": "jest"
}

Monorepo tools like Nx help determine which tests to run with parameters like changedSince. This dramatically reduces build times, especially in large monorepos. For caching test results, we can use the caching mechanisms of CI/CD tools. For example, we can save the results of tests run in a previous build to a file and then skip those tests in the next build using this file.

💡 Test Cache and CI Environment

When caching test results, it's important to ensure the reproducibility of the CI environment. Inconsistencies between different CI runners or environment variables can cause test caches to produce incorrect results. Carefully managing the directories where you store test results and the keys used to validate these results (e.g., commit hash, code coverage) is necessary.

General Approaches for Build Cache Management

In addition to the three main approaches mentioned above, there are also some general strategies that can make build cache management more effective. These strategies offer a holistic approach that covers dependencies, compiled code, and test results.

Firstly, it's crucial to understand when the cache should be invalidated. The cache must be refreshed when dependency files change, when changes in source code affect compiled code, or when test scenarios are updated. Correctly using the cache key mechanisms provided by CI/CD tools automates this invalidation process. For example, including the hash of package-lock.json in the cache key ensures that a new cache is created with any change in this file.

Secondly, consider using distributed cache solutions. Instead of being dependent on a single CI runner, using a central cache server (e.g., Redis, S3 bucket) allows multiple CI agents to share the same cache. This improves performance, especially in large teams and parallel CI/CD pipelines. Object storage services offered by cloud providers (AWS S3, Google Cloud Storage) are frequently used for this purpose.

Thirdly, optimizing cache size and duration is important. Overly large or long-stored caches can lead to disk space issues and may even become corrupted over time. Implementing mechanisms that automatically clean up caches after a certain period or prune caches that exceed a certain size simplifies management.

# Example: Distributed build cache on S3 (GitHub Actions)
# .github/workflows/build.yml snippet

- name: Cache node_modules
  uses: actions/cache@v3
  with:
    path: '**/node_modules'
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-node-

In this GitHub Actions example, the actions/cache action caches the node_modules folder using S3 or a similar storage. The key and restore-keys strategy ensures that the cache is managed intelligently based on dependency changes.

🔥 Cache Polluters

One of the biggest threats in build cache management is "cache polluters." These are factors that unnecessarily invalidate the cache or mistakenly cause old/corrupted data to be cached. For example, code changes that generate random numbers or include timestamps can continuously invalidate the cache by producing different results with each build. Identifying and preventing such situations is vital to maintaining the effectiveness of the cache.

Conclusion: Faster CI/CD, Happier Developers

Implementing build cache management correctly in CI/CD pipelines is not just a technical optimization but a critical factor that directly impacts developer experience and project delivery speed. By intelligently caching dependencies, compiled code, and test results, you can significantly shorten build times, reduce CI/CD infrastructure costs, and most importantly, increase productivity by enabling developers to receive faster feedback.

Remember that the best cache strategy depends on your project's specific requirements, your technology stack, and your CI/CD environment. The three fundamental approaches and general strategies presented in this post will provide a solid starting point for your own pipelines. Experimenting with different cache key strategies, exploring distributed cache solutions, and performing periodic cache cleanups will ensure you achieve the best results in the long run. Reducing your build times is one of the most important steps towards a more agile and efficient development process.