Testing within the CI/CD pipeline, also known as shift-left testing, is a devops best practice that enables agile teams to continually assess the viability of new features at every stage of the development process. Running tests early and often makes it easier to catch issues before they impact your users, reduce technical debt, and foster efficient, cross-team collaboration.

Datadog Synthetic CI/CD Testing offers a flexible approach to shift-left testing that allows you to incorporate your existing Datadog Synthetic tests at any point in your CI/CD pipeline, so you can maintain a positive user experience and minimize downtime. You can use a single Synthetic test suite for a range of testing scenarios—including canary testing and testing in your CI environment—which eliminates the need for environment-specific tests. Your scheduled tests will still run against your production environment as expected, but you’ll be able to run those same tests—with whatever adjustments you see fit—on demand and at every stage of the development process.

Run Synthetic tests in your CI pipeline for earlier issue detection

The rise in popularity of the agile development methodology has made continuous integration testing crucial to the preservation of a healthy production environment. Agile teams frequently deploy new features and application improvements, so it’s critical for companies to ensure that these rapid deployments do not introduce regressions. Tests that are run within the CI pipeline must therefore reliably verify that code changes have not degraded the user experience—without impeding the team’s ability to ship changes quickly.

Traditionally, end-to-end tests have been too brittle and slow to be run in the CI pipelines of teams that ship code on a daily basis. Datadog Synthetic tests, however, are designed with reliability and efficiency in mind. Synthetic tests automatically update in response to UI changes and employ a wait and retry mechanism to ensure that failures are legitimate. And because Synthetic CI/CD Testing runs tests simultaneously rather than sequentially, it won’t slow down your development process.

Datadog’s integrations with popular CI tools such as GitLab, Jenkins, and Azure DevOps already allow you to collect events from your pipelines within the Datadog platform. Synthetic CI/CD Testing takes it a step further by enabling you to incorporate Synthetic tests in your CI pipelines for earlier issue detection. You can even view the results of Synthetic CI test runs directly within your CI, as shown in the following screenshot.

Adding Synthetic tests to your existing workflow is seamless. You can either send requests to a trigger endpoint and a polling endpoint or install an npm package and use the associated CLI as part of your CI build process. The CLI will autodiscover the tests to run in your repositories, execute the tests, and retrieve their results, blocking the deployment pipeline if key Synthetic tests fail in the CI environment. This helps prevent breaking changes from reaching staging or production—and safeguards the core functionality of your application by creating a safety net that only lets viable code progress through the pipeline.

This same CLI tool can be used to upload your frontend JavaScript source maps, which makes the frontend errors you collect with Datadog RUM more meaningful. For example, you will be able to use Error Tracking to investigate when an error first occurred, which line of code fired it, and whether the latest release fixed it.

Run Synthetic tests on deployment to ensure safe releases

In addition to running Synthetic tests throughout the development cycle, you can also use Datadog Synthetic CI/CD Testing to execute tests as part of your CD process. Evaluating the state of your production application immediately after a deployment finishes enables you to detect potential regressions that could impact your users—and automatically trigger a rollback whenever a critical test fails. This capability is particularly useful for highly agile development teams that want to both own features from development to production and deploy frequently without risking extended downtime.

Synthetic CI/CD Testing can be utilized as an automatic safety measure in your canary deployments. If, for instance, you’ve deployed a new feature to a subset of users in a particular location, you can run your Synthetic tests against production in that location immediately after your changes go live. You can then use the API polling endpoint to send the results of your tests to a custom webhook, which, in the event of a failure, will trigger an automated rollback in order to revert to the most recent stable state.

Use a single, customizable test suite for every environment

Datadog Synthetic CI/CD Testing abstracts tests from their parameters, enabling you to leverage your existing, production-level test suite within your CI/CD pipeline, as well as within any environment that requires authentication or direct access to a network. The flexible nature of these Synthetic tests breaks down the silos between development, operations, QA, and product teams by eliminating the need to maintain separate testing scenarios for each stage of development. Simply create a test suite once, and tailor it for every use case.

By default, Synthetic tests that are triggered through Synthetic CI/CD Testing will run according to the same configuration that they have on production. You can, however, specify overrides in order to meet the particular needs of each environment. This can be done for your entire test suite within a global configuration file (as demonstrated in the global section of the following code snippet) or at the individual test level.

{
    "apiKey": "<DATADOG_API_KEY>",
    "appKey": "<DATADOG_APPLICATION_KEY>",
    "global": {
        "basicAuth": { "username": "bits", "password": "labrador" },
        "startUrl": "{{URL}}?static_hash={{STATIC_HASH}}",
    },
    "proxy": {
      "host": "127.0.0.1",
      "port": 3128,
      "protocol": "http"
    }
}

For example, if your testing environment is behind a proxy or basic auth system, you can use overrides to add that specificity to your CI tests without overcomplicating your production-level tests. You can also use environment variables to dynamically configure an override value for your tests’ startUrl. If, for instance, your pipeline generates a static hash for staging purposes whenever an engineer opens a PR, that static hash can be dynamically incorporated into the startUrl parameter, as shown above. For a complete list of customizable parameters, review the documentation.

Shift system-wide troubleshooting to the left

Whenever an issue is surfaced by a Synthetic test, it’s important for users to be able to easily troubleshoot it, regardless of the environment in which the test was run or where the issue originates within the stack. Datadog Synthetic Monitoring enables you to view test results from every environment—along with a wealth of contextual data—in the Datadog UI.

Tests that are run in the CI/CD pipeline have the same level of context as your production-level test runs, so you can use relevant error messages, resources, and screenshots to pinpoint the precise point of failure at any stage of the development cycle. And because Synthetic Monitoring is tightly coupled with APM, each test run is automatically correlated with metrics, traces, and logs from your application and infrastructure, enabling you to conduct efficient investigations without switching contexts.

You can also visualize CI/CD test results in the context of their job executions in the CI Results Explorer, along with test batch statuses and average batch duration over time. This view can be filtered by facets that are relevant to the CI environment, such as branch and commit sha, which makes it easy to focus your investigation on the job executions you care about. The CI Results Explorer also allows you to dive into any test batch in a single click in order to monitor the progress of individual test runs, identify failures, and compare results across different browsers, devices, and locations.

Get started with Synthetic CI/CD Testing

Datadog Synthetic CI/CD Testing allows you to leverage the power of your existing Synthetic test suite at any point in your CI/CD pipeline, so you can catch issues early, run safe deployments, and provide the best possible experience to your users. If you’re already a Datadog customer and you would like more information about how to extend the power of Datadog Synthetic tests to your CI/CD pipeline, check out the documentation. New to Datadog? Get started with a 14-day free trial.