As part of your monitoring and testing strategy, you may run tests on different types of applications that are not publicly available—from local versions of production-level websites to internal applications that directly support your employees. Testing each one requires leveraging tools that allow you to verify functionality across a wide range of devices, browsers, and workflows while maintaining a secure environment. Datadog Synthetic Monitoring already lets you create your own custom probes (on-premise test runners) with private locations to routinely test and monitor all of your internal-facing applications. Now, for on-demand testing, you can also use Datadog’s testing tunnel, a secure tunnel connection that requires little setup.

Private locations and the testing tunnel give you more flexibility over how you test applications in your internal environments, but each tool offers some unique benefits to support different testing goals.

In this post, we’ll look at:

• using the testing tunnel for on-demand testing in local and continuous integration (CI) environments
• creating private locations for durable testing and monitoring

CI and local testing with the testing tunnel

The testing tunnel leverages Datadog’s command line interface (CLI) to create an end-to-end encrypted HTTP proxy between your infrastructure and Datadog. The CLI is an NPM package that enables you to launch Datadog Synthetic tests as part of your CI/CD pipelines, so you can identify and fix regressions in your applications before they impact your users. When used in conjunction with the testing tunnel feature, any test requests you send using the CLI are automatically routed through the datadog-ci client, allowing Datadog to access and test your internal applications.

Datadog’s testing tunnel is designed to support CI pipelines and local development, so you can use it for:

• verifying hotfixes or new features locally before committing code
• running tests in environments reserved for CI pipelines (e.g., staging, user acceptance testing, etc.) or in ephemeral cloud environments

We’ll look at how the tunnel’s unique features and benefits can support these particular testing goals next.

An easy-to-use tool for testing on demand

A key benefit of the testing tunnel is its ease of use within existing infrastructure; it enables you to incorporate API and end-to-end tests into all of your workflows. For example, your teams (e.g., developers, testers) can use this tool out of the box to quickly verify that a hotfix for a time-sensitive issue, such as a service outage, works as expected locally before deploying it to end users. You can also use the tunnel service to run test suites as part of your CI pipelines without launching multiple browsers directly on CI servers, where processing power may be limited.

You can instantly create a tunnel connection to run tests using a simple command:

datadog-ci synthetics run-tests --config synthetics.global.json --tunnel

The example command above will open a WebSocket Secure tunnel connection and launch the suite of tests defined in your local machine’s or CI server’s test configuration files. These files include the public IDs of the tests that you want to run, along with other configuration attributes, such as endpoint URLs, device IDs, and locations.

Since the tunnel is built into Datadog’s CLI, it enables you to quickly start testing your internal applications at any time.

Launch tests with minimal overhead

The tunnel is independent of existing infrastructure, so you can use it without deploying, maintaining, or monitoring additional services. Tests launched via the tunnel are executed from Datadog-managed locations. This means that as long as the host running Datadog’s CI client can create the connections needed to run multiple tests, Datadog will automatically scale to support the increased load as needed. Tunnel connections then end when the Datadog CI client receives all necessary results, so you do not need to track long-running connections to your network.

The tunnel also makes it easy to dynamically override where your tests run with Datadog’s built-in environment variables, so you can continue testing your applications without interruption, even as the environment you are testing changes. This includes environments that rely on ephemeral cloud instances and containers. For example, you can automatically pass the URL of a newly deployed application instance as the starting URL for any tests launched with the tunnel, instead of hard coding that data into your tests.

Datadog shows which tests were launched through the tunnel service so you can monitor them alongside the rest of your synthetic tests. For any test failures, Datadog provides end-to-end visibility for troubleshooting and resolving issues, including details such as screenshots of the UI, JavaScript and network errors, load times for page resources, and APM traces if your test is hitting an instrumented service endpoint.

Now that we’ve covered the benefits of using the testing tunnel for straightforward, on-demand testing, we’ll look at how Datadog’s private locations support your long-term testing and monitoring goals.

Durable testing and monitoring using private locations

As we’ve seen, the testing tunnel offers a turn-key solution for secure, rapid testing in short-lived environments. For organizations who need to regularly test and monitor applications hosted on permanent environments, Datadog provides private locations: Docker containers that you can deploy as custom points of presence (e.g., data centers, geographic locations) inside of your infrastructure using orchestration tools like Docker Compose, Kubernetes, AWS Fargate, and Amazon ECS.

Because private locations are deployed as a durable probing service for launching your tests, they can be useful for:

• customizing and managing a centralized testing tool that is readily available for teams across your organization
• triggering tests on long-running environments (e.g., staging, pre-production) as part of your CI/CD pipelines
• regularly running tests on internal applications that are hosted on private networks to ensure you can maintain your availability SLOs

We’ll look at how you can use private locations to create a customizable, scalable, and easily accessible service in more detail next.

A fully-fledged and customizable testing service for internal applications

Since testing is a crucial part of building resilient applications, you need a system that can support testing a growing network of services as your organization scales. Using private locations, your SRE teams have greater flexibility in not only customizing a probing service for every use case—via their preferred orchestration tool—but also ensuring it can scale to continually verify functionality and monitor application performance.

Private locations come with a number of parameters you can use to match your infrastructure and private network configurations, such as built-in controls to block IPs in order to prevent users from creating synthetic tests on potentially sensitive endpoints in reserved IP ranges. And, as your applications grow, you can horizontally or vertically scale your locations in order to run more synthetic tests concurrently, enabling you to seamlessly test newly added features alongside existing functionality. Leveraging these measures ensure your applications—and your test infrastructure—remain secure and continue supporting your users.

Monitoring private locations

Private locations are designed to regularly test and monitor your applications long term. Because of their longevity—and since tests run on the servers where you’ve deployed private locations—you need to ensure that every location is working as expected. Datadog provides visibility into your entire infrastructure, so you can monitor the performance of your custom locations in one place. For example, you can create custom dashboards to get a high-level overview of all of your private locations and easily monitor usage, as seen below.

You can also use the Datadog Agent to get deeper visibility into the state of your private locations’ underlying containers and confirm that they are performing optimally. If you notice unusual changes in the tests executed by your private location, such as a significant increase in response time, you can then drill down to the affected container in order to troubleshoot further.

Self-service testing for every team

Once deployed, private locations provide a centralized and readily available service for testing, so your teams can create their own tests and assign them to specific locations in one click.

This enables your teams to routinely test applications under a wide variety of conditions. For example, your corporate IT team can launch tests on private locations deployed to multiple data centers to ensure that your company intranet or a key SaaS provider is performing optimally for a growing team of distributed employees, regardless of their location.

Or, your QA team can leverage the same tests and private locations as part of their CI/CD pipelines to verify that key workflows are still accessible to users after a canary deployment of new intranet features.

Your map for comprehensive internal application testing

With private locations and the testing tunnel, you have more options for testing and monitoring your internal-facing applications. Each service offers unique features to help you accomplish your testing goals, whether they require long-running probing services or the ability to quickly launch tests on demand and with little setup. Check out the documentation for private locations and the tunnel service to learn how to get started with both. If you don’t already have a Datadog account, you can sign up for a free 14-day trial.