Datadog Synthetic private locations play a key role in your organization’s test infrastructure by serving as highly customizable points of presence (e.g., data centers, geographic locations) for running synthetic tests on internal services. You can deploy private locations using the orchestrator of your choice, enabling you to seamlessly roll them out and scale them with the rest of your fleet. Together with the testing tunnel, which offers on-demand testing for internal applications, private locations ensure that you are able to test all of the critical applications deployed within your organization’s private network.

Because private locations are integral to testing internal-facing applications, Datadog now offers full visibility into their health and performance with Private Location Monitoring and the Datadog Agent. Your SRE and platform teams, who are often the primary teams responsible for building and supporting test infrastructure, can leverage Datadog to:

• Monitor your private locations with out-of-the-box metrics and alerts in order to respond to critical performance issues
• Monitor your host runtime statistics with the Agent
• Keep track of performance with SLOs

Monitor the state of your private locations

Datadog Synthetic Monitoring provides out-of-the-box metrics that your teams can monitor to ensure that private locations are live, up to date, and have enough resources to support running synthetic tests. For example, by default each private location can support a total of ten concurrent test runs (referred to as slots in Datadog), though you can change that number to fit your needs. To ensure that a private location does not hit its concurrency limit, you can monitor the number of open slots available to run tests.

Datadog also provides out-of-the-box alerts that automatically notify you of issues with your private locations that could quickly become a bottleneck for testing and other CI/CD pipelines, such as a location either using an outdated image or running out of open slots to execute tests.

A private location that consistently hits its concurrency limit is likely underprovisioned, so it’s important to always ensure that a location can support the number of tests assigned to it. For example, you may decide to deploy more workers to distribute the existing load, or update the location’s current concurrency limit to run more tests in parallel.

Get deep visibility into private location host performance

Your SRE teams can easily capture runtime metrics, logs, network activity, and more by deploying the Datadog Agent to the same cluster. Datadog Infrastructure Monitoring enables teams to monitor runtime performance and ensure that private locations have enough resources to run tests.

For example, running a large number of Synthetic browser tests on a private location can cause temporary spikes in the worker’s CPU utilization. This can easily make a private location unresponsive if it does not have enough processing power to support the increased load. If you see this issue in one of your private locations, you may need to assign more resources to it or leverage an orchestrator such as Kubernetes to autoscale it in order to better accommodate the fluctuations in traffic.

Track private location performance with SLOs

Since private locations are a critical part of test infrastructure, your SRE and platform teams need to ensure that they are not only performant but always available for testing. Datadog enables teams to track the performance of all of your deployed private locations and set objectives for improving their reliability over time with highly customizable SLOs.

For example, teams can create an SLO based on Datadog’s out-of-the-box alert for underprovisioned private locations to help ensure that they always have enough open slots to run tests. A triggered alert will cause a dip in SLO status. They can add these SLOs to custom dashboards in order to compare them to changes in private location performance and respond to an issue before it affects an SLO’s status. This allows SRE and platform teams to build reliable test infrastructure, which in turn enables you to deploy performant applications for your employees and customers.

Better visibility into private location performance

Private Location Monitoring gives your SRE and platform teams a better view of the health and performance of private locations, ensuring that they can scale and support your organization’s test goals. The Datadog Agent also provides deeper insights into host and container runtime performance when deployed alongside any private location. Check out our documentation to learn more about private locations and the Datadog Agent. If you don’t already have a Datadog account, you can sign up for a free 14-day trial.