Organizations today rely on cloud object storage to power diverse workloads, from data analytics and machine learning pipelines to content delivery platforms. But as data volumes explode and storage patterns become more complex, teams often struggle to understand and proactively optimize their storage utilization. And when issues arise—such as unexpected costs or performance bottlenecks—these teams frequently lack the visibility needed to quickly identify and resolve the root cause.

To address these issues and provide critical visibility into your cloud storage infrastructure, we’re introducing Datadog Storage Management. By providing both bucket- and prefix-level analytics for Amazon S3 and Google Cloud Storage—with support for more providers to come—Storage Management helps you understand exactly how your cloud storage is being used, detect potential issues before they impact operations, and make data-driven decisions about storage optimization.

In this post, we’ll discuss how you can use Datadog Storage Management to:

• Comprehensively monitor your cloud storage with bucket-level metrics
• Get granular, prefix-level insights into the datasets powering your most important workloads

Comprehensively monitor your cloud storage with bucket-level metrics

Bucket-level metrics enable high-level analysis of your object storage usage, performance, and costs. In Datadog Storage Management, you’ll find a breakdown of your Amazon S3 and Google Cloud Storage resources by bucket (Storage Management metrics are grouped by bucket by default), including metrics for storage consumption, object count distribution, latency patterns, request volume analysis, and more. Storage Management also surfaces and summarizes a range of bucket-level issues:

• Stale prefixes, which indicate unused data that could be inflating storage costs.
• Sharp increases in prefix size, which may indicate unexpected application behavior or security issues.
• Elevated latencies and error counts, which could disrupt application performance and have a negative impact on user experience.

These issue summaries can help you expedite troubleshooting by quickly zeroing in on the affected buckets. You can also select any bucket from this page for a detailed overview of its contents (based on data prefixes), usage, and performance.

Next, we’ll look at how you can use Storage Management for more granular analysis of your storage utilization and performance.

Get granular, prefix-level insights into the datasets powering your most important workloads

In Amazon S3 and Google Cloud Storage, prefixes are used to organize data objects within buckets. With prefix-level analytics, Storage Management enables you to understand the utilization and performance associated with each of the subsets of data stored in your buckets. This type of visibility can be essential for analyzing your storage usage and optimizing the health, performance, and costs of the datasets underpinning your most important workloads. Using prefix-level storage metrics to track prefix growth rates, write patterns, and object update frequencies can help DevOps and other teams stay ahead of a range of issues, from setbacks in application performance to cost overruns.

For example, you can use prefix-level metrics to:

• Manage costs: To stay ahead of rapid prefix growth and preempt spikes in storage costs, you can track the aws.s3.inventory.total_prefix_size and gcp.storage.inventory.total_prefix_size metrics. For example, you might want to use a change alert monitor to ensure that you’re notified of any prefix size increases of more than 50 percent within a 24-hour period.
• Monitor data pipeline health: To detect delays in data delivery that might compromise downstream processing deadlines, you can compare the aws.s3.inventory.prefix_object_count and aws.s3.inventory.total_prefix_size or the gcp.storage.inventory.prefix_object_count and gcp.storage.inventory.total_prefix_size metrics. By using a composite monitor on these metrics, you can ensure that data is flowing as expected in your pipelines by automatically checking for new (non-empty) files on a regular basis.
• Track growth patterns: Abnormal data accumulation may indicate application issues. To help ensure a quick response to these issues, you can track prefix growth rates against historical patterns via anomaly monitors on the aws.s3.inventory.total_prefix_size and aws.s3.inventory.prefix_object_count or gcp.storage.inventory.total_prefix_size and gcp.storage.inventory.prefix_object_count metrics.
• Optimize data organization: To ensure that your data organization is optimized to your workload-specific access patterns, you can use aws.s3.inventory.prefix_object_count and gcp.storage.inventory.prefix_object_count to analyze how different file types and storage tiers are distributed across your prefixes.

Getting started

Storage Management provides actionable visibility into your Amazon S3 and Google Cloud Storage usage with bucket-level and prefix-level metrics that offer a comprehensive picture of your object storage usage. This helps you proactively identify and troubleshoot performance issues, investigate access patterns, and optimize costs. To learn more, see the Storage Management documentation. And if you’re not yet a Datadog user, you can get started with a 14-day free trial.