François Massot
Edith Méndez
Zara Boddula
As AI workloads and cloud-native applications expand, organizations are generating more log data than ever. Each service, container, and model inference produces continuous telemetry that must be stored, secured, and analyzed. As telemetry grows more complex, teams must balance full visibility with new retention and residency needs. Security teams often retain logs for a year or longer to investigate threats and meet stricter compliance requirements, while new regulations increasingly require sensitive data to stay within regional boundaries.
To manage these pressures, many teams turn to a mix of self-hosted log management solutions that are either custom-built or provided by specialized vendors. But this approach comes at the expense of manageability and visibility: Scaling is slow and costly, maintenance is challenging, and data is left fragmented across tools. Additionally, teams who host logs in their own environment normally lose the advantages of a SaaS platform—such as ease of infrastructure maintenance, ease of use, correlation across all telemetry data, AI-powered analysis, and centralized governance control.
Datadog CloudPrem removes the traditional drawbacks of hosting logs in your environment. A hybrid log management solution that runs in your own infrastructure, CloudPrem remains fully integrated with the Datadog SaaS platform. It gives you full control over your data, correlation across all telemetry, AI-powered agents and insights, and high performance based on the Quickwit engine. With CloudPrem, you can store and search logs at petabyte scale in any region while keeping the full Datadog experience that your teams already know.
In this post, we’ll show how CloudPrem helps you:
Self-hosted log management solutions are typically difficult to scale. Adding new nodes often means redistributing large amounts of data across disks, a time-consuming process that impacts performance. Managing local or network-attached storage adds even more overhead, requiring constant tuning and rebalancing to maintain stability.
CloudPrem removes this complexity with a modern architecture that is built to scale easily, flexibly, and cost-effectively. It separates compute from storage, using low-cost object storage as the primary, durable storage layer. This object storage can be located either in your own on-premises cluster or in a cloud storage solution such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. Because compute is truly decoupled from storage in CloudPrem, you can add or remove search nodes in seconds to match demand, scaling up during spikes and down to save costs, all without rebalancing data. Combined with advanced compression algorithms, this scalable architecture significantly reduces infrastructure and operational costs compared to legacy self-hosted systems.
The architecture is also designed to search data directly from object storage with high performance. As logs are ingested, indexers write optimized index pieces called splits directly to your object storage, while a central metastore tracks those splits and makes them available for querying almost instantly. Another advantage is that the search layer is entirely stateless. When you run a query, a coordinator node consults the metastore to locate relevant splits and distributes the request across peer search nodes. Each node fetches data directly from object storage in parallel, without maintaining local index data—making the clusters both resilient and easy to operate.
Managing multiple clusters across regions and cloud providers can make it difficult to maintain a clear picture of system health. CloudPrem simplifies this by bringing management and monitoring together in one place. From the Datadog UI, you can view cluster health, track ingestion rates, and identify performance degradation—all within the same dashboards you already use for metrics and logs. Built-in monitors help you spot issues early so you can take action before they impact performance.
You can also centrally manage indexes for fine-grained data segmentation. With this capability, you can easily separate log data by value, application, or type—such as audit or security logs—and assign specific retention and quota policies to each index. This gives you granular control over storage costs and compliance requirements.
Finally, CloudPrem integrates natively with Datadog Observability Pipelines, eliminating the need to build separate pipelines. Observability Pipelines let you process, enrich, redact, and normalize logs before routing them into CloudPrem for unified search and analysis. With out-of-the-box support for open log formats and integrations, third-party SIEMs, and data lakes, you can ensure that every log stored in CloudPrem is consistent, compliant, and optimized for cost and visibility.
As daily log volumes increase—coming from different teams and systems and in various formats—troubleshooting becomes increasingly difficult. The challenge becomes worse when engineers have to jump between multiple tools, each with its own UI, dashboards, and alerting workflows to understand issues.
With Datadog CloudPrem, teams can search and correlate their self-hosted logs in one central interface while still accessing the full breadth of Datadog’s SaaS platform capabilities. You can investigate logs, metrics, and traces side by side without switching tools or losing context.
Let’s say you’re a developer overseeing a checkout API that suddenly starts timing out. You receive a change throughput alert, and Bits AI SRE immediately steps in as your co-pilot. The agent generates several hypotheses and begins reasoning over all available signals—automatically correlating logs stored in CloudPrem with metrics and traces collected in Datadog SaaS.
Following one of Bits’ investigations, you ask it to show you recent web-store error logs. At the same time, you notice another hypothesis focused on authentication issues and decide to dig into those as well. Within Log Explorer, you use Natural Language Queries (NLQ) to type: “Show error logs for the auth service over the past 30 minutes.”
Datadog automatically translates this request into a structured query—understanding your context, services, tags, and time ranges—and correlates relevant traces and metrics. In the following screenshot, you can see how CloudPrem logs are correlated with APM traces in the same window, facilitating your troubleshooting investigations just like when your logs are stored in Datadog SaaS.
CloudPrem is also MCP-ready, supporting the Model Context Protocol for AI and LLM interoperability. This enables customers to use their own agents—such as those built on OpenAI, Anthropic, or Gemini frameworks—to discover datasets, generate structured queries, and retrieve insights directly from CloudPrem. With MCP, AI systems can reason natively over your observability data, enabling intelligent analytics and faster decision-making without any custom integration.
By correlating your self-hosted logs with metrics and traces in a single view, you can now investigate issues faster, reduce context switching, and make decisions with full visibility across your system.
Security teams are especially familiar with the pains of log growth. Every firewall connection, CDN request, and VPC flow produces chatty, high-volume data that’s too important to discard and too expensive to store in a traditional SIEM. To control costs, many teams stitch together a SIEM for active alerts, a cheap storage tier for long-term retention, and custom scripts to bridge the gap. The result is fragmented data, high operational overhead, and visibility gaps that block investigations.
CloudPrem consolidates these security workloads into one place. You can retain petabytes of security telemetry from Akamai, Palo Alto Firewall, Zscaler, and more without worrying about SIEM ingestion limits or vendor pricing tiers.
For teams that want to preprocess or enrich data before indexing, logs can also flow through Observability Pipelines. Pipelines make it easy to normalize logs with OCSF; enrich logs with GeoIP, threat-intelligence feeds, and contextual metadata; or apply filters to control cost. Out-of-the-box Packs extend these capabilities with prebuilt rules for popular security sources, helping you reduce noisy traffic and standardize enrichment policies across environments. This helps ensure that incoming security data is structured and enriched with the context needed to spot patterns quickly.
Once ingested, CloudPrem automatically organizes this data for investigation. Using AI-based parsing, Datadog detects common security log formats and extracts key fields, while Datadog’s correlation engine links related events across metrics, traces, and logs. Because it shares the same indexing and pipeline framework as Datadog SaaS, security teams get scalable retention and unified visibility without losing the Datadog experience they rely on.
For many organizations, data location is dictated by policy or regulation. Countries such as India, Singapore, Brazil, and China are introducing stricter data-residency laws, requiring organizations to keep sensitive data within national or regional boundaries. Whether you’re processing financial transactions in the European Union, managing patient data in healthcare, or securing government systems, some logs simply can’t leave your network.
CloudPrem helps you meet these data residency, privacy, and security requirements without giving up unified observability and security. Your logs stay where they must—but everything else, from correlation to AI-assisted analysis, works exactly as it normally does in Datadog.
CloudPrem also includes the same enterprise governance features that power Datadog’s SaaS platform:
CloudPrem enables you to retain control of logs in your own infrastructure while still providing full access to Datadog platform capabilities—combining scalability, ease of use, compliance, and AI-powered analysis in one consistent, integrated experience. With CloudPrem, you can manage petabytes of logs efficiently, support regional data-residency requirements, and maintain full visibility across hybrid environments.
CloudPrem is now available in preview. To learn more, visit the CloudPrem documentation or reach out to your Datadog representative.
If you’re new to Datadog, sign up for a 14-day free trial.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。