Exponential log growth doesn’t have to drive exponential cost growth. Storing and analyzing logs at scale can be expensive, but Flex Logs—Datadog’s high-volume, cost-efficient log storage solution—enables teams to store more logs for new use cases while staying within budget.
Now teams using Flex Logs have greater visibility into how their Flex compute is being used. With the new compute usage graphs on the Flex Logs Controls page, you can monitor performance, identify slowdowns, and make informed decisions about scaling or optimizing usage.
In addition to providing a refresher on Flex Logs, this post describes how to:
Flex Logs enables teams to store and query high-volume log data by decoupling the cost of storage and compute. Teams can store vast amounts of logs for up to 15 months while independently choosing a compute size based on their team’s querying needs. Flex Logs works alongside Standard Indexing, giving teams the flexibility to choose which logs are available for real-time troubleshooting use cases and which are retained primarily for ad hoc analysis.
For example, you can use the value of the log to determine which retention tier should be used to balance cost efficiencies and business needs. Application logs from production environments with an ERROR and WARN level should be stored in Standard Indexing first for use in incident response, while logs at an INFO or DEBUG level can be stored directly in Flex Tier.
The different retention tiers can also be determined based on the volume of the log. Noisy logs from sources like CDN, WAF, and DNS services are also good candidates for storing directly in the Flex Tier. Additional recommendations on candidates for the Flex Tier can be found in the documentation.
Datadog now displays Flex query performance on the Flex Logs Controls page. These graphs provide visibility into how your compute is being used, helping you determine whether your current setup meets your needs or if it’s time to optimize or upgrade.
One of the limits of Flex compute is the number of concurrent Flex queries that can be run. When your Flex compute reaches its maximum capacity, new queries must wait for available capacity before executing. To address this, the new graphs on the Flex Logs Controls page enable you to see:
This makes it easier to correlate performance issues with compute capacity and helps teams identify and understand areas of high compute usage.
On the Flex Logs Controls page, you can dig deeper to view the top users and dashboards experiencing query slowdowns. If a dashboard is consistently experiencing slowdowns, it might be time to optimize its performance or move frequently accessed logs into Standard Indexing.
You can also identify if a small group of users or teams are responsible for a disproportionate share of compute usage. Click on top users to view an Audit Trail history of log queries they’ve made, and consider contacting them to understand if they have new workloads or just temporary increases in queries due to testing. This increased visibility into Flex compute usage helps you unearth opportunities to refine log storage throughout your organization.
If you’ve identified areas for optimization, consider the following best practices to improve log query performance and dashboard responsiveness.
To improve query efficiency, specify the log index directly in your queries when you’re working with known datasets. This helps avoid unnecessary scanning and speeds up results.
You can also optimize dashboards to reduce compute load and improve responsiveness. If a widget is only displaying counts of logs with low information density, consider converting those logs into custom metrics and switching to metric-based widgets. Organize widgets into Groups and keep them collapsed until needed to prevent unnecessary queries from being started. During investigations, pause auto-refresh by clicking the “pause” button next to the time window to avoid constant reloading of queries.
If you’re seeing sustained slowdowns or frequent query throttling, consider upgrading your Flex Compute size. This increases your concurrent query limits and improves responsiveness.
The right approach depends on your team’s workflows and priorities. These insights help you fine-tune your configuration to improve performance without unnecessary spend. For more tips, see the Flex Compute usage guide.
Flex Logs offers a flexible, cost-effective way to store and query large volumes of logs. Now, with Flex compute usage insights, you have the transparency needed to manage performance as your usage scales.
To learn more, check out our Flex Logs documentation. If you aren’t yet a Datadog user, you can start exploring compute usage in your own account with a 14-day free trial.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。