We’re excited to learn that our vmagent helped Airbnb migrate its high-volume metrics pipeline from StatsD and Veneur to OpenTelemetry. Airbnb is now handling 100 million samples per second. You can read more about the migration in these articles:

• The Airbnb Tech Blog: Building a high-volume metrics pipeline with OpenTelemetry and vmagent
• InfoQ: Airbnb Migrates High-Volume Metrics Pipeline to OpenTelemetry

In other news, April saw releases across the VictoriaMetrics Observability Stack. We have released several important bugfixes for VictoriaMetrics and many new features in VictoriaLogs.

This release round-up covers updates for:

• VictoriaMetrics
• VictoriaLogs

VictoriaMetrics v1.141.0 and v1.142.0 provide important bugfixes

#

These releases are mostly focused on bug fixes, including two critical issues. Users should avoid affected versions or upgrade to the latest release.

Critical issues

#

OpenTelemetry ingestion issues

There’s a bug in OpenTelemetry ingestion when the -opentelemetry.usePrometheusNaming flag is enabled. Metric names can be incorrectly modified based on the previously processed metric. For example, if a metric http_requests has Unit: seconds, and the next metric has no unit (e.g., cpu_usage), then the latter could be ingested incorrectly using the last unit (e.g., cpu_usage_seconds). For more information, see #10889.

Incorrect binary operation precedence

There is a known serious issue in v1.140.0, v1.136.4, and v1.122.19 versions where binary operation precedence may be evaluated incorrectly, leading to unexpected query results. For example, expressions like 10 - (3 + 3 + 4) are evaluated as 10 - 3 + 3 + 4.

Please avoid upgrading or using these versions in production. Instead, use a fixed version such as v1.136.8 or v1.142.0.

For more details, read the #10856 on GitHub.

Notable improvements in v1.141.0 and v1.142.0

#

• Configurable future sample retention: Added -futureRetention flag. Previously, future-dated samples were capped at 2 days; now this window is configurable. Useful for workloads that ingest forecasts or predictions.
• More accurate alert debugging in VMUI: The UI now preserves alert parameters (e.g., group interval, evaluationTime) when clicking Run Query on Alert view and going to Query execution tab. This makes the Query tab behave much closer to actual alert evaluation and reduces confusion.
• Metadata visibility in cluster Grafana dashboards: Metadata ingestion is now reflected in cluster dashboard. Since metadata processing can take up considerable resources, this information lets you spot potential issues before they impact your cluster.
• Correct tenant attribution for metadata: Metadata ingested via multitenant endpoints or scraped with __tenant_id__ is now properly associated with the correct tenant.

Read the changelogs:

• v1.141.0
• v1.142.0

VictoriaLogs v1.50.0 now supports Splunk Ingestion

#

• Splunk HEC ingestion: VictoriaLogs supports Splunk HEC-compatible endpoints such as /services/collector/event and /insert/splunk/services/collector/event, allowing Splunk HEC clients (for example, Docker’s Splunk logging driver) to send logs to VictoriaLogs with minimal integration changes.
• File‑based collection in vlagent: vlagent can collect logs from files matched by -fileCollector.glob, store read checkpoints, attach file and hostname fields, and support the common create/rename log rotation strategy, making it convenient to collect application logs from paths such as /var/log/nginx/*.log without a separate log shipper.
• Multi‑field LogsQL search: LogsQL now supports field‑prefix filters such as kubernetes.*:nginx and *:error, which simplifies searching across dynamic or nested field groups without listing every field name, while still allowing precise field filters when performance is important.
• Loki JSON message field prefixing: Loki ingestion supports -loki.messageFieldsPrefix and the message_fields_prefix query arg; JSON fields extracted from the log message can be stored with a prefix such as msg., reducing name collisions with stream labels or existing fields.
• Native Datadog Agent paths: VictoriaLogs accepts Datadog’s default /api/v2/logs and /api/v1/validate paths directly, in addition to /insert/datadog/…, which simplifies Datadog Agent and serverless Datadog setups since path rewriting through vmauth is no longer required.

Read the changelog

VictoriaLogs v1.49.0 highlights

#

• CSV export: Logs from /select/logsql/query can now be returned in CSV format, and the web UI supports downloading logs as CSV for use in Excel, Google Sheets, sharing with teams, or as an attachment in tickets.
• Better _stream ingestion: VictoriaLogs can ingest logs with pre-formatted _stream fields, making re-import tasks easier. In addition, VictoriaLogs now validates that _stream matches row fields to catch incorrect metadata early.
• New multitenant native endpoint: Added /insert/multitenant/native for sending native-format logs with mixed tenants from vlagent; recommended over /internal/insert.
• Global filter propagation: options(global_filter=(...)) allows defining a common filter applied across subqueries, reducing repetition in joins, unions, and nested queries.
• Enhanced conditional stats: Added switch(case ..., default ...) in stats pipe to simplify grouped counters like 2xx, 4xx, and 5xx classifications.
• Standard deviation support: Introduced stddev(...) stats function for analyzing variability in values such as latency or payload size.
• Static data in LogsQL: Queries can now use join by (...) rows(...) to enrich logs with predefined static fields or append static rows with union rows(...). Handy for small lookup data, labels, or quick comparisons without preparing extra data outside VictoriaLogs.
• Field discovery improvements: field_names and field_values pipes now support substring filtering, and the field/stream discovery HTTP APIs also support the filter query arg.
• Easier filtering in Web UI: In log details, you can now include or exclude a field value directly from the UI. That saves time during investigation because you do not need to manually edit the query for every small drill-down step.
• LogsQL examples in Web UI: A query example modal was added with categories, so beginners can pick a pattern and run it right away.
• The hits chart was also improved: Clicking legend items now focuses a series or shows all series, while Cmd/Ctrl-click toggles hide/show. Additional actions moved to a hover menu.

Read the changelog

Changelogs

#

• VictoriaMetrics v1.142.0
• VictoriaMetrics v1.141.0
• VictoriaLogs v1.50.0
• VictoriaLogs v1.49.0

#

During April, VictoriaMetrics participated in several community events:

• KubeCon + CloudNativeCon Europe Amsterdam: these events brought together 13,500 attendees, the largest turnout yet. We were happy to talk with our users at our booth, present talks, and participate in panels.
  • We also hosted our very first after party: “After Deploy”. Check out the talks and pictures on our blog.
  • Roman’s and Jiekun’s talk at KubeCon on cutting 90% Distributed Tracing Bandwidth on OpenTelemetry has been published as a blog post. Read it here: Optimizing Tail Sampling in OpenTelemetry with Retroactive Sampling.
• VictoriaMetrics Virtual Meetup: this year’s Virtual Meetup took place in April. We presented roadmap updates for Q1, news on Anomaly Detection, updates on VictoriaMetrics Cloud and VictoriaLogs, and even had time for an AMA. Watch the recording on YouTube.
• LinuxFest Northwest (April 24–26, Bellingham): As Gold sponsor, delivered talks on using VictoriaMetrics for observability in SIEM systems, homelab monitoring setups, intelligent routing optimizations, and cost-optimized Kubernetes observability
• StackConf (April 28–29, Munich): VictoriaMetrics was a Bronze sponsor featuring sessions on aggregating metrics in-flight to reduce cardinality and building full OSS observability stacks combining VictoriaMetrics with OpenTelemetry Collector.

That’s a wrap for April. Here’s to even more developments and exciting events for May. Thank you for reading!