





















@@ -1,47 +1,84 @@
11---
22summary: "Expose OpenClaw diagnostics as Prometheus text metrics through the diagnostics-prometheus plugin"
33title: "Prometheus metrics"
4+sidebarTitle: "Prometheus"
45read_when:
56 - You want Prometheus, Grafana, VictoriaMetrics, or another scraper to collect OpenClaw Gateway metrics
67 - You need the Prometheus metric names and label policy for dashboards or alerts
78 - You want metrics without running an OpenTelemetry collector
89---
91010-OpenClaw can expose diagnostics metrics through the bundled
11-`diagnostics-prometheus` plugin. It listens to trusted internal diagnostics and
12-renders a Prometheus text endpoint at:
11+OpenClaw can expose diagnostics metrics through the bundled `diagnostics-prometheus` plugin. It listens to trusted internal diagnostics and renders a Prometheus text endpoint at:
13121413```text
15-/api/diagnostics/prometheus
14+GET /api/diagnostics/prometheus
1615```
171618-The route uses Gateway authentication. Do not expose it as a public
19-unauthenticated `/metrics` endpoint.
17+Content type is `text/plain; version=0.0.4; charset=utf-8`, the standard Prometheus exposition format.
18+19+<Warning>
20+The route uses Gateway authentication (operator scope). Do not expose it as a public unauthenticated `/metrics` endpoint. Scrape it through the same auth path you use for other operator APIs.
21+</Warning>
22+23+For traces, logs, OTLP push, and OpenTelemetry GenAI semantic attributes, see [OpenTelemetry export](/gateway/opentelemetry).
20242125## Quick start
222623-```json5
24-{
25- plugins: {
26- allow: ["diagnostics-prometheus"],
27- entries: {
28-"diagnostics-prometheus": { enabled: true },
29- },
30- },
31- diagnostics: {
32- enabled: true,
33- },
34-}
35-```
27+<Steps>
28+<Step title="Enable the plugin">
29+<Tabs>
30+ <Tab title="Config">
31+ ```json5
32+ {
33+ plugins: {
34+ allow: ["diagnostics-prometheus"],
35+ entries: {
36+ "diagnostics-prometheus": { enabled: true },
37+ },
38+ },
39+ diagnostics: {
40+ enabled: true,
41+ },
42+ }
43+ ```
44+ </Tab>
45+ <Tab title="CLI">
46+ ```bash
47+ openclaw plugins enable diagnostics-prometheus
48+ ```
49+ </Tab>
50+</Tabs>
51+</Step>
52+<Step title="Restart the Gateway">
53+The HTTP route is registered at plugin startup, so reload after enabling.
54+</Step>
55+<Step title="Scrape the protected route">
56+Send the same gateway auth your operator clients use:
365737-You can also enable the plugin from the CLI:
58+```bash
59+curl -H "Authorization: Bearer $OPENCLAW_GATEWAY_TOKEN" \
60+ http://127.0.0.1:18789/api/diagnostics/prometheus
61+```
386239-```bash
40-openclaw plugins enable diagnostics-prometheus
41-```
63+</Step>
64+<Step title="Wire Prometheus">
65+```yaml
66+# prometheus.yml
67+scrape_configs:
68+ - job_name: openclaw
69+ scrape_interval: 30s
70+ metrics_path: /api/diagnostics/prometheus
71+ authorization:
72+ credentials_file: /etc/prometheus/openclaw-gateway-token
73+ static_configs:
74+ - targets: ["openclaw-gateway:18789"]
75+```
76+</Step>
77+</Steps>
427843-Then scrape the protected Gateway route with the same Gateway authentication you
44-use for operator APIs.
79+<Note>
80+`diagnostics.enabled: true` is required. Without it, the plugin still registers the HTTP route but no diagnostic events flow into the exporter, so the response is empty.
81+</Note>
45824683## Metrics exported
4784@@ -74,16 +111,99 @@ use for operator APIs.
7411175112## Label policy
7611377-Prometheus labels stay bounded and low-cardinality. The exporter does not emit
78-raw diagnostic identifiers such as `runId`, `sessionKey`, `sessionId`, `callId`,
79-`toolCallId`, message IDs, chat IDs, or provider request IDs.
114+<AccordionGroup>
115+<Accordion title="Bounded, low-cardinality labels">
116+Prometheus labels stay bounded and low-cardinality. The exporter does not emit raw diagnostic identifiers such as `runId`, `sessionKey`, `sessionId`, `callId`, `toolCallId`, message IDs, chat IDs, or provider request IDs.
117+118+Label values are redacted and must match OpenClaw's low-cardinality character policy. Values that fail the policy are replaced with `unknown`, `other`, or `none`, depending on the metric.
119+120+</Accordion>
121+<Accordion title="Series cap and overflow accounting">
122+The exporter caps retained time series in memory at **2048** series across counters, gauges, and histograms combined. New series beyond that cap are dropped, and `openclaw_prometheus_series_dropped_total` increments by one each time.
123+124+Watch this counter as a hard signal that an attribute upstream is leaking high-cardinality values. The exporter never lifts the cap automatically; if it climbs, fix the source rather than disabling the cap.
125+126+</Accordion>
127+<Accordion title="What never appears in Prometheus output">
128+- prompt text, response text, tool inputs, tool outputs, system prompts
129+- raw provider request IDs (only bounded hashes, where applicable, on spans — never on metrics)
130+- session keys and session IDs
131+- hostnames, file paths, secret values
132+</Accordion>
133+</AccordionGroup>
134+135+## PromQL recipes
136+137+```promql
138+# Tokens per minute, split by provider
139+sum by (provider) (rate(openclaw_model_tokens_total[1m]))
140+141+# Spend (USD) over the last hour, by model
142+sum by (model) (increase(openclaw_model_cost_usd_total[1h]))
143+144+# 95th percentile model run duration
145+histogram_quantile(
146+ 0.95,
147+ sum by (le, provider, model)
148+ (rate(openclaw_run_duration_seconds_bucket[5m]))
149+)
150+151+# Queue wait time SLO (95p under 2s)
152+histogram_quantile(
153+ 0.95,
154+ sum by (le, lane) (rate(openclaw_queue_lane_wait_seconds_bucket[5m]))
155+) < 2
156+157+# Dropped Prometheus series (cardinality alarm)
158+increase(openclaw_prometheus_series_dropped_total[15m]) > 0
159+```
160+161+<Tip>
162+Prefer `gen_ai_client_token_usage` for cross-provider dashboards: it follows the OpenTelemetry GenAI semantic conventions and is consistent with metrics from non-OpenClaw GenAI services.
163+</Tip>
164+165+## Choosing between Prometheus and OpenTelemetry export
166+167+OpenClaw supports both surfaces independently. You can run either, both, or neither.
168+169+<Tabs>
170+<Tab title="diagnostics-prometheus">
171+- **Pull** model: Prometheus scrapes `/api/diagnostics/prometheus`.
172+- No external collector required.
173+- Authenticated through normal Gateway auth.
174+- Surface is metrics only (no traces or logs).
175+- Best for stacks already standardized on Prometheus + Grafana.
176+</Tab>
177+<Tab title="diagnostics-otel">
178+- **Push** model: OpenClaw sends OTLP/HTTP to a collector or OTLP-compatible backend.
179+- Surface includes metrics, traces, and logs.
180+- Bridges to Prometheus through an OpenTelemetry Collector (`prometheus` or `prometheusremotewrite` exporter) when you need both.
181+- See [OpenTelemetry export](/gateway/opentelemetry) for the full catalog.
182+</Tab>
183+</Tabs>
184+185+## Troubleshooting
8018681-Label values are redacted and must match OpenClaw's low-cardinality character
82-policy. Values that fail the policy are replaced with `unknown`, `other`, or
83-`none`, depending on the metric.
187+<AccordionGroup>
188+<Accordion title="Empty response body">
189+- Check `diagnostics.enabled: true` in config.
190+- Confirm the plugin is enabled and loaded with `openclaw plugins list --enabled`.
191+- Generate some traffic; counters and histograms only emit lines after at least one event.
192+</Accordion>
193+<Accordion title="401 / unauthorized">
194+The endpoint requires the Gateway operator scope (`auth: "gateway"` with `gatewayRuntimeScopeSurface: "trusted-operator"`). Use the same token or password Prometheus uses for any other Gateway operator route. There is no public unauthenticated mode.
195+</Accordion>
196+<Accordion title="`openclaw_prometheus_series_dropped_total` is climbing">
197+A new attribute is exceeding the **2048**-series cap. Inspect recent metrics for an unexpectedly high-cardinality label and fix it at the source. The exporter intentionally drops new series instead of silently rewriting labels.
198+</Accordion>
199+<Accordion title="Prometheus shows stale series after a restart">
200+The plugin keeps state in memory only. After a Gateway restart, counters reset to zero and gauges restart at their next reported value. Use PromQL `rate()` and `increase()` to handle resets cleanly.
201+</Accordion>
202+</AccordionGroup>
8420385-The exporter caps retained time series in memory. If the cap is reached, new
86-series are dropped and `openclaw_prometheus_series_dropped_total` increments.
204+## Related
8720588-For full traces, logs, OTLP export, and OpenTelemetry GenAI semantic attributes,
89-use [OpenTelemetry export](/gateway/opentelemetry).
206+- [Diagnostics export](/gateway/diagnostics) — local diagnostics zip for support bundles
207+- [Health and readiness](/gateway/health) — `/healthz` and `/readyz` probes
208+- [Logging](/logging) — file-based logging
209+- [OpenTelemetry export](/gateway/opentelemetry) — OTLP push for traces, metrics, and logs
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。