@@ -0,0 +1,304 @@
1+---
2+summary: "Export OpenClaw diagnostics to any OpenTelemetry collector via the diagnostics-otel plugin (OTLP/HTTP)"
3+title: "OpenTelemetry export"
4+read_when:
5+ - You want to send OpenClaw model usage, message flow, or session metrics to an OpenTelemetry collector
6+ - You are wiring traces, metrics, or logs into Grafana, Datadog, Honeycomb, New Relic, Tempo, or another OTLP backend
7+ - You need the exact metric names, span names, or attribute shapes to build dashboards or alerts
8+---
9+10+OpenClaw exports diagnostics through the bundled `diagnostics-otel` plugin
11+using **OTLP/HTTP (protobuf)**. Any collector or backend that accepts OTLP/HTTP
12+works without code changes. For local file logs and how to read them, see
13+[Logging](/logging).
14+15+## How it fits together
16+17+- **Diagnostics events** are structured, in-process records emitted by the
18+ Gateway and bundled plugins for model runs, message flow, sessions, queues,
19+ and exec.
20+- **`diagnostics-otel` plugin** subscribes to those events and exports them as
21+ OpenTelemetry **metrics**, **traces**, and **logs** over OTLP/HTTP.
22+- Exporters only attach when both the diagnostics surface and the plugin are
23+ enabled, so the in-process cost stays near zero by default.
24+25+## Quick start
26+27+```json5
28+{
29+ plugins: {
30+ allow: ["diagnostics-otel"],
31+ entries: {
32+"diagnostics-otel": { enabled: true },
33+ },
34+ },
35+ diagnostics: {
36+ enabled: true,
37+ otel: {
38+ enabled: true,
39+ endpoint: "http://otel-collector:4318",
40+ protocol: "http/protobuf",
41+ serviceName: "openclaw-gateway",
42+ traces: true,
43+ metrics: true,
44+ logs: true,
45+ sampleRate: 0.2,
46+ flushIntervalMs: 60000,
47+ },
48+ },
49+}
50+```
51+52+You can also enable the plugin from the CLI:
53+54+```bash
55+openclaw plugins enable diagnostics-otel
56+```
57+58+<Note>
59+`protocol` currently supports `http/protobuf` only. `grpc` is ignored.
60+</Note>
61+62+## Signals exported
63+64+| Signal | What goes in it |
65+| ----------- | --------------------------------------------------------------------------------------------------------------------------------- |
66+| **Metrics** | Counters and histograms for token usage, cost, run duration, message flow, queue lanes, session state, exec, and memory pressure. |
67+| **Traces** | Spans for model usage, model calls, tool execution, exec, webhook/message processing, context assembly, and tool loops. |
68+| **Logs** | Structured `logging.file` records exported over OTLP when `diagnostics.otel.logs` is enabled. |
69+70+Toggle `traces`, `metrics`, and `logs` independently. All three default to on
71+when `diagnostics.otel.enabled` is true.
72+73+## Configuration reference
74+75+```json5
76+{
77+ diagnostics: {
78+ enabled: true,
79+ otel: {
80+ enabled: true,
81+ endpoint: "http://otel-collector:4318",
82+ protocol: "http/protobuf", // grpc is ignored
83+ serviceName: "openclaw-gateway",
84+ headers: { "x-collector-token": "..." },
85+ traces: true,
86+ metrics: true,
87+ logs: true,
88+ sampleRate: 0.2, // root-span sampler, 0.0..1.0
89+ flushIntervalMs: 60000, // metric export interval (min 1000ms)
90+ captureContent: {
91+ enabled: false,
92+ inputMessages: false,
93+ outputMessages: false,
94+ toolInputs: false,
95+ toolOutputs: false,
96+ systemPrompt: false,
97+ },
98+ },
99+ },
100+}
101+```
102+103+### Environment variables
104+105+| Variable | Purpose |
106+| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
107+| `OTEL_EXPORTER_OTLP_ENDPOINT` | Override `diagnostics.otel.endpoint`. If the value already contains `/v1/traces`, `/v1/metrics`, or `/v1/logs`, it is used as-is. |
108+| `OTEL_SERVICE_NAME` | Override `diagnostics.otel.serviceName`. |
109+| `OTEL_EXPORTER_OTLP_PROTOCOL` | Override the wire protocol (only `http/protobuf` is honored today). |
110+| `OTEL_SEMCONV_STABILITY_OPT_IN` | Set to `gen_ai_latest_experimental` to emit the latest experimental GenAI span attribute (`gen_ai.provider.name`) instead of the legacy `gen_ai.system`. GenAI metrics always use bounded, low-cardinality semantic attributes regardless. |
111+| `OPENCLAW_OTEL_PRELOADED` | Set to `1` when another preload or host process already registered the global OpenTelemetry SDK. The plugin then skips its own NodeSDK lifecycle but still wires diagnostic listeners and honors `traces`/`metrics`/`logs`. |
112+113+## Privacy and content capture
114+115+Raw model/tool content is **not** exported by default. Spans carry bounded
116+identifiers (channel, provider, model, error category, hash-only request ids)
117+and never include prompt text, response text, tool inputs, tool outputs, or
118+session keys.
119+120+Set `diagnostics.otel.captureContent.*` to `true` only when your collector and
121+retention policy are approved for prompt, response, tool, or system-prompt
122+text. Each subkey is opt-in independently:
123+124+- `inputMessages` — user prompt content.
125+- `outputMessages` — model response content.
126+- `toolInputs` — tool argument payloads.
127+- `toolOutputs` — tool result payloads.
128+- `systemPrompt` — assembled system/developer prompt.
129+130+When any subkey is enabled, model and tool spans get bounded, redacted
131+`openclaw.content.*` attributes for that class only.
132+133+## Sampling and flushing
134+135+- **Traces:** `diagnostics.otel.sampleRate` (root-span only, `0.0` drops all,
136+`1.0` keeps all).
137+- **Metrics:** `diagnostics.otel.flushIntervalMs` (minimum `1000`).
138+- **Logs:** OTLP logs respect `logging.level` (file log level). Console
139+ redaction does **not** apply to OTLP logs. High-volume installs should
140+ prefer OTLP collector sampling/filtering over local sampling.
141+142+## Exported metrics
143+144+### Model usage
145+146+- `openclaw.tokens` (counter, attrs: `openclaw.token`, `openclaw.channel`, `openclaw.provider`, `openclaw.model`)
147+- `openclaw.cost.usd` (counter, attrs: `openclaw.channel`, `openclaw.provider`, `openclaw.model`)
148+- `openclaw.run.duration_ms` (histogram, attrs: `openclaw.channel`, `openclaw.provider`, `openclaw.model`)
149+- `openclaw.context.tokens` (histogram, attrs: `openclaw.context`, `openclaw.channel`, `openclaw.provider`, `openclaw.model`)
150+- `gen_ai.client.token.usage` (histogram, GenAI semantic-conventions metric, attrs: `gen_ai.token.type` = `input`/`output`, `gen_ai.provider.name`, `gen_ai.operation.name`, `gen_ai.request.model`)
151+- `gen_ai.client.operation.duration` (histogram, seconds, GenAI semantic-conventions metric, attrs: `gen_ai.provider.name`, `gen_ai.operation.name`, `gen_ai.request.model`, optional `error.type`)
152+153+### Message flow
154+155+- `openclaw.webhook.received` (counter, attrs: `openclaw.channel`, `openclaw.webhook`)
156+- `openclaw.webhook.error` (counter, attrs: `openclaw.channel`, `openclaw.webhook`)
157+- `openclaw.webhook.duration_ms` (histogram, attrs: `openclaw.channel`, `openclaw.webhook`)
158+- `openclaw.message.queued` (counter, attrs: `openclaw.channel`, `openclaw.source`)
159+- `openclaw.message.processed` (counter, attrs: `openclaw.channel`, `openclaw.outcome`)
160+- `openclaw.message.duration_ms` (histogram, attrs: `openclaw.channel`, `openclaw.outcome`)
161+- `openclaw.message.delivery.started` (counter, attrs: `openclaw.channel`, `openclaw.delivery.kind`)
162+- `openclaw.message.delivery.duration_ms` (histogram, attrs: `openclaw.channel`, `openclaw.delivery.kind`, `openclaw.outcome`, `openclaw.errorCategory`)
163+164+### Queues and sessions
165+166+- `openclaw.queue.lane.enqueue` (counter, attrs: `openclaw.lane`)
167+- `openclaw.queue.lane.dequeue` (counter, attrs: `openclaw.lane`)
168+- `openclaw.queue.depth` (histogram, attrs: `openclaw.lane` or `openclaw.channel=heartbeat`)
169+- `openclaw.queue.wait_ms` (histogram, attrs: `openclaw.lane`)
170+- `openclaw.session.state` (counter, attrs: `openclaw.state`, `openclaw.reason`)
171+- `openclaw.session.stuck` (counter, attrs: `openclaw.state`)
172+- `openclaw.session.stuck_age_ms` (histogram, attrs: `openclaw.state`)
173+- `openclaw.run.attempt` (counter, attrs: `openclaw.attempt`)
174+175+### Exec
176+177+- `openclaw.exec.duration_ms` (histogram, attrs: `openclaw.exec.target`, `openclaw.exec.mode`, `openclaw.outcome`, `openclaw.failureKind`)
178+179+### Diagnostics internals (memory and tool loop)
180+181+- `openclaw.memory.heap_used_bytes` (histogram, attrs: `openclaw.memory.kind`)
182+- `openclaw.memory.rss_bytes` (histogram)
183+- `openclaw.memory.pressure` (counter, attrs: `openclaw.memory.level`)
184+- `openclaw.tool.loop.iterations` (counter, attrs: `openclaw.toolName`, `openclaw.outcome`)
185+- `openclaw.tool.loop.duration_ms` (histogram, attrs: `openclaw.toolName`, `openclaw.outcome`)
186+187+## Exported spans
188+189+- `openclaw.model.usage`
190+- `openclaw.channel`, `openclaw.provider`, `openclaw.model`
191+- `openclaw.tokens.*` (input/output/cache_read/cache_write/total)
192+- `gen_ai.system` by default, or `gen_ai.provider.name` when the latest GenAI semantic conventions are opted in
193+- `gen_ai.request.model`, `gen_ai.operation.name`, `gen_ai.usage.*`
194+- `openclaw.run`
195+- `openclaw.outcome`, `openclaw.channel`, `openclaw.provider`, `openclaw.model`, `openclaw.errorCategory`
196+- `openclaw.model.call`
197+- `gen_ai.system` by default, or `gen_ai.provider.name` when the latest GenAI semantic conventions are opted in
198+- `gen_ai.request.model`, `gen_ai.operation.name`, `openclaw.provider`, `openclaw.model`, `openclaw.api`, `openclaw.transport`
199+- `openclaw.provider.request_id_hash` (bounded SHA-based hash of the upstream provider request id; raw ids are not exported)
200+- `openclaw.tool.execution`
201+- `gen_ai.tool.name`, `openclaw.toolName`, `openclaw.errorCategory`, `openclaw.tool.params.*`
202+- `openclaw.exec`
203+- `openclaw.exec.target`, `openclaw.exec.mode`, `openclaw.outcome`, `openclaw.failureKind`, `openclaw.exec.command_length`, `openclaw.exec.exit_code`, `openclaw.exec.timed_out`
204+- `openclaw.webhook.processed`
205+- `openclaw.channel`, `openclaw.webhook`, `openclaw.chatId`
206+- `openclaw.webhook.error`
207+- `openclaw.channel`, `openclaw.webhook`, `openclaw.chatId`, `openclaw.error`
208+- `openclaw.message.processed`
209+- `openclaw.channel`, `openclaw.outcome`, `openclaw.chatId`, `openclaw.messageId`, `openclaw.reason`
210+- `openclaw.message.delivery`
211+- `openclaw.channel`, `openclaw.delivery.kind`, `openclaw.outcome`, `openclaw.errorCategory`, `openclaw.delivery.result_count`
212+- `openclaw.session.stuck`
213+- `openclaw.state`, `openclaw.ageMs`, `openclaw.queueDepth`
214+- `openclaw.context.assembled`
215+- `openclaw.prompt.size`, `openclaw.history.size`, `openclaw.context.tokens`, `openclaw.errorCategory` (no prompt, history, response, or session-key content)
216+- `openclaw.tool.loop`
217+- `openclaw.toolName`, `openclaw.outcome`, `openclaw.iterations`, `openclaw.errorCategory` (no loop messages, params, or tool output)
218+- `openclaw.memory.pressure`
219+- `openclaw.memory.level`, `openclaw.memory.heap_used_bytes`, `openclaw.memory.rss_bytes`
220+221+When content capture is explicitly enabled, model and tool spans can also
222+include bounded, redacted `openclaw.content.*` attributes for the specific
223+content classes you opted into.
224+225+## Diagnostic event catalog
226+227+The events below back the metrics and spans above. Plugins can also subscribe
228+to them directly without OTLP export.
229+230+**Model usage**
231+232+- `model.usage` — tokens, cost, duration, context, provider/model/channel,
233+ session ids. `usage` is provider/turn accounting for cost and telemetry;
234+`context.used` is the current prompt/context snapshot and can be lower than
235+ provider `usage.total` when cached input or tool-loop calls are involved.
236+237+**Message flow**
238+239+- `webhook.received` / `webhook.processed` / `webhook.error`
240+- `message.queued` / `message.processed`
241+- `message.delivery.started` / `message.delivery.completed` / `message.delivery.error`
242+243+**Queue and session**
244+245+- `queue.lane.enqueue` / `queue.lane.dequeue`
246+- `session.state` / `session.stuck`
247+- `run.attempt`
248+- `diagnostic.heartbeat` (aggregate counters: webhooks/queue/session)
249+250+**Exec**
251+252+- `exec.process.completed` — terminal outcome, duration, target, mode, exit
253+ code, and failure kind. Command text and working directories are not
254+ included.
255+256+## Without an exporter
257+258+You can keep diagnostics events available to plugins or custom sinks without
259+running `diagnostics-otel`:
260+261+```json5
262+{
263+ diagnostics: { enabled: true },
264+}
265+```
266+267+For targeted debug output without raising `logging.level`, use diagnostics
268+flags. Flags are case-insensitive and support wildcards (e.g. `telegram.*` or
269+`*`):
270+271+```json5
272+{
273+ diagnostics: { flags: ["telegram.http"] },
274+}
275+```
276+277+Or as a one-off env override:
278+279+```bash
280+OPENCLAW_DIAGNOSTICS=telegram.http,telegram.payload openclaw gateway
281+```
282+283+Flag output goes to the standard log file (`logging.file`) and is still
284+redacted by `logging.redactSensitive`. Full guide:
285+[Diagnostics flags](/diagnostics/flags).
286+287+## Disable
288+289+```json5
290+{
291+ diagnostics: { otel: { enabled: false } },
292+}
293+```
294+295+You can also leave `diagnostics-otel` out of `plugins.allow`, or run
296+`openclaw plugins disable diagnostics-otel`.
297+298+## Related
299+300+- [Logging](/logging) — file logs, console output, CLI tailing, and the Control UI Logs tab
301+- [Gateway logging internals](/gateway/logging) — WS log styles, subsystem prefixes, and console capture
302+- [Diagnostics flags](/diagnostics/flags) — targeted debug-log flags
303+- [Diagnostics export](/gateway/diagnostics) — operator support-bundle tool (separate from OTEL export)
304+- [Configuration reference](/gateway/configuration-reference#diagnostics) — full `diagnostics.*` field reference
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。