惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

大猫的无限游戏
大猫的无限游戏
J
Java Code Geeks
小众软件
小众软件
D
Docker
腾讯CDC
H
Hackread – Cybersecurity News, Data Breaches, AI and More
V
V2EX
博客园 - 叶小钗
GbyAI
GbyAI
Microsoft Azure Blog
Microsoft Azure Blog
Stack Overflow Blog
Stack Overflow Blog
B
Blog RSS Feed
酷 壳 – CoolShell
酷 壳 – CoolShell
博客园 - 【当耐特】
IT之家
IT之家
博客园 - 司徒正美
M
MIT News - Artificial intelligence
T
The Blog of Author Tim Ferriss
The GitHub Blog
The GitHub Blog
罗磊的独立博客
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
L
LangChain Blog
阮一峰的网络日志
阮一峰的网络日志
C
Check Point Blog

Recent Commits to openclaw:main

test: merge chat side-result checks · openclaw/openclaw@ddd2c2a test: merge cron history checks · openclaw/openclaw@f7eb746 test: merge responsive navigation shell checks · openclaw/openclaw@c2e4b47 docs(changelog): add codex oauth fixes · openclaw/openclaw@628e6cd test: merge navigation routing cases · openclaw/openclaw@5d8cecb Tests: mock channel registry bundled fallback · openclaw/openclaw@2b08233 Secrets: avoid broad web search discovery for single plugin config · openclaw/openclaw@a464f59 test: merge config view browser checks · openclaw/openclaw@20cf511 fix(status): align oauth health with runtime · openclaw/openclaw@eed7116 feat: add macOS screen snapshots for monitor preview (#67954) thanks … · openclaw/openclaw@f377db1 fix: report shared auth scopes in hello-ok (#67810) thanks @BunsDev · openclaw/openclaw@0b6c39b Auto-reply: avoid eager bundled route fallback · openclaw/openclaw@3ea1bf4 Tests: narrow session binding contract setup · openclaw/openclaw@54e4e16 fix(macOS): enable undo/redo in webchat composer text input (#34962) · openclaw/openclaw@00951dc Tests: speed up channel setup promotion · openclaw/openclaw@82b529a Docs: refresh agent instructions · openclaw/openclaw@5775fe2 fix(auth): serialize OAuth refresh across agents to fix #26322 (#67876) · openclaw/openclaw@8e79080 test: allow ollama public surface boundary test · openclaw/openclaw@7d4f1a6 Docs: add test performance guardrails · openclaw/openclaw@89706d3 Tests: restore context-engine usage proof · openclaw/openclaw@e4c4f95 Tests: slim context engine runtime coverage · openclaw/openclaw@74c198f ci: retry failed custom checkouts · openclaw/openclaw@0ee5baf test: trim duplicate provider auth onboarding cases · openclaw/openclaw@1ffc02e matrix: fix sessions_spawn --thread subagent session spawning (#67643) · openclaw/openclaw@1ce2596 test: reduce auth choice fixture churn · openclaw/openclaw@857b9cd test: mock health status config boundaries · openclaw/openclaw@9d5ab4a test: mock onboard config io boundary · openclaw/openclaw@299694d test: mock legacy state plugin boundaries · openclaw/openclaw@2713089 test: mock channel install boundaries · openclaw/openclaw@b945248 test: mock doctor preview channel boundaries · openclaw/openclaw@b1a3ad4
fix(gateway): capture opt-in memory pressure snapshots (#...
steipete · 2026-05-17 · via Recent Commits to openclaw:main
Original file line numberDiff line numberDiff line change

@@ -21,6 +21,7 @@ Docs: https://docs.openclaw.ai

2121
2222

### Fixes

2323
24+

- Gateway/diagnostics: add opt-in critical memory pressure stability snapshots with gateway logs, V8 heap, cgroup, active-resource, and redacted large session-file evidence. Fixes #82518.

2425

- CLI/config: show concise human config-write output with an indented backup path instead of printing checksum-heavy overwrite audit details by default.

2526

- CLI/docs: call the canonical lowercase docs MCP search tool and surface MCP errors instead of returning empty search results. Fixes #82702. (#82704) Thanks @hclsys.

2627

- QA-Lab: ignore heartbeat-only operational transcripts when capturing runtime parity cells so background checks cannot replace the scenario reply. (#80323) Thanks @100yenadmin.

Original file line numberDiff line numberDiff line change

@@ -1,4 +1,4 @@

1-

c79ffc4fd2a7aa7e33a3fa12f22b3d42658a3148f9927c638afdc0fee5512f00 config-baseline.json

2-

b2c0fbfdbfc23bd617233e9b78740aa2b5e0f272f5c666e90c2504a8887ae6b8 config-baseline.core.json

1+

240af35b9ff0d5f6cdef89ddea23ddbec55e8c152648eb039814ed2bd3985161 config-baseline.json

2+

58c0e8c24b8811be2727a3fbf4848807b047cfed17076d3997f24a0636db5ac0 config-baseline.core.json

33

fe4f1cb00d7d1dee9746779ec3cf14236e5f672c91502268a12ad6e467a2c4ad config-baseline.channel.json

44

e9049ce0154f484f44bb0ac174a44198269256044da5ba62a6e107e78bfd7a70 config-baseline.plugin.json

Original file line numberDiff line numberDiff line change

@@ -107,6 +107,10 @@

107107

"source": "Gateway",

108108

"target": "Gateway 网关"

109109

},

110+

{

111+

"source": "Gateway health",

112+

"target": "Gateway 健康"

113+

},

110114

{

111115

"source": "Gateway RPC reference",

112116

"target": "Gateway RPC 参考"

Original file line numberDiff line numberDiff line change

@@ -1032,6 +1032,7 @@ Notes:

10321032

flags: ["telegram.*"],

10331033

stuckSessionWarnMs: 30000,

10341034

stuckSessionAbortMs: 300000,

1035+

memoryPressureSnapshot: false,

10351036
10361037

otel: {

10371038

enabled: false,

@@ -1072,6 +1073,7 @@ Notes:

10721073

- `flags`: array of flag strings enabling targeted log output (supports wildcards like `"telegram.*"` or `"*"`).

10731074

- `stuckSessionWarnMs`: no-progress age threshold in ms for classifying long-running processing sessions as `session.long_running`, `session.stalled`, or `session.stuck`. Reply, tool, status, block, and ACP progress reset the timer; repeated `session.stuck` diagnostics back off while unchanged.

10741075

- `stuckSessionAbortMs`: no-progress age threshold in ms before eligible stalled active work may be abort-drained for recovery. When unset, OpenClaw uses the safer extended embedded-run window of at least 5 minutes and 3x `stuckSessionWarnMs`.

1076+

- `memoryPressureSnapshot`: captures a redacted pre-OOM stability snapshot when memory pressure reaches `critical` (default: `false`). Set to `true` to add the stability bundle file scan/write while keeping normal memory pressure events.

10751077

- `otel.enabled`: enables the OpenTelemetry export pipeline (default: `false`). For the full configuration, signal catalog, and privacy model, see [OpenTelemetry export](/gateway/opentelemetry).

10761078

- `otel.endpoint`: collector URL for OTel export.

10771079

- `otel.tracesEndpoint` / `otel.metricsEndpoint` / `otel.logsEndpoint`: optional signal-specific OTLP endpoints. When set, they override `otel.endpoint` for that signal only.

Original file line numberDiff line numberDiff line change

@@ -189,6 +189,21 @@ diagnostic event collection:

189189

Disabling diagnostics reduces bug-report detail. It does not affect normal

190190

Gateway logging.

191191
192+

Critical memory pressure snapshots are off by default. To keep diagnostics

193+

events and also capture the pre-OOM stability snapshot:

194+
195+

```json5

196+

{

197+

diagnostics: {

198+

memoryPressureSnapshot: true,

199+

},

200+

}

201+

```

202+
203+

Use this only on hosts that can tolerate the extra file-system scan and snapshot

204+

write during critical memory pressure. Normal memory pressure events still

205+

record RSS, heap, threshold, and growth facts when the snapshot is off.

206+
192207

## Related

193208
194209

- [Health checks](/gateway/health)

Original file line numberDiff line numberDiff line change

@@ -30,7 +30,7 @@ health commands above for live connectivity checks.

3030

- Creds on disk: `ls -l ~/.openclaw/credentials/whatsapp/<accountId>/creds.json` (mtime should be recent).

3131

- Session store: `ls -l ~/.openclaw/agents/<agentId>/sessions/sessions.json` (path can be overridden in config). Count and recent recipients are surfaced via `status`.

3232

- Relink flow: `openclaw channels logout && openclaw channels login --verbose` when status codes 409–515 or `loggedOut` appear in logs. (Note: the QR login flow auto-restarts once for status 515 after pairing.)

33-

- Diagnostics are enabled by default. The gateway records operational facts unless `diagnostics.enabled: false` is set. Memory events record RSS/heap byte counts, threshold pressure, and growth pressure. Liveness warnings record event-loop delay, event-loop utilization, CPU-core ratio, and active/waiting/queued session counts when the process is running but saturated. Oversized-payload events record what was rejected, truncated, or chunked, plus sizes and limits when available. They do not record the message text, attachment contents, webhook body, raw request or response body, tokens, cookies, or secret values. The same heartbeat starts the bounded stability recorder, which is available through `openclaw gateway stability` or the `diagnostics.stability` Gateway RPC. Fatal Gateway exits, shutdown timeouts, and restart startup failures persist the latest recorder snapshot under `~/.openclaw/logs/stability/` when events exist; inspect the newest saved bundle with `openclaw gateway stability --bundle latest`.

33+

- Diagnostics are enabled by default. The gateway records operational facts unless `diagnostics.enabled: false` is set. Memory events record RSS/heap byte counts, threshold pressure, and growth pressure. Critical memory pressure logs through the gateway logger. When `diagnostics.memoryPressureSnapshot: true` is set, critical memory pressure also writes a pre-OOM stability bundle with V8 heap stats, Linux cgroup counters when available, active resource counts, and the largest session/transcript files by redacted relative path. Liveness warnings record event-loop delay, event-loop utilization, CPU-core ratio, and active/waiting/queued session counts when the process is running but saturated. Oversized-payload events record what was rejected, truncated, or chunked, plus sizes and limits when available. They do not record the message text, attachment contents, webhook body, raw request or response body, tokens, cookies, or secret values. The same heartbeat starts the bounded stability recorder, which is available through `openclaw gateway stability` or the `diagnostics.stability` Gateway RPC. Fatal Gateway exits, shutdown timeouts, and restart startup failures persist the latest recorder snapshot under `~/.openclaw/logs/stability/` when events exist; critical memory pressure does too only when `diagnostics.memoryPressureSnapshot: true` is set. Inspect the newest saved bundle with `openclaw gateway stability --bundle latest`.

3434

- For bug reports, run `openclaw gateway diagnostics export` and attach the generated zip. The export combines a Markdown summary, the newest stability bundle, sanitized log metadata, sanitized Gateway status/health snapshots, and config shape. It is meant to be shared: chat text, webhook bodies, tool outputs, credentials, cookies, account/message identifiers, and secret values are omitted or redacted. See [Diagnostics Export](/gateway/diagnostics).

3535
3636

## Health monitor config

Original file line numberDiff line numberDiff line change

@@ -375,6 +375,42 @@ Related:

375375

- [Configuration](/gateway/configuration)

376376

- [Doctor](/gateway/doctor)

377377
378+

## Gateway exits during high memory use

379+
380+

Use this when the Gateway disappears under load, the supervisor reports an OOM-style restart, or logs mention `critical memory pressure bundle written`.

381+
382+

```bash

383+

openclaw gateway status --deep

384+

openclaw logs --follow

385+

openclaw gateway stability --bundle latest

386+

openclaw gateway diagnostics export

387+

```

388+
389+

Look for:

390+
391+

- `Reason: diagnostic.memory.pressure.critical` in the latest stability bundle.

392+

- `Memory pressure:` with `critical/rss_threshold`, `critical/heap_threshold`, or `critical/rss_growth`.

393+

- `V8 heap:` values near the heap limit.

394+

- `Largest session files:` entries such as `agents/<agent>/sessions/<session>.jsonl` or `sessions/<session>.jsonl`.

395+

- Linux cgroup memory counters when the gateway runs inside a container or memory-limited service.

396+
397+

Common signatures:

398+
399+

- `critical memory pressure bundle written` appears shortly before restart → OpenClaw captured a pre-OOM stability bundle. Inspect it with `openclaw gateway stability --bundle latest`.

400+

- `memory pressure: level=critical ... memoryPressureSnapshot=disabled` appears in gateway logs → OpenClaw detected critical memory pressure, but the pre-OOM stability snapshot is off.

401+

- `Largest session files:` points at a very large redacted transcript path → reduce retained session history, inspect session growth, or move old transcripts out of the active store before restarting.

402+

- `V8 heap:` used bytes are close to the heap limit → lower prompt/session pressure, reduce concurrent work, or raise the Node heap limit only after confirming the workload is expected.

403+

- `Memory pressure: critical/rss_growth` → memory grew quickly inside one sampling window. Check the latest logs for a large import, runaway tool output, repeated retries, or a batch of queued agent work.

404+

- Critical memory pressure appears in logs but no bundle exists → this is the default. Set `diagnostics.memoryPressureSnapshot: true` to capture the pre-OOM stability bundle on future critical memory pressure events.

405+
406+

The stability bundle is payload-free. It includes operational memory evidence and redacted relative file paths, not message text, webhook bodies, credentials, tokens, cookies, or raw session ids. Attach the diagnostics export to bug reports instead of copying raw logs.

407+
408+

Related:

409+
410+

- [Gateway health](/gateway/health)

411+

- [Diagnostics export](/gateway/diagnostics)

412+

- [Sessions](/cli/sessions)

413+
378414

## Gateway rejected invalid config

379415
380416

Use this when Gateway startup fails with `Invalid config` or hot reload logs say

Original file line numberDiff line numberDiff line change

@@ -233,6 +233,41 @@ describe("gateway-cli coverage", () => {

233233

uptimeMs: 2000,

234234

},

235235

host: { hostname: "test-host" },

236+

evidence: {

237+

memoryPressure: {

238+

level: "critical",

239+

reason: "rss_threshold",

240+

memory: {

241+

rssBytes: 4096,

242+

heapTotalBytes: 2048,

243+

heapUsedBytes: 1536,

244+

externalBytes: 128,

245+

arrayBuffersBytes: 64,

246+

},

247+

thresholdBytes: 3000,

248+

heapStatistics: {

249+

totalHeapSizeBytes: 2048,

250+

totalHeapSizeExecutableBytes: 256,

251+

totalPhysicalSizeBytes: 2048,

252+

totalAvailableSizeBytes: 8192,

253+

usedHeapSizeBytes: 1536,

254+

heapSizeLimitBytes: 4096,

255+

mallocedMemoryBytes: 32,

256+

externalMemoryBytes: 128,

257+

},

258+

activeResources: {

259+

total: 2,

260+

byType: { Timeout: 2 },

261+

},

262+

topSessionFiles: [

263+

{

264+

relativePath: "agents/<agent>/sessions/<session>.jsonl",

265+

sizeBytes: 4096,

266+

mtimeMs: Date.parse("2026-04-22T12:00:00.000Z"),

267+

},

268+

],

269+

},

270+

},

236271

snapshot: {

237272

generatedAt: "2026-04-22T12:00:00.000Z",

238273

capacity: 1000,

@@ -274,6 +309,10 @@ describe("gateway-cli coverage", () => {

274309

expect(callGateway).not.toHaveBeenCalled();

275310

expect(output).toContain("Stability bundle");

276311

expect(output).toContain("gateway.restart_startup_failed");

312+

expect(output).toContain("Memory pressure");

313+

expect(output).toContain("rss_threshold");

314+

expect(output).toContain("Largest session files");

315+

expect(output).toContain("agents/<agent>/sessions/<session>.jsonl");

277316

expect(output).toContain("payload.large");

278317

expect(output).toContain("gateway.http.json");

279318

} finally {

Original file line numberDiff line numberDiff line change

@@ -325,6 +325,42 @@ function renderStabilityBundleSummary(params: {

325325

lines.push(`${colorize(rich, theme.muted, "Error:")} ${errorParts.join(" ")}`);

326326

}

327327

}

328+

const memoryPressure = bundle.evidence?.memoryPressure;

329+

if (memoryPressure) {

330+

lines.push(

331+

`${colorize(rich, theme.muted, "Memory pressure:")} ${memoryPressure.level}/${

332+

memoryPressure.reason

333+

} rss=${formatBytes(memoryPressure.memory.rssBytes)} heap=${formatBytes(

334+

memoryPressure.memory.heapUsedBytes,

335+

)} threshold=${formatBytes(memoryPressure.thresholdBytes)}`,

336+

);

337+

if (memoryPressure.heapStatistics) {

338+

lines.push(

339+

`${colorize(rich, theme.muted, "V8 heap:")} used=${formatBytes(

340+

memoryPressure.heapStatistics.usedHeapSizeBytes,

341+

)} limit=${formatBytes(

342+

memoryPressure.heapStatistics.heapSizeLimitBytes,

343+

)} available=${formatBytes(memoryPressure.heapStatistics.totalAvailableSizeBytes)}`,

344+

);

345+

}

346+

if (memoryPressure.activeResources) {

347+

const resources = Object.entries(memoryPressure.activeResources.byType)

348+

.map(([type, count]) => `${type}=${count}`)

349+

.join(", ");

350+

lines.push(

351+

`${colorize(rich, theme.muted, "Active resources:")} total=${

352+

memoryPressure.activeResources.total

353+

}${resources ? ` · ${resources}` : ""}`,

354+

);

355+

}

356+

if (memoryPressure.topSessionFiles?.length) {

357+

const files = memoryPressure.topSessionFiles

358+

.slice(0, 5)

359+

.map((file) => `${file.relativePath}=${formatBytes(file.sizeBytes)}`)

360+

.join(", ");

361+

lines.push(`${colorize(rich, theme.muted, "Largest session files:")} ${files}`);

362+

}

363+

}

328364

lines.push("", ...renderStabilitySummary(snapshot, rich));

329365

return lines;

330366

}

Original file line numberDiff line numberDiff line change

@@ -133,6 +133,75 @@ describe("legacy session parent fork migrate", () => {

133133

});

134134

});

135135
136+

describe("legacy diagnostics memory pressure snapshot migrate", () => {

137+

it("renames the boolean toggle", () => {

138+

const res = migrateLegacyConfigForTest({

139+

diagnostics: {

140+

enabled: true,

141+

memoryPressureBundle: false,

142+

},

143+

});

144+
145+

expect(res.config?.diagnostics).toEqual({

146+

enabled: true,

147+

memoryPressureSnapshot: false,

148+

});

149+

expect(res.changes).toStrictEqual([

150+

"Moved diagnostics.memoryPressureBundle → memoryPressureSnapshot.",

151+

]);

152+

});

153+
154+

it("preserves the renamed toggle when both keys are present", () => {

155+

const res = migrateLegacyConfigForTest({

156+

diagnostics: {

157+

memoryPressureBundle: false,

158+

memoryPressureSnapshot: true,

159+

},

160+

});

161+
162+

expect(res.config?.diagnostics).toEqual({

163+

memoryPressureSnapshot: true,

164+

});

165+

expect(res.changes).toStrictEqual([

166+

"Removed diagnostics.memoryPressureBundle (memoryPressureSnapshot already set).",

167+

]);

168+

});

169+
170+

it("moves nested enabled to the renamed boolean", () => {

171+

const res = migrateLegacyConfigForTest({

172+

diagnostics: {

173+

enabled: true,

174+

memoryPressureBundle: {

175+

enabled: false,

176+

},

177+

},

178+

});

179+
180+

expect(res.config?.diagnostics).toEqual({

181+

enabled: true,

182+

memoryPressureSnapshot: false,

183+

});

184+

expect(res.changes).toStrictEqual([

185+

"Moved diagnostics.memoryPressureBundle → memoryPressureSnapshot.",

186+

]);

187+

});

188+
189+

it("moves empty object form to the renamed default boolean", () => {

190+

const res = migrateLegacyConfigForTest({

191+

diagnostics: {

192+

memoryPressureBundle: {},

193+

},

194+

});

195+
196+

expect(res.config?.diagnostics).toEqual({

197+

memoryPressureSnapshot: true,

198+

});

199+

expect(res.changes).toStrictEqual([

200+

"Moved diagnostics.memoryPressureBundle → memoryPressureSnapshot.",

201+

]);

202+

});

203+

});

204+
136205

describe("legacy thread binding spawn migrate", () => {

137206

it("moves matching split spawn flags to unified spawnSessions", () => {

138207

const res = migrateLegacyConfigForTest({