@@ -0,0 +1,156 @@
1+# Long-context progress watchdog
2+3+```yaml qa-scenario
4+id: long-context-progress-watchdog
5+title: Long-context progress watchdog
6+surface: runtime
7+runtimeParityTier: live-only
8+coverage:
9+primary:
10+ - runtime.gateway-log-sentinel.codex-progress
11+secondary:
12+ - runtime.long-context
13+ - runtime.codex-app-server
14+objective: Fail live proof when long-context activity triggers Codex app-server timeout or stalled-progress sentinels.
15+successCriteria:
16+ - Gateway config routes the selected QA model through the Codex app-server runtime.
17+ - Agent reads through the seeded long-context fixture and replies with the marker found at the tail.
18+ - Gateway logs since the scenario cursor contain no app-server timeout or stalled-progress sentinel.
19+docsRefs:
20+ - docs/concepts/qa-e2e-automation.md
21+ - qa/scenarios/index.md
22+codeRefs:
23+ - extensions/qa-lab/src/gateway-log-sentinel.ts
24+ - extensions/codex/src/app-server
25+execution:
26+kind: flow
27+summary: Seed a large workspace fixture, complete a read turn, and scan for Codex app-server progress failures.
28+config:
29+requiredProviderMode: live-frontier
30+harnessRuntime: codex
31+fixtureFile: LONG_CONTEXT_SENTINEL_FIXTURE.txt
32+expectedMarker: LONG-CONTEXT-WATCHDOG-OK
33+repeatCount: 2000
34+```
35+36+```yaml qa-flow
37+steps:
38+ - name: catches app-server timeout or stalled progress during long-context activity
39+actions:
40+ - call: waitForGatewayHealthy
41+args:
42+ - ref: env
43+ - 60000
44+ - call: waitForQaChannelReady
45+args:
46+ - ref: env
47+ - 60000
48+ - call: readConfigSnapshot
49+saveAs: originalSnapshot
50+args:
51+ - ref: env
52+ - set: originalModelEntry
53+value:
54+expr: originalSnapshot.config.agents?.defaults?.models?.[env.primaryModel]
55+ - set: originalPluginAllow
56+value:
57+expr: originalSnapshot.config.plugins?.allow
58+ - set: originalCodexPluginEntry
59+value:
60+expr: originalSnapshot.config.plugins?.entries?.codex
61+ - try:
62+actions:
63+ - call: patchConfig
64+args:
65+ - env:
66+ref: env
67+patch:
68+plugins:
69+allow:
70+expr: "Array.from(new Set([...(Array.isArray(originalPluginAllow) ? originalPluginAllow : []), 'codex']))"
71+entries:
72+codex:
73+expr: "({ ...((originalCodexPluginEntry && typeof originalCodexPluginEntry === 'object') ? originalCodexPluginEntry : {}), enabled: true })"
74+agents:
75+defaults:
76+models:
77+expr: "({ [env.primaryModel]: { agentRuntime: { id: config.harnessRuntime } } })"
78+ - call: waitForGatewayHealthy
79+args:
80+ - ref: env
81+ - 60000
82+ - call: waitForQaChannelReady
83+args:
84+ - ref: env
85+ - 60000
86+ - call: readConfigSnapshot
87+saveAs: snapshot
88+args:
89+ - ref: env
90+ - assert:
91+expr: "snapshot.config.agents?.defaults?.models?.[env.primaryModel]?.agentRuntime?.id === config.harnessRuntime"
92+message:
93+expr: "`expected ${env.primaryModel} agentRuntime.id=${config.harnessRuntime}, got ${JSON.stringify(snapshot.config.agents?.defaults?.models?.[env.primaryModel]?.agentRuntime)}`"
94+ - call: reset
95+ - set: logCursor
96+value:
97+expr: markGatewayLogCursor()
98+ - set: fixturePath
99+value:
100+expr: "path.join(env.gateway.workspaceDir, config.fixtureFile)"
101+ - call: fs.writeFile
102+args:
103+ - ref: fixturePath
104+ - expr: "`START LONG-CONTEXT-WATCHDOG\\n${Array.from({ length: config.repeatCount }, (_entry, index) => `context row ${index + 1}: alpha beta gamma`).join('\\n')}\\nTAIL ${config.expectedMarker}\\n`"
105+ - utf8
106+ - set: startIndex
107+value:
108+expr: state.getSnapshot().messages.length
109+ - call: runAgentPrompt
110+args:
111+ - ref: env
112+ - sessionKey:
113+expr: "`agent:qa:long-context-watchdog:${randomUUID().slice(0, 8)}`"
114+message:
115+expr: "`Read ${fixturePath}, find the marker on the TAIL line, and reply with that marker only.`"
116+timeoutMs:
117+expr: liveTurnTimeoutMs(env, 90000)
118+ - call: waitForOutboundMessage
119+saveAs: outbound
120+args:
121+ - ref: state
122+ - lambda:
123+params: [candidate]
124+expr: "candidate.conversation.id === 'qa-operator' && normalizeLowercaseStringOrEmpty(candidate.text).includes(normalizeLowercaseStringOrEmpty(config.expectedMarker))"
125+ - expr: liveTurnTimeoutMs(env, 45000)
126+ - sinceIndex:
127+ref: startIndex
128+ - call: assertNoGatewayLogSentinels
129+args:
130+ - since:
131+ref: logCursor
132+kinds:
133+ - codex-app-server-timeout
134+ - stalled-agent-run
135+finally:
136+ - call: patchConfig
137+args:
138+ - env:
139+ref: env
140+patch:
141+plugins:
142+allow:
143+expr: "originalPluginAllow === undefined ? null : originalPluginAllow"
144+entries:
145+codex:
146+expr: "originalCodexPluginEntry === undefined ? null : { ...originalCodexPluginEntry, enabled: originalCodexPluginEntry.enabled === undefined ? null : originalCodexPluginEntry.enabled }"
147+agents:
148+defaults:
149+models:
150+expr: "({ [env.primaryModel]: originalModelEntry === undefined ? null : { ...originalModelEntry, agentRuntime: originalModelEntry.agentRuntime === undefined ? null : originalModelEntry.agentRuntime } })"
151+ - call: waitForGatewayHealthy
152+args:
153+ - ref: env
154+ - 60000
155+detailsExpr: outbound.text
156+```
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。