@@ -0,0 +1,182 @@
1+# Dreaming shadow trial report
2+3+```yaml qa-scenario
4+id: dreaming-shadow-trial-report
5+title: Dreaming shadow trial report
6+surface: memory
7+coverage:
8+primary:
9+ - memory.dreaming
10+secondary:
11+ - memory.promotion
12+ - qa.artifact-safety
13+risk: medium
14+capabilities:
15+ - tools.read
16+ - tools.write
17+ - channel.reply
18+objective: Verify a dreaming shadow-trial handoff writes a useful report that compares a candidate memory against a baseline before promotion.
19+successCriteria:
20+ - Agent reads the shadow-trial brief and candidate evidence before writing the report.
21+ - Report compares baseline and candidate outcomes without changing MEMORY.md.
22+ - Report records a helpful, neutral, or harmful verdict with reason and risk flags.
23+ - Final reply points to the report and does not claim the candidate was promoted.
24+docsRefs:
25+ - docs/concepts/dreaming.md
26+ - docs/concepts/memory.md
27+codeRefs:
28+ - extensions/memory-core/src/dreaming.ts
29+ - extensions/memory-core/src/dreaming-phases.ts
30+ - extensions/qa-lab/src/providers/mock-openai/server.ts
31+execution:
32+kind: flow
33+summary: Verify a report-only dreaming shadow trial compares candidate memory utility before promotion.
34+config:
35+sessionKey: agent:qa:dreaming-shadow-trial
36+reportName: dreaming-shadow-trial-report.md
37+safeMarker: DREAMING-SHADOW-TRIAL-OK
38+seededMemory: "# Memory\n\n"
39+workspaceFiles:
40+DREAMING_SHADOW_TRIAL_BRIEF.md: |-
41+ # Dreaming shadow trial brief
42+43+ Write a report-only shadow trial for a candidate memory. Do not edit MEMORY.md.
44+45+ Required report contract:
46+ 1. Read DREAMING_SHADOW_TRIAL_BRIEF.md.
47+ 2. Read DREAMING_CANDIDATE_EVIDENCE.md.
48+ 3. Write ./dreaming-shadow-trial-report.md.
49+ 4. Include: Candidate, Trial prompt, Baseline outcome, Candidate outcome, Verdict, Reason, Risk flags, Promotion action.
50+ 5. For this seeded evidence, Verdict must be helpful.
51+ 6. Promotion action must be report-only.
52+ DREAMING_CANDIDATE_EVIDENCE.md: |-
53+ # Candidate evidence
54+55+ Candidate memory: The user prefers release reports that include exact verification commands and remaining risk.
56+ Trial prompt: Prepare a release readiness reply for a local OpenClaw QA change.
57+ Baseline outcome: mentions tests passed but omits the exact command and remaining risk.
58+ Candidate outcome: includes the exact verification command and calls out the remaining review risk.
59+ Risk flags: no secret exposure; no outdated preference conflict; no over-personalization.
60+ prompt: |-
61+ Dreaming shadow trial report check. Read DREAMING_SHADOW_TRIAL_BRIEF.md and DREAMING_CANDIDATE_EVIDENCE.md first.
62+ Then write ./dreaming-shadow-trial-report.md as a report-only shadow trial.
63+ For this seeded evidence, use Verdict: helpful and Promotion action: report-only.
64+ Do not edit MEMORY.md and do not claim the candidate was promoted.
65+ Reply with the report path and exact marker DREAMING-SHADOW-TRIAL-OK.
66+ expectedReportAll:
67+ - "candidate:"
68+ - "exact verification commands and remaining risk"
69+ - "trial prompt:"
70+ - "baseline outcome:"
71+ - "omits the exact command and remaining risk"
72+ - "candidate outcome:"
73+ - "calls out the remaining review risk"
74+ - "verdict: helpful"
75+ - "reason:"
76+ - "risk flags:"
77+ - "no secret exposure"
78+ - "promotion action: report-only"
79+forbiddenReplyNeedles:
80+ - "candidate was promoted to MEMORY.md"
81+ - "I updated MEMORY.md"
82+ - "promotion complete"
83+```
84+85+```yaml qa-flow
86+steps:
87+ - name: writes a report-only shadow trial for a candidate memory
88+actions:
89+ - call: reset
90+ - forEach:
91+items:
92+expr: "Object.entries(config.workspaceFiles ?? {})"
93+item: workspaceFile
94+actions:
95+ - call: fs.writeFile
96+args:
97+ - expr: "path.join(env.gateway.workspaceDir, String(workspaceFile[0]))"
98+ - expr: "`${String(workspaceFile[1] ?? '').trimEnd()}\\n`"
99+ - utf8
100+ - set: reportPath
101+value:
102+expr: "path.join(env.gateway.workspaceDir, config.reportName)"
103+ - set: memoryPath
104+value:
105+expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
106+ - call: fs.writeFile
107+args:
108+ - ref: memoryPath
109+ - expr: config.seededMemory
110+ - utf8
111+ - call: waitForGatewayHealthy
112+args:
113+ - ref: env
114+ - 60000
115+ - call: waitForQaChannelReady
116+args:
117+ - ref: env
118+ - 60000
119+ - set: requestCountBefore
120+value:
121+expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
122+ - call: runAgentPrompt
123+args:
124+ - ref: env
125+ - sessionKey:
126+expr: config.sessionKey
127+message:
128+expr: config.prompt
129+timeoutMs:
130+expr: liveTurnTimeoutMs(env, 40000)
131+ - call: waitForCondition
132+saveAs: report
133+args:
134+ - lambda:
135+async: true
136+expr: "(() => { const normalize = (value) => normalizeLowercaseStringOrEmpty(value); const matches = (value) => { const normalized = normalize(value); return normalized && config.expectedReportAll.every((needle) => normalized.includes(normalize(needle))); }; return fs.readFile(reportPath, 'utf8').then((value) => matches(value) ? value : undefined).catch(() => undefined); })()"
137+ - expr: liveTurnTimeoutMs(env, 30000)
138+ - expr: "env.providerMode === 'mock-openai' ? 100 : 250"
139+ - set: normalizedReport
140+value:
141+expr: "normalizeLowercaseStringOrEmpty(report)"
142+ - assert:
143+expr: "config.expectedReportAll.every((needle) => normalizedReport.includes(normalizeLowercaseStringOrEmpty(needle)))"
144+message:
145+expr: "`shadow trial report missing expected fields: ${report}`"
146+ - call: fs.readFile
147+saveAs: memoryAfter
148+args:
149+ - ref: memoryPath
150+ - utf8
151+ - assert:
152+expr: "String(memoryAfter) === config.seededMemory"
153+message:
154+expr: "`shadow trial modified durable memory instead of staying report-only: ${memoryAfter}`"
155+ - call: waitForCondition
156+saveAs: outbound
157+args:
158+ - lambda:
159+expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-operator' && candidate.text.includes(config.safeMarker) && candidate.text.includes(config.reportName)).at(-1)"
160+ - expr: liveTurnTimeoutMs(env, 30000)
161+ - expr: "env.providerMode === 'mock-openai' ? 100 : 250"
162+ - assert:
163+expr: "!config.forbiddenReplyNeedles.some((needle) => normalizeLowercaseStringOrEmpty(outbound.text).includes(normalizeLowercaseStringOrEmpty(needle)))"
164+message:
165+expr: "`shadow trial reply overclaimed promotion: ${outbound.text}`"
166+ - set: shadowTrialDebugRequests
167+value:
168+expr: "env.mock ? [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].slice(requestCountBefore).filter((request) => /dreaming shadow trial report check/i.test(String(request.allInputText ?? ''))) : []"
169+ - assert:
170+expr: "!env.mock || shadowTrialDebugRequests.filter((request) => request.plannedToolName === 'read').length >= 2"
171+message:
172+expr: "`expected two shadow-trial reads before write, saw plannedToolNames=${JSON.stringify(shadowTrialDebugRequests.map((request) => request.plannedToolName ?? null))}`"
173+ - assert:
174+expr: "!env.mock || shadowTrialDebugRequests.some((request) => request.plannedToolName === 'write')"
175+message:
176+expr: "`expected shadow-trial report write, saw plannedToolNames=${JSON.stringify(shadowTrialDebugRequests.map((request) => request.plannedToolName ?? null))}`"
177+ - assert:
178+expr: "!env.mock || (() => { const readIndices = shadowTrialDebugRequests.map((r, i) => r.plannedToolName === 'read' ? i : -1).filter(i => i >= 0); const firstWrite = shadowTrialDebugRequests.findIndex((r) => r.plannedToolName === 'write'); return readIndices.length >= 2 && firstWrite >= 0 && readIndices[1] < firstWrite; })()"
179+message:
180+expr: "`expected shadow-trial reads before write, saw plannedToolNames=${JSON.stringify(shadowTrialDebugRequests.map((request) => request.plannedToolName ?? null))}`"
181+detailsExpr: outbound.text
182+```
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。