fix(voice): reuse preflight transcripts across channels · openclaw/openclaw@6a67f65
steipete
·
2026-04-26
·
via Recent Commits to openclaw:main
| Original file line number | Diff line number | Diff line change |
|---|
@@ -96,6 +96,14 @@ Docs: https://docs.openclaw.ai
|
96 | 96 | before agent dispatch and keep raw Feishu `file_key` payloads out of message |
97 | 97 | text. Fixes #67120 and #61876. |
98 | 98 | - Tasks: terminalize async Gateway agent task records from the Gateway run result while preserving aborted, failed, and cancelled outcomes instead of leaving completed runs stuck as active or lost. (#71905) Thanks @likewen-tech. |
| 99 | +- WhatsApp: let authorized group voice-note transcripts satisfy mention gating |
| 100 | + before reply dispatch, while keeping unmentioned transcripts in pending group |
| 101 | + history. Fixes #44908. |
| 102 | +- Media understanding: carry channel voice-note preflight state into attachment |
| 103 | + selection so WhatsApp, Feishu, Telegram, and Discord do not transcribe the |
| 104 | + same inbound audio twice. Fixes #70580. |
| 105 | +- TTS/BlueBubbles: deliver compatible auto-TTS audio as iMessage voice memo |
| 106 | + bubbles instead of plain MP3/CAF file attachments. Fixes #16848. |
99 | 107 | - ACP: send subagent and async-task completion wakes to external ACP harnesses as |
100 | 108 | plain prompts instead of OpenClaw internal runtime-context envelopes, while |
101 | 109 | keeping those envelopes out of ACP transcripts. |
|
| Original file line number | Diff line number | Diff line change |
|---|
@@ -21,6 +21,8 @@ need a separate `openclaw plugins install` step.
|
21 | 21 | - OpenClaw talks to it through its REST API (`GET /api/v1/ping`, `POST /message/text`, `POST /chat/:id/*`). |
22 | 22 | - Incoming messages arrive via webhooks; outgoing replies, typing indicators, read receipts, and tapbacks are REST calls. |
23 | 23 | - Attachments and stickers are ingested as inbound media (and surfaced to the agent when possible). |
| 24 | +- Auto-TTS replies that synthesize MP3 or CAF audio are delivered as iMessage |
| 25 | + voice memo bubbles instead of plain file attachments. |
24 | 26 | - Pairing/allowlist works the same way as other channels (`/channels/pairing` etc) with `channels.bluebubbles.allowFrom` + pairing codes. |
25 | 27 | - Reactions are surfaced as system events just like Slack/Telegram so agents can "mention" them before replying. |
26 | 28 | - Advanced features: edit, unsend, reply threading, message effects, group management. |
|
| Original file line number | Diff line number | Diff line change |
|---|
@@ -244,6 +244,7 @@ content and identifiers.
|
244 | 244 | |
245 | 245 | - explicit WhatsApp mentions of the bot identity |
246 | 246 | - configured mention regex patterns (`agents.list[].groupChat.mentionPatterns`, fallback `messages.groupChat.mentionPatterns`) |
| 247 | +- inbound voice-note transcripts for authorized group messages |
247 | 248 | - implicit reply-to-bot detection (reply sender matches bot identity) |
248 | 249 | |
249 | 250 | Security note: |
@@ -296,6 +297,11 @@ When the linked self number is also present in `allowFrom`, WhatsApp self-chat s
|
296 | 297 | - `<media:document>` |
297 | 298 | - `<media:sticker>` |
298 | 299 | |
| 300 | +Authorized group voice notes are transcribed before mention gating when the |
| 301 | +body is only `<media:audio>`, so saying the bot mention in the voice note can |
| 302 | +trigger the reply. If the transcript still does not mention the bot, the |
| 303 | +transcript is kept in pending group history instead of the raw placeholder. |
| 304 | + |
299 | 305 | Location bodies use terse coordinate text. Location labels/comments and contact/vCard details are rendered as fenced untrusted metadata, not inline prompt text. |
300 | 306 | |
301 | 307 | </Accordion> |
|
| Original file line number | Diff line number | Diff line change |
|---|
@@ -58,6 +58,10 @@ Video and music generation run as background tasks because provider processing t
|
58 | 58 | |
59 | 59 | Deepgram, ElevenLabs, Mistral, OpenAI, SenseAudio, and xAI can all transcribe |
60 | 60 | inbound audio through the batch `tools.media.audio` path when configured. |
| 61 | +Channel plugins that preflight a voice note for mention gating or command |
| 62 | +parsing mark the transcribed attachment on the inbound context, so the shared |
| 63 | +media-understanding pass reuses that transcript instead of making a second STT |
| 64 | +call for the same audio. |
61 | 65 | Deepgram, ElevenLabs, Mistral, OpenAI, and xAI also register Voice Call |
62 | 66 | streaming STT providers, so live phone audio can be forwarded to the selected |
63 | 67 | vendor without waiting for a completed recording. |
|
| Original file line number | Diff line number | Diff line change |
|---|
@@ -330,7 +330,7 @@ export const bluebubblesPlugin: ChannelPlugin<ResolvedBlueBubblesAccount, BlueBu
|
330 | 330 | }, |
331 | 331 | sendMedia: async (ctx) => { |
332 | 332 | const runtime = await loadBlueBubblesChannelRuntime(); |
333 | | -const { cfg, to, text, mediaUrl, accountId, replyToId } = ctx; |
| 333 | +const { cfg, to, text, mediaUrl, accountId, replyToId, audioAsVoice } = ctx; |
334 | 334 | const { mediaPath, mediaBuffer, contentType, filename, caption } = ctx as { |
335 | 335 | mediaPath?: string; |
336 | 336 | mediaBuffer?: Uint8Array; |
@@ -349,6 +349,7 @@ export const bluebubblesPlugin: ChannelPlugin<ResolvedBlueBubblesAccount, BlueBu
|
349 | 349 | caption: caption ?? text ?? undefined, |
350 | 350 | replyToId: replyToId ?? null, |
351 | 351 | accountId: accountId ?? undefined, |
| 352 | +asVoice: audioAsVoice === true, |
352 | 353 | }); |
353 | 354 | }, |
354 | 355 | }, |
|
| Original file line number | Diff line number | Diff line change |
|---|
@@ -323,4 +323,27 @@ describe("sendBlueBubblesMedia local-path hardening", () => {
|
323 | 323 | ); |
324 | 324 | expect(sendBlueBubblesAttachmentMock).toHaveBeenCalledTimes(1); |
325 | 325 | }); |
| 326 | + |
| 327 | +it("passes asVoice through to attachment delivery", async () => { |
| 328 | +runtimeMocks.fetchRemoteMedia.mockResolvedValueOnce({ |
| 329 | +buffer: new Uint8Array([1, 2, 3]), |
| 330 | +contentType: "audio/mpeg", |
| 331 | +fileName: "voice.mp3", |
| 332 | +}); |
| 333 | + |
| 334 | +await sendBlueBubblesMedia({ |
| 335 | +cfg: createConfig(), |
| 336 | +to: "chat:123", |
| 337 | +mediaUrl: "https://example.com/voice.mp3", |
| 338 | +asVoice: true, |
| 339 | +}); |
| 340 | + |
| 341 | +expect(sendBlueBubblesAttachmentMock).toHaveBeenCalledWith( |
| 342 | +expect.objectContaining({ |
| 343 | +asVoice: true, |
| 344 | +contentType: "audio/mpeg", |
| 345 | +filename: "voice.mp3", |
| 346 | +}), |
| 347 | +); |
| 348 | +}); |
326 | 349 | }); |
| Original file line number | Diff line number | Diff line change |
|---|
@@ -1686,6 +1686,7 @@ async function processMessageAfterDedupe(
|
1686 | 1686 | caption: caption ?? undefined, |
1687 | 1687 | replyToId: replyToMessageGuid || null, |
1688 | 1688 | accountId: account.accountId, |
| 1689 | +asVoice: payload.audioAsVoice === true, |
1689 | 1690 | }); |
1690 | 1691 | } catch (err) { |
1691 | 1692 | forgetPendingOutboundMessageId(pendingId); |
|
| Original file line number | Diff line number | Diff line change |
|---|
@@ -405,6 +405,7 @@ describe("preflightDiscordMessage", () => {
|
405 | 405 | ); |
406 | 406 | expect(result).not.toBeNull(); |
407 | 407 | expect(result?.isDirectMessage).toBe(true); |
| 408 | +expect(result?.preflightAudioTranscript).toBe("hello openclaw from dm audio"); |
408 | 409 | }); |
409 | 410 | |
410 | 411 | it("falls back to the default discord account for omitted-account dm authorization", async () => { |
@@ -1096,6 +1097,7 @@ describe("preflightDiscordMessage", () => {
|
1096 | 1097 | ); |
1097 | 1098 | expect(result).not.toBeNull(); |
1098 | 1099 | expect(result?.wasMentioned).toBe(true); |
| 1100 | +expect(result?.preflightAudioTranscript).toBe("hey openclaw"); |
1099 | 1101 | }); |
1100 | 1102 | |
1101 | 1103 | it("does not transcribe guild audio from unauthorized members", async () => { |
|
| Original file line number | Diff line number | Diff line change |
|---|
@@ -1112,6 +1112,7 @@ export async function preflightDiscordMessage(
|
1112 | 1112 | commandAuthorized, |
1113 | 1113 | baseText, |
1114 | 1114 | messageText, |
| 1115 | + ...(preflightTranscript !== undefined ? { preflightAudioTranscript: preflightTranscript } : {}), |
1115 | 1116 | wasMentioned, |
1116 | 1117 | route: effectiveRoute, |
1117 | 1118 | threadBinding, |
|
| Original file line number | Diff line number | Diff line change |
|---|
@@ -56,6 +56,7 @@ export type DiscordMessagePreflightContext = DiscordMessagePreflightSharedFields
|
56 | 56 | commandAuthorized: boolean; |
57 | 57 | baseText: string; |
58 | 58 | messageText: string; |
| 59 | +preflightAudioTranscript?: string; |
59 | 60 | wasMentioned: boolean; |
60 | 61 | |
61 | 62 | route: ReturnType<typeof resolveAgentRoute>; |
|
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。