@@ -1,91 +1,134 @@
11---
2-summary: "Unified landing page for media generation, understanding, and speech capabilities"
2+summary: "Image, video, music, speech, and media-understanding capabilities at a glance"
33read_when:
4- - Looking for an overview of media capabilities
4+ - Looking for an overview of OpenClaw's media capabilities
55 - Deciding which media provider to configure
66 - Understanding how async media generation works
77title: "Media overview"
8+sidebarTitle: "Media overview"
89---
91010-# Media Generation and Understanding
11-12-OpenClaw generates images, videos, and music, understands inbound media (images, audio, video), and speaks replies aloud with text-to-speech. All media capabilities are tool-driven: the agent decides when to use them based on the conversation, and each tool only appears when at least one backing provider is configured.
13-14-## Capabilities at a glance
15-16-| Capability | Tool | Providers | What it does |
17-| -------------------- | ---------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
18-| Image generation | `image_generate` | ComfyUI, fal, Google, MiniMax, OpenAI, Vydra, xAI | Creates or edits images from text prompts or references |
19-| Video generation | `video_generate` | Alibaba, BytePlus, ComfyUI, fal, Google, MiniMax, OpenAI, Qwen, Runway, Together, Vydra, xAI | Creates videos from text, images, or existing videos |
20-| Music generation | `music_generate` | ComfyUI, Google, MiniMax | Creates music or audio tracks from text prompts |
21-| Text-to-speech (TTS) | `tts` | ElevenLabs, Google, Gradium, Local CLI, Microsoft, MiniMax, OpenAI, Vydra, xAI, Xiaomi MiMo | Converts outbound replies to spoken audio |
22-| Media understanding | (automatic) | Any vision/audio-capable model provider, plus CLI fallbacks | Summarizes inbound images, audio, and video |
11+OpenClaw generates images, videos, and music, understands inbound media
12+(images, audio, video), and speaks replies aloud with text-to-speech. All
13+media capabilities are tool-driven: the agent decides when to use them based
14+on the conversation, and each tool only appears when at least one backing
15+provider is configured.
16+17+## Capabilities
18+19+<CardGroup cols={2}>
20+<Card title="Image generation" href="/tools/image-generation" icon="image">
21+Create and edit images from text prompts or reference images via
22+`image_generate`. Synchronous — completes inline with the reply.
23+</Card>
24+<Card title="Video generation" href="/tools/video-generation" icon="video">
25+Text-to-video, image-to-video, and video-to-video via `video_generate`.
26+Async — runs in the background and posts the result when ready.
27+</Card>
28+<Card title="Music generation" href="/tools/music-generation" icon="music">
29+Generate music or audio tracks via `music_generate`. Async on shared
30+providers; ComfyUI workflow path runs synchronously.
31+</Card>
32+<Card title="Text-to-speech" href="/tools/tts" icon="microphone">
33+Convert outbound replies to spoken audio via the `tts` tool plus
34+`messages.tts` config. Synchronous.
35+</Card>
36+<Card title="Media understanding" href="/nodes/media-understanding" icon="eye">
37+Summarize inbound images, audio, and video using vision-capable model
38+providers and dedicated media-understanding plugins.
39+</Card>
40+<Card title="Speech-to-text" href="/nodes/audio" icon="ear-listen">
41+Transcribe inbound voice messages through batch STT or Voice Call
42+streaming STT providers.
43+</Card>
44+</CardGroup>
23452446## Provider capability matrix
254726-This table shows which providers support which media capabilities across the platform.
27-28-| Provider | Image | Video | Music | TTS | STT / Transcription | Realtime Voice | Media Understanding |
29-| ----------- | ----- | ----- | ----- | --- | ------------------- | -------------- | ------------------- |
30-| Alibaba | | Yes | | | | | |
31-| BytePlus | | Yes | | | | | |
32-| ComfyUI | Yes | Yes | Yes | | | | |
33-| Deepgram | | | | | Yes | Yes | |
34-| ElevenLabs | | | | Yes | Yes | | |
35-| fal | Yes | Yes | | | | | |
36-| Google | Yes | Yes | Yes | Yes | | Yes | Yes |
37-| Gradium | | | | Yes | | | |
38-| Local CLI | | | | Yes | | | |
39-| Microsoft | | | | Yes | | | |
40-| MiniMax | Yes | Yes | Yes | Yes | | | |
41-| Mistral | | | | | Yes | | |
42-| OpenAI | Yes | Yes | | Yes | Yes | Yes | Yes |
43-| Qwen | | Yes | | | | | |
44-| Runway | | Yes | | | | | |
45-| SenseAudio | | | | | Yes | | |
46-| Together | | Yes | | | | | |
47-| Vydra | Yes | Yes | | Yes | | | |
48-| xAI | Yes | Yes | | Yes | Yes | | Yes |
49-| Xiaomi MiMo | Yes | | | Yes | | | Yes |
48+| Provider | Image | Video | Music | TTS | STT | Realtime voice | Media understanding |
49+| ----------- | :---: | :---: | :---: | :-: | :-: | :------------: | :-----------------: |
50+| Alibaba | | ✓ | | | | | |
51+| BytePlus | | ✓ | | | | | |
52+| ComfyUI | ✓ | ✓ | ✓ | | | | |
53+| Deepgram | | | | | ✓ | ✓ | |
54+| ElevenLabs | | | | ✓ | ✓ | | |
55+| fal | ✓ | ✓ | | | | | |
56+| Google | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ |
57+| Gradium | | | | ✓ | | | |
58+| Local CLI | | | | ✓ | | | |
59+| Microsoft | | | | ✓ | | | |
60+| MiniMax | ✓ | ✓ | ✓ | ✓ | | | |
61+| Mistral | | | | | ✓ | | |
62+| OpenAI | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ |
63+| Qwen | | ✓ | | | | | |
64+| Runway | | ✓ | | | | | |
65+| SenseAudio | | | | | ✓ | | |
66+| Together | | ✓ | | | | | |
67+| Vydra | ✓ | ✓ | | ✓ | | | |
68+| xAI | ✓ | ✓ | | ✓ | ✓ | | ✓ |
69+| Xiaomi MiMo | ✓ | | | ✓ | | | ✓ |
50705171<Note>
52-Media understanding uses any vision-capable or audio-capable model registered in your provider config. The table above highlights providers with dedicated media-understanding support; most LLM providers with multimodal models (Anthropic, Google, OpenAI, etc.) can also understand inbound media when configured as the active reply model.
72+Media understanding uses any vision-capable or audio-capable model registered
73+in your provider config. The matrix above lists providers with dedicated
74+media-understanding support; most multimodal LLM providers (Anthropic, Google,
75+OpenAI, etc.) can also understand inbound media when configured as the active
76+reply model.
5377</Note>
547855-## How async generation works
79+## Async vs synchronous
80+81+| Capability | Mode | Why |
82+| --------------- | ------------ | ------------------------------------------------------------------ |
83+| Image | Synchronous | Provider responses return in seconds; completes inline with reply. |
84+| Text-to-speech | Synchronous | Provider responses return in seconds; attached to the reply audio. |
85+| Video | Asynchronous | Provider processing takes 30 s to several minutes. |
86+| Music (shared) | Asynchronous | Same provider-processing characteristic as video. |
87+| Music (ComfyUI) | Synchronous | Local workflow runs inline against the configured ComfyUI server. |
568857-Video and music generation run as background tasks because provider processing typically takes 30 seconds to several minutes. When the agent calls `video_generate` or `music_generate`, OpenClaw submits the request to the provider, returns a task ID immediately, and tracks the job in the task ledger. The agent continues responding to other messages while the job runs. When the provider finishes, OpenClaw wakes the agent so it can post the finished media back into the original channel. Image generation and TTS are synchronous and complete inline with the reply.
89+For async tools, OpenClaw submits the request to the provider, returns a task
90+id immediately, and tracks the job in the task ledger. The agent continues
91+responding to other messages while the job runs. When the provider finishes,
92+OpenClaw wakes the agent so it can post the finished media back into the
93+original channel.
94+95+## Speech-to-text and Voice Call
58965997Deepgram, ElevenLabs, Mistral, OpenAI, SenseAudio, and xAI can all transcribe
6098inbound audio through the batch `tools.media.audio` path when configured.
6199Channel plugins that preflight a voice note for mention gating or command
62100parsing mark the transcribed attachment on the inbound context, so the shared
63-media-understanding pass reuses that transcript instead of making a second STT
64-call for the same audio.
101+media-understanding pass reuses that transcript instead of making a second
102+STT call for the same audio.
103+65104Deepgram, ElevenLabs, Mistral, OpenAI, and xAI also register Voice Call
66105streaming STT providers, so live phone audio can be forwarded to the selected
67106vendor without waiting for a completed recording.
6810769-Google maps to OpenClaw's image, video, music, batch TTS, backend realtime
70-voice, and media-understanding surfaces. OpenAI maps to OpenClaw's image,
71-video, batch TTS, batch STT, Voice Call streaming STT, backend realtime voice,
72-and memory embedding surfaces. xAI currently maps to OpenClaw's image, video,
73-search, code-execution, batch TTS, batch STT, and Voice Call streaming STT
74-surfaces. xAI Realtime voice is an upstream capability, but it is not
75-registered in OpenClaw until the shared realtime voice contract can represent
76-it.
77-78-## Quick links
79-80-- [Image Generation](/tools/image-generation) -- generating and editing images
81-- [Video Generation](/tools/video-generation) -- text-to-video, image-to-video, and video-to-video
82-- [Music Generation](/tools/music-generation) -- creating music and audio tracks
83-- [Text-to-Speech](/tools/tts) -- converting replies to spoken audio
84-- [Media Understanding](/nodes/media-understanding) -- understanding inbound images, audio, and video
108+## Provider mappings (how vendors split across surfaces)
109+110+<AccordionGroup>
111+<Accordion title="Google">
112+Image, video, music, batch TTS, backend realtime voice, and
113+media-understanding surfaces.
114+</Accordion>
115+<Accordion title="OpenAI">
116+Image, video, batch TTS, batch STT, Voice Call streaming STT, backend
117+realtime voice, and memory-embedding surfaces.
118+</Accordion>
119+<Accordion title="xAI">
120+Image, video, search, code-execution, batch TTS, batch STT, and Voice
121+Call streaming STT. xAI Realtime voice is an upstream capability but is
122+not registered in OpenClaw until the shared realtime-voice contract can
123+represent it.
124+</Accordion>
125+</AccordionGroup>
8512686127## Related
8712888129- [Image generation](/tools/image-generation)
89130- [Video generation](/tools/video-generation)
90131- [Music generation](/tools/music-generation)
91132- [Text-to-speech](/tools/tts)
133+- [Media understanding](/nodes/media-understanding)
134+- [Audio nodes](/nodes/audio)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。