Understanding what’s happening in an audio clip is a deceptively hard problem. Transcribing spoken words is the easy part. A truly capable system also needs to recognize who is speaking, detect their emotional state, interpret background sounds, analyze musical content, and answer time-grounded questions like ‘what did the speaker say at the 2-minute mark?’. Tackling all of that required stitching together multiple specialized systems.
Tthe OpenMOSS team, MOSI.AI, and Shanghai Innovation Institute released MOSS-Audio: an open-source audio understanding model designed to unify all of those capabilities inside a single foundation model.
MOSS-Audio supports speech understanding, environmental sound understanding, music understanding, audio captioning, time-aware QA, and complex reasoning over real-world audio. Its capability set breaks down into several distinct areas. Speech & Content Understanding accurately recognizes and transcribes spoken content, supporting both word-level and sentence-level timestamp alignment. Speaker, Emotion & Event Analysis identifies speaker characteristics, analyzes emotional states based on tone, timbre, and context, and detects key acoustic events within the audio. Scene & Sound Cue Extraction pulls meaningful signals from background sounds, environmental noise, and non-speech signals to infer scene context and atmosphere. Music Understanding analyzes musical style, emotional progression, and instrumentation. Audio Question Answering & Summarization handles questions and summaries across speech, podcasts, meetings, and interviews. Finally, Complex Reasoning performs multi-hop reasoning over audio content, powered by both chain-of-thought training and reinforcement learning.
In practical terms, a single MOSS-Audio model can do all of the above without switching between different specialized systems.
The team released four variants at launch: MOSS-Audio-4B-Instruct, MOSS-Audio-4B-Thinking, MOSS-Audio-8B-Instruct, and MOSS-Audio-8B-Thinking. The naming convention is worth understanding if you’re deciding which to use. The Instruct variants are optimized for direct instruction following, making them well-suited for production pipelines where you want predictable, structured outputs. The Thinking variants provide stronger chain-of-thought reasoning capabilities, better suited for tasks requiring multi-hop inference. The 4B models use Qwen3-4B as the LLM backbone, and the 8B models use Qwen3-8B, resulting in total model sizes of approximately 4.6B and 8.6B parameters respectively.
MOSS-Audio follows a modular design comprising three components: an audio encoder, a modality adapter, and a large language model. Raw audio is first encoded by the MOSS-Audio-Encoder into continuous temporal representations at 12.5 Hz. Those representations are then projected into the language model’s embedding space through the adapter, and finally consumed by the LLM for auto-regressive text generation.
The research team trained the encoder from scratch rather than relying on off-the-shelf audio frontends. Their reasoning: a dedicated encoder delivers more robust speech representations, tighter temporal alignment, and better extensibility across acoustic domains.
Two architectural innovations inside MOSS-Audio are worth understanding in detail.
DeepStack Cross-Layer Feature Injection: A common weakness in audio models is that relying only on the encoder’s top-layer features tends to lose low-level acoustic information, things like prosody, transient events, and local time-frequency structure. MOSS-Audio addresses this with a DeepStack-inspired cross-layer injection module between the encoder and the language model: in addition to the encoder’s final-layer output, features from earlier and intermediate layers are selected, independently projected, and injected into the language model’s early layers. This preserves multi-granularity information ranging from low-level acoustic details to high-level semantic abstractions, helping the model retain rhythm, timbre, transients, and background structure that a single high-level representation cannot fully capture.
Time-Aware Representation: Time is a critical dimension in audio that text models aren’t naturally equipped to handle. MOSS-Audio addresses this through a time-marker insertion strategy during pretraining: explicit time tokens are inserted between audio frame representations at fixed time intervals to indicate temporal positions. This lets the model learn ‘what happened when’ within a unified text generation framework, naturally supporting timestamp ASR, event localization, time-based QA, and long-audio retrospection — without requiring a separate localization head or post-processing pipeline.
The numbers are strong. On general audio understanding, MOSS-Audio-8B-Thinking achieves an average accuracy of 71.08 across four benchmarks — 77.33 on MMAU, 64.92 on MMAU-Pro, 66.53 on MMAR, and 75.52 on MMSU, outperforming majority of open-source models. That includes larger models: Step-Audio-R1 at 33B scores 70.67, and Qwen3-Omni-30B-A3B-Instruct at 30B scores 67.91. For further context, Kimi-Audio (7B) scores 61.14 and MiMo-Audio-7B scores 62.97 on the same average. The 4B Thinking variant scores 68.37, meaning the smaller model with chain-of-thought training beats all larger open-source instruct-only competitors.
On speech captioning, evaluated with an LLM-as-a-Judge methodology across 13 fine-grained dimensions including gender, age, accent, pitch, volume, speed, texture, clarity, fluency, emotion, tone, personality, and summary, MOSS-Audio-Instruct variants lead across 11 out of 13 dimensions, with MOSS-Audio-8B-Instruct achieving the best overall average score of 3.7252.
On automatic speech recognition (ASR) spanning 12 evaluation dimensions — including health condition, code-switching, dialect, singing, and non-speech scenarios — MOSS-Audio-8B-Instruct achieves the lowest overall CER (Character Error Rate) of 11.30 across all tested models.
Check out the Model Weights and Repo. Also, feel free to follow us on Twitter and don’t forget to join our 130k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。