惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

罗磊的独立博客
Martin Fowler
Martin Fowler
J
Java Code Geeks
The GitHub Blog
The GitHub Blog
C
Check Point Blog
H
Help Net Security
Google DeepMind News
Google DeepMind News
人人都是产品经理
人人都是产品经理
博客园 - 聂微东
P
Proofpoint News Feed
V
Visual Studio Blog
Stack Overflow Blog
Stack Overflow Blog
雷峰网
雷峰网
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
Vercel News
Vercel News
S
SegmentFault 最新的问题
L
LangChain Blog
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
The Cloudflare Blog
Hugging Face - Blog
Hugging Face - Blog
有赞技术团队
有赞技术团队
博客园_首页
小众软件
小众软件
aimingoo的专栏
aimingoo的专栏

cs.SD updates on arXiv.org

Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generation WhisperPipe: A Resource-Efficient Streaming Architecture for Real-Time Automatic Speech Recognition Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech Praxy Voice: Voice-Prompt Recovery + BUPS for Commercial-Class Indic TTS from a Frozen Non-Indic Base at Zero Commercial-Training-Data Cost Korean aegyo speech shows systematic F1 increase to signal childlike qualities All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation RAS: a Reliability Oriented Metric for Automatic Speech Recognition Speech Enhancement Based on Drifting Models HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling Robust Audio-Text Retrieval via Cross-Modal Attention and Hybrid Loss Spectro-Temporal Modulation Representation Framework for Human-Imitated Speech Detection UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition Materialistic RIR: Material Conditioned Realistic RIR Generation SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation ONOTE: Benchmarking Omnimodal Notation Processing for Expert-level Music Intelligence From Image to Music Language: A Two-Stage Structure Decoding Approach for Complex Polyphonic OMR ATIR: Towards Audio-Text Interleaved Contextual Retrieval Enhancing Speaker Verification with Whispered Speech via Post-Processing Environmental Sound Deepfake Detection Using Deep-Learning Framework Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Model BEAT: Tokenizing and Generating Symbolic Music by Uniform Temporal Steps Deep Supervised Contrastive Learning of Pitch Contours for Robust Pitch Accent Classification in Seoul Korean HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India Tadabur: A Large-Scale Quran Audio Dataset
Mechanisms of Multimodal Synchronization: Insights from D...
Akshita Gupta, Tatiana Likhomanenko, Karren Dai Yang, Richard He · 2024-11-27 · via cs.SD updates on arXiv.org

Unified decoder-only transformers have shown promise for multimodal generation, yet the mechanisms by which they synchronize modalities with heterogeneous sampling rates remain underexplored. We investigate these mechanisms through video-text-to-speech (VTTS) synthesis-a controlled task requiring fine-grained temporal alignment between sparse text, video, and continuous speech. Using a unified decoder-only transformer, dubbed Visatronic, trained on VoxCeleb2, we study: (i) how modalities contribute complementary information, (ii) how positional encoding strategies enable synchronization across heterogeneous rates, (iii) how modality ordering shapes the trade-off between in-domain performance and cross-domain transfer, (iv) how phoneme-level synchronization metrics provide diagnostic insight into per-phoneme timing errors. Our findings reveal that both "global sequential indexing'' (unique position IDs across modalities) and "co-temporal ordered indexing'' (identical IDs for temporally corresponding tokens) achieve strong synchronization performance, with co-temporal ordered indexing providing a simple mechanism without explicit timestamp metadata. Both text and video contribute complementary signals: text ensures intelligibility while video provides temporal cues and emotional expressiveness. Modality ordering reveals a consistent trade-off: video-first ordering achieves stronger in-domain performance while text-first ordering generalizes more robustly to unseen domains. Our findings also reveal, that diverse large-scale training enables transferable synchronization strategies. To enable fine-grained analysis, we also introduce TimeSync, a phoneme-level metric that reveals temporal misalignments overlooked by frame-level metrics. These insights establish VTTS as a valuable testbed for understanding temporal synchronization in unified multimodal decoders.