惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Y
Y Combinator Blog
V
V2EX
Jina AI
Jina AI
爱范儿
爱范儿
M
MIT News - Artificial intelligence
量子位
L
LangChain Blog
Google DeepMind News
Google DeepMind News
酷 壳 – CoolShell
酷 壳 – CoolShell
罗磊的独立博客
腾讯CDC
MongoDB | Blog
MongoDB | Blog
P
Proofpoint News Feed
宝玉的分享
宝玉的分享
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
Last Week in AI
Last Week in AI
H
Hackread – Cybersecurity News, Data Breaches, AI and More
F
Fortinet All Blogs
The GitHub Blog
The GitHub Blog
Engineering at Meta
Engineering at Meta
博客园 - 聂微东
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
Vercel News
Vercel News
T
The Blog of Author Tim Ferriss

cs.SD updates on arXiv.org

Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generation WhisperPipe: A Resource-Efficient Streaming Architecture for Real-Time Automatic Speech Recognition Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech Praxy Voice: Voice-Prompt Recovery + BUPS for Commercial-Class Indic TTS from a Frozen Non-Indic Base at Zero Commercial-Training-Data Cost Korean aegyo speech shows systematic F1 increase to signal childlike qualities All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation RAS: a Reliability Oriented Metric for Automatic Speech Recognition Speech Enhancement Based on Drifting Models HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling Robust Audio-Text Retrieval via Cross-Modal Attention and Hybrid Loss Spectro-Temporal Modulation Representation Framework for Human-Imitated Speech Detection UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition Materialistic RIR: Material Conditioned Realistic RIR Generation SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation ONOTE: Benchmarking Omnimodal Notation Processing for Expert-level Music Intelligence From Image to Music Language: A Two-Stage Structure Decoding Approach for Complex Polyphonic OMR ATIR: Towards Audio-Text Interleaved Contextual Retrieval Enhancing Speaker Verification with Whispered Speech via Post-Processing Environmental Sound Deepfake Detection Using Deep-Learning Framework Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Model BEAT: Tokenizing and Generating Symbolic Music by Uniform Temporal Steps Deep Supervised Contrastive Learning of Pitch Contours for Robust Pitch Accent Classification in Seoul Korean HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India Tadabur: A Large-Scale Quran Audio Dataset
OmniFlow: Any-to-Any Generation with Multi-Modal Rectifie...
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Zichun Liao, · 2024-12-02 · via cs.SD updates on arXiv.org

We introduce OmniFlow, a novel generative model designed for any-to-any generation tasks such as text-to-image, text-to-audio, and audio-to-image synthesis. OmniFlow advances the rectified flow (RF) framework used in text-to-image models to handle the joint distribution of multiple modalities. It outperforms previous any-to-any models on a wide range of tasks, such as text-to-image and text-to-audio synthesis. Our work offers three key contributions: First, we extend RF to a multi-modal setting and introduce a novel guidance mechanism, enabling users to flexibly control the alignment between different modalities in the generated outputs. Second, we propose a novel architecture that extends the text-to-image MMDiT architecture of Stable Diffusion 3 and enables audio and text generation. The extended modules can be efficiently pretrained individually and merged with the vanilla text-to-image MMDiT for fine-tuning. Lastly, we conduct a comprehensive study on the design choices of rectified flow transformers for large-scale audio and text generation, providing valuable insights into optimizing performance across diverse modalities. The Code will be available at https://github.com/jacklishufan/OmniFlows.