惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

WordPress大学
WordPress大学
GbyAI
GbyAI
P
Proofpoint News Feed
B
Blog
MyScale Blog
MyScale Blog
V
V2EX
B
Blog RSS Feed
Microsoft Security Blog
Microsoft Security Blog
量子位
Jina AI
Jina AI
博客园 - 叶小钗
Recent Announcements
Recent Announcements
有赞技术团队
有赞技术团队
罗磊的独立博客
L
LangChain Blog
I
InfoQ
云风的 BLOG
云风的 BLOG
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
人人都是产品经理
人人都是产品经理
小众软件
小众软件
V
Visual Studio Blog
月光博客
月光博客
The Cloudflare Blog
雷峰网
雷峰网

cs.IR updates on arXiv.org

From Top-1 to Top-K: A Reproducibility Study and Benchmarking of Counterfactual Explanations for Recommender Systems Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI Diagnosable ColBERT: Debugging Late-Interaction Retrieval Models Using a Learned Latent Space as Reference Enhancing Unsupervised Keyword Extraction in Academic Papers through Integrating Highlights with Abstract CAST: Modeling Semantic-Level Transitions for Complementary-Aware Sequential Recommendation IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text Think Before Writing: Feature-Level Multi-Objective Optimization for Generative Citation Visibility RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora Personalized Benchmarking: Evaluating LLMs by Individual Preferences Modular Representation Compression: Adapting LLMs for Efficient and Effective Recommendations JFinTEB: Japanese Financial Text Embedding Benchmark UsefulBench: Towards Decision-Useful Information as a Target for Information Retrieval SIMMER: Cross-Modal Food Image--Recipe Retrieval via MLLM-Based Embedding Rethinking the Necessity of Adaptive Retrieval-Augmented Generation through the Lens of Adaptive Listwise Ranking BioHiCL: Hierarchical Multi-Label Contrastive Learning for Biomedical Retrieval with MeSH Labels Learning Behaviorally Grounded Item Embeddings via Personalized Temporal Contexts Collaborative Filtering Through Weighted Similarities of User and Item Embeddings IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning Metric-agnostic Learning-to-Rank via Boosting and Rank Approximation GenRec: A Preference-Oriented Generative Framework for Large-Scale Recommendation Uncertainty-aware Generative Learning Path Recommendation with Cognition-Adaptive Diffusion CPGRec+: A Balance-oriented Framework for Personalized Video Game Recommendations Don't Retrieve, Navigate: Distilling Enterprise Knowledge into Navigable Agent Skills for QA and RAG NewsTorch: A PyTorch-based Toolkit for Learner-oriented News Recommendation Controlling Authority Retrieval: A Missing Retrieval Objective for Authority-Governed Knowledge APEX-MEM: Agentic Semi-Structured Memory with Temporal Reasoning for Long-Term Conversational AI ID and Graph View Contrastive Learning with Multi-View Attention Fusion for Sequential Recommendation Large Language Models to Enhance Business Process Modeling: Past, Present, and Future Trends Dual-Enhancement Product Bundling: Bridging Interactive Graph and Large Language Model Evaluation of Agents under Simulated AI Marketplace Dynamics
Bridging High-Quality Audio and Video via Language for So...
Julia Wilkins, Justin Salamon, Magdalena Fuentes, Juan Pablo Bel · 2023-08-18 · via cs.IR updates on arXiv.org

Finding the right sound effects (SFX) to match moments in a video is a difficult and time-consuming task, and relies heavily on the quality and completeness of text metadata. Retrieving high-quality (HQ) SFX using a video frame directly as the query is an attractive alternative, removing the reliance on text metadata and providing a low barrier to entry for non-experts. Due to the lack of HQ audio-visual training data, previous work on audio-visual retrieval relies on YouTube (in-the-wild) videos of varied quality for training, where the audio is often noisy and the video of amateur quality. As such it is unclear whether these systems would generalize to the task of matching HQ audio to production-quality video. To address this, we propose a multimodal framework for recommending HQ SFX given a video frame by (1) leveraging large language models and foundational vision-language models to bridge HQ audio and video to create audio-visual pairs, resulting in a highly scalable automatic audio-visual data curation pipeline; and (2) using pre-trained audio and visual encoders to train a contrastive learning-based retrieval system. We show that our system, trained using our automatic data curation pipeline, significantly outperforms baselines trained on in-the-wild data on the task of HQ SFX retrieval for video. Furthermore, while the baselines fail to generalize to this task, our system generalizes well from clean to in-the-wild data, outperforming the baselines on a dataset of YouTube videos despite only being trained on the HQ audio-visual pairs. A user study confirms that people prefer SFX retrieved by our system over the baseline 67% of the time both for HQ and in-the-wild data. Finally, we present ablations to determine the impact of model and data pipeline design choices on downstream retrieval performance. Please visit our project website to listen to and view our SFX retrieval results.