Abstract:Social interactions play a crucial role in shaping human behavior, relationships, and societies. It encompasses various forms of communication, such as verbal conversation, non-verbal gestures, facial expressions, and body language. In this work, we develop a novel computational approach to detect face-to-face verbal conversations, a foundational aspect of human social interactions. We leverage multimodal data captured by a commodity smartwatch, specifically synchronizing microphone audio with 6-axis inertial signals (accelerometer and gyroscope). We design, train, and evaluate convolutional and attention-based neural networks using three different fusion methods to integrate the audio and motion modalities. To validate this framework, we conduct a lab study with 11 participants and a semi-naturalistic study with 24 participants. Our comprehensive evaluation demonstrates that fusing inertial data with audio significantly improves detection performance by capturing non-verbal conversational dynamics. Overall, our framework achieved 82.0$\pm$3.0% macro F1-score when detecting conversations in the lab and 77.2$\pm$1.8% in the semi-naturalistic setting. Lastly, we demonstrate real-time conversation detection by deploying our trained model to a user application running on a commercial smartwatch.
| Comments: | Accepted to ACM Transactions on Intelligent Systems and Technology |
| Subjects: | Machine Learning (cs.LG) |
| ACM classes: | I.2.0; J.4 |
| Cite as: | arXiv:2507.12002 [cs.LG] |
| (or arXiv:2507.12002v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2507.12002 arXiv-issued DOI via DataCite |
From: Alice Zhang [view email]
[v1]
Wed, 16 Jul 2025 07:57:15 UTC (5,835 KB)
[v2]
Tue, 12 May 2026 02:53:11 UTC (6,324 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。