Abstract:Many SLT systems quietly assume that brief chunks of signing map directly to spoken-language words. That assumption breaks down because signers often create meaning on the fly using context, space, and movement. We revisit SLT and argue that it is mainly a cross-modal reasoning task, not just a straightforward video-to-text conversion. We thus introduce a reasoning-driven SLT framework that uses an ordered sequence of latent thoughts as an explicit middle layer between the video and the generated text. These latent thoughts gradually extract and organize meaning over time. On top of this, we use a plan-then-ground decoding method: the model first decides what it wants to say, and then looks back at the video to find the evidence. This separation improves coherence and faithfulness. We also built and released a new large-scale gloss-free SLT dataset with stronger context dependencies and more realistic meanings. Experiments across several benchmarks show consistent gains over existing gloss-free methods. Our code and data are available at this https URL.
| Comments: | Accepted to ACL 2026 Main |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2604.15301 [cs.CV] |
| (or arXiv:2604.15301v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2604.15301 arXiv-issued DOI via DataCite |
From: Yiyang Jiang [view email]
[v1]
Thu, 16 Apr 2026 17:57:24 UTC (2,494 KB)
[v2]
Fri, 17 Apr 2026 07:02:15 UTC (2,494 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。