

















Abstract:Image-to-fMRI encoding is important for both neuroscience research and practical applications. However, such "Brain-Encoders" have been typically trained per-subject and per fMRI-dataset, thus restricted to very limited training data. In this paper we propose a Universal Brain-Encoder, which can be trained jointly on data from many different subjects/datasets/machines. What makes this possible is our new voxel-centric Encoder architecture, which learns a unique "voxel-embedding" per brain-voxel. Our Encoder trains to predict the response of each brain-voxel on every image, by directly computing the cross-attention between the brain-voxel embedding and multi-level deep image features. This voxel-centric architecture allows the functional role of each brain-voxel to naturally emerge from the voxel-image cross-attention. We show the power of this approach to (i) combine data from multiple different subjects (a "Crowd of Brains") to improve each individual brain-encoding, (ii) quick & effective Transfer-Learning across subjects, datasets, and machines (e.g., 3-Tesla, 7-Tesla), with few training examples, and (iii) use the learned voxel-embeddings as a powerful tool to explore brain functionality (e.g., what is encoded where in the brain).
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2406.12179 [cs.CV] |
| (or arXiv:2406.12179v4 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2406.12179 arXiv-issued DOI via DataCite |
From: Roman Beliy [view email]
[v1]
Tue, 18 Jun 2024 01:17:07 UTC (28,227 KB)
[v2]
Wed, 19 Mar 2025 23:24:48 UTC (44,719 KB)
[v3]
Sat, 8 Nov 2025 20:02:15 UTC (23,663 KB)
[v4]
Sun, 24 May 2026 14:47:20 UTC (22,315 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。