Paper deep dive

BAT-CLIP: Trimodal Alignment of Brain, Audio and Text

Suhyun Kim, Jinmo Han, Danny Dongyeop Han, Ahhyun Lucy Lee, Jewoon Lee, Yonghyeon Gwon, Zach Paris, Chun Kee Chung, Saewoong Bahk, Nam Soo Kim, Seong Jae Hwang, Jiook Cha

Year: 2026Venue: arXiv preprintArea: cs.SDType: PreprintEmbeddings: 31

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Decoding and interpreting naturalistic speech from the brain increasingly relies on alignment to pretrained speech and language representation spaces. However, current CLIP-style brain-speech alignment ground neural activity to a single anchor modality-audio or text-despite the brain's inherently multimodal speech processing. This induces a trade-off: audio anchoring preserves temporal structure but weakens linguistic separability, while text anchoring captures semantics yet discards acoustic detail. We propose BAT-CLIP, the first CLIP-style trimodal alignment framework for iEEG that jointly aligns neural embeddings to both pretrained audio and text anchors in a shared, frozen audio-text manifold. On the naturalistic Podcast benchmark, BAT-CLIP yields more robust representations than bimodal CLIP baselines. We also highlight the importance of using self-supervised foundation models for CLIP training.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

29,096 characters extracted from source content.

Expand or collapse full text

BAT-CLIP: Trimodal Alignment of Brain, Audio and Text ∗These authors contributed equally to this work. †Corresponding author: connectome@snu.ac.kr. Abstract Decoding and interpreting naturalistic speech from the brain increasingly relies on alignment to pretrained speech and language representation spaces. However, current CLIP-style brain–speech alignment ground neural activity to a single anchor modality—audio or text—despite the brain’s inherently multimodal speech processing. This induces a trade-off: audio anchoring preserves temporal structure but weakens linguistic separability, while text anchoring captures semantics yet discards acoustic detail. We propose BAT-CLIP, the first CLIP-style trimodal alignment framework for iEEG that jointly aligns neural embeddings to both pretrained audio and text anchors in a shared, frozen audio–text manifold. On the naturalistic Podcast benchmark, BAT-CLIP yields more robust representations than bimodal CLIP baselines. We also highlight the importance of using self-supervised foundation models for CLIP training. Suhyun Kim1,∗, Jinmo Han2,∗, Danny Dongyeop Han2,∗, Ahhyun Lucy Lee2, Jewoon Lee2, Yonghyeon Gwon2, Zach Paris3, Chun Kee Chung2, Saewoong Bahk2, Nam Soo Kim2, Seong Jae Hwang1, Jiook Cha2,† 1 Yonsei University, Seoul, Republic of Korea 2 Seoul National University, Seoul, Republic of Korea 3 Dartmouth College, Hanover, NH, USA Index Terms— Speech Representation Learning, Intracranial EEG (iEEG/ECoG), Contrastive Learning, Audio–Text Alignment 1 Introduction Brain–speech alignment is a central objective in neurolinguistics, spanning encoding models that predict neural responses from speech/language representations [5] and decoding models that infer speech or linguistic variables from brain activity [3, 12]. It matters for applications because stronger alignment improves downstream decoding, enabling simple readouts to recover linguistic variables and more expressive decoders to generate coherent text. It is also a scientific target, offering a principled way to test which computational representations best match the brain and where different representational levels emerge across cortex [16, 8]. Recently, large-scale speech and language foundation models have reshaped brain–speech alignment: rather than relying on hand-crafted features or task-specific labels, neural signals can be aligned end-to-end to pretrained embedding spaces via CLIP-style contrastive objectives [17, 10]. This approach has driven strong gains in both encoding and decoding—including naturalistic speech and vision in fMRI and electrophysiology (EEG/MEG/iEEG)—by leveraging the semantic structure and inductive biases of pretrained audio–text models [16]. Fig. 1: Overview of BAT-CLIP. A trainable brain encoder fθf_θ, initialized from brain foundation model DIVER-1, maps neural windows to brain embeddings zBz^B. Frozen AudioCLIP encoders provide audio anchors zAz^A and text anchors zTz^T. BAT-CLIP optimizes a trimodal contrastive objective ℒTotalL_Total to align neural embeddings with the joint audio–text representation space. However, current CLIP-style brain–speech methods anchor neural activity to a single modality (audio or text). This neglects the multimodal nature of speech comprehension, where the brain integrates linguistic and acoustic information across distributed networks [11]. Encoding work further shows that audio/speech and text/language models capture complementary aspects of neural language comprehension: speech models better explain activity in earlier auditory regions, whereas text models better explain later language regions linked to higher-level linguistic processing [14, 1]. Motivated by this complementarity of audio and text model representations, we propose BAT-CLIP (Brain–Audio–Text CLIP), which aligns a pretrained brain foundation model to a frozen audio–text manifold (AudioCLIP [6]) via a CLIP-style trimodal contrastive objective, anchoring brain representations to both audio and text embeddings (Fig. 1). Furthermore, we assess the value of self-supervised pretraining itself by comparing random initialization to a strong SSL-pretrained brain foundation model, showing that SSL provides a critical prior for reliable trimodal alignment. We further perform embedding-space analyses that characterize how trimodal alignment better organizes neural representations relative to audio and text anchors. 2 Methods We propose BAT-CLIP, a trimodal contrastive alignment framework that grounds self-supervised pretrained brain representations in a pretrained audio–text manifold. Specifically, we initialize from a brain foundation model (DIVER-1 [7]) and adapt with a trimodal objective using paired iEEG–audio–text data. 2.1 Problem Formulation Let =(xiB,xiA,xiT)i=1ND=\(x_i^B,x_i^A,x_i^T)\_i=1^N denote time-aligned triplets, where xiBx_i^B is an iEEG segment with C channels over T time points, and xiAx_i^A and xiTx_i^T are the corresponding audio and transcript segments. We downsample iEEG to match the DIVER-1 checkpoint, resample audio to 44.1 kHz for AudioCLIP, and tokenize transcripts with the AudioCLIP tokenizer. Our CLIP alignment objective is to train a brain encoder fθ:B→ℝdf_θ:X^B ^d that maps neural windows xiBx^B_i to a brain embedding ziB=fθ(xiB)z^B_i=f_θ(x^B_i) in a joint embedding space (d=1024d=1024) shared with audio and text representations. 2.2 Alignment Prior: DIVER-1 Initialization We initialize the brain encoder from DIVER-1, a iEEG foundation model pretrained with self-supervised masked reconstruction on large amounts of data. This initialization acts as an alignment prior, reducing overfitting in our limited paired brain–audio–text dataset. During CLIP alignment, we flatten the brain encoder’s embeddings and project them to the shared embedding space with a lightweight two-layer MLP, followed by ℓ2 _2 normalization. 2.3 Shared Audio-Linguistic Space: AudioCLIP Anchors We ground neural representations in a shared multimodal space using the pretrained AudioCLIP audio and text encoders. For each time-aligned triplet, we encode the audio waveform and transcript with gϕ∗g_φ^* and hψ∗h_ψ^* to obtain anchor embeddings ziA=norm⁡(gϕ∗(xiA))z_i^A=norm(g_φ^*(x_i^A)) and ziT=norm⁡(hψ∗(xiT))z_i^T=norm(h_ψ^*(x_i^T)), where norm⁡(⋅)norm(·) denotes ℓ2 _2 normalization. We keep ϕ∗φ^* and ψ∗ψ^* frozen throughout training so that the brain encoder is trained to match a stable, well-regularized audio–text manifold rather than co-adapting the anchor spaces in the low-data setting. 2.4 Trimodal Contrastive Alignment We align brain embeddings to both audio and text anchors with a symmetric Brain–Text and Brain–Audio contrastive objective. Over the global batch of size NbN_b, we compute similarity logit matrices BA,BT∈ℝNb×NbS_BA,S_BT ^N_b× N_b: BA=αBA⊤,BT=αBT⊤,S_BA=α\,Z_BZ_A ,\ S_BT=α\,Z_BZ_T , where B,A,T∈ℝNb×dZ_B,Z_A,Z_T ^N_b× d stack the ℓ2 _2-normalized embeddings ziB,ziA,ziTz^B_i,z^A_i,z^T_i for the batch. The constant α=14.3α=14.3 is a scaling factor applied to sharpen the logit distribution and stabilize optimization. Following CLIP, we compute a bidirectional cross-entropy loss for each pair. Given diagonal ground-truth labels =[0,1,…,Nb−1]y=[0,1,…,N_b-1], the Brain-Audio loss ℒBAL_BA and the Brain-Text loss ℒBTL_BT are formulated as: ℒBA _BA =12(CE⁡(BA,)+CE⁡(BA⊤,)), = 12 (CE(S_BA,y)+CE(S_BA ,y) ), (1) ℒBT _BT =12(CE⁡(BT,)+CE⁡(BT⊤,)). = 12 (CE(S_BT,y)+CE(S_BT ,y) ). (2) The final contrastive objective is the sum: ℒTotal=12(ℒBA+ℒBT)L_Total= 12 (L_BA+L_BT ) (3) Method Type Content Onset POS GPT Surprisal Word Emb Whisper AUC Δ AUC Δ AUC Δ AUC Δ AvgAUC Δ Acc Δ CNN(Baseline)† Sup 0.5498 – 0.7519 – 0.5653 – 0.5019 – 0.6260 – 0.6634 – BrainBERT SSL 0.5505 +0.13% 0.6341 -15.67% 0.5325 -5.80% 0.5157 +2.75% 0.5533 -11.61% 0.5805 -12.49% PopT SSL 0.4950 -9.97% 0.4654 -38.10% 0.5031 -11.00% 0.5022 +0.06% 0.5002 -20.10% 0.5014 -24.42% DIVER-1 SSL 0.5645 +2.67% 0.7729 +2.79% 0.5656 +0.05% 0.5155 +2.71% 0.6450 +3.04% 0.7022 +5.85% BA-CLIP Align 0.5840 +6.31% 0.6560 -12.61% 0.5673 +0.32% 0.5169 +3.02% 0.6468 +4.26% 0.7141 +7.75% BT-CLIP Align 0.5813 +5.82% 0.6562 -12.56% 0.5709 +0.97% 0.5203 +3.69% 0.6510 +4.94% 0.7157 +7.98% BAT-CLIP(Ours) Align 0.5819 +5.96% 0.6589 -12.18% 0.5740 +1.50% 0.5229 +4.21% 0.6512 +5.01% 0.7162 +8.09% Table 1: Overall decoding performance (averaged over 9 subjects). We report AUC for Content, Onset, POS, GPT surprisal, and Word Emb decoding, and accuracy for Whisper. Δ denotes relative gain over the supervised CNN baseline (baseline Δ : “–”). † indicates full fine-tuning (CNN); all other methods are evaluated via linear probing. BA-CLIP, BT-CLIP, and BAT-CLIP apply CLIP-style contrastive alignment on top of the pretrained DIVER-1 brain encoder, using brain–audio, brain–text, and trimodal brain–audio–text alignment, respectively. Overall, DIVER-1 is the strongest foundation-model baseline, CLIP alignment generally improves over DIVER-1, and BAT-CLIP achieves the best result among the aligned variants on five of six probe tasks. Bold and underline denote the best and second-best results within each column. 3 Experiments and Results We evaluate BAT-CLIP with the goal of isolating the effect of trimodal alignment on iEEG representations under realistic low-data regimes. We first describe the experimental setup—data, preprocessing, baselines, and training configurations for alignment learning. We then report quantitative results on the Podcast Benchmark probe suite and analyze ablations that attribute performance to specific design choices. 3.1 Experimental Settings Data and benchmark. We use the Podcast intracranial electrophysiology dataset[18], which provides iEEG recordings during naturalistic passive listening to a spoken narrative, along with word-level time-aligned transcripts and derived linguistic/acoustic features. All reported results follow the Podcast Benchmark † Paris, Z., Sim, S., Bhattacharjee, A., Jalon, I., Peeper, G., Han, J., & Hasson, U. ECoG Podcast Benchmark [Computer software]. Available at https://github.com/hassonlab/podcast-benchmark. evaluation suite built on top of this dataset, which standardizes probe tasks, metrics, and data splits for comparable decoding-based assessment. Baselines and model variants. We compare against (i) a supervised CNN baseline trained end-to-end with full finetuning on each downstream probe task and (i) recent brain foundation models (BrainBERT[15], PopT[2], and DIVER-1[7]). Since unaligned DIVER-1 provides the strongest off-the-shelf performance in this setting, we adopt its pretrained 12-layer transformer backbone (dmodel=256d_model=256) as the common initialization for our aligned variants (BA-CLIP (Brain-Audio CLIP), BT-CLIP (Brain-Text CLIP), and BAT-CLIP (Brain-Audio-Text CLIP)). This controls for architecture and initialization, attributing gains to multimodal alignment. Preprocessing. For each subject, we exclude bad channels and retain the top 10% most statistically responsive electrodes (selected using training data only to avoid leakage). Raw iEEG is standardized and resampled to 500 Hz, then segmented into 1.0 s windows anchored to benchmark-defined target events. Following DIVER-1, each window is tokenized into non-overlapping patches of 50 samples (0.1 s) to form the transformer input sequence. Alignment. Aligned models are trained with the contrastive objectives described in Sec. 2, using frozen audio/text anchor encoders. We optimize the brain encoder and its projection head; the pretrained anchor encoders remain fixed throughout training. Hyperparameters for alignment training (e.g., learning rate and weight decay) are selected on the validation split and held fixed when reporting test performance. Evaluation tasks, metrics and protocol. We evaluate each model on six probe tasks that test what information can be decoded from neural activity around word/sentence events: (1) content vs. non-content word classification (binary), (2) coarse part-of-speech classification (5 classes), (3) sentence-onset detection (binary; negatives are sampled away from true onsets within the same sentence), (4) GPT Surprisal (GPT-2 XL; 3-bin: low/medium/high), (5) word-embedding decoding (GPT-2 XL/GloVe), and (6) Whisper latent decoding[5]. For classification probes (1)–(4), we report AUC (binary ROC-AUC or one-vs-rest multiclass AUC). For word-embedding decoding, we report AvgAUC: one-vs-rest AUC is computed per vocabulary item and then averaged across eligible words. For Whisper-latent decoding, we report pairwise matching accuracy: for each predicted segment, we test whether it is more similar to its true matched target than to mismatched targets. We train supervised head for each task under an identical finetuning protocol: a fixed temporal 60%/20%/20% train/validation/test split over five contiguous blocks, with the first block held out for testing and the remaining blocks split sequentially into training and validation, lag fixed at 00 ms, and early stopping with a maximum of 30 epochs and patience =5=5. Method Init. Content Onset POS GPT Surprisal Word Emb Whisper AUC Δ AUC Δ AUC Δ AUC Δ AvgAUC Δ Acc Δ BA-CLIP Scratch 0.5671 +3.14% 0.7792 +3.75% 0.5624 -0.35% 0.5143 +2.55% 0.6059 -2.33% 0.6609 +0.05% DIVER-1 0.5840 +6.31% 0.6560 -12.61% 0.5673 +0.32% 0.5169 +3.02% 0.6468 +4.26% 0.7141 +7.75% BT-CLIP Scratch 0.5650 +2.85% 0.7822 +4.19% 0.5635 -0.18% 0.5120 +2.10% 0.6296 +1.52% 0.6607 -0.02% DIVER-1 0.5813 +5.82% 0.6562 -12.56% 0.5709 +0.97% 0.5203 +3.69% 0.6510 +4.94% 0.7157 +7.98% BAT-CLIP Scratch 0.5586 +1.70% 0.7600 +0.94% 0.5480 -2.94% 0.5119 +2.09% 0.6123 -1.32% 0.6463 -2.36% DIVER-1 0.5819 +5.96% 0.6589 -12.18% 0.5740 +1.50% 0.5229 +4.21% 0.6512 +5.01% 0.7162 +8.09% Table 2: Effect of pretraining across methods. We compare CLIP-aligned models trained from scratch (Scratch) versus the same models initialized with pretrained DIVER-1 weights, across unimodal alignments (BA-CLIP: brain–audio; BT-CLIP: brain–text) and trimodal alignment (BAT-CLIP: brain–audio–text). As in Table 1, Δ denotes relative gain over the supervised CNN baseline (baseline Δ : “–”). Pretrained DIVER-1 initialization is key to realizing the gains from trimodal CLIP alignment; training from scratch largely fails to capitalize on brain–audio–text supervision. Fig. 2: Visualization of learned word-level representations. 2D t-SNE projection of brain embeddings extracted by various models for Sub03. Each point represents a single word-level embedding. The color of the point denotes the sentence to which the word belongs, and the shape indicates the speaker. Compared to the baselines, the proposed trimodal alignment model, BAT-CLIP, exhibits tighter clustering of words from the same sentence (shared colors), consistent with the sentence-level clustering results in Table 3. 3.2 Benefits of Trimodal Alignment. Table 1 compares our BAT-CLIP against bimodal alignment variants built on the same DIVER-1 brain encoder—BA-CLIP (Brain–Audio) and BT-CLIP (Brain–Text)—as well as a supervised CNN baseline and pretrained foundation models (BrainBERT, PopT, DIVER-1). Among the foundation-model and CNN baselines, DIVER-1 is the strongest overall, despite being evaluated via linear probing rather than full fine-tuning; this motivates using DIVER-1 pretrained weights as the initialization for CLIP-style alignment. Applying CLIP alignment on top of DIVER-1 generally improves decoding performance, and BAT-CLIP is the best aligned model overall. Trimodal alignment outperforms the bimodal variants on POS, GPT surprisal, word-embedding decoding, and Whisper decoding; on the Content probe, however, it falls behind the bimodal BA-CLIP. A notable exception is the Onset task, where both the CNN and unaligned DIVER-1 baselines remain strongest and CLIP alignment degrades performance. We hypothesize this reflects a mismatch between the objective and the task structure. Onset detection is timing-critical and can depend on transient, highly localized neural signatures (sometimes confined to a small subset of channels) [13], whereas CLIP alignment encourages temporal invariances and relies on globally pooled window-level embeddings. Such pooling can wash out sharp transient cues essential for precise onset detection. 3.3 SSL Initialization for Trimodal Alignment. Table 2 compares CLIP-aligned models trained from scratch (Random Init.) with the same objectives initialized from pretrained DIVER-1 weights. Pretrained initialization substantially amplifies the benefits of CLIP alignment, yielding markedly stronger downstream decoding. This matches the standard recipe in other domains, where CLIP-style objectives are typically applied on top of strong self-supervised or foundation-model priors [9, 4]. In contrast, prior brain CLIP-alignment work has rarely combined SSL pretraining with subsequent CLIP grounding. Taken together, these results position brain foundation models as a strong substrate for multimodal grounding and suggest that scaling and reusing pretrained neural representations will be central to future progress in brain–audio–text alignment, in line with recent results EEG [10]. Notably, the advantage of trimodal alignment is only clearly realized when starting from a strong SSL prior: with DIVER-1 initialization, BAT-CLIP benefits from joint brain–audio–text supervision, whereas under random initialization the trimodal setting does not reliably separate from unimodal variants, suggesting pretrained representations are needed to effectively absorb and exploit trimodal CLIP supervision. 3.4 Embedding-space analysis We analyzed how SSL pretraining and CLIP-style grounding shape the geometry of neural representations by comparing test-fold neural embedding spaces from DIVER-1, BA-CLIP, BT-CLIP, and BAT-CLIP. For each test segment, we extracted the ℓ2 _2-normalized brain backbone embedding ziBz_i^B and visualized the space with t-SNE (Fig. 2). SSL-only DIVER-1 exhibits weak sentence-level structure, with embeddings from different sentences substantially mixed, whereas CLIP clearly re-organizes the space, with BAT-CLIP showing the strongest sentence-level grouping. To quantify this, we evaluate sentence-level clustering of word-aligned neural embeddings using two metrics (Table 3), over K=31K=31 test sentences with at least three words (N=1,025N=1,025). Cosine Coherence is computed as C=1K∑s[cos¯intra(s)−cos¯inter(s)]C= 1K _s[ _intra(s)- _inter(s)], where the terms denote mean cosine similarity among embeddings within sentence s and between embeddings in s and those in all other sentences, respectively. Separation Ratio is computed in PCA-reduced 50-dimensional space as SR=D¯inter/D¯intraSR= D_inter/ D_intra, where D¯inter D_inter is the mean distance between sentence centroids and D¯intra D_intra is the mean within-sentence spread. BAT-CLIP attains the highest scores on both metrics, consistent with tighter intra-sentence grouping and stronger inter-sentence separation than DIVER-1 and the bimodal variants. Method Cosine Coherence (↑ ) Separation Ratio (↑ ) DIVER-1 (SSL) 0.0004 0.95 BA-CLIP 0.0789 1.09 BT-CLIP 0.0954 1.04 BAT-CLIP (Ours) 0.1572 1.23 Table 3: Quantitative clustering evaluation of word-level embeddings. We evaluate the structure of the learned embedding space for a representative subject (Sub03; Fig. 2) using sentence-level clustering metrics, where higher values indicate tighter grouping of word embeddings within the same sentence. The proposed trimodal alignment (BAT-CLIP) yields the strongest clustering compared to the unaligned foundation model (DIVER-1) and unimodal alternatives. 4 Discussion and Conclusion In this work, we introduce BAT-CLIP, a trimodal brain–audio–text alignment framework for learning semantically grounded representations from naturalistic iEEG. By aligning iEEG embeddings to a shared audio–text manifold with contrastive objectives against both audio and text anchors, our results suggest two takeaways: (1) trimodal grounding is effective, yielding stronger performance on linguistic/semantic probes than single-anchor alignment; and (2) SSL-pretrained neural features are key for alignment, as initializing from SSL (DIVER-1) and then applying CLIP-style grounding consistently outperforms training from scratch and enables the most reliable gains from trimodal supervision. Taken together, these findings motivate a new paradigm for brain–speech alignment: pretrain neural representations with SSL, then ground them with trimodal contrastive alignment into a shared audio–text manifold. Concretely, SSL supplies a strong neural prior learned from massive unlabeled recordings (e.g., DIVER-1 pretrained on ∼ 5.3k hours), while the trimodal objective uses both audio and text anchors to inject complementary acoustic-to-semantic structure from only limited paired data (here, ∼ 30 minutes). Although we focus on probe tasks to isolate the strengths of the learned representations, this does not directly test end-to-end decoding; a natural next step is to use BAT-CLIP as a brain–speech alignment module and attach a downstream LLM decoder to evaluate settings such as open-vocabulary decoding or sequence-level prediction beyond linear probes. In addition, iEEG remains low-resource for naturalistic speech because large datasets are scarce and rarely open, unlike MEG/EEG where larger public corpora are available. How BAT-CLIP scales with substantially larger iEEG collections is therefore unclear and may depend on community efforts to curate shared benchmarks, while experiments on large-scale MEG speech datasets would provide a complementary test of cross-modality generalization and scaling. 5 Acknowledgments Work supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2021R1C1C1006503, RS-2023-00266787, RS-2023-00265406, RS-2024-00421268, RS-2024-00342301, RS-2024-00435727, RS-2025-25457239, RS-2026-25490183, RS-2026-25524667, RS-2026-25518389, RS-2021-NR061370, NRF-2021M3E5D2A01022515, and NRF-2021S1A3A2A02090597), by the Researchers Program through Seoul National University (No. 200-20250071, 200-20250049, 200-20240057, 0670-20260027, 200-20260009, 200-20250116, 200-20260081, 200-20250115, 200-20250113, 0670-20250039, 200-20260009). Additional support provided by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [No. RS-2021-I211343, Artificial Intelligence Graduate School Program, Seoul National University] and by the Global Research Support Program in the Digital Field (RS-2024-00421268). Also supported by the Artificial Intelligence Industrial Convergence Cluster Development Project funded by the Ministry of Science and ICT and Gwangju Metropolitan City, by the Korea Brain Research Institute (KBRI) basic research program (25-BR-05-01), by the Korea Health Industry Development Institute (KHIDI) and the Ministry of Health and Welfare, Republic of Korea (HR22C1605), and by the Korea Basic Science Institute (National Research Facilities and Equipment Center) grant funded by the Ministry of Education (RS-2024-00435727). We acknowledge the National Supercomputing Center for providing supercomputing resources and technical support (KSC-2023-CRE-0568, KSC-2024-CRE-0198, KSC-2025-CRE-0340). References [1] R. J. Antonello, A. Vaidya, and A. G. Huth (2023) Scaling laws for language encoding models in fMRI. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), p. 21895–21907. Cited by: §1. [2] G. Chau, C. Wang, S. Talukder, V. Subramaniam, S. Soedarmadji, Y. Yue, B. Katz, and A. Barbu (2025) Population transformer: learning population-level representations of neural activity. In Proceedings of the International Conference on Learning Representations (ICLR), Cited by: §3.1. [3] A. Défossez, C. Caucheteux, J. Rapin, O. Kabeli, and J. King (2023) Decoding speech perception from non-invasive brain recordings. Nature Machine Intelligence 5 (10), p. 1060–1070. Cited by: §1. [4] B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang (2023) CLAP learning audio concepts from natural language supervision. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Cited by: §3.3. [5] A. Goldstein, H. Wang, L. Niekerken, M. Schain, Z. Zada, B. Aubrey, T. Sheffer, S. A. Nastase, H. Gazula, A. Singh, A. Rao, G. Choe, C. Kim, W. Doyle, D. Friedman, S. Devore, P. Dugan, A. Hassidim, M. Brenner, Y. Matias, O. Devinsky, A. Flinker, and U. Hasson (2025) A unified acoustic-to-speech-to-language embedding space captures the neural basis of natural language processing in everyday conversations. Nature Human Behaviour 9 (5), p. 1041–1055. External Links: Document Cited by: §1, §3.1. [6] A. Guzhov, F. Raue, J. Hees, and A. Dengel (2022) AudioCLIP: extending CLIP to image, text and audio. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Cited by: §1. [7] D. D. Han, Y. Gwon, A. L. Lee, T. Lee, S. J. Lee, J. Choi, S. Lee, J. Bang, S. Lee, D. K. Park, S. Yoo, C. K. Chung, and J. Cha (2025) DIVER-1: scaling intracranial eeg foundation models for transferable representations. arXiv preprint arXiv:2512.19097. Cited by: §2, §3.1. [8] S. Jain, V. A. Vo, L. Wehbe, and A. G. Huth (2024) Computational language modeling and the promise of in silico experimentation. Neurobiology of Language 5 (1), p. 80–106. Cited by: §1. [9] J. Li, D. Li, C. Xiong, and S. Hoi (2022) BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning (ICML), p. 12888–12900. Cited by: §3.3. [10] R. Liu, Z. Chen, S. Peng, W. You, Z. Huang, J. Wu, and K. C. Tan (2026) MindMix: a multimodal foundation model for auditory perception decoding via deep neural-acoustic alignment. In Proceedings of the International Conference on Learning Representations (ICLR), Cited by: §1, §3.3. [11] C. McGettigan, A. Faulkner, I. Altarelli, J. Obleser, H. Baverstock, and S. K. Scott (2012) Speech comprehension aided by multiple modalities: behavioural and neural interactions. Neuropsychologia 50 (5), p. 762–776. Cited by: §1. [12] A. Mishra, S. Shukla, J. Torres, J. Gwizdka, and S. Roychowdhury (2025) Thought2Text: text generation from EEG signal using large language models (LLMs). In Findings of the Association for Computational Linguistics: NAACL 2025, p. 3560–3576. Cited by: §1. [13] Y. Oganian and E. F. Chang (2019) A speech envelope landmark for syllable encoding in human superior temporal gyrus. Science Advances 5 (11), p. eaay6279. Cited by: §3.2. [14] S. R. Oota, E. Çelik, F. Deniz, and M. Toneva (2024) Speech language models lack important brain-relevant semantics. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), p. 8503–8528. Cited by: §1. [15] C. Wang, V. Subramaniam, A. U. Yaari, G. Kreiman, B. Katz, I. Cases, and A. Barbu (2023) BrainBERT: self-supervised representation learning for intracranial recordings. In Proceedings of the International Conference on Learning Representations (ICLR), Cited by: §3.1. [16] Y. Wang, H. Liu, Y. Wang, C. Xuan, Y. Hou, S. Feng, H. Liu, Y. Liao, and Y. Wang (2025) Progress, challenges and future of linguistic neural decoding with deep learning. Communications Biology 8, p. 1350. Cited by: §1, §1. [17] Y. Yang, Y. Duan, Q. Zhang, H. Jo, J. Zhou, W. H. Lee, R. Xu, and H. Xiong (2024) NeuSpeech: decode neural signal as speech. arXiv preprint arXiv:2403.01748. Cited by: §1. [18] Z. Zada, S. A. Nastase, B. Aubrey, I. Jalon, S. Michelmann, H. Wang, L. Hasenfratz, W. Doyle, D. Friedman, P. Dugan, L. Melloni, S. Devore, A. Flinker, O. Devinsky, A. Goldstein, and U. Hasson (2025) The “podcast” ECoG dataset for modeling neural activity during natural language comprehension. Scientific Data 12 (1), p. 1135. Cited by: §3.1.

Instant research discovery