Paper deep dive
MultiModalPFN: Extending Prior-Data Fitted Networks for Multimodal Tabular Learning
Wall Kim, Chaeyoung Song, Hanul Kim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 3:35:42 PM
Summary
The paper introduces MultiModalPFN (MMPFN), an extension of the TabPFN foundation model designed to handle multimodal tabular data by integrating non-tabular modalities like images and text. MMPFN utilizes per-modality encoders and a novel modality projector consisting of a Multi-head Gated MLP (MGM) and a Cross-Attention Pooler (CAP) to align non-tabular embeddings with the tabular space, addressing issues of overcompression and attention imbalance. Extensive experiments on medical and general-purpose datasets demonstrate that MMPFN outperforms state-of-the-art methods.
Entities (12)
Relation Signals (13)
MultiModalPFN â extends â TabPFN
confidence 95% ¡ MMPFN extends TabPFN to handle tabular and non-tabular modalities in a unified manner.
MultiModalPFN â comprises â Multi-head Gated MLP
confidence 92% ¡ MMPFN comprises per-modality encoders, modality projectors... introduce a multi-head gated MLP
MultiModalPFN â comprises â Cross-Attention Pooler
confidence 92% ¡ MMPFN comprises... cross-attention pooler
MultiModalPFN â evaluatedon â CBIS-DDSM
confidence 90% ¡ We evaluate MMPFN using... CBIS-DDSM
MultiModalPFN â evaluatedon â PetFinder
confidence 90% ¡ We evaluate MMPFN using... PetFinder
MultiModalPFN â evaluatedon â Airbnb
confidence 90% ¡ We evaluate MMPFN using... Airbnb
MultiModalPFN â evaluatedon â Salary
confidence 90% ¡ We evaluate MMPFN using... Salary
MultiModalPFN â evaluatedon â Cloth
confidence 90% ¡ We evaluate MMPFN using... Cloth
MultiModalPFN â evaluatedon â PAD-UFES-20
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recently, TabPFN has gained attention as a foundation model for tabular data. However, it struggles to integrate heterogeneous modalities such as images and text, which are common in domains like healthcare and marketing, thereby limiting its applicability. To address this, we present the Multi-Modal Prior-data Fitted Network (MMPFN), which extends TabPFN to handle tabular and non-tabular modalities in a unified manner. MMPFN comprises per-modality encoders, modality projectors, and pre-trained foundation models. The modality projectors serve as the critical bridge, transforming non-tabular embeddings into tabular-compatible tokens for unified processing. To this end, we introduce a multi-head gated MLP and a cross-attention pooler that extract richer context from non-tabular inputs while mitigates attention imbalance issue in multimodal learning. Extensive experiments on medical and general-purpose multimodal datasets demonstrate that MMPFN consistently outperforms competitive state-of-the-art methods and effectively exploits non-tabular modalities alongside tabular features. These results highlight the promise of extending prior-data fitted networks to the multimodal setting, offering a scalable and effective framework for heterogeneous data learning. The source code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2602.20223v3
- Canonical: https://arxiv.org/abs/2602.20223v3
Trouble viewing inline? Open PDF directly â
Full Text
62,454 characters extracted from source content.
Expand or collapse full text
MultiModalPFN: Extending Prior-Data Fitted Networks for Multimodal Tabular Learning Wall Kim Samsung Electronics Hwaseong, South Korea wall.kim@samsung.com Chaeyoung SongHanul Kim Seoul National University of Science and Technology Seoul, South Korea cysong, hukim@seoultech.ac.kr Abstract Recently, TabPFN has gained attention as a foundation model for tabular data.However, it struggles to inte- grate heterogeneous modalities such as images and text, which are common in domains like healthcare and mar- keting, thereby limiting its applicability. To address this, we present the Multi-Modal Prior-data Fitted Network (MMPFN), which extends TabPFN to handle tabular and non-tabular modalities in a unified manner. MMPFN com- prises per-modality encoders, modality projectors, and pre- trained foundation models. The modality projectors serve as the critical bridge, transforming non-tabular embed- dings into tabular-compatible tokens for unified process- ing. To this end, we introduce a multi-head gated MLP and a cross-attention pooler that extract richer context from non-tabular inputs while mitigates attention imbalance is- sue in multimodal learning. Extensive experiments on med- ical and general-purpose multimodal datasets demonstrate that MMPFN consistently outperforms competitive state-of- the-art methods and effectively exploits non-tabular modal- ities alongside tabular features. These results highlight the promise of extending prior-data fitted networks to the multimodal setting, offering a scalable and effective framework for heterogeneous data learning. The source code is available at https://github.com/too- z/MultiModalPFN . 1. Introduction Tabular data is one of the most widely used data formats across domains such as healthcare, finance, and marketing. Traditionally, gradient-boosted decision trees [5, 34, 47] have dominated this field, owing to their fast training and strong predictive performance. However, recent progress in modern tabular deep learning models [2, 52] has shown that deep architectures can learn more expressive tabular repre- sentations and often surpass traditional tree-based methods. These advances have also broadened the scope of tabular data analysis to multimodal settings, where structured fea- tures are combined with unstructured modalities such as im- ages and text [20]; for example, diagnostic tasks may jointly leverage structured test results and medical images [27, 50], while marketing applications may integrate numerical sales records with textual product reviews [9, 54]. Despite this growing interest, attempts to extend gradient-boosted de- cision trees to heterogeneous data types have yielded only modest gains, and deep learning models that jointly em- bed tabular data with images or text, while promising, often suffer from limited performance in data-scarce regimes and slow training [8, 62]. More recently, TabPFN [25, 26] has attracted consid- erable attention as a tabular foundation model that treats supervised learning on tables as amortized Bayesian infer- ence, achieving strong performance on small- and medium- sized datasets in a single forward pass. While TabPFN es- tablishes a powerful prior over purely tabular distributions, its pretraining is restricted to synthetic tabular data, and no principled extensions to unstructured modalities have been explored. Consequently, despite its strong performance on purely tabular tasks, TabPFN does not address the growing need to jointly model tabular features with image and text modalities in practical multimodal applications. In this work, we propose the Multi-Modal Prior-data Fit- ted Network (MMPFN), an extension of TabPFN that pro- cesses tabular and non-tabular modalities in a unified man- ner. MMPFN first extracts features with per-modality en- coders for tabular, image, and text inputs. A modality pro- jector then aligns the non-tabular embeddings with the tab- ular embedding space. The resulting multimodal embed- dings are fed into the pretrained TabPFN backbone, so that its tabular prior can be directly reused while incorporating information from images and text through light fine-tuning. In addition, MMPFN explicitly tackles two common fail- ure modes in multimodal learners: overcompressed non- tabular embeddings and attention imbalance from token- count disparities, by introducing a multi-head gated MLP arXiv:2602.20223v3 [cs.LG] 9 Apr 2026 (MGM) that expands non-tabular representations into mul- tiple tokens and a cross-attention pooler (CAP) that com- presses them into a compact, balanced set. MGM and CAP constitute our modality projector. We evaluate MMPFN on multiple benchmarks [30â33, 45, 49] that pair tabu- lar inputs with images or text inputs. Across nearly all datasets, MMPFN surpasses recent state-of-the-art meth- ods [3, 13, 20, 24, 42, 58]. Extensive experiments demon- strate that MGM and CAP effectively mitigate the identified failure modes, while MMPFN scales positively as modali- ties are added, preserves the strengths of TabPFNâs model- ing in low-data regimes. Our main contributions are sum- marized as follows: ⢠We propose MMPFN, the first framework to extend TabPFN, pretrained on synthetic tabular distributions, to heterogeneous inputs (tabular + image/text) through a unified pathway. ⢠We identify two failure modes: overcompressed non- tabular embeddings and token-countâinduced attention imbalance, and introduce MGM and CAP as components of the modality projector to address them. ⢠Through experiments on medical and general-purpose datasets, we show that MMPFN outperforms competi- tive baselines, scales positively as modalities are added, and maintains robust performance under data scarcity and limited compute. 2. Related works VisionâLanguage Multimodal Models. Early research in multimodal learning developed fusion and conditioning mechanisms for integrating text and images. FiLM [46] in- troduced feature-wise modulation for language-conditioned visual reasoning, while early transformer-based models such as ViLBERT, VisualBERT, VL-BERT, LXMERT, and UNITER [6, 36, 41, 53, 56] explored co-attention and unified architectures, achieving state-of-the-art results on visionâlanguage benchmarks. A major shift came with CLIP [48], which used large-scale contrastive pretraining for scalable zero-shot transfer. More recent approaches, such as BLIP-2 [35] and LLaVA [38], integrated large lan- guage models for generalizable multimodal reasoning. Tabular and Multimodal Models. The pretraining- driven paradigm has since expanded to structured data. In the tabular domain, approaches typically adopt either a row- as-text strategy, serializing entire rows for large language model (LLM) processing [23], or a per-column embed- ding strategy with modality-specific encoders. Methods such as Tab2Text [37] transform rows into textual narra- tives for improved alignment, while others [3] demonstrate that careful design of fusion layers substantially improves benchmarks. LANISTR [15] extended this direction with similarity-based multimodal masking, enabling joint learn- ing from language, images, and structured inputs even with missing modalities. Unstructuredâstructured integration has also been ex- plored in image-centric datasets. Representative works in- cluded MMCL [20], which aligned tabular and image em- beddings through contrastive learning; TIP [13], which im- proved robustness to missing features; STiL [14], which leveraged unlabeled data through semi-supervised pseudo- labeling; TIME [42], which used TabPFN [25] as a tabular encoder; and Turbo [29], which strengthened cross-modal reasoning. Beyond individual models, toolkits such as Au- toGluon [57] and modular pipelines [19] provided practi- cal infrastructure for multimodal integration. Despite this progress, most work remained focused on visionâlanguage tasks, and systematic treatment of structured data remained limited. Fusion strategies were often heuristic and less reli- able under low-data regimes or modality imbalance. These gaps motivated more general multimodal tabular systems. General-Purpose Pre-trained Models. Pretraining large foundation models has transformed representation learning across domains. In NLP, models progressed from masked language modeling to more efficient self-supervised strate- gies such as ELECTRA [7] and DeBERTa [21]. Later re- finements including DeBERTaV3 [22], ModernBERT [60], and multilingual encoders such as BGE/M3 [4] introduced architectural improvements (e.g., disentangled embeddings, FlashAttention-2, optimized tokenization) and broadened applications to retrieval and cross-lingual tasks. In computer vision, self-supervised pretraining has be- come a dominant paradigm.DINOv2 [44] and DI- NOv3 [51] showed scalable self-distillation for robust vi- sual features, EVA [16, 17] advanced masked image mod- eling with large Vision Transformers, and iBOT [63] com- bined masking and self-distillation for effective ViT repre- sentations. For structured data, TabPFN [25, 26] extended this idea by pretraining on large synthetic datasets to learn a general prior over tabular distributions. It achieved strong performance on small and medium-sized datasets in a sin- gle forward pass without fine-tuning, making it a foundation model for tabular learning. Yet pretraining for multimodal tabular data remained underexplored relative to NLP and vision. Bridging this gap is important for multimodal foun- dation models with structured inputs. 3. Proposed Method Figure 1 (a) illustrates the overall architecture of our mul- timodal PFN (MMPFN) that extends TabPFN [25, 26] to the multimodal setting, where image or text modalities ac- company tabular inputs. Therefore, we begin by briefly reviewing TabPFN. We then describe the proposed mul- timodal PFN architecture, including the per-modality en- coders and the modality projector that aligns non-tabular Tabular Encoder Image Encoder Text Encoder Multi-head Gated MLP Cross- Attention Pooler Learnable embeddings TabPFN Backbone instance í train instance í ... Per-modality Encoders Modality Projector Healthy playful young kitten TabPFN Decoder train instance 1 train instance 0 test instance í ... test instance 1 test instance 0 pred 0 pred 1 pred í MGM tokens MLP head 1 MLP head íâ1 MLP head í+1 MLP head í Encoder [CLS] ... ... LayerNorm Linear GLU Dropout Linear c ítokens í - dim MGM tokens Learnable embeddings Q K, V Multi-Layer Perceptron MGM tokens Linear LayerNorm LayerNorm Multi - Head Cross Attention + LinearLinear GELU Dropout ížtokens í - dim Non-tabular tokens LayerNorm (a) Overall architecture (b) Multi-head Gated MLP(c) Cross-Attention Pooler í tokens ížtokens Trainable Frozen cConcatenation + Add c Figure 1. An overview of MMPFN. MMPFN extends TabPFN by incorporating per-modality encoders and a modality projector to extract features from non-tabular data. Newly developed components are highlighted in color, while existing ones appear in gray. Layers marked as âfrozenâ remain fixed during fine-tuning, whereas all others are trainable. Encoded target labels are part of the training inputs but are omitted from the diagram for clarity. embeddings with the tabular feature space, followed by the training protocol for fine-tuning MMPFN on downstream multimodal tasks. Finally, we analyze the phenomenon of attention imbalance that arises from token-count disparities across modalities. 3.1. Preliminary: TabPFN TabPFN is a tabular foundation model that treats tabular learning as amortized Bayesian inference. More specifi- cally, a transformer is pretrained on a large collection of synthetic tabular datasets sampled from structural causal model priors. During pretraining, it learns to map a small labeled training set and an accompanying query set directly to posterior-predictive label distributions in a single forward pass. Once pretrained, TabPFN can be applied to new tabu- lar tasks without task-specific optimization, simply by feed- ing the new trainingâtest pairs to the network. Architecturally, TabPFN stacks 2D TabPFN blocks. Each block splits attention into two stages: feature atten- tion, where each feature attends to other features within the same sample, and sample attention, where the same feature attends across all samples. This design yields permutation invariance over both samples and features and scales effi- ciently to larger tables than those encountered during pre- training. For in-context inference, TabPFN processes the concatenated training and test rows with masks that allow self-attention within labeled training rows and restrict test rows to cross-attend only to training rows. An MLP head then maps the test embeddings to the predictions. 3.2. Multimodal PFN: Architecture As shown in Figure 1 (a), MMPFN consists of per-modality encoders, a modality projector, and a TabPFN backbone. The per-modality encoders map each input modality to a feature representation, while the modality projector aligns image and text embeddings with the shared tabular embed- ding space. The TabPFN backbone then jointly processes the resulting multimodal embeddings, and a lightweight de- coder head produces predictions for the test samples. Per-Modality Encoders. The per-modality encoders comprise tabular, image, and text branches. The tabular branch is identical to the TabPFN v2 encoder [26] and re- mains frozen during fine-tuning. For images, we employ the DINOv2 ViT-B/14 backbone [12, 44]: input images are re- sized so that both height and width are divisible by 14, and the final [CLS] token is used as a global image represen- tation. For text, we adopt an ELECTRA-based encoder [7], chosen based on preliminary experiments in which it con- sistently outperformed DeBERTa variants [21]. Text inputs are tokenized and truncated to a maximum length of 512 to- kens, and the corresponding [CLS] embedding is used as the text representation. Modality Projector. The modality projector transforms image and text embeddings into tabular-like representa- tions, which share d-dimensional space compatible with the TabPFN backbone.It comprises two sublayers: a multi-head gated MLP (MGM) and a cross-attention pooler (CAP). MGM addresses the limitation of a single [CLS] embedding, which can overly compress image/text infor- mation, by expanding it into N parallel d-dimensional pro- jections. Figure 1 (b) illustrates the detailed structure of MGM. Specifically, the [CLS] embedding is fed into N MLP heads that project the encoder output dimension to d and produce candidate modality-specific tokens. A Gated Linear Unit (GLU) [10] modulates the contribution of each head, encouraging head-wise specialization and preserving diverse aspects of the original non-tabular representation in the resulting token set. CAP then balances tabular and non-tabular cues before fusion in the TabPFN backbone. As shown in Figure 1 (c), it takes the N MGM tokens as keys and values and introduces K learnable query vectors that cross-attend to them, yield- ing K representative d-dimensional embeddings per modal- ity. The pooled tokens are refined by an MLP. These K tokens form a compact, calibrated summary of image/text information and are concatenated with the tabular tokens along the feature dimension to construct the multimodal input table for TabPFN. Without pooling, too many non- tabular tokens can induce attention imbalance, where the modality with more tokens dominates the attention bud- get and suppresses tabular signal. CAP mitigates this by producing a compact, calibrated set of embeddings for the TabPFN backbone. We analyze this in Section 3.4. 3.3. Multimodal PFN: Training Since TabPFN is pre-trained on large corpora of synthetic tabular data, its representations can be misaligned with im- age/text embeddings. We therefore freeze all modality en- coders and train the modality projector, the TabPFN back- bone, and the decoder. Note that all components are pre- trained, except for the modality projector.To leverage TabPFNâs in-context inference, we follow its standard pro- tocol: split the multimodal data into training and test sets, concatenate their embeddings into a single table, and feed it to the backbone. The model then produces predictions for the test samples to obtain supervisory signals for training. 3.4. Attention Imbalance in MMPFN We study how the number of non-tabular tokens affects at- tention mechanism. Consider a query token q attending to two sets of keys: non-tabular tokens k (I) 1 ,¡ , k (I) N I and tab- ular tokens k (T) 1 ,¡ , k (T) N T , where N I and N T are their re- spective counts. The scaled dot-product attention scores are given by s (I) i = q ⤠k (I) i / â d,s (T) j = q ⤠k (T) j / â d.(1) Let w (I) i = e s (I) i and w (T) j = e s (T) j be the unnormal- ized attention weights, and define the per-token expecta- tions c I =E[w (I) i ] and c T =E[w (T) j ], where the expec- tation is over token indices and any randomness in (q, k). Also, let a I denote the total attention weight allocated to the non-tabular set, defined by a I = N I X i=1 w (I) i P N I u=1 w (I) u + P N T v=1 w (T) v .(2) Then its expectation is approximated by E[a I ]â N I c I N I c I + N T c T (3) Hence, when per-token quality is comparable (c I â c T ), token-count imbalance (N I > N T ) induces attention im- balance, potentially degrading performance. Consequently, MMPFNâs performance might vary with the modality token ratio. This suggests the importance of CAP. In Section 4, we validate this observation by varying K in the CAP. 4. Experiments 4.1. Experimental Setup Dataset. We evaluate MMPFN using well-established multimodal datasets that have been extensively validated in previous studies [3, 28, 43, 57, 58]: ⢠PAD-UFES-20(PU20) [45] contains 2,298 samples from six skin lesion types, each paired with a clinical image and up to 26 metadata features. ⢠CBIS-DDSM [49] is a curated subset of DDSM with digi- tized mammograms, annotated regions of interest for cal- cifications and masses, biopsy-verified benign/malignant labels, and lesion-level metadata. ⢠Airbnb [30] provides a detailed snapshot of Melbourneâs homestay activity as of December 2018; following the preprocessing strategy of T, we discretize the target into ten quantile-based groups of equal size. Table 1. Statistics of the multimodal datasets used in our experiments. â# imagesâ and â# textâ denote the number of image and text fields per sample, respectively. PetFinder variants indicate which modalities (tabular (T), image (I), text (t)) are used. Dataset# train samples# test samples# features# numeric features# categorical features# images# text# classes PAD-UFES-20 [45]183846021318106 CBIS-DDSM(Mass) [49]1318378835302 CBIS-DDSM(Calc) [49]1545326835302 Airbnb [30]1831645795027230110 Salary [33] 158413961413036 Cloth [31]187884698523035 PetFinder-I (T+I) [32]11721293119514105 PetFinder-t (T+t) [32] 11721293119514015 PetFinder-A (T+I+t) [32]11721293119514115 ⢠Salary [33] contains 15,841 training and 3,961 test job postings from India, each with company, years of expe- rience, job description, designation, job type, key skills, and location, with the task of predicting the salary range. ⢠Cloth [31] comprises 23,486 customer reviews with text fields and 10 tabular features for multimodal sentiment and recommendation prediction. ⢠PetFinder [32], released for the Kaggle PetFinder.my challenge, contains over 14,000 pet profiles with images, descriptive text, and structured attributes for predicting a five-class adoption-speed outcome. For each dataset, we randomly split the data into training and test sets, except for CBIS-DDSM, for which we use the predefined train-test split. Table 1 summarizes the statistics of these datasets. More details about datasets are provided in the supplementary material. Implementation Details. We fine-tune MMPFN for 100 iterations using cross-entropy loss. We use AdamW [40] with a learning rate of 1Ă 10 â5 and batch size 1. For all experiments, we use random seeds0, 1, 2, 3, 4 and report average accuracy. More details are provided in the supple- mentary material. 4.2. Main Results Results on TabularâImage Modality Datasets. Table 2 summarizes the classification accuracy of MMPFN and state-of-the-art baselines on four tabularâimage datasets. Compared with fine-tuned TabPFN [26], which uses only tabular inputs, MMPFN consistently improves performance by leveraging image features. MMCL [20], TIP [13], and HEALNet [24] show inconsistent results, likely due to the small dataset size and low-dimensional tabular features. In contrast, MMPFN achieves the best results on all datasets except Mass, where it remains competitive with the top models. Compared with TIME [42], which uses TabPFN 0 We cite all results of Luo et al. [42] directly. Although TIME used the CBIS-DDSM dataset without specifying subtype, the reported sample size matches the calcification subset, so we list it under CBIS-DDSM calcifica- tion in Table 2. Since the code is unavailable, reproduction was infeasible. Table 2. Comparison with state-of-the-art on tabular-image multimodal datasets. Results are reported as accuracy (rank), averaged over five random seeds, where lower rank indicates better performance. âAvg.â denotes the mean accuracy across datasets. Best and second-best results are marked in bold and underline . MethodPU20MassCalcPetfinderAvg. TabPFN [25]82.17 (2)71.27 (5)73.31 (2)36.33 (8)4.25 Catboost [47]80.43 (4)78.31 (1)72.09 (4)38.69 (4)3.25 AutoGluon [57]81.09 (3)76.28 (2)71.04 (6)38.81 (3)3.50 MMCL [20]76.61 (7)57.62 (7)60.12 (8)36.61 (7)7.25 TIP [13]78.75 (6)73.12 (4)67.96 (7)37.28 (5)5.50 HEALNet [24]74.65 (8)68.10 (6)71.83 (5)37.03 (6)6.25 TIME [42] 80.35 (5)-72.70 (3) 1 39.25 (2)3.33 MMPFN85.22 (1)74.53 (3)75.40 (1)40.74 (1)1.50 as its tabular encoder, MMPFN delivers substantial gains, suggesting that our modality projection strategies are more effective than simple fusion. CatBoost [47] uses image em- beddings as raw input features and achieves strong perfor- mance. AutoGluon [57], an AutoML framework for multi- modal data, also performs competitively on several bench- marks. However, MMPFN achieves a better average rank than both models. Results on TabularâText Modality Datasets. Table 3 re- ports results on tabularâtext datasets. As in the image set- ting, adding text features consistently improves over the fine-tuned TabPFN baseline. MMPFN is particularly strong on Airbnb, which includes 50+ tabular features and a sin- gle text field, allowing tabular-specialized models to cap- ture most of the predictive signal. Accordingly, MMPFN substantially outperforms language modelâbased methods, such as TFN [61] and MulT [59], which struggle to ex- ploit abundant tabular features. By contrast, Cloth has few informative tabular features, while the review text carries most of the signal. This is reflected in the weak perfor- mance of tabular-only models and the strong results of All- TextBERT [3], indicating that text-specialized models excel in such cases. Even so, among methods that explicitly pre- serve tabular structure, MMPFN achieves the best overall Table 3. Comparison with state-of-the-art on tabular-text mul- timodal datasets. Results are reported as accuracy (rank), aver- aged over five random seeds, where lower rank indicates better performance. âAvg.â denotes the mean accuracy across datasets. Best and second-best results are marked in bold and underline . MethodAirbnbSalaryClothPetfinderAvg. TabPFN [25]46.96 (2)44.96 (6)55.07 (9)36.33 (7)6.00 Catboost [47]43.56 (4)40.36 (9)59.24 (8)35.47 (8)7.25 AutoGluon [57]44.60 (3)45.24 (5)72.07 (1)37.96 (4)3.25 AllTextBERT [3]30.9 (9)44.0 (7)68.0 (3)34.6 (9)7.00 TFN [61] 35.7 (8)45.8 (3)60.1 (7)36.8 (6)6.00 MulT [59]36.3 (7)45.4 (4)63.6 (6)37.6 (5)5.50 T [3]38.3 (6)47.2 (1)65.5 (5)38.9 (3)3.75 TabSTAR [1]40.06 (5)43.75 (8)71.75 (2)41.53 (1)4.00 MMPFN47.78 (1)46.17 (2)66.26 (4)39.04 (2)2.25 performance, trailing the text-specialized baseline by only a small margin. This contrasts with prior multimodal tabular studies, which focused on tabular-dominant datasets [20]. Overall, MMPFN effectively handles both tabular and un- structured modalities. 4.3. Analysis MMPFN as an Image and Text Classifier. Figure 2 (a) evaluates MMPFN with non-tabularâonly inputs.As a baseline, we use pre-trained DINOv2 [44] or Electra [7] [CLS] embeddings with an attached MLP classifier, a widely used and often near-optimal setup [55]. MMPFN uses only MGM to generate non-tabular embeddings from the CLS token. With these embeddings as the sole in- puts, MMPFN remains within âź1% of DINOv2 (69.30% vs. 69.89%). Accuracy increases with the number of image tokens, suggesting that additional tokens capture comple- mentary, higher-resolution information. Although trained on synthetic tabular data, MMPFN effectively classifies im- age embeddings mapped to tabular-like features, showing that it is not limited to native tabular inputs. CAP further provides a modest gain. Attention Imbalance. We empirically analyze attention imbalance in multimodal processing.In Figure 2 (a), MMPFN uses only non-tabular inputs. Accuracy increases with the number of MGM heads, showing that additional non-tabular tokens improve performance. CAP is not used here.These results show that a PFN trained on syn- thetic tabular data can classify projected non-tabular fea- tures competitively with standard baselines. TabPFN is therefore not limited to native tabular inputs. 1 AllTextBertconvertsalltabularfeaturesintostrings, concatenatesthem,andinputstheresultingsequenceinto DistilBERT-base-uncased for modeling, as described in [3]. í 32641282561664128256 MGM+CAP MGM MGM Baseline 71.0 70.5 68.0 68.5 70.0 69.5 69.0 85.2 84.0 84.8 84.4 83.6 328 Figure 2. Performance on PU20 versus the number of non- tabular tokens. (a) Image-only results with DINOv2 and an MLP baseline. (b) Multimodal results under token imbalance. The y- axis shows accuracy and the x-axis shows the number of MGM heads. In (b), MGM+CAP uses 24 CAP heads. 24 8 163264 0.8 0.6 0.4 0.2 Tabular Image Tabular Text 24816 3264 128256 Figure 3. Token count and attention mass. Attention mass for tabular and non-tabular tokens is measured as the number of non- tabular tokens varies, without CAP. Values are averaged over 12 self-attention layers in TabPFN. PU20 and Salary use 11 and 4 tabular tokens. The x-axis is in log scale. In Figure 2 (b), the trend changes when tabular and non- tabular features are used together. MMPFN performs best when the two modalities have similar token counts, and per- formance drops as one modality dominates the sequence. This pattern does not appear in Figure 2 (a), where only non-tabular inputs are used. The contrast is consistent with attention imbalance: the modality with more tokens absorbs more of the attention budget, while the other receives less attention and contributes less signal. MGM+CAP mitigates this problem by extracting non-tabular features with enough MGM heads and then compressing them into 24 CAP to- kens. As the number of MGM heads increases, MGM+CAP improves steadily and outperforms MGM alone. We further examine attention allocation by varying the number of non-tabular tokens while keeping the number of tabular tokens fixed. In Figure 3, attention mass shifts monotonically toward the non-tabular modality as its token count increases, while the mass on tabular tokens decreases. This result supports the same explanation: a modality with more tokens receives a larger share of the attention bud- get. Table 4 then separates the effects of token count and representation quality. Increasing token count alone re- Table 4. Token count and representation quality on PU20 and Cloth. We compare non-tabular token configurations on PU20 (left) and Cloth (right). Columns 32 and 128 denoteN=32 and N=128 MGM heads, and 4Ă32 denotes four repetitions of the N=32 setting. For MGM+CAP,K is fixed to 24 and 4, respec- tively. Method324Ă32128 MGM84.1782.4383.91 MGM+CAP84.1083.5784.59 Method324Ă32128 MGM63.1261.7661.86 MGM+CAP64.2064.2265.03 duces MGM by nearly 2%p despite identical representa- tions, whereas MGM+CAP remains stable. When represen- tation quality improves at the same total token count (4Ă32 and 128), MGM still suffers from the larger token set, but MGM+CAP benefits from the stronger representations by compressing them into a compact set of tokens. More de- tails are provided in the supplementary material. Modality Projector. Table 5 ablates the modality pro- jector, comparing single-head (Linear, MLP) and multi- head (MLP, MoE, MGM) variants.Parameter budgets are provided in the supplementary material. Single-head baselines apply one linear or MLP projection to the non- tabular [CLS] embedding. Although they improve over the tabular-only backbone, they yield the lowest average accu- racies, suggesting that a single projection overcompresses non-tabular information. Expanding [CLS] into multi- ple tokens consistently improves performance, showing the benefit of capturing diverse aspects of image/text features. Within this family, MoE is less effective and less stable across datasets, suggesting that sparse expert routing is hard to exploit in the low-data regime. By contrast, MGM com- bines multi-head projection with GLU-based gating and achieves the best accuracy on every dataset and the best overall average. We next examine how the modality projector incorpo- rates non-tabular information. Specifically, we compare CAP, which pools non-tabular features into representative tokens, with Feature-wise Linear Modulation (FiLM) [46], which uses non-tabular representations to generate feature- wise affine parameters for tabular tokens. As shown in Ta- ble 6, CAP consistently outperforms FiLM on all datasets. This result suggests that controlling token count is impor- tant for tabularânon-tabular fusion because it alleviates at- tention imbalance between the two modalities. FiLM ap- plies the same channel-wise transformation to all tabular to- kens, limiting token-specific use of non-tabular cues. CAP instead compresses non-tabular features into representative tokens that the TabPFN encoder attends jointly with tabular tokens, preserving relevant cross-modal cues while reduc- ing token imbalance. Figure 4. Cosine similarity between multimodal feature em- beddings. Axes denote all tabular and text/image features. From left to right and top to bottom, it shows the correlations between features in the experiments on the PU20, Calc, Cloth, Mass, Petfinder, and Airbnb datasets. Cross-Modal Correlation. Figure 4 visualizes cosine similarities among TabPFNâbackbone embeddings on all tabularâimage and tabularâtext datasets. These similari- ties illustrates the predictive relationships between features learned by MMPFN. As expected, within-modality blocks exhibit high similarity. However, several tabularâimage/text pairs are also strongly aligned, indicating that MMPFN models cross-modal interactions rather than only within- modality structure. Details of the cosine similarity com- putation are provided in supplementary material. Robustness in Low-Data Regimes. Tabular datasets of- ten require expert annotation, leading to limited sample sizes and sparse labels [13, 14]. Models that remain ro- bust under such data scarcity are therefore desirable. In this setting, MMPFN performs strongly. Table 7 compares MMPFN and TIP when trained on only 10% of randomly selected samples from each dataset. Although TIP uses self- supervised pretraining on all unlabeled data, we focus on supervised finetuning with limited labeled data. Although MMPFN shows a larger relative drop, it con- sistently outperforms TIP on all datasets, even with only 10% of the data. On CBIS-DDSM Mass, performance even improves under subsampling. This result suggests that the PFN, pretrained on synthetic priors, can better capture dis- criminative characteristics when finetuned on fewer labeled examples. More details on low-data behavior are provided in the supplementary material. Table 5. Ablation of the design of MGM. We compare single-head and multi-head feature extraction mechanisms (Linear/MLP, MoE, and the proposed MGM) across all datasets. MoE denotes Mixture-of-Experts. Best results are highlighted in bold. CategoryMethodPU20MassCalcClothSalaryAirbnbPetFinder-IPetFinder-TPetFinder-AAvg. Single-head Linear83.4866.1974.1758.0644.6747.0237.2236.6437.3053.86 MLP83.7864.8773.8760.4444.3346.8537.2236.6437.3054.14 Multi-head MLP84.3967.9973.9964.3945.9346.7940.6438.1240.0455.81 MoE83.2266.6773.1355.0744.2546.1236.8236.8336.9453.23 MGM85.2274.5375.4066.2646.1747.7840.7039.0441.1957.37 Table 6. Ablation of the design of CAP. We compare two mechanisms for incorporating non-tabular information (FiLM and the proposed CAP) on all datasets. FiLM denotes Feature-wise Linear Modulation. Best results are highlighted in bold. MethodPU20MassCalcClothSalaryAirbnbPetFinder-IPetFinder-TPetFinder-AAvg. FiLM [46]80.0073.0273.2565.8242.2746.0639.8438.3140.7755.48 CAP85.2274.5375.4066.2646.1747.7840.7039.0441.1957.37 Table 7. Performance in Low-data Regime. Each method uses two rows: accuracy (top) and percentage change vs. full-data (bot- tom). The âAvg.â column averages percentage changes only. Best results are highlighted in bold. PU20MassCalcPetFinderAvg. TIP [13] 10%70.4468.3162.2734.8658.97 (-10.6)(-6.58)(-8.37)(-6.49)(-8.00) MMPFN 10%72.8776.1372.0935.7364.21 (-14.14)(+0.75)(-5.23)(-12.30)(-10.27) Scaling with Added Modalities. We assess MMPFN as multiple non-tabular modalities are added. On Petfinder, we compare against AutoGluon, a multimodal AutoML system supporting image and text modalities. As shown in Figure 5, MMPFNâs accuracy increases monotonically from tabular â tabular+text â tabular+image â tabu- lar+image+text (39%â 40%â 41%), indicating comple- mentary signal from both image and text. These results have particular significance for tabular modeling, where perfor- mance improvements from architectural changes alone are often saturated. Adding complementary modalities offers a practical route to further gains. Moreover, MMPFN out- performs AutoGluon under every combination. Unlike Au- toGluonâs large ensembles, MMPFN achieves higher accu- racy with a lightweight and specialized architecture. 5. Conclusion We introduced MMPFN, a multimodal extension of TabPFN that unifies tabular, image, and text inputs with per- modality encoders, a modality projector, and the TabPFN backbone.We developed MGM and CAP, which map non-tabular embeddings to the tabular space and mitigate token-countâinduced attention imbalance. By leveraging Figure 5. Accuracy of AutoGluon vs. MMPFN on PetFinder under different modality combinations: tabular, +text, +image, +image+text. pretrained foundation models and fine-tuning lightweight components, MMPFN achieved strong accuracy with sub- stantially lower training costs. Across medical and general- purpose benchmarks, it consistently outperformed competi- tive state-of-the-art methods, scaled positively as modalities were added, and maintained robust performance in low-data regimes. Acknowledgements This work was supported in part by the National Research Foundation of Korea (NRF) funded by the Korean gov- ernment under Grants RS-2023-00221365 and RS-2024- 00352566. References [1] Alan Arazi, Eilam Shapira, and Roi Reichart. Tabstar: A foundation tabular model with semantically target-aware rep- resentations. In NeurIPS, 2025. 6 [2] Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. In ICLR, 2022. 1 [3] Thomas Bonnier. Revisiting multimodal transformers for tabular data with text fields. In Findings of ACL, pages 1481â 1500, 2024. 2, 4, 5, 6 [4] Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu.M3-embedding:Multi- linguality, multi-functionality, multi-granularity text embed- dings through self-knowledge distillation. In Findings of ACL, pages 2318â2335, Bangkok, Thailand, 2024. ACL. 2 [5] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., pages 785â794, 2016. 1 [6] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning.In ECCV, pages 104â120, 2020. 2 [7] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christo- pher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020. 2, 4, 6, 3 [8] Can Cui, Haichun Yang, Yaohong Wang, Shilin Zhao, Zuhayr Asad, Lori A Coburn, Keith T Wilson, Bennett A Landman, and Yuankai Huo. Deep multimodal fusion of im- age and non-image data in disease diagnosis and prognosis: a review. Progress in Biomedical Engineering, 5(2):022001, 2023. 1 [9] Ronnie Das, Wasim Ahmed, Kshitij Sharma, Mariann Hardey, Yogesh K Dwivedi, Ziqi Zhang, Chrysostomos Apostolidis, and Raffaele Filieri. Towards the development of an explainable e-commerce fake review index: An at- tribute analytics approach. European Journal of Operational Research, 317(2):382â400, 2024. 1 [10] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional net- works. In ICML, pages 933â941, 2017. 4 [11] Aaron Defazio, Xingyu Yang, Ahmed Khaled, Konstantin Mishchenko, Harsh Mehta, and Ashok Cutkosky. The road less scheduled. In NeurIPS, pages 9974â10007, 2024. 4 [12] Alexey Dosovitskiy. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021. 4 [13] Siyi Du, Shaoming Zheng, Yinsong Wang, Wenjia Bai, De- clan P OâRegan, and Chen Qin. Tip: Tabular-image pre- training for multimodal classification with incomplete data. In ECCV, pages 478â496, 2024. 2, 5, 7, 8, 4 [14] Siyi Du, Xinzhe Luo, Declan P OâRegan, and Chen Qin. Stil: Semi-supervised tabular-image learning for comprehensive task-relevant information exploration in multimodal classifi- cation. In CVPR, pages 15549â15559, 2025. 2, 7, 4 [15] Sayna Ebrahimi, Sercan O Arik, Yihe Dong, and Tomas Pfis- ter. Lanistr: Multimodal learning from structured and un- structured data. arXiv preprint arXiv:2305.16556, 2023. 2 [16] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representa- tion learning at scale. In CVPR, pages 19358â19369, 2023. 2 [17] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xin- long Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 149:105171, 2024. 2 [18] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bern- hard Sch Ě olkopf, and Alexander Smola. A kernel two-sample test. The journal of machine learning research, 13(1):723â 773, 2012. 1 [19] Ken Gu and Akshay Budhkar. A package for learning on tabular and text data with transformers. In Proceedings of the Third Workshop on Multimodal Artificial Intelligence, pages 69â73, 2021. 2 [20] Paul Hager, Martin J Menten, and Daniel Rueckert. Best of both worlds: Multimodal contrastive learning with tabular and imaging data. In CVPR, pages 23924â23935, 2023. 1, 2, 5, 6, 4 [21] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In ICLR, 2021. 2, 4 [22] Pengcheng He, Jianfeng Gao, and Weizhu Chen.De- BERTav3: Improving deBERTa using ELECTRA-style pre- training with gradient-disentangled embedding sharing. In ICLR, 2023. 2 [23] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Mon- ica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few- shot classification of tabular data with large language mod- els. In AISTATS, pages 5549â5581, 2023. 2 [24] Konstantin Hemker, Nikola Simidjievski, and Mateja Jam- nik. Healnet: multimodal fusion for heterogeneous biomed- ical data. In NeurIPS, 2024. 2, 5 [25] Noah Hollmann, Samuel M Ě uller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. In ICLR, 2023. 1, 2, 5, 6 [26] Noah Hollmann, Samuel M Ě uller, Lennart Purucker, Ar- jun Krishnakumar, Max K Ě orfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637 (8045):319â326, 2025. 1, 2, 4, 5 [27] Shih-Cheng Huang, Anuj Pareek, Saeed Seyyedi, Imon Banerjee, and Matthew P Lungren. Fusion of medical imag- ing and electronic health records using deep learning: a sys- tematic review and implementation guidelines. NPJ digital medicine, 3(1):136, 2020. 1 [28] Jun-Peng Jiang, Han-Jia Ye, Leye Wang, Yang Yang, Yuan Jiang, and De-Chuan Zhan. Tabular insights, visual impacts: transferring expertise from tables to images. In ICML, 2024. 4 [29] Jun-Peng Jiang, Yu Xia, Hai-Long Sun, Shiyin Lu, Qing- Guo Chen, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, and Han-Jia Ye. Multimodal tabular reasoning with privileged structured information. In NeurIPS, 2025. 2 [30] Kaggle.Melbourne airbnb open data. kaggle.com/ datasets/tylerx/melbourne- airbnb- open- data, 2018. Accessed: September 24, 2025. 2, 4, 5 [31] Kaggle.Womenâs e-commerce clothing reviews. kaggle.com/datasets/nicapotato/womens- ecommerce-clothing-reviews, 2019.Accessed: September 24, 2025. 5 [32] Kaggle.Petfinder.my adoption prediction. https: //w.kaggle.com/competitions/petfinder- adoption-prediction, 2019. Accessed: September 24, 2025. 5 [33] Kaggle. Predict the data scientistâs salary in india. kaggle. com/datasets/ankitkalauni/predict- the- data-scientists-salary-in-india, 2021. Ac- cessed: September 24, 2025. 2, 5 [34] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Light- gbm: A highly efficient gradient boosting decision tree. In NeurIPS, 2017. 1 [35] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730â19742, 2023. 2 [36] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.Visualbert: A simple and perfor- mant baseline for vision and language.arXiv preprint arXiv:1908.03557, 2019. 2 [37] Tong Lin, Jason Yan, David Jurgens, and Sabina J Tomkins. Tab2Text - a framework for deep learning with tabular data. In Findings of EMNLP, pages 12925â12935, 2024. 2 [38] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, pages 34892â34916, 2023. 2 [39] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation net- works. In ICML, pages 2208â2217, 2017. 2 [40] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 5 [41] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS, 2019. 2 [42] Jiaqi Luo, Yuan Yuan, and Shixin Xu.Time: Tabpfn- integrated multimodal engine for robust tabular-image learn- ing. arXiv preprint arXiv:2506.00813, 2025. 2, 5 [43] Martin Mr Ě az, Breenda Das, Anshul Gupta, Lennart Purucker, and Frank Hutter. Towards benchmarking foundation models for tabular data with text. In ICML Workshop, 2025. 4 [44] Maxime Oquab, Timoth Ě e Darcet, Th Ě eo Moutakanni, Huy V Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without su- pervision. Trans. Mach. Learn. Res., 2024. 2, 4, 6, 3 [45] Andre GC Pacheco, Gustavo R Lima, Amanda S Salo- mao, Breno Krohling, Igor P Biral, Gabriel G De Angelo, F Ě abio CR Alves Jr, Jos Ě e GM Esgario, Alana C Simora, Pe- dro BC Castro, et al. Pad-ufes-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones. Data in Brief, 32:106221, 2020. 2, 4, 5 [46] Ethan Perez, Florian Strub, Harm De Vries, Vincent Du- moulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018. 2, 7, 8 [47] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: Un- biased boosting with categorical features. In NeurIPS, 2018. 1, 5, 6 [48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748â8763, 2021. 2 [49] R. Sawyer-Lee, F. Gimenez, A. Hoogi, and D. Rubin. Cu- rated breast imaging subset of digital database for screening mammography (cbis-ddsm). The Cancer Imaging Archive, 2016. Data set. 2, 4, 5 [50] J Ě org Schilcher, Alva Nilsson, Oliver Andlid, and Anders Eklund.Fusion of electronic health records and radio- graphic images for a multimodal deep learning prediction model of atypical femur fractures. Computers in Biology and Medicine, 168:107704, 2024. 1 [51] Oriane Sim Ě eoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Micha Ě el Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025. 2 [52] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342, 2021. 1 [53] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual- linguistic representations. In ICLR, 2020. 2 [54] Maarten Sukel, Stevan Rudinac, and Marcel Worring. Mul- timodal temporal fusion transformers are good product de- mand forecasters. IEEE Trans. Multimedia, 31(2):48â60, 2024. 1 [55] Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for text classification? In China national con- ference on Chinese computational linguistics, pages 194â 206, 2019. 6 [56] Hao Tan and Mohit Bansal.LXMERT: Learning cross- modality encoder representations from transformers.In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 5100â5111, 2019. 2 [57] Zhiqiang Tang, Haoyang Fang, Su Zhou, Taojiannan Yang, Zihan Zhong, Cuixiong Hu, Katrin Kirchhoff, and George Karypis. Autogluon-multimodal (automm): Supercharging multimodal automl with foundation models.In ICAML, pages 15/1â35, 2024. 2, 4, 5, 6 [58] Zhiqiang Tang, Zihan Zhong, Tong He, and Gerald Fried- land. Bag of tricks for multimodal automl with image, text, and tabular data. arXiv preprint arXiv:2412.16243, 2024. 2, 4 [59] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In ACL, 2019. 5, 6 [60] Benjamin Warner, Antoine Chaffin, Benjamin Clavi Ě e, Orion Weller, Oskar Hallstr Ě om, Said Taghadouini, Alexis Gal- lagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Grif- fin Thomas Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional en- coder for fast, memory efficient, and long context finetuning and inference. In ACL, pages 2526â2547, Vienna, Austria, 2025. Association for Computational Linguistics. 2 [61] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for mul- timodal sentiment analysis. In EMNLP, pages 1103â1114, 2017. 5, 6 [62] Fei Zhao, Chengcui Zhang, and Baocheng Geng. Deep mul- timodal data fusion. ACM Comput. Surv., 56(9):1â36, 2024. 1 [63] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022. 2 [64] Xuran Zhu. Cross-modal domain adaptation in brain disease diagnosis: Maximum mean discrepancy-based convolutional neural networks. In Int. Conf. Commun., Inf. Syst. Comput. Eng., pages 1515â1519, 2024. 1 MultiModalPFN: Extending Prior-Data Fitted Networks for Multimodal Tabular Learning Supplementary Material S1. Additional Analysis Attention Imbalance. Figure S1 illustrates the relation- ship between the number of non-tabular input features gen- erated by MGM and CAP and the resulting performance across the evaluated datasets. In all cases, the best perfor- mance is achieved when the number of non-tabular features is similar to the number of tabular features. Performance tends to degrade when the number of non-tabular features is either substantially smaller or larger than the number of tabular features. These results experimentally support our analysis of attention imbalance. Activation Choice in MGM. We study the impact of us- ing GLU as the activation function in MGM on the CBISâ DDSM (MASS) and Salary datasets. Table S1 compares GLU against a GELU baseline in terms of accuracy and mean output-vector orthogonality. Since GLU reduces the dimensionality by half after the gating operation, the GELU baseline is configured with more parameters. Despite this advantage, GLU consistently yields higher accuracy than GELU on both datasets, while also increasing the orthog- onality measure. This suggests that the gating mechanism in GLU not only improves predictive performance but also promotes more diverse output representations, aligning with our objective of learning complementary non-tabular fea- tures. Consequently, we adopt GLU as the default activation in MGM. Table S1. Effect of activation choice in MGM. Comparison of accuracy and mean output-vector orthogonality when using GELU versus GLU, with and without orthogonality loss, on CBISâ DDSM (MASS) and Salary. CBISâDDSM (MASS)Salary ActivationAccuracyOrthogonalityAccuracyOrthogonality GELU72.090.056545.040.04876 GLU75.100.091345.870.05831 Parameter budgets in Table 5. In Table 5, the parameter counts of the modality projector vary depending on the ar- chitectural choice. Let N denote the number of MGM heads and d the token dimension. The parameter counts scale as follows: single-head + linear O(d), single-head + MLP O(d 2 + d), multi-head + MLP O N(d 2 + d) , and multi- head + MGM O N(d 2 + d/2) . MGM requires fewer pa- rameters than the multi-head MLP due to its linear gating with channel splitting. These results indicate that the per- formance trends in Table 5 are not solely explained by in- creased model capacity. Moreover, the additional parame- ters introduced by the projector contribute only marginally to inference latency. Robustness of Low-Data Regimes. In Table 7, MMPFN achieves a higher average performance (64.21) than TIP (58.97), despite larger relative drops on PU20 and Petfinder. This robustness primarily stems from strong priors learned during large-scale meta-training on synthetic datasets[25, 26], which capture a broad range of plausible tabular dis- tributions and enable effective generalization from few real samples. Fine-tuning then provides a light task-specific adaptation on top of this Bayesian inference. Because the model requires only light adaptation, it avoids overfitting and remains stable in low-sample settings. Together with the inductive bias of the Per-Feature Transformer, this ex- plains the superior performance of our MMPFN across low- data experiments. Replacing Modality Encoders. MMPFN combines pre- trained models, making the framework naturally extensible as newer and stronger encoders become available. This modular design allows components such as TabPFN or DINO to be replaced with more recent architectures with- out altering the overall pipeline. Leveraging improved pre- trained models enhances the quality of feature representa- tions and can lead to measurable downstream gains. For example, substituting DINOv2 with the recently released DINOv3 yields consistent improvements across datasets, as shown in Table S2; on PU20, accuracy increases by approx- imately 0.74 percentage points. We also examine the effect of replacing the text encoder. As shown in Table S3, switching between ELECTRA and DeBERTa results in only minor performance differences across the evaluated datasets. These results suggest that while stronger encoders can provide modest improvements, the overall performance of MMPFN remains relatively sta- ble across different encoder choices. Distribution Alignment Across Modalities. We inves- tigate whether applying embedding-space alignment tech- niquesâcommonly used in multimodal learningâcan fur- ther improve the performance of MMPFN. In particular, we incorporate Maximum Mean Discrepancy (MMD) [18], a standard measure of distributional discrepancy [64], into (a) PU20(b) Calc (c) Cloth(d) PetFinder-A Figure S1. Effect of the ratio between tabular and non-tabular features. The black dots and vertical lines show the mean and variance across five random seeds. Darker black dots correspond to a larger number of MGM heads (i.e., more non-tabular features generated by MGM), ranging from 8 to 128. The red dot indicates the average result across all MGM-head settings. The x-axis shows the number of non-tabular features generated by CAP, and the y-axis denotes accuracy. The blue line represents the number of tabular features. Dataset names are shown above each subfigure. Table S2. Effect of replacing the image encoder. Performance of MMPFN when substituting DINOv2 with ResNet50 and DINOv3. Results are reported as averaged accuracy over five random seeds. EncoderPU20MassCalcPetfinder ResNet5083.26-73.94- DINOv285.2274.5375.4040.74 DINOv385.6175.4876.7540.57 our framework. We apply MMD to the embeddings gen- erated for each feature to reduce the distributional gap be- tween representations from different modalities. For tabu- lar data, where feature distributions can differ substantially Table S3. Effect of replacing the text encoder. Performance of MMPFN when using different pretrained text encoders. Results are reported as averaged accuracy over five random seeds. EncoderAirbnbSalary Electra47.7846.17 DeBERTa47.8245.69 across dimensions, we additionally employ Joint MMD (JMMD) [39] to capture discrepancies at the level of the joint feature distribution. We train the multi-head MLP module that produces un- structured feature embeddings by adding the discrepancy 0.8 0.6 0.4 0.2 1.0 0.0 24816612244896192768384612244896192384 Figure S2. Additional results of token-count and attention mass. We extend the analysis in Figure 3 to additional datasets. For Mass and Calc, each sample contains three images, and thus the number of non-tabular tokens is given by the number of mgm heads times 3. The number of tabular tokens is 25, 4 and 4 for Airbnb, Mass, and Calc, respectively. (a) Image-only MGM variants. (b) Full MMPFN (tabular + image) variants. Figure S3. Effect of distribution-alignment and orthogonality regularization. Under the same experimental conditions as Fig- ure 2, we compare the performance of variants that incorporate orthogonality and MMD-based regularization constraints to the MGM baseline, for both image-only and tabular+image settings. between tabular embeddings and image/text embeddings as an auxiliary loss term. However, as shown in Figure S3, this alignment-based regularization consistently underperforms the MGM baseline, and incorporating the same loss directly into MGM also fails to yield improvements. Together with our cosine-similarity analysis, these negative results suggest that embeddings extracted by MGM from image and text modalities are already mapped into a semantically compat- ible space with tabular embeddings, enabling effective in- teraction through the attention module. Moreover, while MMD-style losses can reduce distributional gaps, they may also suppress discriminative variations, leading to perfor- mance degradation. We therefore infer that once MGM and CAP produce sufficiently aligned embeddings, enforcing additional distributional alignment does not provide further gains and can even be detrimental. Comparison with Patch-Token Features. In MMPFN, the text encoder [7] and the image encoder [44] produce output embeddings for every token (e.g., text tokens or im- age patches). In principle, one could replace the [CLS]- based MGM features with ViT patch-token outputs and use all token embeddings directly for feature generation. How- ever, this design has both practical and empirical draw- backs. From a memory perspective, using all patch tokens is substantially more expensive than using the aggregated [CLS] token. For example, when resizing PAD-UFES-20 images to 336 = 14Ă24 pixels and encoding them with the DINOv2 ViT-B/14 backbone, the model produces 576 token embeddings per imageâmore than four times the number of MGM heads (128) used in our experiments. In terms of storage, the [CLS] embedding requires only 7.1 MB, whereas retaining all patch-token outputs occupies 4.1 GB, leading to a prohibitive increase in memory consumption. A similar issue arises for text: in the Cloth dataset, many text attributes approach the maximum input length of 512 tokens, so storing all token embeddings again results in ex- cessive memory usage. Empirically, we also observe that models using ViT patch-token outputs underperform those relying on the [CLS]-driven MGM features.On PU20, replacing the [CLS]-based MGM heads with patch-token features de- creases accuracy by 0.85 percentage points (from 85.22% to 84.02%). We attribute this degradation to the fact that the [CLS] representation of a well-trained foundation model already encodes task-relevant global information, while raw token-level outputs contain substantial redundancy and noise. Consequently, patch-token features are less suitable than [CLS]-based MGM heads for constructing compact, tabular-like feature representations. Additional qualitative results of attention imbalance. To complement the analysis in the main paper, we further examine the relationship between token count and attention mass on additional datasets. As shown in Figure S2, we ob- serve a consistent trend across all datasets: as the number of non-tabular tokens increases, the attention mass assigned to the non-tabular modality increases monotonically, while the attention to tabular tokens decreases accordingly. For datasets such as Mass and Calc, each sample con- tains multiple images, and the number of non-tabular to- kens is given by the number of MGM heads multiplied by the number of images. Despite variations in the number of tabular tokens across datasets, such as 25 for Airbnb and 4 for Mass and Calc, the same qualitative behavior consis- tently emerges. These results further indicate that attention allocation is primarily determined by the relative proportion of tokens within the input sequence. Implementation Details. For the training procedure, we adopt the official TabPFN repository 2 and modify it to sup- port MMPFN by extending the MMPFNClassifier and MMPFNRegressor classes. Fine-tuning is performed by splitting the available data into training and validation sets and updating model parameters based on the validation loss. We use a small learning rate of 1 Ă 10 â5 , a fixed bud- get of 100 training steps, and ScheduleFree [11] for learn- ing rate scheduling. Figure S4 shows the learning curve on four different datasets. For contrastive-pretraining base- lines [13, 14, 20], we train for 500 epochs using a cosine- annealing scheduler with a 10-epoch warmup. Text Data Pre-Processing. We adopt the text pre- processing pipeline of T [3], following the implemen- tations provided in the official codebase. For the Salary dataset, the original source URL referenced in Bonnier [3] 2 https://github.com/LennartPurucker/finetune_ tabpfn_v2 (a) PU20(b) Mass (c) Calc(d) PetFinder-A Figure S4. Fine-tuning loss curves of MMPFN across datasets. Each subfigure shows the validation loss over training steps for four different datasets, illustrating stable convergence behavior un- der our fine-tuning setup. is no longer accessible, so we use a Kaggle-hosted copy in- stead. Applying the official T scripts to this version does not reproduce the exact dataset size reported in the paper, suggesting minor discrepancies. We therefore re-evaluate the T baseline on this revised dataset; the resulting ac- curacy (46.5) closely matches the originally reported per- formance, confirming that the new version is suitable for evaluating our model. Both ELECTRA and DeBERTa text encoders are lim- ited to 512 input tokens, so longer sequences are truncated. For datasets with multiple text attributes, we extract em- beddings for each attribute separately and incorporate them as additional text features, whereas T concatenates all text columns into a single sequence; this yields small but consistent accuracy gains, although the improvements re- main within the error margin and are not reported in the main comparison table. The Airbnb and PetFinder datasets contain Chinese characters in their text fields; because the ELECTRA variant we use is not pretrained on Chinese, these characters are replaced with empty strings before en- coding. For CatBoost and AutoGluon, we rely on the li- brariesâ built-in text handling capabilities. Cosine Similarity Computation. We provide the de- tailed procedure used for the cosine-similarityâbased cor- relation analysis in Figure 4 and Sec. 4.3. The TabPFN encoder for tabular data normally groups multiple features into a single embedding to reduce memory usage, but such grouping can introduce noise when comparing cosine sim- ilarity between tabular and image (or text) embeddings. To obtain more precise relationships, we set the tabular group size to 1, generate an individual embedding for each fea- ture, and then compare these feature-wise embeddings with image embeddings. Cosine similarity between input features is computed at the instance level and subsequently averaged across in- stances. Averaging embeddings over the entire dataset be- fore computing similarity vectors would be cheaper but tends to underrepresent the contribution of individual sam- ples, whereas computing similarity for every token embed- ding is prohibitively expensive and makes global patterns difficult to visualize. We therefore adopt an intermediate strategy, which balances computational cost and fidelity.