Paper deep dive
LoRA-Based Cascaded Multimodal Fusion for Action Recognition in Medical Training Environments
Divya Mereddy, Jeevan Beedareddy
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 7/18/2026, 3:05:12 PM
Summary
The paper proposes a cascaded Low-Rank Adaptation (LoRA)-based multimodal fusion framework for action and activity recognition in healthcare training environments. The method sequentially integrates modalities (e.g., skeleton, RGB, text, sensor data) using modality-specific LoRA adapters without retraining previous components. It is evaluated on the NurViD and Nurse Training datasets, demonstrating improved accuracy over unimodal models and competitive performance against state-of-the-art baselines.
Entities (16)
Relation Signals (13)
Proposed Framework → evaluatedon → NurViD
confidence 98% · We evaluate the framework on two healthcare-oriented training environment datasets: NurViD
Proposed Framework → evaluatedon → Nurse Training Dataset
confidence 98% · and the Nurse Training dataset.
LoRA → usedin → Cascaded Fusion
confidence 95% · proposes a novel LoRA-Based Cascaded Multimodal Fusion Architecture
RGB → fusedwith → Skeleton Data
confidence 90% · Stage 1, which combines skeleton and acceleration modalities... first integrates more closely related modalities... skeleton representations are derived from RGB frames
Proposed Framework → outperforms → Unimodal Models
confidence 90% · improves over individual modality models
STGCN → processes → Skeleton Data
confidence 90% · skeleton-only unimodel using STGCN
TimeSformer → processes → RGB
confidence 90% · RGB video features are modeled using TimeSformer
BERT → processes → Text
confidence 90% · text is encoded with BERT
Text → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This paper presents a cascaded Low-Rank Adaptation (LoRA)-based multimodal fusion framework for action and activity recognition in healthcare-oriented training environments. The proposed architecture combines parameter-efficient modality-specific adaptation with sequential fusion, enabling modalities to be integrated in stages without retraining previously learned components. Rather than assuming a fixed fusion structure, the framework first integrates more closely related modalities and then incorporates additional heterogeneous modalities, supporting scalable adaptation across datasets with different modality this http URL evaluate the framework on two healthcare-oriented training environment datasets: NurViD and the Nurse Training dataset. Across these datasets, preliminary results suggest that the proposed cascaded fusion strategy improves over individual modality models and provides competitive performance relative to previously reported dataset-specific baselines. Overall, these findings indicate that cascaded LoRA-based fusion is a promising parameter-efficient approach for integrating heterogeneous modalities in medical training action and activity recognition tasks. github: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2607.11839v1
- Canonical: https://arxiv.org/abs/2607.11839v1
Trouble viewing inline? Open PDF directly →
Full Text
25,440 characters extracted from source content.
Expand or collapse full text
LoRA-Based Cascaded Multimodal Fusion for Action Recognition in Medical Training Environments Divya Mereddy ✁! divya.mereddy@vanderbilt.edu Vanderbilt University Nashville, TN, USA Jeevan Beedareddy Quince Mountainview, CA, USA Abstract This paper presents a cascaded Low-Rank Adaptation (LoRA)-based multimodal fusion framework for action and activity recognition in healthcare-oriented training environments. The proposed archi- tecture combines parameter-e!cient modality-speci"c adaptation with sequential fusion, enabling modalities to be integrated in stages without retraining previously learned components. Rather than as- suming a "xed fusion structure, the framework "rst integrates more closely related modalities and then incorporates additional hetero- geneous modalities, supporting scalable adaptation across datasets with di#erent modality sets.We evaluate the framework on two healthcare-oriented training environment datasets: NurViD and the Nurse Training dataset. Across these datasets, preliminary results suggest that the proposed cascaded fusion strategy improves over individual modality models and provides competitive performance relative to previously reported dataset-speci"c baselines. Overall, these "ndings indicate that cascaded LoRA-based fusion is a promis- ing parameter-e!cient approach for integrating heterogeneous modalities in medical training action and activity recognition tasks. github: https://github.com/anonymous0-ai/LoRA-Based-Cascaded- Multimodal-Fusion-.git. CCS Concepts •Computing methodologies→Neural networks;Computer vision;Learning latent representations;Machine learning algorithms. Keywords Multimodal learning, Cascaded fusion, Low-Rank Adaptation, LoRA, Action recognition, Activity recognition, Medical training environ- ments ACM Reference Format: Divya Mereddy and Jeevan Beedareddy. 2018. LoRA-Based Cascaded Multi- modal Fusion for Action Recognition in Medical Training Environments . InProceedings of Make sure to enter the correct conference title from your rights con!rmation email (Conference acronym ’X).ACM, New York, NY, USA, 5 pages. https://doi.org/X.X Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for pro"t or commercial advantage and that copies bear this notice and the full citation on the "rst page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speci"c permission and/or a fee. Request permissions from permissions@acm.org. Conference acronym ’X, Woodstock, NY ©2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-X-X/2018/06 https://doi.org/X.X 1 Introduction The ability to accurately recognize and classify human actions is critical across numerous domains, including surveillance, health- care, education, and training environments. With the increasing complexity of real-world environments, action recognition tasks often require the integration of diverse data modalities, such as skeletal, $ow, RGB, and audio information. Multimodal fusion mod- els have emerged as a promising solution to these challenges by leveraging complementary modalities to provide a richer and more nuanced understanding of activities. However, existing approaches face several limitations, including high computational costs, ine!- ciency in adapting to new modalities, and the need for retraining entire models when datasets evolve. To address these challenges, we propose a novelLoRA-Based Cascaded Multi modal Fusion Architecturethat combines the strengths of cascaded fusion with the computational e!ciency of LoRA-based "ne-tuning. The framework introduces a modular ar- chitecture that enables the sequential integration of modalities without retraining previously trained components, o#ering signi"- cant $exibility and scalability. For example, our approach allows for incremental modality addition, such as combining skeleton and RGB data (Stage 1) and subsequently integrating text or other modalities (Stage 2), while preserving the integrity and performance of earlier stages. The framework is evaluated using two benchmark datasets— NurViD, a large-scale video audio dataset focused on action local- ization in clinical environments, and theNurse Training Dataset, a sensors based domain-speci"c dataset for healthcare action recog- nition. By leveraging the complementary strengths of di#erent modalities (such as skeleton, RGB, and textual) in a cascaded fu- sion setup, our approach demonstrates promising results (specially with limited resources) compared to state-of-the-art fusion meth- ods. Speci"cally, our model achieved comparable results to existing benchmarks while showcasing the advantages of lightweight and sequential fusion strategies. In summary, our contributions are twofold: (1)We propose a Cascaded LoRA-based Fusion Framework for sequential and modular modality integration. (2)We validate the proposed architecture on two datasets, along with a comparative analysis against state-of-the-art methods to demonstrate proposed method performance. The proposed framework establishes a foundational approach for dynamic sequential multi-modal learning. Integrating LoRA into this framework will allow for e!cient and straightforward adaptation, minimizing the computational cost associated with introducing additional modalities while maintaining robust perfor- mance across tasks. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. 2 Related Work: The growing interest in multimodal models stems from their capac- ity to capture a holistic understanding of complex environments by leveraging multiple data types or modalities. Such models enable a more nuanced interpretation and have been widely applied across diverse tasks. Notably, considerable research has been dedicated to the development of e#ective multimodal fusion techniques [11] [6]. Leveraging the advantages of cascaded network architectures, cascaded fusion techniques have been implemented across both transformer-based models [3][8], as well as in non-transformer models,[2]. In parallel, the popular "ne-tuning approach Low-Rank Adaptation (LoRA) [4] has emerged as a highly e!cient method for multimodal tuning, especially due to its low computational demands and adaptability. LoRA-based tuning models are now widely used in various fusion architectures, encompassing late, early and mid-level including hierarchical fusion setups [13] [14] [9] [9] [12] [10]. Despite these advances, cascaded fusion models using LoRA have not yet been fully explored. We propose to develop a LoRA- based cascaded fusion model to leverage the $exibility of cascaded networks, which inherently support the incremental addition of new modalities. Apart from above, Our approach is close to hier- archical fusion method type, which share structural similarities with cascaded networks. Our approach even specially related to LoRA-based hierarchical multimodal techniques [15] [1] [16] [16]. 3 Proposed Methodology 3.1 Background Low-Rank Adaptation (LoRA): LoRA is a "ne-tuning method that focuses on updating low-rank matrices, o#ering an e!cient and computationally lightweight alternative to traditional "ne-tuning techniques. Instead of modifying the full model parameters, LoRA introduces low-rank updates to pretrained weights, signi"cantly reducing memory and computational costs. In the context of multimodality, LoRA brings distinct modality embeddings (e.g., text, video, and audio representations) closer together in the feature space, enabling e#ective modality-speci"c tuning. This capability is particularly advantageous for multimodal systems, where it allows e!cient "ne-tuning while maintaining the modularity of individual modalities. Cascaded Multimodal Fusion: Cascaded fusion is a sequential integration technique that enables the step-by-step addition of modalities, such as퐿+푀→퐿푀→퐿푀+푁→퐿푀푁.The key advantage of this approach is that it avoids retraining the entire system when a new modality is added, thereby preserving the previously learned representations and reducing computational overhead. This approach is particularly well-suited for evolving datasets where new modalities may be introduced over time. The cascaded fusion framework allows for $exibility in sequential modality addi- tion, making it adaptable to dynamic multimodal learning scenarios. 3.2 Proposed Method In this project, we aim to develop a LoRA-based cascaded fusion model that leverages the modularity and adaptability of cascaded Figure 1: Proposed Methodology-Cascaded Low-Rank Adap- tation Multi-modal Fusion networks to accommodate multiple modalities e!ciently with the help of LORA. We conducted a detailed literature review to iden- tify key multimodal fusion techniques, focusing on LoRA-based approaches and cascaded fusion architectures. While cascaded ar- chitectures available, their open source is code is not available, we are developing cascaded LORA architecture using three di#erent singular modalities. The model architecture is based on cascaded fusion principles, where modality-speci"c LoRA adapters are se- quentially integrated. Each modality (e.g., skeleton, RGB, text) is "rst processed through a modality-speci"c LoRA adapter for "ne tuning purpose. Each LoRA adapter will be designed to process an individual modality. So overall all modalities are added sequentially (e.g., A+B→AB, then AB+C→ABC) without retraining prior components. 4 Experiments 4.1 Experimental Setup: 4.1.1 Data:We evaluate the proposed cascaded LoRA-based mul- timodal fusion framework on two healthcare-oriented action and activity recognition datasets: NurViD [5] and the Nurse Training dataset [7]. NurViD contains 1,538 untrimmed clinical videos total- ing 144 hours, segmented into 5,608 clips with 177 action-step anno- tations. We focus on classi"cation over these 177 action categories using synchronized video, audio, and MMPose-derived skeletal representations collected from real-world clinical environments. The Nurse Training dataset[7] is a domain-speci"c nursing ac- tivity recognition benchmark containing 240 activity sequences from eight professional nurses performing six prede"ned activi- ties. It includes motion-capture trajectories, Meditag-based indoor localization, and smartphone accelerometer data, capturing body movement, spatial location, air pressure, and motion dynamics. Due LoRA-Based Cascaded Multimodal Fusion for Action Recognition in Medical Training EnvironmentsConference acronym ’X, June 03–05, 2018, Woodstock, NY Figure 2: Proposed Methodology-Cascaded Low-Rank Adap- tation Multimodal Fusion to its limited size and imbalanced activity durations, we use 3-fold cross-validation for evaluation. 4.2 Implementation Details For Nurvid dataset, we developed a with a skeleton-only unimodel using STGCN "netuned model; RGB video features are modeled using TimeSformer; and audio is transcribed into text using google apIs and this modlaities processed using BERT.In case of Nurse training dataset, for position data, we developed a Meditag-based position transformer, inspired by the structure used for acceleration data. And [6] has already provided Acceleration Model(accelaration modalaity) and Spatio-Temporal Skeleton Model(skeleton sensors points modalaity). To validate our cascaded LoRA fusion model, we plan a series of experiments to evaluate its performance and $exi- bility. To validate our cascaded LoRA fusion model, we designed experiments tailored to the unique characteristics of two datasets: NurViD and Nurse Training Challenge. Each dataset utilizes dis- tinct modalities and architectures, emphasizing the adaptability and performance of our proposed approach. NurViD Dataset: The NurViD dataset, which focuses on action localization in untrimmed videos, incorporates the following three modalities: Skeleton Data:Skeleton data is processed using the Nurse Training Transformer (skeleton- only variant) "ne-tuned with LoRA layers, capturing spatial-temporal patterns critical for action localization. Video Frames (RGB)" Video frames are modeled using TimeSformer, which e#ectively extracts spatial-temporal features from video data, aiding in precise motion representation. Text Data: Audio data is transcribed into text using the Google API, and the resulting text is encoded with BERT, "ne- tuned with LoRA layers to extract semantic information relevant to activity localization. Nurse Training Challenge Dataset: The Nurse Training Challenge dataset, focused on nursing activity recognition, utilizes the following three modalities: Skeleton Data: Skeleton data is modeled using the Skeleton Transformer from the Nurse Training Transformer paper, capturing movement patterns associated with nursing tasks. Acceleration Data: Acceleration data, collected via smartphones placed in nurses’ chest pockets, is pro- cessed using a basic transformer architecture, "ne-tuned with LoRA layers to detect activity-speci"c motion.Position Data: Positional data, obtained via Meditag-based indoor localization, is processed using a Meditag-based position transformer, designed to extract spatial insights similar to the acceleration transformer. 4.2.1 Evaluation Plan:We utilize accuracy as the primary metric to evaluate the e#ectiveness of our model in classifying activities based on sequentially fused modalities. 4.2.2 Baseline Comparisons:We will establish a baseline by testing the unimodal version of the model and comparing it to fully "ne- tuned multimodal models that integrate all available modalities without LoRA. This provides a reference for the bene"ts of cascaded LoRA fusion. 4.3 Experiments and Results: 4.3.1 Nurse Training Dataset.Table 1 reports the performance of individual modalities, late fusion, and the proposed two-stage cas- caded fusion strategy on the Nurse Training dataset. Among the unimodal models, skeleton features achieve the strongest perfor- mance with 0.71 accuracy, while acceleration and Meditag position features obtain 0.38 and 0.30 accuracy, respectively. Although ac- celeration and position are weaker as standalone modalities, their contribution becomes evident when integrated with skeleton fea- tures. Stage 1 fusion, which combines skeleton and acceleration modalities, improves accuracy to 0.7586, matching the reported cross-modal fusion baseline. Stage 2 further incorporates Meditag position features and improves performance to 0.7931, outperform- ing both unimodal and late-fusion con"gurations. These results suggest that the proposed LoRA-based cascaded fusion framework e#ectively integrates complementary sensor modalities while main- taining a lightweight and sequential fusion structure. Table 1: Performance Metrics for Di!erent Model Stages Nurse Care Activity Recognition Challenge Dataset ModelAccuracy (%) Simple Fusion(Late Fusion of Three Modalities)0.73 Skeleton Sensors Data Only (Modlaity1)0.71 Acceleration Sensor Data Only (Modlaity2)0.38 Position Sensor Data Only - Meditag (Modality3)0.3 Stage1 Fusion (Modality1+Modlaity2)0.7586 Stage2 Fusion (Stage1+Modlaity3)0.7931 Note.all the models are "netuned on Nurse care activity recognition challenge dataset Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. 4.3.2 NurViD Dataset.For the NurViD dataset, we evaluate the proposed framework using individual modality models and two sequential fusion stages. As shown in Table 2, RGB achieves the strongest unimodal performance, followed by text and skeleton representations. In the "rst stage, we fuse the skeleton and RGB modalities, following multimodal learning literature that suggests integrating more closely related modalities before incorporating more heterogeneous modalities. This design choice is also moti- vated by the fact that skeleton representations are derived from RGB frames and therefore share stronger spatial-temporal corre- spondence with the visual modality than with audio-derived text. In the second stage, we incorporate the text modality obtained from transcribed audio. Accuracy improves from 40.46% for the best unimodal model to 61.87% after Stage 1 fusion and 83.02% after all-modality fusion, indicating that the proposed parameter- e!cient cascaded fusion strategy provides a promising direction for sequentially fusing multiple modalities in the context of action recognition for medical training environments. Table 2: Performance Metrics for Di!erent Model Stages ModelTest Accuracy (%) Skeleton Model11.25 RGB Model37.37 Text Model38.11 Stage 1 Fusion( skeleton & Vision)61.70 Stage 2 Fusion (All Modalities)83.02 Note.all the models are "netuned on nurvid data 5 State-of-the-Art Comparison Table 3: Comparison with state-of-the-art methods on Nurse Training Dataset ModelAccuracy(%) Cross Fusion Multi-model Transformer (MNT [7])75.8 LoRA-Based Cascaded Multimodal Fusion (Ours)79.31 Table 4: Comparison with state-of-the-art methods on Nurvid Dataset ModelAccuracy(%) SlowFast * (baseline from [5])14.83 C3D * (baseline from [5])14.90 I3D * (baseline from [5])14.83 LoRA-Based Cascaded Multimodal Fusion (Ours)83.02 Note.An asterisk ( * ) denotes the initialization from the model pre-trained on Kinetics 400 For the Nurse Training dataset, the prior MNT model [7] re- ports 75.8% accuracy using skeleton and acceleration modalities. Our Stage 1 cascaded fusion model uses the same modalities and achieves comparable performance at 75.86%. As shown in Table 3 by further incorporating Meditag-based position data in Stage 2, the proposed model improves accuracy to 79.31%, demonstrating the bene"t of sequentially integrating complementary sensor modali- ties. For NurViD as shown in Table 4 , we compare against the base- lines reported in [5]. SlowFast, C3D, and I3D achieve 14.83%, 14.90%, and 14.83% accuracy, respectively, whereas our LoRA-based cas- caded multimodal fusion model achieves 83.02% by integrating skeleton, RGB, and audio-derived text modalities. Overall, these results show that the proposed framework provides competitive or improved performance while supporting parameter-e!cient and sequentially scalable multi-modal fusion. 6 Conclusion In this paper, we presented a cascaded Low-Rank Adaptation (LoRA)- based multimodal fusion framework for action and activity recog- nition in healthcare-oriented training environments. The proposed framework combines cascaded fusion with parameter-e!cient LoRA "ne-tuning, enabling modalities to be integrated sequentially with- out retraining previously learned components. This design sup- ports modularity, computational e!ciency, and adaptation across datasets with di#erent modality con"gurations. We evaluated the framework on NurViD and the Nurse Train- ing dataset. The results suggest that sequentially integrating com- plementary modalities can improve performance over individual modality models and provide competitive performance relative to previously reported dataset-speci"c baselines. In particular, the framework demonstrates the value of "rst integrating closely re- lated modalities and then incorporating more heterogeneous modal- ities in later fusion stages. Following LoRA, MixLoRA, and LoRA-based fusion literature, modality-speci"c adapters encourage representations to become more closer in the latent space, reducing the need for costly cross- fusion mechanisms and enabling simple fusion operations such as concatenation. Future ablation studies will further examine the e#ects of LoRA adaptation, fusion order, resource-utilization and runtime optimizations. Overall, the proposed cascaded LoRA-based fusion framework o#ers a promising direction for scalable and parameter-e!cient multimodal fusion in medical training environ- ments. 7 Limitations and Applicability The proposed cascaded LoRA-based multimodal fusion framework e!ciently aggregates complementary information from di#erent modalities; however, it has several limitations. First, it relies on implicit co-adaptation of modality representations by fusing output embeddings from modality-speci"c architectures. Thus, the ap- proach may be most suitable when moderate representation re"ne- ment is su!cient; settings requiring stronger modality alignment need further ablation studies. Second, the sequential fusion strategy may introduce a representation bottleneck, since later modalities must adapt to partially "xed representations from earlier stages. While this improves modularity and e!ciency, it may limit globally optimal multimodal representation learning when strong cross- modal dependencies exist.Despite these limitations, the framework LoRA-Based Cascaded Multimodal Fusion for Action Recognition in Medical Training EnvironmentsConference acronym ’X, June 03–05, 2018, Woodstock, NY is well suited for classi"cation-oriented tasks with complementary modalities, evolving multimodal inputs, and computational con- straints. Future work should examine alignment-aware mechanisms for tasks requiring explicit cross-modal interaction modeling. References [1]AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis- Philippe Morency. 2018. Multimodal Language Analysis in the Wild: CMU- MOSEI Dataset and Interpretable Dynamic Fusion Graph. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, 2236–2246. doi:10.18653/v1/P18-1208 [2]Debapriya Banerjee, Fotios Lygerakis, and Fillia Makedon. 2021. Sequential Late Fusion Technique for Multi-modal Sentiment Analysis. Version Number: 1. doi:10.48550/ARXIV.2106.11473 [3]Sihan Chen, Jiajia Tang, Li Zhu, and Wanzeng Kong. 2023. A multi-stage dynami- cal fusion network for multimodal emotion recognition.Cognitive Neurodynamics 17, 3 (June 2023), 671–680. doi:10.1007/s11571-022-09851-w [4]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-Rank Adaptation of Large Language Models. Version Number: 2. doi:10.48550/ARXIV.2106.09685 [5]Ming Hu, Lin Wang, Siyuan Yan, Don Ma, Qingli Ren, Peng Xia, Wei Feng, Peibo Duan, Lie Ju, and Zongyuan Ge. 2023. NurViD: a large expert-level video database for nursing procedure activity understanding. InProceedings of the 37th International Conference on Neural Information Processing Systems(New Orleans, LA, USA)(NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 798, 19 pages. [6]Momal Ijaz, Renato Diaz, and Chen Chen. 2022. Multimodal Transformer for Nursing Activity Recognition. Version Number: 1. doi:10.48550/ARXIV.2204. 04564 [7]Momal Ijaz, Renato Diaz, and Chen Chen. 2022. Multimodal Transformer for Nursing Activity Recognition. arXiv:2204.04564 [cs.CV] https://arxiv.org/abs/ 2204.04564 [8]Chenyu Li, Bing Zhang, Danfeng Hong, Jun Zhou, Gemine Vivone, Shutao Li, and Jocelyn Chanussot. 2024. CasFormer: Cascaded transformers for fusion-aware computational hyperspectral imaging.Information Fusion108 (Aug. 2024), 102408. doi:10.1016/j.in#us.2024.102408 [9]Dengchun Li, Yingzi Ma, Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. 2024. MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts. Version Number: 3. doi:10.48550/ARXIV.2404.15159 [10]Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2018. E!cient Low-rank Multimodal Fusion with Modality-Speci"c Factors. Version Number: 1. doi:10.48550/ARXIV. 1806.00064 [11]Shruti Palaskar, Oggi Rudovic, Sameer Dharur, Florian Pesce, Gautam Krishna, Aswin Sivaraman, Jack Berkowitz, Ahmed Hussen Abdelaziz, Saurabh Adya, and Ahmed Tew"k. 2024. Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection. Version Number: 1. doi:10.48550/ARXIV.2406.09617 [12]Saurav Sahay, Eda Okur, Shachi H Kumar, and Lama Nachman. 2020. Low Rank Fusion based Transformers for Multimodal Sequences. Version Number: 1. doi:10.48550/ARXIV.2007.02038 [13]Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, and Lifu Huang. 2024. Multimodal Instruction Tuning with Conditional Mixture of LoRA. Version Number: 1. doi:10.48550/ARXIV.2402.15896 [14]Zhiyi Shi, Junsik Kim, Wanhua Li, Yicong Li, and Hanspeter P"ster. 2024. MoRA: LoRA Guided Multi-Modal Disease Diagnosis with Missing Modality. Version Number: 1. doi:10.48550/ARXIV.2408.09064 [15]Zhao Tong, Qiang Liu, Haichao Shi, Yuwei Xia, Shu Wu, and Xiao-Yu Zhang. 2024. Semantics Fusion of Hierarchical Transformers for Multimodal Named Entity Recognition. InAdvanced Intelligent Computing Technology and Applications, De-Shuang Huang, Zhanjun Si, and Qinhu Zhang (Eds.). Vol. 14877. Springer Nature Singapore, Singapore, 414–426. Series Title: Lecture Notes in Computer Science. doi:10.1007/978-981-97-5669-8_34 [16]Linhui Xiao, Xiaoshan Yang, Fang Peng, Yaowei Wang, and Changsheng Xu. 2024. HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding. InProceedings of the 32nd ACM International Conference on Multimedia. ACM, Melbourne VIC Australia, 5460–5469. doi:10.1145/3664647.3681071