Paper deep dive
USPLIT-VQA: U-Shaped Split Learning for Visual Question Answering with Contribution-Aware Weighted Aggregation
Md Khalid Syfullah, Alvi Ataur Khalil
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/16/2026, 8:12:14 AM
Summary
The paper introduces USplit-VQA, a U-shaped split learning framework for Visual Question Answering (VQA) that preserves privacy by keeping raw data and labels on client devices while offloading intermediate layers to a server. It proposes Contribution-Aware Weighted Aggregation (CAWA), a gradient-similarity-based mechanism to mitigate malicious updates. Experiments on four datasets show USplit-VQA achieves accuracy comparable to or better than Federated Learning (FL) and Centralized training, with significant reductions in client memory (up to 5.8x) and communication (up to 10.8x). CAWA reduces the influence of malicious clients by over 98%.
Entities (11)
Relation Signals (11)
USplit-VQA → comparedwith → Federated Learning
confidence 95% · show accuracy gains over Federated Learning
USplit-VQA → evaluatedon → VQA-RAD
confidence 95% · Experiments on four VQA datasets (VQA-RAD...)
USplit-VQA → evaluatedon → SLAKE
confidence 95% · Experiments on four VQA datasets (...SLAKE...)
USplit-VQA → evaluatedon → PathVQA
confidence 95% · Experiments on four VQA datasets (...PathVQA...)
USplit-VQA → evaluatedon → VizWiz
confidence 95% · Experiments on four VQA datasets (...and VizWiz)
USplit-VQA → isbasedon → Split Learning
confidence 95% · We propose USPLIT-VQA, a U-shaped split learning framework
USplit-VQA → reduces → communication
confidence 95% · communication reductions of up to 10.8X
USplit-VQA → reduces → Client Memory
confidence 95% · client memory reductions of up to 5.8X
USplit-VQA → supportsbackbone → BiomedCLIP
confidence 95% · with two backbones: BiomedCLIP [4] and a lightweight custom architecture
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Visual Question Answering (VQA) systems, jointly interpreting images and natural language queries, hold significant promise across many domains, yet the privacy-sensitive nature of user data creates a fundamental barrier. Centralized training requires access to all data, while federated learning requires each client to host the full model. We propose USPLIT-VQA, a U-shaped split learning framework for privacy-preserving VQA in which each client retains the initial layers and the classification head while the server hosts the computationally heavy intermediate layers, keeping raw inputs and labels on the client device. We further introduce Contribution-Aware Weighted Aggregation (CAWA), a gradientsimilarity-based client scoring mechanism designed to reduce the influence of malicious updates. Experiments on four VQA datasets (VQA-RAD, SLAKE, PathVQA, and VizWiz) with two backbones show accuracy gains over Federated Learning for the Custom model and reduced accuracy for BiomedCLIP under the evaluated fixed split, alongside client memory reductions of up to 5.8X and communication reductions of up to 10.8X. With one malicious client, CAWA reduces the attacker's influence by over 98%, while experiments at higher corruption levels identify its limitations. Reconstruction experiments further show lower inversion quality under the evaluated attacks.
Tags
Links
- Source: https://arxiv.org/abs/2609.12168v1
- Canonical: https://arxiv.org/abs/2609.12168v1
Trouble viewing inline? Open PDF directly →
Full Text
33,841 characters extracted from source content.
Expand or collapse full text
USplit-VQA: U-Shaped Split Learning for Visual Question Answering with Contribution-Aware Weighted Aggregation Md Khalid Syfullah Alvi Ataur Khalil Affiliation: Transformative Innovation for Trustworthy AI and Network Security (TITANS) Lab, Affiliation: Computer Science, Southern Illinois University Carbondale, USA Affiliation: mdkhalid.syfullah, a.khalil@siu.edu Abstract Visual Question Answering (VQA) systems, jointly interpreting images and natural language queries, hold significant promise across many domains, yet the privacy-sensitive nature of user data creates a fundamental barrier. Centralized training requires access to all data, while federated learning requires each client to host the full model. We propose USplit-VQA, a U-shaped split learning framework for privacy-preserving VQA in which each client retains the initial layers and the classification head while the server hosts the computationally heavy intermediate layers, keeping raw inputs and labels on the client device. We further introduce Contribution-Aware Weighted Aggregation (CAWA), a gradient-similarity-based client scoring mechanism designed to reduce the influence of malicious updates. Experiments on four VQA datasets (VQA-RAD, SLAKE, PathVQA, and VizWiz) with two backbones show accuracy gains over Federated Learning for the Custom model and reduced accuracy for BiomedCLIP under the evaluated fixed split, alongside client memory reductions of up to 5.8× and communication reductions of up to 10.8×. With one malicious client, CAWA reduces the attacker’s influence by over 98%, while experiments at higher corruption levels identify its limitations. Reconstruction experiments further show lower inversion quality under the evaluated attacks. Index Terms: Split Learning, federated learning, visual question answering, medical AI, privacy-preserving learning I Introduction With the advancement of Large Language Models (LLMs), the use of Visual Question Answering (VQA) has increased significantly, yet Vision Language Models (VLMs) remain among the most computationally expensive models, often deemed “AI-complete” or “AI-hard” [1]. Their high cost complicates deployment, and prompt-based access through central servers raises concerns for privacy-sensitive data. In the medical domain, Med-VQA enables automated clinical decision support over radiology scans, pathology slides, and retinal photographs [2, 3]. Recent VLMs achieve strong Med-VQA performance [4], yet rely on centralized training that aggregates patient data at a single server, conflicting with HIPAA, GDPR, and institutional data governance policies [5]. Distributed learning partially addresses these issues. Federated Learning (FL) keeps raw data local yet forces clients to host the full VLM, imposing prohibitive demands on clinical edge devices [6], while Standard Split Learning (SL) offloads computation yet exposes labels to the server [7]. Bidirectional Contrastive Split Learning (BiCSL) [8] decouples a VQA model into representation and contrastive modules with bidirectional gradient sharing. None of these simultaneously provides the resource distribution, label localization, and robustness targeted here. We propose USplit-VQA, a U-shaped split learning framework for privacy-sensitive VQA. The client hosts encoders and the classification head while the server handles multimodal fusion, keeping raw images, questions, and clinical labels local. Distributing gradient computation across untrusted clients, however, introduces a vulnerability: malicious or low-quality updates can silently corrupt the shared server model, and the server cannot validate them without client data. We address this with Contribution-Aware Weighted Aggregation (CAWA), a gradient-similarity mechanism scoring each client by its contribution to convergence. The Custom model also enables split-point analysis for server-client load balancing. Our contributions are four-fold: • We propose USplit-VQA, a U-shaped split learning framework tailored to VQA that keeps raw inputs and labels on the client. • We introduce CAWA, a split-learning aggregation mechanism that reduces the influence of data poisoning and backdoor attacks through adaptive, streak-based client reputation scoring. • We evaluate USplit-VQA on four privacy-sensitive VQA datasets (VQA-RAD, SLAKE, PathVQA, VizWiz) with two backbones: BiomedCLIP [4] and a lightweight custom architecture (∼ 8.5M parameters) for resource-constrained edge devices. • We benchmark against centralized and FL baselines and evaluate model inversion, gradient inversion, data poisoning, split-point selection, and client scalability. We investigate the following Research Questions (RQs): • RQ1: Can USplit-VQA match centralized training and FL while preserving data and label privacy? • RQ2: How does USplit-VQA compare with FL and BiCSL in accuracy, communication, and client burden? • RQ3: How robust is USplit-VQA against model inversion and gradient inversion attacks? • RQ4: How well does CAWA mitigate data poisoning and backdoor attacks across corruption levels? The remainder of this paper is organized as follows. Section I presents preliminaries, Section I reviews related work and research gaps, Section IV describes the proposed framework, models, and datasets, Section V reports our experimental setup and findings, and Section VI concludes. I Background In this section, we discuss several preliminary concepts to facilitate an understanding of distributed learning, attacks, and the positioning of our proposed USplit-VQA framework. I-A Distributed Learning Paradigms Distributed learning is a broad AI approach that trains models across multiple computing nodes, typically involving one server and multiple client nodes. Figure 1 illustrates the three distributed learning paradigms relevant to this work. Federated Learning (FL): In FL [6], each client maintains a full copy of the model, trains on private data, and transmits parameter updates to a central server for aggregation. While raw data remains local, clients must host the entire architecture and transmit full model weights. Standard Split Learning (SL): SL [7] partitions the model at a cut layer; the client processes data up to this layer and transmits intermediate activations to the server, which completes the forward pass and computes the loss. This reduces client computation however requires the server to access ground-truth labels, violating privacy. Fig. 1: Comparison of architectural layouts. U-Shaped Split Learning: U-shaped SL [9] splits the model twice: the client hosts the bottom encoders and the top classification layers, while the server hosts the middle layers. The forward pass flows client→ → , and loss computation occurs locally. Consequently, neither raw data nor labels are exposed to the server. I-B Threat Models in Distributed Learning Even without explicit data sharing, distributed systems remain vulnerable to various adversarial attacks such as: Gradient Inversion Attacks (GIA): A passively adversarial server reconstructs private training data from transmitted activations or gradients by optimizing a dummy input to match the observed signals [10]. Model Inversion Attacks: An adversary trains a decoder to reconstruct raw inputs from intermediate representations intercepted at the split boundary, exploiting spatial structures retained in early-layer features [11]. Data Poisoning: A Byzantine client intentionally corrupts local data or labels (e.g., label flipping or backdoor injection) to submit adversarial gradients, thereby degrading global model performance or installing targeted backdoors [12]. I Related Work VQA has advanced rapidly through vision-language pretraining: BiomedCLIP [4] improves biomedical representations by pretraining on 15M image-text pairs, and Dong et al. [13] survey recent generative approaches. Existing VQA systems commonly assume centralized data access, leaving a substantial gap in privacy-preserving methodologies. FL is well studied for unimodal medical imaging [5], though its application to multimodal Med-VQA remains nascent. Zhu et al. [14] proposed prompt-based personalized FL with learnable prompts and a reliability parameter that downweights low-performing clients, and Lu et al. [15] introduced FedMedVLP to unify disparate medical datasets. Both require clients to host the full VQA architecture, which can be prohibitive for edge devices. SL offers a more resource-efficient alternative and has primarily been explored for unimodal tasks. SplitFed [16] combined split and federated learning for health applications, restricted to single-modality models. For multimodal VQA, Sun et al. introduced BiCSL [8], which relies on contrastive self-supervised learning rather than direct supervised classification, does not employ U-shaped splitting, and is not evaluated on clinical datasets. Prior work established U-shaped configurations in unimodal distributed settings: RoS-FL [17] partitions U-shaped medical image networks across clients and a server while addressing model drift, UVSL [18] protects labels in vertical split learning via local differential privacy, and Lyu et al. [19] optimize resource allocation for U-shaped parallel split learning. We tailor this topology to supervised multimodal VQA with contribution-aware aggregation. The distributed nature of these paradigms also demands robust security measures. Split learning is vulnerable to inversion attacks such as FSHA [20], while FL counters data poisoning [21] through Byzantine-robust aggregation [22]. Three gaps therefore remain: U-shaped split learning is largely unstudied for supervised multimodal VQA, client contribution scoring in multimodal split learning is underexplored, and a comprehensive cost analysis against FL for VQA is absent. USplit-VQA addresses these gaps. IV Methodology In this section, we discuss the technical details of our work. Table I summarizes the key symbols used throughout. TABLE I: Definitions of used notations and symbols. Sym. Definition Sym. Definition K Number of clients simksim_k Weighted cosine similarity of k R,rR,r Total rounds, current round ckc_k Streak counter for client k B Batch size τ± _± Adaptive thresholds (μ±λσμ±λσ) D Hidden dimension (256, 768) ρ Temporal factor (r/R)ϕ(r/R)^φ L Text sequence length α,βα,β Reward / penalty step sizes NvN_v Visual tokens (49, 196) γ Streak amplification factor C Number of answer classes λ Threshold sensitivity parameter V,TV,T Visual / text sequences T Softmax temperature gkg_k Server gradient (client k) ϕφ Temporal exponent RkR_k Reputation score (client k) θs,θc _s, _c Server / client parameters WkW_k Trust weight for client k cuta_cut Activation at the cut layer IV-A Model Architectures We use a pretrained model (BiomedCLIP [4]) and our defined lightweight custom model, allowing us to observe how USplit-VQA performs across different architectures. IV-A1 BiomedCLIP-Based Model This architecture leverages BiomedCLIP [4] (D=768D=768), pretrained on PMC-15M. The vision encoder uses ViT-B/16 (196 patch tokens) and the text encoder uses PubMedBERT [23]. Same cross-modal fusion and classification architecture is applied, scaled to D=768D=768. IV-A2 Custom Lightweight Model We use a fully trainable architecture (∼ 8.5M parameters, D=256D=256) that contains: (i) Vision Encoder: A four-stage convolutional pipeline performs aggressive spatial downsampling- 1 _1 =MaxPool(ϕ(BN(Conv7×7s=2()))) =MaxPool\! (φ\! (BN\! (Conv_7× 7^s=2(x) ) ) ) (1) l+1 _l+1 =ϕ(BN(Conv3×3s=2(l))),l=1,2,3 =φ\! (BN\! (Conv_3× 3^s=2(h_l) ) ), l=1,2,3 (2) =LN(Reshape(4))∈ℝB×49×D =LN\! (Reshape(h_4) ) ^B× 49× D (3) where ϕ(⋅)φ(·) is the SiLU activation and the output is a sequence of 49 spatial tokens from a 7×77×7 feature grid. (i) Text Encoder: Token embeddings with learned positional encoding are processed through Nce=2N_ce=2 pre-norm Transformer Blocks (TB): =LN(TB(LN(tok[]+pos),Nce))∈ℝB×L×DT=LN (TB (LN(E_tok[q]+E_pos),N_ce ) ) ^B× L× D (4) (i) CBAM Refinement: Visual tokens undergo refinement through NCB=3N_CB=3 CBAM blocks [24], each applying channel attention cM_c and spatial attention sM_s: (l+1)=LN((l)+Flat(s⊙c⊙s))+FFN(⋅)V^(l+1)=LN\! (V^(l)+Flat(V_s _c _s) )+FFN(·) (5) (iv) Text Refinement: An additional NT=2N_T=2 pre-norm transformer blocks deepen text understanding: (l+1)=LN((l)+MHA((l)))+FFN(⋅)T^(l+1)=LN\! (T^(l)+MHA(T^(l)) )+FFN(·) (6) (v) Cross-Modal Fusion: Four bidirectional cross-attention layers for vision-to-text and text-to-vision fusion: ′ =LN(+MHAV→T(,,))+FFN(⋅) =LN\! (V+MHA_V→ T(V,T,T) )+FFN(·) (7) ′ =LN(+MHAT→V(,′,′))+FFN(⋅) =LN\! (T+MHA_T→ V(T,V ,V ) )+FFN(·) (8) A learnable query p∈ℝ1×Dq_p ^1× D extracts a fused representation via cross-attention on concatenated sequence [′;′][V ;T ]. (vi) Classification Head: The fused vector is mapped to class logits through a two-layer MLP with GELU activation, dropout, and a residual projection. Fig. 2: The architectural layout of USplit-VQA. IV-B U-Shaped Split Learning Framework As illustrated in Figure 2, the model is partitioned into three components following the U-shaped data flow: (i) Client Head (forward pass): The client encodes raw images and questions through local encoders, producing visual tokens ∈ℝB×Nv×DV ^B× N_v× D and text tokens ∈ℝB×L×DT ^B× L× D. These feature tokens are transmitted to the server as activations. (i) Server Body (processing): The server applies visual refinement blocks, text processing blocks, and a multi-modal fusion transformer comprising bidirectional cross-attention, question-aware visual attention, and learnable pooling. The server returns only the fused feature vector ∈ℝB×Df ^B× D. (i) Client Tail (backward pass): The client maps f through the classification head, computes cross-entropy loss with its private labels, and backpropagates. The logit gradient returns to the server, which updates its parameters through the CAWA module and sends back activation gradients. Raw images, questions, and labels never leave the client. BiomedCLIP uses a fixed split after the first vision and text transformer blocks, placing 38.8M parameters on the client and 187.9M on the server; alternative positions were not evaluated. Training freezes client encoders for 4 rounds, then unfreezes them. IV-C Contribution-Aware Weighted Aggregation (CAWA) In multi-client split learning, uniformly trusting all clients leaves the model vulnerable to adversarial or low-quality updates. CAWA dynamically evaluates gradient alignment against adaptive statistical bounds and scales trust via historical momentum. The protocol is detailed in Algorithm 1. (i) Reputation and Loss Weighting: Each client maintains an unbounded reputation RkR_k, converted to a bounded trust weight via temperature-scaled softmax: Wk=exp(Rk−maxjRjT)W_k= \! ( R_k- _jR_jT ) (9) This weight directly scales the client’s loss: ℒw=Wk⋅ℒL_w=W_k·L, suppressing detrimental updates at the gradient level. (i) Reputation-Weighted Gradient Similarity: To prevent a cartel of malicious clients from dominating consensus, the similarity score for client k is the reputation-weighted average cosine similarity with all peers: simk=∑j≠kW~jk⋅j‖k‖‖j‖sim_k= _j≠ k W_j g_k·g_j\|g_k\|\|g_j\| (10) where W~j=Wj/∑i≠kWi W_j=W_j/ _i≠ kW_i normalizes peer weights. (i) Adaptive Thresholds and Streak-Based Updates: Dynamic bounds τ±=μ±λσ _±=μ±λσ (where μ,σμ,σ are computed from the current round’s similarity distribution) replace static thresholds. A temporal scaling factor ρ=(r/R)ϕρ=(r/R)^φ dampens early-round volatility. Reputation is updated using streak-amplified rewards/penalties: Rk(r+1)=Rk(r)+ραeγ(ck−1)if simk>τ+Rk(r)−ρβeγ(|ck|−1)if simk<τ−Rk(r)otherwiseR_k^(r+1)= casesR_k^(r)+ρα e^γ(c_k-1)&if sim_k> _+\\ R_k^(r)-ρβ e^γ(|c_k|-1)&if sim_k< _-\\ R_k^(r)&otherwise cases (11) where streak counter ckc_k tracks consecutive positive/negative evaluations. Default hyperparameters are: α=0.1α=0.1, β=0.05β=0.05, γ=0.5γ=0.5, λ=0.5λ=0.5, T=1.0T=1.0, and ϕ=2.0φ=2.0. Algorithm 1 USplit-VQA with CAWA 0: Clients K, Rounds R, α,β,γ,λ,T,ϕα,β,γ,λ,T,φ 1: Init: Rk=0R_k=0, ck=0,∀kc_k=0,∀ k 2: for r=1,…,Rr=1,…,R do 3: Rmax←maxjRjR_ ← _jR_j 4: for each client k∈Kk∈ K do 5: Wk←exp((Rk−Rmax)/T)W_k← ((R_k-R_ )/T) 6: for batch (,,,)(x,q,m,y) in k’s data do 7: ^←USplitFwd(,,) y (x,q,m) 8: ℒw←Wk⋅CE(,^)L_w← W_k·CE(y, y); update θs,θc _s, _c 9: end for 10: k←g_k← avg server gradient over batches 11: end for 12: for each client k∈Kk∈ K do 13: simk←∑j≠kW~jcos(k,j)sim_k← _j≠ k W_j (g_k,g_j) 14: end for 15: μ,σ←mean,stdμ,σ ,std of simk\sim_k\; ρ←(r/R)ϕρ←(r/R)^φ 16: for each client k∈Kk∈ K do 17: if simk>μ+λσsim_k>μ+λσ then 18: ck←max(1,ck+1)c_k← (1,c_k+1); Rk+=ραeγ(ck−1)R_k +=ρα e^γ(c_k-1) 19: else if simk<μ−λσsim_k<μ-λσ then 20: ck←min(−1,ck−1)c_k← (-1,c_k-1); Rk-=ρβeγ(|ck|−1)R_k -=ρβ e^γ(|c_k|-1) 21: else 22: ck←0c_k← 0 23: end if 24: end for 25: end for IV-D Datasets Utilized We evaluate our approach on VQA-RAD [2], SLAKE [3], PathVQA [25], and VizWiz [26], as summarized in Table I. We use the English SLAKE subset, apply majority voting to VizWiz annotations, and normalize answers into classification categories. Distributed training uses IID client partitions, while test sets are maintained centrally. TABLE I: Summary of the used datasets. Dataset Images Train Test Class Domain VQA-RAD 315 3,064 451 490 Radiology SLAKE 642 4,923 1,055 218 Medical PathVQA 4,998 19,755 6,279 4033 Pathology VizWiz 4,319 3,455 864 175 Accessibility IV-E Split Point & Client Scalability Analysis The split boundary position determines the privacy-efficiency trade-off. For the Custom model, we evaluate four configurations (V1-V4) that progressively shift model capacity from the server to the client, as summarized in Table I. For measuring client scalability, the number of clients was varied from 5 to 15. All preserve the same architecture; only the partitioning differs. TABLE I: Details for varying model split points. Version Client Components Cl.% Sv.% Transmitted V1 Vision Enc. 2.0 98.0 Visual tokens V2 V1 + Text Embed 42.6 57.4 Vis. + text tokens V3 V2 + CBAM + Text Ref. 63.4 36.6 Refined tokens V4 V3 + CrossAttn + Fusion 81.4 18.6 Fused tokens V Experimental Results In this section, we present our experimental setup, results, and the strengths and weaknesses of our contribution. V-A Experimental Setup All experiments use PyTorch, with the Custom model on a Tesla T4 GPU and BiomedCLIP on an NVIDIA RTX PRO 6000. We benchmark USplit-VQA against centralized training (full data access) and Federated Learning (3 local epochs per round), using K=5K=5 clients with IID partitioning over 20 rounds, extended to 30 rounds for the scalability study. Hyperparameters: server LR 3×10−43×10^-4, client LR 3×10−53×10^-5, dropout 0.15, weight decay 10−410^-4, batch size 24 (Custom) and 16 (BiomedCLIP); centralized models train up to 30 epochs with early stopping (patience 16). Security and split analyses (Sections V-D–V-G) use the Custom model on SLAKE. V-B Accuracy Comparison with Centralized and FL Baselines TABLE IV: Comparison of test accuracy (%) across different datasets, model architectures, and training paradigms. BiomedCLIP Custom Dataset Cent. FL USplit-VQA Cent. FL USplit-VQA VQA-RAD 43.68 44.57 37.92 39.25 36.36 43.90 SLAKE 81.24 83.51 74.18 78.23 65.88 67.77 PathVQA 68.42 69.18 58.25 46.88 46.60 48.25 VizWiz 60.89 62.62 61.01 62.25 60.02 62.75 Table IV shows that with the Custom model, USplit-VQA exceeds FL on all four datasets and matches or exceeds centralized performance on three, with the largest FL gain of 7.54 percentage points on VQA-RAD. Under the fixed BiomedCLIP split, FL performs better on all datasets, with the largest gap of 10.93 points on PathVQA and the smallest of 1.61 on VizWiz. The accuracy impact therefore depends on the backbone and split boundary. Fig. 3: Evolution of client reputation weights and overall model performance under various data poisoning intensities. V-C Comparison with Federated Learning TABLE V: Resource and performance comparison between FL and USplit-VQA. BiomedCLIP Custom Metric FL USplit-VQA FL USplit-VQA Client params 226.7M 38.8M 8.5M 1.5M Server params n/a 187.9M n/a 7.0M Client memory 2,720 MB 465 MB 102 MB 18 MB Comm./round 907 MB 84 MB 34 MB 5.6 MB Time/round (s) 139.6 99.5 68.1 36.3 Avg. accuracy (%) 64.97 57.84 52.22 55.67 Table V shows that USplit-VQA cuts client memory by 5.8×5.8× (BiomedCLIP) and 5.7×5.7× (Custom), and communication by 10.8×10.8× and 6.1×6.1× respectively. Average per-round time falls by 47% for the Custom model and 29% for BiomedCLIP, although Figure 4 shows a BiomedCLIP exception on VQA-RAD, where USplit-VQA takes 36.2 s against 25.4 s for FedAvg. We estimate client memory as Pclient×4B×3P_client× 4\,B× 3 and communication from full parameter exchange for FL or cut-layer activations and gradients for USplit-VQA. USplit-VQA keeps labels local, a property FL shares and conventional SL lacks, and CAWA adds Byzantine robustness. Fig. 4: Comparison of time requirements per round. V-D Byzantine Robustness Evaluation We evaluate Byzantine robustness against a compound attack in which malicious clients flip 30% of labels and apply an 8×88×8 white trigger to 20% of samples (yt=1y_t=1). Figure 3 shows that CAWA suppresses one attacker’s weight by 98.2% while maintaining 65.2% accuracy, although performance degrades as the malicious-client proportion rises. Table VI compares USplit-VQA, USplit-VQA with uniform aggregation (No CAWA), and FL. With one malicious client, USplit-VQA records 65.2% accuracy and 0.6% ASR against 61.2% and 44.1% for the ablated variant. With two or more malicious clients, all frameworks exhibit high ASR, while USplit-VQA retains the highest clean accuracy under three malicious clients. TABLE VI: Clean accuracy and Attack Success Rate (ASR) under varying numbers of malicious clients. Method Metric No Att. 1 Mal. 2 Mal. 3 Mal. USplit-VQA Acc. 65.7 65.2 62.6 60.9 ASR 0.6 0.6 83.5 94.2 No CAWA Acc. 62.5 61.2 60.6 59.2 ASR 0.6 44.1 95.3 96.3 FL Acc. 64.2 62.7 61.3 58.0 ASR 0.6 3.9 93.0 97.0 V-E Robustness Against Privacy Attacks We evaluate model inversion with a convolutional decoder and apply Deep Leakage from Gradients (DLG) [10] to 20 test samples. Table VII shows that USplit-VQA produces 10×10× higher model-inversion MSE, a 9.9 dB lower PSNR, and lower SSIM [27] than centralized training. For DLG, all PSNR values remain below 8 dB and USplit-VQA records the lowest at 5.70 dB, indicating lower reconstruction quality within this evaluation rather than a general privacy guarantee. TABLE VII: Summary of model & gradient inversion attacks. Model Inversion Gradient Inversion Metric Cent. FL USplit-VQA Cent. FL USplit-VQA MSE ↑ 0.0018 0.0013 0.0175 0.248 0.253 0.272 PSNR ↓ 27.42 28.80 17.56 6.08 6.00 5.70 SSIM ↓ 0.989 0.988 0.796 - - - V-F Comparison with BiCSL Figure 5 compares USplit-VQA and BiCSL across four normalized axes. USplit-VQA achieves higher clean accuracy and a smaller drop under a 30% label-flip attack, while keeping ground truth on the client incurs a 2×2× communication overhead from the U-shaped round trip. Fig. 5: Trade-offs between USplit-VQA and BiCSL. Fig. 6: Split-point (a, c) and client-scalability (b, d) analysis: test accuracy and per-round time. V-G Split-Point and Client Scalability Analysis Figure 6 combines the split-configuration study of Table I (a, c) with the client-scalability analysis (b, d). V3 gives the highest accuracy (69.75%), and V4 is nearly as accurate (69.18%) with the lowest round time, since a larger client share reduces the activations and gradients exchanged each step. The accuracy and efficiency optima therefore need not coincide, and V3 remains preferable when client resources bind. All K∈5,…,15K∈\5,…,15\ converge to a 65–70% accuracy band within 30 rounds (b), while the average per-round time on SLAKE (d) grows sub-linearly from 23.1 s (K=5K=5) to 35.0 s (K=15K=15), as fixed server-side computation is amortized across clients. V-H Summary of Findings USplit-VQA cuts client memory 5.75.7–5.8×5.8× versus FL on both backbones. For RQ1 and RQ2, the Custom model exceeds FL in average accuracy, whereas BiomedCLIP falls behind under the evaluated fixed split, indicating a backbone-dependent trade-off. For RQ3, the evaluated inversion attacks reconstruct inputs less faithfully under USplit-VQA than under centralized training or FL. For RQ4, CAWA suppresses one malicious client’s weight by 98%, although backdoor protection degrades with two or more attackers. Several limitations remain. The IID assumption does not reflect clinical heterogeneity, large pretrained backbones lose accuracy under the evaluated U-shaped split, and CAWA weakens as the malicious-client proportion increases. Future work will focus on non-IID adaptation and split-geometry optimization for large-scale transformers. VI Conclusion We introduced USplit-VQA, a U-shaped split learning framework tailored to VQA that keeps raw images, queries, and labels on the client while reducing client memory by up to 5.8×5.8× relative to FL. The Custom model improves average accuracy over FL, while BiomedCLIP exhibits an accuracy trade-off under the evaluated fixed split. CAWA strongly suppresses one malicious client, although its backdoor protection declines as the malicious-client proportion increases. The evaluated inversion attacks also produce lower reconstruction quality for USplit-VQA. Future research will consider non-IID data and improved split selection for large pretrained models. References [1] J. Zhang, J. Huang, S. Jin, and S. Lu (2024) Vision-language models for vision tasks: a survey. IEEE transactions on pattern analysis and machine intelligence 46 (8), p. 5625–5644. Cited by: §I. [2] J. J. Lau, S. Gayen, A. Ben Abacha, and D. Demner-Fushman (2018) A dataset of clinically generated visual questions and answers about radiology images. Scientific data 5 (1), p. 180251. Cited by: §I, §IV-D. [3] B. Liu, L. Zhan, L. Xu, L. Ma, Y. Yang, and X. Wu (2021) SLAKE: a semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), p. 1650–1654. Cited by: §I, §IV-D. [4] S. Zhang, Y. Xu, N. Usuyama, H. Xu, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, et al. (2025) A multimodal biomedical foundation model trained from fifteen million image–text pairs. NEJM AI 2 (1), p. AIoa2400640. Cited by: 3rd item, §I, §I, §IV-A1, §IV-A. [5] N. Rieke, J. Hancox, W. Li, F. Milletari, H. R. Roth, S. Albarqouni, S. Bakas, M. N. Galtier, B. A. Landman, K. Maier-Hein, et al. (2020) The future of digital health with federated learning. npj Digital Medicine 3 (1), p. 119. Cited by: §I, §I. [6] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas (2017) Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, Cited by: §I, §I-A. [7] P. Vepakomma, O. Gupta, T. Swedish, and R. Raskar (2018) Split learning for health: distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.00564. Cited by: §I, §I-A. [8] Y. Sun and H. Ochiai (2024) Bidirectional contrastive split learning for visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 21602–21609. Cited by: §I, §I. [9] O. Gupta and R. Raskar (2018) Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications 116, p. 1–8. Cited by: §I-A. [10] L. Zhu, Z. Liu, and S. Han (2019) Deep leakage from gradients. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: §I-B, §V-E. [11] Z. He, T. Zhang, and R. B. Lee (2019) Model inversion attacks against collaborative inference. In Proceedings of the 35th annual computer security applications conference, p. 148–162. Cited by: §I-B. [12] V. Tolpegin, S. Truex, M. E. Gursoy, and L. Liu (2020) Data poisoning attacks against federated learning systems. In European symposium on research in computer security, p. 480–501. Cited by: §I-B. [13] W. Dong, S. Shen, Y. Han, T. Tan, J. Wu, and H. Xu (2025) Generative models in medical visual question answering: a survey. Applied Sciences 15 (6), p. 2983. Cited by: §I. [14] H. Zhu, R. Togo, T. Ogawa, and M. Haseyama (2024) Prompt-based personalized federated learning for medical visual question answering. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1821–1825. Cited by: §I. [15] S. Lu, Z. Liu, T. Liu, and W. Zhou (2023) Scaling-up medical vision-and-language representation learning with federated learning. Engineering Applications of Artificial Intelligence 126, p. 107037. Cited by: §I. [16] C. Thapa, P. C. M. Arachchige, S. Camtepe, and L. Sun (2022) SplitFed: when federated learning meets split learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 36. Cited by: §I. [17] Z. Yang, Y. Chen, H. Huangfu, M. Ran, H. Wang, X. Li, and Y. Zhang (2022) Robust split federated learning for U-shaped medical image networks. arXiv preprint arXiv:2212.06378. Cited by: §I. [18] L. Wang, H. Chen, L. Zuo, and H. Liu (2024) U-shaped vertical split learning with local differential privacy for privacy preserving. In International Conference on Intelligent Computing, p. 72–81. Cited by: §I. [19] S. Lyu, Z. Lin, G. Qu, X. Chen, X. Huang, and P. Li (2023) Optimal resource allocation for U-shaped parallel split learning. In 2023 IEEE Globecom Workshops (GC Wkshps), p. 197–202. Cited by: §I. [20] D. Pasquini, G. Ateniese, and M. Bernaschi (2021) Unleashing the tiger: inference attacks on split learning. In Proceedings of the 2021 ACM SIGSAC conference on computer and communications security, p. 2113–2129. Cited by: §I. [21] M. Fang, X. Cao, J. Jia, and N. Gong (2020) Local model poisoning attacks to Byzantine-Robust federated learning. In 29th USENIX security symposium (USENIX Security 20), p. 1605–1622. Cited by: §I. [22] P. Blanchard, E. M. El Mhamdi, R. Guerraoui, and J. Stainer (2017) Machine learning with adversaries: byzantine tolerant gradient descent. Advances in neural information processing systems 30. Cited by: §I. [23] Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon (2021) Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH) 3 (1), p. 1–23. Cited by: §IV-A1. [24] S. Woo, J. Park, J. Lee, and I. S. Kweon (2018) CBAM: convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV), p. 3–19. Cited by: §IV-A2. [25] X. He, Y. Zhang, L. Mou, E. Xing, and P. Xie (2020) PathVQA: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286. Cited by: §IV-D. [26] D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham (2018) VizWiz grand challenge: answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 3608–3617. Cited by: §IV-D. [27] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli (2004) Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13 (4), p. 600–612. Cited by: §V-E.