Paper deep dive
Don't Send What You Don't Need: Question-Guided Token Pruning as a Privacy Defense for Vision-Language Models
Md Khalid Syfullah, Alvi Ataur Khalil
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Visual Question Answering (VQA) with Vision-Language Models (VLMs) is increasingly used in privacy-sensitive and bandwidth-constrained settings. Federated Learning (FL), Split Learning (SL), and U-Shaped Split Learning (USL) keep raw data local, but transmitting all visual tokens across a model partition remains costly and can expose private information. We propose QPriv-VL, a question-guided, privacy-aware token-pruning framework for FL, SL, and USL that prunes visual tokens before transmission based on task utility and privacy sensitivity. Its core component is a lightweight Dynamic Threshold Predictor (DTP) that jointly estimates a sample-specific pruning ratio and a token-level retention mask in one forward pass. DTP combines question relevance, computed from cross-modal similarity between visual patches and the pooled question embedding, with a sensitivity signal derived from frozen DINOv2 features. This allows the model to suppress potentially sensitive regions while preserving patches useful for answering the question, without requiring sensitivity labels. We evaluate QPriv-VL on GQA, OK-VQA, VQAv2, SLAKE, VQA-RAD, and PathVQA against four privacy attack families: FSHA, FORA, iDLG, and attribute-inference membership inference attacks. DTP matches or outperforms fixed-ratio pruning while using substantially fewer transmitted tokens. On VQA-RAD, it reduces membership-inference attack success from 0.99 to 0.76-0.79, lowers FSHA and FORA reconstruction PSNR relative to fixed-ratio pruning, and preserves competitive VQA accuracy using about 40% of the original visual-token budget. A sensitivity exclusion ratio of 1.20 +/- 0.18 indicates preferential removal of privacy-sensitive patches, while explainability analysis shows that retention adapts to question semantics rather than generic visual saliency.
Tags
Links
- Source: https://arxiv.org/abs/2609.15671v1
- Canonical: https://arxiv.org/abs/2609.15671v1
Trouble viewing inline? Open PDF directly →
Full Text
78,714 characters extracted from source content.
Expand or collapse full text
Don’t Send What You Don’t Need: Question-Guided Token Pruning as a Privacy Defense for Vision-Language Models Md Khalid Syfullah Alvi Ataur Khalil Affiliation: Transformative Innovation for Trustworthy AI and Network Security (TITANS) Lab, Affiliation: Computer Science, Southern Illinois University Carbondale, USA Affiliation: mdkhalid.syfullah, a.khalil@siu.edu Abstract Visual Question Answering (VQA) with Vision-Language Models (VLMs) is increasingly deployed in privacy-sensitive and bandwidth-constrained settings. Distributed paradigms such as Federated Learning (FL), Split Learning (SL), and U-Shaped Split Learning (USL) keep raw data local; however, transmitting all visual tokens across the cut layer remains costly and vulnerable to privacy attacks. We propose QPriv-VL, a question-guided and privacy-aware token-pruning framework that operates uniformly across all three distributed setups and serves as a privacy defense by pruning visual tokens before they cross the cut layer using a combined measure of utility and sensitivity. At the core of QPriv-VL is a novel lightweight Dynamic Threshold Predictor (DTP), which, as an adaptive defense module, jointly predicts a per-sample prune ratio and a per-token retention mask in a single forward pass. DTP balances question relevance, estimated through a crossmodal comparison between each visual patch representation and the pooled question embedding, against a content-sensitivity signal derived from frozen features of DINOv2, a self-supervised vision transformer. DINOv2 enables DTP to identify visually sensitive regions while preserving question-relevant content via object-aware patch representations without labelled supervision. We evaluate QPriv-VL on three general-domain benchmarks (GQA, OKVQA, VQAv2), three medical benchmarks (SLAKE, VQA-RAD, Path-VQA), and four attack families (FSHA, FORA, iDLG, attribute-inference MIA). DTP matches or surpasses fixed-ratio pruning as a defense method while retaining substantially more tokens, reducing MIA attack success on VQA-RAD from 0.990.99 to 0.760.76–0.790.79, lowering FSHA and FORA reconstruction PSNR below the linear fixed-ratio trend, and preserving competitive VQA accuracy at roughly 40%40\% of the original token budget. A sensitivity exclusion ratio of 1.20±0.181.20± 0.18 shows that DTP preferentially removes privacy-sensitive patches, while explainability analysis confirms retention adapts to question semantics rather than generic saliency. Index Terms: Visual question answering, security in vision-language models, distributed learning, knowledge distillation I Introduction Visual Question Answering (VQA) is a prominent multimodal reasoning task where a model answers a natural language question based on the content of an image [1, 2]. Modern VQA systems are primarily powered by large Vision Language Models (VLMs) such as LLaVA [3, 4], BLIP2 [5], and Flamingo [6], which combine a Vision Transformer (ViT) [7] encoder with a large language model via a cross-modal projector. While these models deliver high accuracy, their compute demands make on-device inference difficult. Beyond raw compute, deployment is further complicated by privacy and throughput: many practical VQA applications operate in privacy-sensitive and bandwidth-constrained environments where transmitting raw visual data or deploying large models on-device is infeasible, such as multi-institutional medical VQA [8, 9, 10] and on-device assistive systems. These challenges have motivated distributed training paradigms, notably Federated Learning (FL) [11] and Split Learning (SL) [12]. FL preserves data locality by aggregating model updates without sharing raw data, whereas SL partitions the model between client and server, transmitting only intermediate activations across a cut layer. U-Shaped Split Learning (USL) extends SL with a second cut layer, keeping both labels and the question head on the client side [13]. While suited for VLM-based VQA, these paradigms introduce three key challenges. First, communication overhead is substantial: a CLIP-ViT-L/14 encoder at 336×336336×336 resolution produces 576576 visual tokens with 10241024-dimensional embeddings, making transmission across the SL cut prohibitively expensive [14, 15]. Second, intermediate activations and gradient updates are vulnerable to feature-inversion, gradient-leakage, and membership-inference attacks [16, 17, 18, 19], against which existing activation-level defenses reduce utility while ignoring that many transmitted tokens carry no question-relevant information. Third, VLM analyses show that only a small subset of visual tokens contributes after the early language-model layers [20, 21], meaning transmitting the full prefix inflates both communication cost and attack surface. These challenges motivate a simple observation: only a subset of visual tokens is necessary to cross the cut layer. Selecting task-relevant and privacy-preserving tokens before transmission removes redundancy, shrinks the attack surface, and preserves semantically meaningful information. Recent question-guided and adaptive token-pruning approaches for VLMs reduce the visual prefix by two-thirds or more [20, 21, 22, 23, 24, 25, 26]. However, these methods are evaluated as centralized inference accelerators and, to our knowledge, remain unexplored as privacy defenses or jointly trained with content-sensitivity signals. Existing SL compression approaches are largely task-agnostic and do not exploit the client-side question features. To bridge this gap, we introduce QPriv-VL, a Question-guided and Privacy-aware token-pruning framework for Vision-Language models, built on LLaVA-1.5-7B [4], which combines a CLIP-ViT visual encoder with a Llama-2 language backbone via a lightweight Multi Layer Perception (MLP) projector. QPriv-VL operates as a unified client-side pruning framework for Centralized Training (CL), FL, SL, and USL, using the same decision mechanism regardless of cut placement. A single checkpoint shortens the visual prefix, reduces communication cost across the cut, and limits the information exposed to curious or malicious servers. The framework centers on a lightweight client-side Dynamic Threshold Predictor (DTP), where one forward pass produces a per-sample pruning ratio (how many tokens leave the client) and a per-token keep-mask (which tokens are retained). Our contribution in this work is four-fold: • We introduce a lightweight, privacy-aware DTP module that decides how many and which visual tokens to keep, jointly weighing question relevance against a learned content-sensitivity signal. DTP generalizes across general and medical domains without per-domain retraining. • We present a unified implementation of QPriv-VL on LLaVA-1.5-7B that supports both fixed and dynamic thresholds, letting the same client-side module act as a communication-efficiency layer and a privacy defense under distributed settings. • We provide a systematic privacy evaluation of QPriv-VL, showing robustness against four representative attack families: FSHA [16], FORA [17], iDLG [18], and MIA [19], on both general-domain and medical VQA benchmarks. • We investigate the utilised VLM model’s internal decision process via transformer-based attention visualization [27], using case studies with in and out-of-context queries. This work answers four research questions: • RQ1: Does QPriv-VL simultaneously reduce communication cost, shrink the attack surface, and preserve accuracy across all four learning settings? • RQ2: Does a privacy-aware dynamic threshold predictor yield a better accuracy/communication/privacy Pareto trade-off than fixed-ratio or training-free pruning? • RQ3: Does QPriv-VL specifically remove tokens that are both reconstructable and privacy-relevant, rather than merely degrading performance? • RQ4: Can its effectiveness be verified through explainability analysis? The remainder of this paper is organized as follows. Section I presents the background. Section I discusses related work. Section IV introduces the proposed framework. Section V outlines the methodology. Section VI reports the experimental findings and section VII concludes the paper. I Background In this section, we discuss preliminary concepts to facilitate an understanding of distributed learning, attacks, and the positioning of our proposed QPriv-VL framework. I-A VQA and VLMs VQA takes an image together with a natural-language question and produces a free-form textual answer [1]. Modern systems such as LLaVA [3, 4], BLIP-2 [5], and Flamingo [6] realise this mapping with a VLM that pairs a visual encoder with a language model through a small projection module, such that visual features are translated into a form the language model can attend to alongside the question tokens. We use LLaVA-1.5-7B [4] as our reference architecture because its CLIP-ViT-L/14 encoder, MLP projector, and Vicuna language model expose a clean interface between modalities and have an established literature on visual-token pruning [20, 21, 22, 28]. I-B Self-Supervised Visual Representations Self-supervised visual representation learning trains a ViT on a large unlabelled image corpus by predicting one view of an image from another. The resulting per-patch features carry strong semantic locality and transfer well across visual domains, including medical imagery, without retraining. We adopt DINOv2 [29] as the self-supervised backbone behind our sensitivity branch, since it offers high-quality patch features at a small computational cost and remains frozen. Fig. 1: Overview of the four attacks considered in this work. I-C Training Paradigms We compare four training paradigms differing in how data and the model are partitioned between client and server. CL The classical setting where a single trainer holds the entire dataset and updates a single copy of the model. CL serves as the no-distribution upper bound: nothing crosses a network boundary, eliminating any leakage channel, though raw data must be aggregated in one place. FL [11] Each client keeps its data local and trains a local copy. Clients periodically send parameter updates to a server, which aggregates them and broadcasts the result. FL preserves data locality; transmitted updates can still leak information through gradient-inversion attacks such as DLG and iDLG [30, 18]. SL [12] The model is partitioned at a single cut layer. The client owns the lower part and the input data; the server owns the upper part and the loss computation. Only the intermediate activations at the cut and the back-flowing gradients are exchanged. The smashed activations become the primary target for feature-inversion attacks [16, 17]. USL [13] USL extends SL with a second cut so both ends of the network remain on the client side: the client retains the embedding layers, the language-model head, and the labels, while the server operates only on the middle block. This topology removes several leakage channels at the cost of an extra round of communication per step. I-D Attacks on VLMs A central privacy concern in distributed learning is that an honest-but-curious or malicious server can recover the input, its label, or a sensitive attribute from exchanged information (e.g., parameter updates in FL or smashed activations and back-flowing gradients at the cut layer(s) in SL and USL). We adopt four attacks spanning the threat space (Figure 1). FSHA Pasquini et al. [16] introduce an active attack where an attacker-controlled module replaces the legitimate server and hijacks training: a discriminator on the attacker side covertly aligns the client’s smashed representations with embeddings of the attacker’s public proxy images, while an inverter trained jointly against this aligned space reconstructs the client’s private input (Figure 1(a)). The attack defeats distance-correlation defenses because the alignment objective collapses the distances those defenses regularise. FORA Xu et al. [17] propose a passive reconstruction attack in which an honest-but-curious server collects smashed activations from a trained client model and trains an inverter on a same-domain public auxiliary set, using its copy of the client encoder to generate matched feature/image pairs (Figure 1(b)). Because the inverter is fit against unpruned, undefended representations, FORA acts as an oracle attacker that bounds what any adversary could recover from the deployed system. MIA Following Shokri et al. [19], an adversary trains shadow models on data from the target’s distribution and uses their confidence patterns to build a binary classifier that predicts whether a candidate sample was in the training set (Figure 1(c)). The signal exploited is the well-known tendency of models to be more confident on samples they have seen during training, providing a leakage channel orthogonal to reconstruction. iDLG DLG [30] optimises a dummy input whose synthetic gradient matches an observed gradient update, iteratively refining a noise initialisation until the reconstructed gradient agrees with the intercepted one and the dummy input converges to the private image; iDLG [18] additionally recovers the label analytically from the sign of the final-layer gradient. The attack applies to FL on transmitted parameter updates and to SL by inverting the gradient flowing back across the cut (Figure 1(d)). I Literature Review QPriv-VL sits at the intersection of two largely separate lines: question-guided token pruning for VLMs targeting centralised inference efficiency, and privacy-preserving distributed deep learning, whose defenses operate on activations or gradients without semantic token control. Question-guided and adaptive token pruning is directly related. FastV [20] prunes low-attention tokens in deeper LLM layers; SparseVLM [21] and ATP-LLaVA [22] add text-conditioned per-sample budgets; QG-VTC [23] learns question-conditioned token scores; and CROP [24] adapts retention to query-relevant regions. PyramidDrop [25] and VisionZip [26] interleave dropping with progressive selection across LLM depths, while LLaVA-PruMerge [28] combines outlier-based selection with attention-guided merging. These methods show two-thirds or more of the visual prefix is redundant for VQA, yet place the decision inside the LLM and serve only as centralised inference accelerators. None positions it before a network boundary, co-trains it with a content-sensitivity signal, or measures its effect on inversion-based or membership-inference attackers. Privacy in distributed deep learning has been extensively studied for CNN-style architectures. Feature-inversion attacks [16, 17] reconstruct private inputs from smashed activations, and gradient-leakage attacks [30, 18] invert gradient updates; defenses such as NoPeek [31], DP-CutMixSL [32], and PATROL [33] mitigate leakage via distance-correlation regularisation, sample mixing, or channel pruning. Communication-efficient SL methods such as BottleNet [14] and ADC [34] cut activation bandwidth via feature compression or attention-driven dropping, though without question conditioning. For multimodal systems, FedMultimodal [35] and pFL-MedVQA [36] focus on FL parameter aggregation, while BiCSL [37] combines SL and FL for VQA under a contrastive privacy objective without token pruning, leaving the visual prefix and its reconstructible surface untouched. Overall, existing defenses target CNN-style activations or gradients without exploiting question semantics, so multimodal extensions still transmit the full visual prefix and leave its reconstructible surface largely unaddressed. Self-supervised vision transformers DINO [38] and DINOv2 [29] learn label-free, object-aware patch representations transferable across natural and medical domains. For faithfulness evaluation, transformer-aware attention propagation [27] and deletion-based probes [39, 40] are commonly used in VLM explainability analysis. Together, these tools provide a label-free signal for identifying privacy-relevant regions and a principled means of verifying that pruning decisions reflect question-driven reasoning rather than generic saliency. Fig. 2: The proposed QPriv-VL framework. Prior work studies question-guided VLM pruning, privacy-preserving SL/FL, and self-supervised sensitivity modelling separately. VLM pruning methods focus on inference cost in centralised settings, while privacy defenses protect CNN-style features or gradients without controlling semantic tokens. Distributed multimodal methods transmit the full visual sequence. QPriv-VL addresses this gap by combining question-guided token pruning with learned per-sample ratios and token-level keep-masks, integrating DINOv2-based patch sensitivity, evaluating a unified policy across CL, FL, SL, and USL, and analysing robustness against four attacks. IV Proposed Framework The QPriv-VL framework performs Question-Guided Token Pruning (QGTP) and sits between the frozen vision-language encoders and the LLaVA multi-modal projector (Figure 2). Given an image-question pair, it determines both the quantity and selection of visual patches to pass forward, blocking question-irrelevant or privacy-sensitive content from leaving the client. This same framework operates unchanged across CL, FL, SL, and USL, requiring no architectural adjustments. IV-A Frozen Encoders Three pre-trained backbones produce fixed feature representations. The CLIP vision encoder (CLIP-V) turns the image into a grid of patch tokens and a pooled image summary. The CLIP text encoder (CLIP-T) turns the question into a pooled vector. A self-supervised DINOv2 encoder produces a grid of patch features emphasising object-level structure independent of text supervision; these feed the privacy-sensitivity signal. IV-B Utility Branch The utility branch assigns a question-relevance score to each CLIP patch. The pooled question embedding and CLIP patch tokens are projected into a shared low-dimensional space via query and key projections. A scaled dot-product attention followed by a sigmoid then produces a utility score per patch, emphasising regions most relevant to the question. IV-C DTP Module The DTP module fuses utility with privacy sensitivity, predicts the per-sample prune ratio, and is the only learnable component of the pipeline. It groups three sub-components: Sensitivity Branch DINOv2 patch features are passed through a small per-patch MLP (the sensitivity head), reshaped onto the DINOv2 grid, and bilinearly upsampled onto the CLIP grid. Every CLIP patch therefore carries a utility and a sensitivity score on the same spatial layout. Differentiable Mask Path A combined score is computed by subtracting weighted sensitivity from weighted utility, followed by a sigmoid with a learnable sharpness parameter to generate a soft per-patch keep-mask in (0,1)(0,1). During training, this mask acts as a differentiable distillation target; at inference it is replaced with a discrete top-k token selection. Threshold Holder A compact summary of the sensitivity field (mean, maximum, fraction above a threshold) is concatenated with the pooled image and question descriptors, and a small MLP outputs the per-sample prune ratio, the only quantity varying per sample at inference. IV-D Pruning Decision Branch The pruning decision branch selects one of three policies for each forward pass: off (no pruning, ρ=0ρ=0, used as the no-defense baseline), Fixed (a manually chosen ρ∈0.3,0.5,0.7,0.9ρ∈\0.3,0.5,0.7,0.9\), or DTP. The chosen ratio drives a top-k selection over the utility scores produced by the utility branch. IV-E Output The retained patches are assembled into a variable-length visual prefix, which is passed through LLaVA’s multi-modal projector and combined with the question token embeddings before being processed by the language model. To preserve sufficient visual information under aggressive pruning, a safety threshold of kmin=16k_ =16 tokens is enforced, ensuring the answer generator always receives a minimum amount of image context. From the LLM’s perspective, the primary modification is the shorter, more selectively curated visual prefix. During downstream fine-tuning, only the language model is adapted using LoRA adapters [41], the pruning module remains fixed. V Methodology This section gives the technical specification of every component including the QPriv-VL framework, the DTP network and its preparation pipeline, the four training configurations, and the robustness and privacy probes used in the evaluation. Table I summarises the notation used throughout the paper. TABLE I: Summary of notation used throughout the paper. Symbol Description Ib,qb,ab⋆I_b,q_b,a _b Image, question, and answer for sample b N Number of CLIP-V patch tokens (N=576N=576, 24×2424×24 grid) VbV_b CLIP-V patch tokens, Vb∈ℝN×DvV_b ^N× D_v, Dv=1024D_v=1024 vbpoolv_b^pool CLIP-V pooled image feature, vbpool∈ℝDvv_b^pool ^D_v tbt_b CLIP-T pooled question embedding, tb∈ℝDtt_b ^D_t, Dt=768D_t=768 dbd_b DINOv2 patch tokens, db∈ℝ256×Ddd_b ^256× D_d, Dd=384D_d=384 ψ LLaVA multi-modal projector Wq,WkW_q,W_k Scorer projections to shared dim P=256P=256 ub,nu_b,n Question-conditioned utility score for patch n HsensH_sens Per-patch sensitivity MLP sb,mdinos^dino_b,m Per-patch sensitivity on DINOv2 grid sbaligneds^aligned_b Sensitivity upsampled to the CLIP 24×2424×24 grid cb,nc_b,n Combined score αub,n−βsb,nalignedα u_b,n-β s^aligned_b,n mb,nm_b,n Soft keep-mask, mb,n=σ(γcb,n)m_b,n=σ(γ c_b,n) α,βα,β Utility/sensitivity weights (buffers, α=β=1α=β=1) γ Learnable mask sharpness (initialised to 3.03.0) ρb,ρ^b _b, ρ_b True/predicted per-sample prune ratio ρb⋆ρ _b Oracle prune ratio from LLaVA-in-the-loop search kb,kmink_b,k_ Number of retained tokens; safety floor (kmin=16k_ =16) bS_b Index set of retained patches MLPτMLP_τ Threshold predictor producing ρ^b ρ_b θ DTP parameters Wq,Wk,Hsens,MLPτ,γ\W_q,W_k,H_sens,MLP_τ,γ\ ℒsens,ℒτL_sens,L_τ Teacher sensitivity and threshold losses ℒρ,ℒm,ℒKDL_ρ,L_m,L_KD Student ratio, mask, and combined KD losses vqa,privD_vqa,D_priv VQA and auxiliary privacy training sets L Number of LLM decoder layers (L=32L=32) C,CA,CBC,C_A,C_B Split cut layers (SL: C=16C=16; USL: CA=8C_A=8, CB=24C_B=24) NC,TlN_C,T_l Number of clients and local steps per round G Oracle prune-ratio search grid V-A DTP Preparation The DTP preparation pipeline (Algorithm 1) has three stages: (i) feature pre-computation, (i) oracle prune-ratio search, and (i) teacher-student Knowledge Distillation. Training uses six VQA benchmarks vqaD_vqa spanning natural images (VQAv2 [42], GQA [2], OK-VQA [43]) and medical imaging (SLAKE [9], VQA-RAD [8], PathVQA [10]), each capped at 3,0003,000 samples. The sensitivity branch HsensH_sens is trained on a separate auxiliary set privD_priv with image-level privacy labels drawn from CelebA, NIH Chest X-Ray, EuroSAT (positive bags) and MNIST, CIFAR-10 (negative bags); these labels are used only for HsensH_sens and never enter the VQA loop. Feature pre-computation For each sample (Ib,qb,ab⋆)∈vqa(I_b,q_b,a _b) _vqa, the three frozen encoders are evaluated once and the tuple (Vb,vbpool,tb,db)(V_b,v_b^pool,t_b,d_b) is stored in half precision for reuse in later stages. Oracle prune-ratio search For each sample we identify the smallest prune ratio under which LLaVA-1.5-7B still recovers the gold answer. Patches are ranked by a fixed, randomly initialised scorer u~ u. For each ρ∈=0.10,0.20,0.30,0.50,0.70ρ =\0.10,0.20,0.30,0.50,0.70\ in ascending order, the top-⌈(1−ρ)N⌉ (1-ρ)N patches are passed through LLaVA’s projector ψ and language model; the prediction is matched against ab⋆a _b under a first-token substring criterion. The smallest matching ρ defines ρb⋆ρ _b; if no value matches, ρb⋆=1.0ρ _b=1.0. The search is capped at 500500 samples per dataset. Teacher-student knowledge distillation This stage is the largest and consists of three parts: teacher training, the rationale for distillation, and student distillation. Teacher training: The teacher shares DTP’s forward equations, training only HsensH_sens and MLPτMLP_τ; (Wq,Wk)(W_q,W_k) remain at random initialisation and mask sharpness is fixed at γ=3γ=3. The two heads are trained sequentially. With image-level labels yb∈0,1y_b∈\0,1\ from privD_priv and y^b=maxmsb,mdino y_b= _ms^dino_b,m, HsensH_sens minimises a binary cross-entropy loss for 1010 epochs (AdamW, lr 3×10−43×10^-4, weight decay 10−410^-4): ℒsens=−1B∑b[yblogy^b+(1−yb)log(1−y^b)].L_sens=- 1B _b [\,y_b y_b+(1-y_b) (1- y_b)\, ]. (1) With HsensH_sens frozen, MLPτMLP_τ then regresses the oracle ratio on vqaD_vqa for 2020 epochs (AdamW, lr 10−410^-4, weight decay 10−210^-2, dropout 0.10.1, batch size 6464): ℒτ=1B∑b(ρ^b−ρb⋆)2.L_τ= 1B _b ( ρ_b-ρ _b )^2. (2) Rationale: The oracle ratio ρb⋆ρ _b is costly because it requires a per-sample LLaVA-based grid search over |||G| candidates and is discrete since it can only take values on the search grid. In addition, the privacy-supervised teacher head HsensTH^T_sens is trained on a separate auxiliary set privD_priv that is not available to the student. As a result, neither signal can directly serve as the sole supervision for a unified differentiable student that must jointly predict ρ^b ρ_b and a soft keep-mask mbm_b in a single forward pass. Knowledge distillation addresses this by transferring both signals through end-to-end training: the soft teacher outputs provide smoother supervision beyond the discrete oracle ratios, while the hard oracle targets regularise the model against teacher miscalibration and oracle saturation cases. Student distillation: DTP is initialised from the teacher’s HsensH_sens and MLPτMLP_τ, with Wq,WkW_q,W_k random and γ=3.0γ=3.0, and is trained end-to-end against both the teacher’s outputs (ρbT,mbT)(ρ^T_b,m^T_b) and the hard oracle ρb⋆ρ _b: ℒρ _ρ =12(ρ^b−ρbT)2+12(ρ^b−ρb⋆)2, = 12( ρ_b-ρ^T_b)^2+ 12( ρ_b-ρ _b)^2, (3) ℒm _m =BCE(mb,mbT), =BCE(m_b,m^T_b), (4) ℒKD _KD =λρℒρ+λmℒm, = _ρ\,L_ρ+ _m\,L_m, (5) with λρ=λm=1 _ρ= _m=1. The mask term drives (Wq,Wk)(W_q,W_k) to mimic the teacher’s saliency. Training runs for 2020 epochs (AdamW, lr 3×10−43×10^-4 with cosine annealing, batch size 128128). Algorithm 1 DTP preparation pipeline. Input : vqaD_vqa, privD_priv, prune-ratio grid G Output : DTP param. θ=Wq,Wk,Hsens,MLPτ,γθ=\W_q,W_k,H_sens,MLP_τ,γ\ 1 foreach (Ib,qb)∈vqa(I_b,q_b) _vqa do 2 (Vb,vbpool,tb,db)←Enc(Ib,qb)(V_b,v_b^pool,t_b,d_b) (I_b,q_b) 3 end foreach 4 foreach b∈vqab _vqa do 5 ρb⋆←1.0ρ _b← 1.0 6 foreach ρ∈ρ in ascending order do 7 k←⌈(1−ρ)N⌉k← (1-ρ)\,N 8 ←top-k(u~(Vb,tb))S -k ( u(V_b,t_b) ) 9 if LLaVA(ψ(Vb[]),qb)≈ab⋆LLaVA (ψ(V_b[S]),q_b )≈ a _b then 10 ρb⋆←ρ _b←ρ 11 break 12 end if 13 end foreach 14 end foreach 15 HsensT←argminHsensℒsens(priv)H_sens^T← _H_sensL_sens(D_priv) 16 MLPτT←argminMLPτℒτ(vqa)MLP_τ^T← _MLP_τL_τ(D_vqa) 17 θ←init(HsensT,MLPτT,rand(Wq,Wk),γ=3.0)θ (H_sens^T,\,MLP_τ^T,\,rand(W_q,W_k),\,γ=3.0 ) 18 θ←argminθℒKD(vqa)θ← _θL_KD(D_vqa) 19 return θ V-B QPriv-VL Framework Architecture Fig. 3: Dataset distribution, backbone model, and system architecture across the four training configurations. We summarise the dimensions and equations defining the QPriv-VL framework from Section IV. CLIP-V produces Vb∈ℝN×DvV_b ^N× D_v from its penultimate layer (N=576N=576, Dv=1024D_v=1024), along with pooled features vbpool∈ℝDvv_b^pool ^D_v. CLIP-T [44] produces tb∈ℝDtt_b ^D_t with Dt=768D_t=768. DINOv2 [29] produces db∈ℝ256×Ddd_b ^256× D_d on a 16×1616×16 grid with Dd=384D_d=384. The cross-modal scorer projects features into a shared P=256P=256 space via bias-free projections with a scaled dot-product: qb=Wqtb,Kb=VbWk⊤,ub,n=σ(qb⊤Kb,n/P).q_b=W_qt_b,\ K_b=V_bW_k ,\ u_b,n=σ (q_b K_b,n/ P ). (6) The sensitivity head Hsens:ℝDd→(0,1)H_sens\!:R^D_d\!→\!(0,1) has hidden widths 384→128→128→1384→128→128→1 with GELU activations [45], producing sb,mdinos^dino_b,m, which is bilinearly upsampled onto the CLIP grid as sbaligned∈(0,1)Ns^aligned_b∈(0,1)^N. The fused score and soft mask are defined as cb,n=αub,n−βsb,naligned,mb,n=σ(γcb,n),c_b,n=α\,u_b,n-β\,s^aligned_b,n, m_b,n=σ(γ\,c_b,n), (7) where α=β=1α=β=1 are fixed buffers and γ is a learnable sharpness parameter initialised to 3.03.0. The threshold predictor concatenates global descriptors into xb∈ℝ1795x_b ^1795 and applies a 1795→512→512→11795→512→512→1 MLP (GELU, dropout 0.10.1): xb=[vbpool;tb;summ(sbdino)],ρ^b=σ(MLPτ(xb)),x_b=[\,v_b^pool;\,t_b;\,summ(s^dino_b)\,], ρ_b=σ (MLP_τ(x_b) ), (8) where summ(sbdino)=[s¯,maxs,1256|sb,m>0.5|]summ(s^dino_b)=[\, s,\, s,\, 1256|\s_b,m>0.5\|\,]. The controller performs a safety-floored top-k selection: kb=max(kmin,round((1−ρb)N)),b=sort(topk(ub,kb)),k_b= \! (k_ ,\,round((1- _b)N) ),\ S_b=sort\! (topk(u_b,k_b) ), (9) with kmin=16k_ =16. The selected patches are passed through LLaVA’s projector ψ and concatenated with surrounding text embeddings as Eb=[ebpre;ψ(Vb[b,:]);ebpost]E_b=[\,e^pre_b;\,ψ(V_b[S_b,:]);\,e^post_b\,]. The learnable parameters θ=Wq,Wk,Hsens,MLPτ,γθ=\W_q,W_k,H_sens,MLP_τ,γ\ form the DTP network, which jointly predicts ρ^b ρ_b and mbm_b in a single forward pass and contains ≈1.7≈ 1.7M parameters (mainly MLPτMLP_τ with ≈1.18≈ 1.18M and scorer projections with ≈459≈ 459K). During downstream fine-tuning, only LoRA adapters [41] are trained on the seven attention and MLP projections Wq,Wk,Wv,Wo,Wgate,Wup,Wdown\W_q,W_k,W_v,W_o,W_gate,W_up,W_down\ in each decoder layer at rank 3232, while base LLaVA and QPriv-VL remain frozen. V-C Training Configurations Figure 3(a) summarises the data distribution across the six VQA benchmarks together with the shared LLaVA-1.5-7B backbone and the LoRA placement that is common to every configuration; Figure 3(b)-(e) shows the four training paradigms evaluated in this work. All paradigms use the same backbone, LoRA configuration, and optimisation schedule, and differ only in how the model and data are divided between client and server. CL A single trainer holds the entire training set and updates the LoRA adapters via mini-batch SGD with gradient accumulation; no partitioning is performed (Figure 3(b)). FL NCN_C clients hold non-overlapping shards. Each round, the server broadcasts the global LoRA state, every client trains locally for TlT_l micro-batches, and the server applies FedAvg [11] to aggregate the updates (Figure 3(c)). Optimiser state is not shared; a single global cosine schedule is used to remain comparable to CL. SL The Llama-2-7B decoder is cut at layer C=16C=16, with the client owning [0,C)[0,C) together with the QPriv-VL framework, projector, and embeddings, and the server owning [C,L)[C,L) together with the LM head (Figure 3(d)). Within each round, NCN_C clients train sequentially against the shared server. The cut is implemented via forward hooks on the boundary layer, preserving rotary embeddings and causal masking. USL The decoder is cut at CA=8C_A=8 and CB=24C_B=24, splitting it into a client-head, a server-body, and a client-tail (Figure 3(e)). The server therefore never sees the raw image, question, answer labels, or LM-head logits; only the activations at the two cut points are server-visible. Client-head and client-tail LoRA weights are private per client, while the server-body LoRA is shared; per-client AdamW states prevent statistical leakage across clients. V-D Robustness and Privacy Probes Beyond standard VQA accuracy, we design two probes targeting robustness and privacy sensitivity. The irrelevant-question probe pairs each test image with three categories of off-topic queries: (i) cross-image questions, where a question from one sample is paired with a different image, (i) manually constructed nonsensical questions such as “What is the capital of France?”, and (i) false-premise questions referring to objects absent from the image. Responses containing refusal or uncertainty phrases from a curated set ℛR (e.g., “not visible”, “cannot determine”) are classified as refusals; remaining non-empty responses are treated as hallucinations. We compute hallucination and refusal rates over ℐ∗⊆ℐI^* , the samples with a non-empty generation: Halluc. . =1|ℐ∗|∑i∈ℐ∗[y^i∉ℛ],Refusal = 1|I^*| _i ^*1\! [ y_i ],Refusal =1|ℐ∗|∑i∈ℐ∗[y^i∈ℛ]. = 1|I^*| _i ^*1\! [ y_i ]. (10) To analyse privacy-sensitive behaviour, we additionally construct a privacy-relevant subset per dataset by ranking test images according to the maximum DINOv2-derived sensitivity score smax(x)=maxpsp(x)s_ (x)= _ps_p(x) across image patches and selecting the top-N samples (the sensitivity-training threshold smax>0.5s_ >0.5 is met by essentially all retained samples on medical datasets). On this subset, we evaluate the sensitivity exclusion ratio: ExclRatio=p∈[sp]p∈[sp],ExclRatio= E_p \! [s_p ]E_p \! [s_p ], (11) where K and D denote the sets of retained and pruned patches, respectively. Values above one indicate that the policy preferentially removes highly sensitive regions, whereas values near one suggest content-agnostic dropping. We further report closed-question exact-match accuracy and open-question recall on the same privacy-sensitive subset to assess the trade-off between privacy preservation and downstream task performance. VI Experimental Results This section reports the experimental setup, then walks through the qualitative behaviour of the proposed pruning policy, the VQA accuracy of QPriv-VL across the four training paradigms and its robustness to four representative privacy attacks. Later it presents an explainability analysis of the learned policy, followed by an aggregated probe on off-topic robustness and sensitivity-aware pruning. Finally, the section concludes through a discussion of remaining limitations. VI-A Experimental Setup All experiments are implemented in PyTorch on two NVIDIA RTX PRO 6000 Blackwell GPUs, benchmarking QPriv-VL against CL, FL, SL, and USL under a shared training configuration: LoRA adapters (r=32r=32, α=64α=64, dropout 0.050.05); AdamW (β1=0.9 _1=0.9, β2=0.999 _2=0.999, weight decay 00) with peak learning rate 1×10−41×10^-4, cosine decay, 5%5\% warmup, and ℓ2 _2 gradient clipping at 1.01.0. The budget is R=20R=20 rounds of 100100 micro-batches, applied as 100100 sequential updates in CL or distributed across NC=5N_C=5 clients with Tl=20T_l=20 local steps in FL/SL/USL. We use micro-batch size 1616 with accumulation factor 22 (effective 3232) in bfloat16, greedy decoding up to 2020 tokens, and early stopping (patience 55, minimum improvement 0.0050.005 on validation accuracy) with the best-validation checkpoint restored for testing. VI-B Qualitative Visualisation of DTP Intermediate Signals Figure 4 visualises four intermediate signals from one randomly selected sample in each of five VQA datasets: the per-patch sensitivity map s from the DINOv2-supervised head, the question-conditioned utility score u, the combined score αu−βsα u-β s, and the final keep-mask m. Three patterns emerge: (i) low-sensitivity natural images (VQAv2, maxpsp=0.80 _ps_p=0.80, 565565/576576 retained) yield diffuse distributions and high retention; (i) privacy-sensitive natural images such as the OK-VQA example with identifiable faces (maxpsp=0.99 _ps_p=0.99, 479479/576576 retained) produce a tighter keep-mask despite the generic question, indicating that DTP also responds to image content; (i) medical samples (SLAKE, VQA-RAD, PathVQA, maxpsp≈1.00 _ps_p≈1.00) show concentrated sensitivity peaks and reduce the keep-mask to 44–2020 diagnostically relevant patches. Across all five examples, the DTP keep-mask closely follows the high-response regions of the combined score αu−βsα u-β s in Figure 4, suggesting that DTP preserves the content-aware pruning behaviour while maintaining lightweight inference cost for deployment. Fig. 4: DTP intermediate signals (sensitivity, utility, combined score, keep-mask) on one sample per dataset. VI-C Adaptive vs. Fixed Token Pruning In this part, we evaluate DTP against fixed-rate top-k pruning at comparable token counts to show whether its per-sample, content-aware budget retains more semantically useful regions. For each input image, the frozen CLIP-ViT-L/14 encoder produces N=576N=576 visual patch tokens on a 24×2424×24 grid, and the QPriv-VL framework assigns a question-conditioned utility score to each token while predicting a per-sample pruning ratio ρ ρ via the student head. Figure 5 visualises the keep-masks for representative samples from VQA-RAD, GQA, OK-VQA, and PathVQA, comparing the learned DTP policy with the off baseline and fixed-rate settings ρ∈0.3,0.5,0.7,0.9ρ∈\0.3,0.5,0.7,0.9\, with each panel reporting the retained token count k and the predicted ρ ρ. DTP retains 305305–342342 tokens (ρ^≈0.41 ρ≈0.41–0.470.47) and consistently preserves semantically relevant regions, abdominal structures in CT, clothing in GQA, foreground subjects in OK-VQA, and tissue regions in PathVQA, whereas fixed-rate pruning removes informative regions uniformly without regard to question relevance. VI-D Accuracy-Token-Budget Trade-off over Training Paradigm Tables I and I report test accuracy on three general-domain benchmarks (GQA, OKVQA, VQAv2) and three medical benchmarks (SLAKE, Path-VQA, VQA-RAD) under Direct, Federated, Split, and U-Split, with token budgets ranging from 576576 to nearly 10%10\% of the visual prefix. Both Fixed and DTP rank tokens by the same question-conditioned utility score with top-k selection, differing only in budget: Fixed applies a manual ratio, whereas DTP predicts a sample-specific ρ^b ρ_b via the sensitivity-conditioned MLPτMLP_τ. DTP is deterministic at inference for a specific sample, so each row is a single reproducible operating point since the same testing sample is used. Bold values indicate the best paradigm within each row. On the medical benchmarks, U-Split is strongest, dominating Path-VQA and VQA-RAD across closed, open, and average accuracy, while Split is competitive on SLAKE. Fixed at ρ=0.3ρ=0.3 (403403 tokens) tracks the unpruned baseline within 11–22 points because top-k retention keeps the patches most relevant for answering. DTP instead lets the sensitivity branch dictate the budget: because medical diagnostic regions are small and sensitivity-rich, DTP allocates only 155155–259259 tokens on these datasets. The trade-off is a few accuracy points, e.g. Path-VQA U-Split DTP keeps just 155155 tokens at 55.9%55.9\% average accuracy (5.95.9 points below Off), in exchange for the privacy-aware behaviour quantified in the attack sections. Fig. 5: Question-guided token pruning under different setups. General-domain benchmarks tell a similar story with budgets shifted upward. Although called “natural images”, many samples carry privacy-relevant content, identifiable faces in OK-VQA, person and clothing attributes in GQA, foreground subjects in VQAv2, which DTP picks up via its DINOv2-derived sensitivity signal. The budget is sharply sample-adaptive: OK-VQA collapses to 194194 tokens (33.7%33.7\%, near the medical regime) due to dense face and person content, while GQA (318318) and VQAv2 (296296) remain more permissive. Federated and Split take turns leading the unpruned setting, U-Split dominates aggressive fixed budgets at ρ∈0.3,0.5ρ∈\0.3,0.5\, and DTP stays within a few points of the matched Fixed rows. Viewed as a communication-cost study, all DTP rows lie between Fixed ρ=0.5ρ=0.5 (288288 tokens) and Fixed ρ=0.7ρ=0.7 (173173 tokens), averaging ≈40%≈ 40\% retention across the six datasets. DTP achieves this without manual tuning by adapting per sample: clean, low-sensitivity images keep more tokens (where the attacker had little to gain), and sensitive medical or face-heavy images are pruned harder (where the attacker had most to gain), yielding a sample-adaptive Pareto operating point that no single fixed ratio reproduces at the same accuracy. TABLE I: Final test accuracy (%) on general-domain VQA datasets. Dataset Model Tokens Accuracy (%) Avg. Kept % Direct Fed. Split U-Split GQA Off 576 100.0 62.4 68.6 68.6 66.8 DTP 318 55.2 60.4 61.0 60.3 60.1 Fixed (ρ=.3ρ=.3) 403 70.0 59.9 66.8 69.4 65.0 Fixed (ρ=.5ρ=.5) 288 50.0 66.0 62.7 66.4 68.2 Fixed (ρ=.7ρ=.7) 173 30.0 51.1 57.6 59.0 56.1 Fixed (ρ=.9ρ=.9) 58 10.0 52.9 50.0 50.7 50.1 OKVQA Off 576 100.0 52.5 56.2 54.5 49.0 DTP 194 33.7 41.1 42.6 41.3 37.0 Fixed (ρ=.3ρ=.3) 403 70.0 50.4 50.7 50.4 46.3 Fixed (ρ=.5ρ=.5) 288 50.0 47.9 49.1 48.4 47.0 Fixed (ρ=.7ρ=.7) 173 30.0 43.1 44.2 42.3 37.5 Fixed (ρ=.9ρ=.9) 58 10.0 37.2 37.1 36.0 35.2 VQAv2 Off 576 100.0 69.0 68.2 69.3 67.7 DTP 296 51.4 59.1 59.0 56.1 53.6 Fixed (ρ=.3ρ=.3) 403 70.0 66.2 65.0 66.3 66.6 Fixed (ρ=.5ρ=.5) 288 50.0 61.3 59.8 60.2 62.6 Fixed (ρ=.7ρ=.7) 173 30.0 54.3 54.4 53.6 53.5 Fixed (ρ=.9ρ=.9) 58 10.0 45.7 47.4 46.5 45.3 TABLE I: Closed, open, and overall accuracy (%) on medical VQA datasets. Dataset Model Tokens Direct Federated Split U-Split Avg. Kept % Closed Open Avg Closed Open Avg Closed Open Avg Closed Open Avg SLAKE Off 576 100.0 84.1 72.3 78.2 86.0 76.5 81.3 87.2 78.1 82.7 85.1 81.1 82.4 DTP 259 45.0 79.0 69.8 74.6 77.2 68.5 73.8 78.4 69.2 74.3 74.9 73.1 73.7 Fixed (ρ=.3ρ=.3) 403 70.0 85.3 74.8 80.2 83.1 72.9 78.6 86.0 75.4 80.7 83.9 79.4 80.9 Fixed (ρ=.5ρ=.5) 288 50.0 84.8 74.1 79.8 79.2 67.8 71.0 82.1 71.4 75.4 78.3 71.6 73.8 Fixed (ρ=.7ρ=.7) 173 30.0 78.5 65.2 72.1 76.0 63.8 69.2 79.4 67.0 73.4 74.3 73.9 74.0 Fixed (ρ=.9ρ=.9) 58 10.0 70.2 58.7 64.5 68.1 56.5 62.3 74.5 61.3 70.2 57.6 69.3 65.4 Path-VQA Off 576 100.0 66.5 50.2 58.4 65.9 49.6 58.0 68.8 52.1 60.5 70.2 53.4 61.8 DTP 155 26.9 57.2 43.0 50.1 60.4 47.0 53.7 62.1 48.7 55.4 63.5 48.3 55.9 Fixed (ρ=.3ρ=.3) 403 70.0 63.9 50.3 57.1 63.4 49.7 56.8 67.1 53.6 60.6 69.4 54.8 62.1 Fixed (ρ=.5ρ=.5) 288 50.0 62.1 48.2 55.1 61.8 48.0 55.0 65.5 51.4 58.5 67.2 52.6 59.9 Fixed (ρ=.7ρ=.7) 173 30.0 61.4 47.1 55.0 59.9 46.3 53.1 64.2 50.0 57.1 65.8 51.2 58.5 Fixed (ρ=.9ρ=.9) 58 10.0 60.7 46.5 55.5 59.8 45.9 54.6 63.1 49.2 56.2 64.7 50.5 57.6 VQA-RAD Off 576 100.0 62.8 45.7 54.3 64.0 47.3 55.7 60.5 43.8 52.1 65.3 48.1 56.7 DTP 221 38.4 60.4 44.1 52.9 57.6 42.0 50.1 59.8 43.3 52.4 61.7 44.0 53.0 Fixed (ρ=.3ρ=.3) 403 70.0 64.3 46.5 55.4 62.1 44.9 53.2 65.0 47.8 57.4 66.8 48.9 58.5 Fixed (ρ=.5ρ=.5) 288 50.0 61.9 45.0 54.0 60.5 43.2 52.9 56.1 41.8 48.5 63.4 46.2 55.4 Fixed (ρ=.7ρ=.7) 173 30.0 58.4 44.7 51.5 55.1 41.9 48.5 56.2 43.0 49.6 60.1 45.3 52.9 Fixed (ρ=.9ρ=.9) 58 10.0 55.0 44.1 49.6 50.3 39.8 45.4 52.0 41.6 46.8 56.8 42.7 49.9 VI-E QPriv-VL Robustness Against FSHA We next ask whether QPriv-VL limits what an adversarial split-learning server can recover from intermediate activations via FSHA [16]. The attack targets cut layer ℓ=16 =16 of LLaVA-1.5-7B. With the victim LoRA adapters trained and frozen, we simulate FSHA by training a shadow autoencoder (f~,f~−1)( f, f^-1) on 800800 public VQA-RAD images while a discriminator D adversarially aligns the output distribution of f~ f with the real client activations (λadv=0.2 _adv=0.2, λrec=1.0 _rec=1.0). The trained inverse f~−1 f^-1 is applied to 400400 held-out private activations to reconstruct medical images at 112×112112×112 under off, fixed ρ∈0.3,0.5,0.7,0.9ρ∈\0.3,0.5,0.7,0.9\, and the learned DTP policy. Figure 6 shows that reconstructions remain low fidelity across every defense, with PSNR in [14.0,15.2][14.0,15.2] dB and cosine in [0.86,0.88][0.86,0.88]. Outputs degrade into coarse anatomical silhouettes, faint cranial outlines in brain MRIs or blurred thoracic regions in chest X-rays, without revealing lesion-level details, indicating that FSHA fails to recover diagnostically meaningful information from layer-1616 representations even without pruning. The fixed setting ρ=0.9ρ=0.9 transmits the fewest tokens (5858 of 576576) and consequently yields the lowest reconstruction quality, while DTP attains comparable PSNR (≈14.0≈ 14.0 dB, cosine ≈0.86≈ 0.86) while retaining roughly 221221 tokens on VQA-RAD, placing it below the linear PSNR-vs-tokens trend. Fig. 6: Qualitative FSHA reconstructions from layer-16 smashed activations under different QPriv-VL defense settings. VI-F QPriv-VL Robustness Against FORA We stress-test QPriv-VL against FORA, a stronger inversion attack whose decoders are trained once on unpruned public activations and reused against every defended target, providing an upper bound on recoverable information. The setting is USL: the client keeps the embeddings and top decoder head; the malicious server controls the middle block between cuts a=8a=8 and b=24b=24. For each victim checkpoint we extract 800800 public and 400400 private layer-1616 activations and train (i) a feature-space decoder targeting the 576×1024576×1024 CLIP-V grid and (i) a pixel-space decoder targeting 112×112112×112 RGB images (AdamW, 66 epochs, batch size 1616, lr 10−410^-4). We sweep ρ∈0,0.3,0.5,0.7,0.9ρ∈\0,0.3,0.5,0.7,0.9\ together with DTP, and compare a victim trained without the QPriv-VL framework against one co-trained with DTP on SLAKE and VQA-RAD. Table IV reports pixel-space PSNR and cosine similarity; feature-space PSNR stays at or below 2.32.3 dB across all settings, indicating that layer-1616 representations are too entangled to invert into CLIP feature space. Aggressive pruning reduces attacker PSNR by roughly 1.51.5 dB on SLAKE (15.7615.76 to 14.2814.28) and 2.02.0 dB on VQA-RAD (14.3914.39 to 12.4312.43) between off and ρ=0.9ρ=0.9. DTP at its operating point (ρ=0.42ρ=0.42, keeping ∼ 335335 tokens on SLAKE; ρ=0.61ρ=0.61, keeping ∼ 225225 tokens on VQA-RAD) achieves the lowest PSNR in every column and sits 1.41.4–1.71.7 dB below the linearly interpolated fixed-ρ trend at matched token cost. On VQA-RAD, DTP even matches the PSNR of the most aggressive fixed pruning (ρ=0.9ρ=0.9, only 5858 tokens) while retaining roughly 4×4× as many tokens, a Pareto improvement. Co-training the victim encoder with DTP adds a dataset-dependent defense: on VQA-RAD it weakens the attacker even at ρ=0ρ=0 (14.39→13.8514.39→ 13.85 dB) and further at the DTP operating point (12.42→12.3412.42→ 12.34 dB), suggesting co-training shifts representations toward a harder-to-invert geometry; on SLAKE the inference-time pruning dominates and co-training has little additional effect. TABLE IV: FORA reconstruction under USL. Policy ρ SLAKE VQA-RAD w/o QPriv-VL co-trained w/o QPriv-VL co-trained PSNR Cos PSNR Cos PSNR Cos PSNR Cos Off 0 15.76 0.892 15.76 0.890 14.39 0.885 13.85 0.887 Fixed 0.3 15.31 0.878 15.68 0.885 14.90 0.878 14.78 0.879 Fixed 0.5 14.94 0.862 15.39 0.876 14.40 0.864 14.46 0.863 Fixed 0.7 14.74 0.852 15.11 0.868 13.62 0.845 13.66 0.842 Fixed 0.9 14.28 0.841 14.55 0.855 12.43 0.801 12.36 0.793 DTP 0.42 13.51 0.824 14.07 0.851 - - - - DTP 0.61 - - - - 12.42 0.786 12.34 0.784 VI-G QPriv-VL Robustness Against MIA We examine whether QPriv-VL limits attribute leakage. The adversary is a malicious server observing layer-1616 smashed activations of LLaVA-1.5-7B to infer a sensitive attribute: the binary open/closed question type on VQA-RAD, and the three-class imaging modality (CT, MRI, X-ray) on SLAKE from BoKelvin/SLAKE annotations matched by (question,answer)(question,answer). To avoid trivial length leakage, the attacker uses only the image-token portion and mean-pools it into a 40964096-dimensional vector; a frozen CLIP-T question embedding is optionally concatenated as prior knowledge. An MLP classifier is trained on 800800 public and evaluated on 400400 private samples; leakage is measured as AUCfull−AUCquestion-onlyAUC_full-AUC_question-only. We sweep all four training paradigms against six defenses: off, fixed ρ∈0.3,0.5,0.7,0.9ρ∈\0.3,0.5,0.7,0.9\, and DTP at ρ=0.61ρ=0.61 on VQA-RAD and ρ=0.47ρ=0.47 on SLAKE. Figure 7 summarises the attack. On VQA-RAD, undefended accuracy is near saturation (∼ 0.990.99) and drops to 0.530.53–0.650.65 at ρ=0.9ρ=0.9, with Direct, Split, and U-Split bottoming near 0.550.55 while Federated leaks more (∼ 0.650.65). DTP at ρ=0.61ρ=0.61 (retaining ∼ 225225 tokens) sits at 0.760.76–0.790.79, reducing leakage while keeping nearly four times as many tokens as the strongest fixed setting. On SLAKE, accuracy remains near saturation (≥0.95≥ 0.95) under every defense including DTP, because the modality label is encoded in coarse image statistics (e.g. global intensity distributions separating MRI from X-ray) surviving heavy pruning, and because modality-specific question terms already give the attacker a strong CLIP-T prior. The persistent leakage on SLAKE reflects the low-entropy modality attribute rather than a limitation of the pruning policy. Fig. 7: Attribute-inference attack accuracy across four training paradigms and six defense settings on VQA-RAD and SLAKE. VI-H QPriv-VL Robustness Against iDLG Attack We test whether QPriv-VL leaks visual content through LoRA gradients during federated training. An honest-but-curious FedAvg server observes per-client LoRA gradients from one forward-backward pass over a real (image,question,answer)(image,question,answer) tuple and tries to reconstruct the image using DLG [30] and iDLG [18]. The attacker initialises a dummy CLIP-V feature tensor of shape (576,1024)(576,1024) with Gaussian noise, applies the client’s pruning policy, and optimises the dummy input with the true answer fixed, minimising ∑i∥∇θiℒdummy−∇θiℒreal∥2 _i _ _iL_dummy- _ _iL_real ^2 over the LoRA parameters (Adam, lr 1.01.0, 300300 steps, with a small total-variation regulariser). LoRA parameters are cast to fp32 for double-backward through bf16-LLaVA. defenses include off, fixed ρ∈0.3,0.5,0.7,0.9ρ∈\0.3,0.5,0.7,0.9\, and DTP at ρ=0.70ρ=0.70. Quality is measured in CLIP feature space using PSNR and cosine similarity averaged over N=8N=8 samples. Figure 8 shows that gradient inversion is essentially ineffective here. Even without any defense, recovery is poor (PSNR ≈−3.7≈-3.7 dB, cosine ≈0.13≈ 0.13), reflecting how diffusely visual information spreads across the LoRA adapters of a 77B VLM. Pruning shifts PSNR only into −3.7-3.7 to −1.6-1.6 dB and cosine into 0.050.05–0.160.16, all well below useful image quality. DTP at ρ=0.70ρ=0.70 lies inside this noise band (−2.0-2.0 dB, ∼ 0.130.13 cosine) while retaining more tokens than fixed ρ=0.9ρ=0.9, preserving downstream utility without weakening the (already saturated) defense. VI-I Explainability Analysis of QPriv-VL We now ask whether QPriv-VL genuinely conditions on the question rather than on generic saliency. To answer this, we visualise the DTP utility map u∈ℝ24×24u ^24× 24 and the corresponding keep/drop decisions for a single image paired with five questions: the original GQA query, two visually grounded in-context questions, and two out-of-context (OOC) questions (arithmetic or factual queries unrelated to the image). The utility map is bilinearly upsampled and rendered with the magma colormap; retained patches carry a cyan contour and dropped regions are darkened. We compare DTP against the off baseline and fixed ratios ρ∈0.3,0.5,0.7,0.9ρ∈\0.3,0.5,0.7,0.9\, and report the retained token count k and the answer from the LoRA-adapted LLaVA-1.5-7B model trained Direct on GQA. TABLE V: Aggregated robustness and privacy probe results across 7272 experimental dataset-setting pairs. Mode Halluc. ↓ Refusal ↑ Rel. Acc ↑ Excl. ratio ↑ Sens. Closed ↑ Sens. Open ↑ off 0.955±0.0620.955± 0.062 0.045±0.0620.045± 0.062 0.570±0.1410.570± 0.141 - 0.797±0.1270.797± 0.127 0.468±0.2150.468± 0.215 DTP 0.977±0.0270.977± 0.027 0.023±0.0270.023± 0.027 0.505±0.1450.505± 0.145 1.201±0.1771.201± 0.177 0.736±0.1550.736± 0.155 0.378±0.2150.378± 0.215 fixed_0.5 0.937±0.0550.937± 0.055 0.063±0.0550.063± 0.055 0.502±0.1360.502± 0.136 0.998±0.0010.998± 0.001 0.734±0.1510.734± 0.151 0.420±0.2090.420± 0.209 Fig. 8: Gradient inversion on federated VQA-RAD. Figure 9 shows that DTP adapts both the size and location of retained regions to question semantics. For the colour query (“What colour is the blouse?”), DTP keeps k=265k=265 patches focused on the blouse and predicts black, while off returns red; both fixed ρ=0.3ρ=0.3 and ρ=0.9ρ=0.9 return white, confirming that uniform pruning at these budgets discards blouse-relevant context. For the OOC arithmetic query (“What is two plus two?”) the policy collapses to the safety floor (k=16k=16), signalling that no visual evidence is needed. Together these cases provide a behavioural verification that the learned policy is genuinely question-conditioned rather than a fixed saliency template. Fig. 9: Question-conditioned utility maps and retained regions under multiple in-context and out-of-context questions. VI-J Behavioural Probes: Off-Topic Queries and Sensitivity-Aware Pruning The four preceding subsections each tested resistance to a specific privacy attack. This subsection instead aggregates two behavioural probes: an off-topic robustness probe that measures how the model behaves when asked an irrelevant question, and a sensitivity-aware pruning probe that measures whether DTP preferentially drops patches that DINOv2 marks as privacy-relevant. Table V summarises both probes aggregated over all 7272 dataset × defense combinations. For the off-topic probe, hallucination rates stay high (0.940.94–0.980.98) and refusal rates low (0.020.02–0.060.06) under every pruning mode, confirming that the tendency to answer nonsensical questions is set primarily by the underlying language model rather than by visual token pruning. Relevant-question accuracy is essentially flat (0.500.50–0.570.57), meaning aggressive pruning at ρ=0.5ρ=0.5 or the DTP policy preserves most of the task-relevant features. On the sensitivity-aware pruning probe, DTP achieves an exclusion ratio of 1.20±0.181.20± 0.18, meaning dropped patches carry on average about 20%20\% higher sensitivity than retained ones, whereas fixed-rate pruning at ρ=0.5ρ=0.5 stays at 0.9980.998, indistinguishable from content-agnostic dropping. The cost is a moderate utility hit on the same sensitive subset (closed accuracy 0.797→0.7360.797→ 0.736, open recall 0.468→0.3780.468→ 0.378), an expected trade-off because privacy-rich regions overlap with the most informative parts of the image. Overall, DTP gives targeted suppression of sensitive content while leaving off-topic robustness untouched. VI-K Discussion and Limitations The four research questions can now be addressed. For RQ1, DTP compresses the visual prefix to nearly 40%40\% of its original size while reducing the effectiveness of all evaluated attack families, including FSHA, FORA, MIA, and iDLG. Despite this reduction, VQA performance remains close to the unpruned baseline across CL, FL, SL, and USL settings. These results demonstrate that communication efficiency and privacy preservation can be improved jointly without causing significant degradation in task accuracy. For RQ2, the dynamic threshold predictor consistently outperforms both fixed-ratio and training-free pruning approaches on the privacy-utility trade-off. Under the same token budget, it achieves 1.41.4–1.71.7 dB lower FORA reconstruction quality than interpolated fixed-ρ pruning, and on VQA-RAD it reduces MIA accuracy by nearly half at operating points unattainable by any single fixed ratio. For RQ3, the observed sensitivity exclusion ratio of 1.20±0.181.20± 0.18 (compared to approximately 1.001.00 for fixed pruning), together with stronger pruning behaviour on face-rich and medical images, suggests that DTP preferentially suppresses patches that are privacy-sensitive and easier to reconstruct instead of applying uniform pruning. For RQ4, the explainability analysis in Section VI-I further confirms this behaviour. The policy collapses to the safety floor for out-of-context questions while preserving question-relevant regions for in-context queries, consistent with the optimisation objective. Several limitations remain. First, low-entropy attributes such as imaging modality still leak comparatively heavily even under aggressive pruning, since coarse global statistics and modality-specific question terms may already expose the target before visual features are transmitted. Second, the experiments are limited to LLaVA-1.5-7B with fixed cut-layer configurations (SL at ℓ=16 =16 and USL at a=8a=8, b=24b=24), meaning that the resulting privacy-utility trade-offs may vary for deeper cuts, larger backbones, or different fine-tuning strategies. Third, DTP relies on a fixed DINOv2-based sensitivity model during training and may generalise poorly against adaptive attackers that explicitly exploit its token-retention behaviour. VII Conclusion We presented QPriv-VL, a question-guided and privacy-aware token-pruning framework for distributed VLM-based VQA, where DTP jointly determines both the number and selection of CLIP visual tokens transmitted across the cut layer before reaching a LoRA-adapted LLaVA-1.5-7B backbone. Across CL, FL, SL, and USL settings on six general-domain and medical VQA benchmarks, DTP achieves a strong privacy-utility trade-off, matching or outperforming the best fixed-ratio defenses against FSHA, FORA, DLG/iDLG, and attribute-inference MIA while preserving substantially more tokens than aggressive uniform pruning approaches. However, token pruning alone is insufficient to guarantee formal privacy protection. Future work will explore extending DTP with alternative mechanisms, adversarial co-training against adaptive attackers, broader generalisation across VLM architectures and modalities such as 3D medical volumes and video, and integration with differential privacy and secure aggregation to transform the empirical protection demonstrated here into formal guarantees. VIII LLM Usage Statement Large language models (LLMs) were used during preparation of this manuscript as assistive tools for writing and code debugging. All AI-assisted outputs were reviewed, verified, and corrected by the authors, who take full responsibility for the correctness, originality, and integrity of the work. We detail the role of LLMs below in accordance with the three criteria of originality, transparency, and responsibility- Originality LLMs were used for editorial purposes, and all outputs were inspected by the authors to ensure accuracy and originality. LLMs assisted in drafting and refining portions of the text, while every generated passage was carefully reviewed, fact-checked, and revised by the authors to align with the underlying research findings. All figures, tables, and quantitative results were produced entirely by the authors without AI assistance. The authors remain fully responsible for the content of the paper, including all written text, technical descriptions, and reported numerical values. Transparency LLMs were also used during code development, primarily for debugging and identifying implementation issues. However, the experimental code required substantial manual design, step-by-step correction, and multiple iterations of testing before reaching a working state, as AI-generated suggestions alone were insufficient to produce the final pipeline. Every result reported in this manuscript was obtained by running the author-verified code, and all outputs were validated against expected behaviour before inclusion. LLMs were not used to design the core methodology, model architecture, or experimental protocol of QPriv-VL, which were conceived and validated by the authors. Responsibility LLMs were used responsibly throughout the preparation of this work. No sensitive, private, or proprietary information, including dataset content, or participant data was shared with any AI tool during writing or debugging. The authors ensured that all interactions with LLMs respected ethical considerations regarding data ownership and intellectual property. The use of LLMs was limited to general-purpose writing assistance and code debugging, and did not influence the scientific contributions or claims. References [1] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh (2015) Vqa: visual question answering. In Proceedings of the IEEE international conference on computer vision, p. 2425–2433. Cited by: §I, §I-A. [2] D. A. Hudson and C. D. Manning (2019) Gqa: a new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 6700–6709. Cited by: §I, §V-A. [3] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. Advances in neural information processing systems 36, p. 34892–34916. Cited by: §I, §I-A. [4] H. Liu, C. Li, Y. Li, and Y. J. Lee (2024) Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 26296–26306. Cited by: §I, §I, §I-A. [5] J. Li, D. Li, S. Savarese, and S. Hoi (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, p. 19730–19742. Cited by: §I, §I-A. [6] J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. (2022) Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35, p. 23716–23736. Cited by: §I, §I-A. [7] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Cited by: §I. [8] J. J. Lau, S. Gayen, A. Ben Abacha, and D. Demner-Fushman (2018) A dataset of clinically generated visual questions and answers about radiology images. Scientific data 5 (1), p. 180251. Cited by: §I, §V-A. [9] B. Liu, L. Zhan, L. Xu, L. Ma, Y. Yang, and X. Wu (2021) Slake: a semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), p. 1650–1654. Cited by: §I, §V-A. [10] X. He, Y. Zhang, L. Mou, E. Xing, and P. Xie (2020) Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286. Cited by: §I, §V-A. [11] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas (2017) Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, p. 1273–1282. Cited by: §I, §I-C, §V-C. [12] O. Gupta and R. Raskar (2018) Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications 116, p. 1–8. Cited by: §I, §I-C. [13] P. Vepakomma, O. Gupta, T. Swedish, and R. Raskar (2018) Split learning for health: distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.00564. Cited by: §I, §I-C. [14] A. E. Eshratifar, A. Esmaili, and M. Pedram (2019) Bottlenet: a deep learning architecture for intelligent mobile cloud computing services. In 2019 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), p. 1–6. Cited by: §I, §I. [15] M. Hassan, S. Davy, O. B. Zuber, and N. Ashraf (2025) SpikeBottleNet: spike-driven feature compression architecture for edge-cloud co-inference. In Intelligent Systems Conference, p. 342–355. Cited by: §I. [16] D. Pasquini, G. Ateniese, and M. Bernaschi (2021) Unleashing the tiger: inference attacks on split learning. In Proceedings of the 2021 ACM SIGSAC conference on computer and communications security, p. 2113–2129. Cited by: §-A, 3rd item, §I, §I-C, §I-D, §I, §VI-E. [17] X. Xu, M. Yang, W. Yi, Z. Li, J. Wang, H. Hu, Y. Zhuang, and Y. Liu (2024) A stealthy wrongdoer: feature-oriented reconstruction attack against split learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 12130–12139. Cited by: 3rd item, §I, §I-C, §I-D, §I. [18] B. Zhao, K. R. Mopuri, and H. Bilen (2020) Idlg: improved deep leakage from gradients. arXiv preprint arXiv:2001.02610. Cited by: 3rd item, §I, §I-C, §I-D, §I, §VI-H. [19] R. Shokri, M. Stronati, C. Song, and V. Shmatikov (2017) Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), p. 3–18. Cited by: 3rd item, §I, §I-D. [20] L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang (2024) An image is worth 1/2 tokens after layer 2: plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, p. 19–35. Cited by: §-A, TABLE VI, §I, §I, §I-A, §I. [21] Y. Zhang, C. Fan, J. Ma, W. Zheng, T. Huang, K. Cheng, D. Gudovskiy, T. Okuno, Y. Nakata, K. Keutzer, et al. (2024) Sparsevlm: visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417. Cited by: §-A, TABLE VI, §I, §I, §I-A, §I. [22] X. Ye, Y. Gan, Y. Ge, X. Zhang, and Y. Tang (2025) Atp-llava: adaptive token pruning for large vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 24972–24982. Cited by: §I, §I-A, §I. [23] S. Li, J. Xu, X. Li, C. Deng, and L. Huang (20 25) Qg-vtc: question-guided visual token compression in mllms for efficient vqa. arXiv preprint arXiv:2504.00654. Cited by: §I, §I. [24] J. Guo, F. Zhai, P. Jian, Q. Wei, and Y. Zhou (2025) CROP: contextual region-oriented visual token pruning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 9767–9783. Cited by: §I, §I. [25] L. Xing, Q. Huang, X. Dong, J. Lu, P. Zhang, Y. Zang, Y. Cao, C. He, J. Wang, F. Wu, et al. (2024) Pyramiddrop: accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247. Cited by: §-A, TABLE VI, §I, §I. [26] S. Yang, Y. Chen, Z. Tian, C. Wang, J. Li, B. Yu, and J. Jia (2025) Visionzip: longer is better but not necessary in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 19792–19802. Cited by: §-A, TABLE VI, §I, §I. [27] H. Chefer, S. Gur, and L. Wolf (2021) Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 782–791. Cited by: 4th item, §I. [28] Y. Shang, M. Cai, B. Xu, Y. J. Lee, and Y. Yan (2025) Llava-prumerge: adaptive token reduction for efficient large multimodal models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 22857–22867. Cited by: §I-A, §I. [29] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2023) Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: §I-B, §I, §V-B. [30] L. Zhu, Z. Liu, and S. Han (2019) Deep leakage from gradients. Advances in neural information processing systems 32. Cited by: §I-C, §I-D, §I, §VI-H. [31] P. Vepakomma, A. Singh, O. Gupta, and R. Raskar (2020) NoPeek: information leakage reduction to share activations in distributed deep learning. In 2020 International Conference on Data Mining Workshops (ICDMW), p. 933–942. Cited by: §-A, §I. [32] S. Baek, J. Park, P. Vepakomma, R. Raskar, M. Bennis, and S. Kim (2022) Visual transformer meets cutmix for improved accuracy, communication efficiency, and data privacy in split learning. arXiv preprint arXiv:2207.00234. Cited by: §I. [33] S. Ding, L. Zhang, M. Pan, and X. Yuan (2024) Patrol: privacy-oriented pruning for collaborative inference against model inversion attacks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 4716–4725. Cited by: §I. [34] F. Alvetreti, J. Pomponi, P. Di Lorenzo, and S. Scardapane (2025) Communication efficient split learning of vits with attention-based double compression. arXiv preprint arXiv:2509.15058. Cited by: §I. [35] T. Feng, D. Bose, T. Zhang, R. Hebbar, A. Ramakrishna, R. Gupta, M. Zhang, S. Avestimehr, and S. Narayanan (2023) Fedmultimodal: a benchmark for multimodal federated learning. In Proceedings of the 29th ACM SIGKDD conference on knowledge discovery and data mining, p. 4035–4045. Cited by: §I. [36] H. Zhu, R. Togo, T. Ogawa, and M. Haseyama (2024) Which client is reliable?: a reliable and personalized prompt-based federated learning for medical image question answering. arXiv preprint arXiv:2410.17484. Cited by: §I. [37] Y. Sun and H. Ochiai (2024) Bidirectional contrastive split learning for visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 21602–21609. Cited by: §I. [38] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin (2021) Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, p. 9650–9660. Cited by: §I. [39] V. Petsiuk, A. Das, and K. Saenko (2018) Rise: randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421. Cited by: §I. [40] J. DeYoung, S. Jain, N. F. Rajani, E. Lehman, C. Xiong, R. Socher, and B. C. Wallace (2020) ERASER: a benchmark to evaluate rationalized nlp models. In Proceedings of the 58th annual meeting of the association for computational linguistics, p. 4443–4458. Cited by: §I. [41] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. Iclr 1 (2), p. 3. Cited by: §IV-E, §V-B. [42] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh (2017) Making the v in vqa matter: elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 6904–6913. Cited by: §V-A. [43] D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi (2022) A-okvqa: a benchmark for visual question answering using world knowledge. In European conference on computer vision, p. 146–162. Cited by: §V-A. [44] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §V-B. [45] D. Hendrycks and K. Gimpel (2016) Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. Cited by: §V-B. -A Comparison with Prior Work Table VI compares QPriv-VL with recent approaches from two separate research directions: visual-token pruning for LLaVA-style VLMs (FastV [20], SparseVLM [21], PyramidDrop [25], VisionZip [26]) and representation-level privacy protection for split or split-federated learning (NoPeek [31]). The token-pruning methods focus on inference efficiency in centralized settings and do not evaluate robustness against feature-reconstruction or membership-inference attacks. NoPeek and ResSFL are designed for small CNN-based models on datasets such as CIFAR and CelebA, have not been extended to VLMs, and NoPeek remains vulnerable to FSHA attacks [16]. To the best of our knowledge, no prior work jointly integrates question-guided token pruning and privacy-aware defence within a distributed VLM framework. Under the closest comparable setting on GQA using LLaVA-1.5-7B, QPriv-VL achieves 60.4%60.4\% accuracy with 318318 retained tokens in Direct mode, comparable to VisionZip (59.3%59.3\%) and PyramidDrop (60.1%60.1\%) using 192192 tokens. The higher token budget reflects the sensitivity-driven nature of DTP, which prioritizes privacy alongside efficiency. Beyond utility, QPriv-VL reduces FORA reconstruction PSNR by 1.41.4–1.71.7 dB on SLAKE and VQA-RAD and decreases VQA-RAD membership-inference accuracy from 0.990.99 to 0.760.76, metrics not reported by existing token-pruning approaches. TABLE VI: Comparison of QPriv-VL with other related works. Method Venue Backbone Q-guided Sens-aware GQA FastV [20] ECCV’24 LLaVA-1.5 ✓ — 52.752.7 SparseVLM [21] ICML’25 LLaVA-1.5 ✓ — 57.657.6 VisionZip [26] CVPR’25 LLaVA-1.5 — — 59.359.3 PyramidDrop [25] CVPR’25 LLaVA-1.5 ✓ — 60.160.1 QPriv-VL (Ours) — LLaVA-1.5 ✓ ✓ 60.460.4 -B Ablation Study To analyse the influence of the utility weight α and privacy weight β on the proposed dynamic threshold predictor, we sweep the simplex α∈0.0,0.1,…,1.0α∈\0.0,0.1,…,1.0\ while enforcing β=1−αβ=1-α, resulting in eleven student models. For each (α,β)(α,β), the student is retrained with teacher mask targets recomputed under the same weighting, ensuring consistency between the distillation objective and the deployment-time scoring function α−βα\,u-β\,s. Each trained student is then integrated into the QGTP controller and used for LoRA fine-tuning of LLaVA-1.5-7B on VQA-RAD and SLAKE under identical schedules, learning rates, and early-stopping settings, isolating the effect of (α,β)(α,β). We report test accuracy and the average number of retained visual tokens per image as complementary measures of task performance and information exposure. The resulting Pareto trade-off is illustrated in Figure 10. Fig. 10: Ablation study of the utility-privacy trade-off. Figure 10 highlights two observations. First, the best performance on both datasets is achieved at α=0α=0, where token ranking is determined entirely by the privacy-sensitivity signal ββ\,s. Under this setting, VQA-RAD achieves an accuracy of 0.5660.566 and SLAKE reaches 0.8170.817, both outperforming the pure-utility configuration at α=1α=1 (0.5270.527 and 0.7740.774, respectively). This suggests that sensitivity-aware token selection naturally preserves the most diagnostically relevant regions while discarding irrelevant or noisy content that utility-only ranking tends to retain. Second, the average number of retained tokens remains nearly constant across the sweep, varying only from approximately 220220–224224 on VQA-RAD and 256256–262262 on SLAKE. This stability indicates that ρ ρ is controlled mainly by the global predictor inputs (pool,,summary)(c_pool,t,s_summary) rather than directly by (α,β)(α,β). Consequently, the trade-off is driven not by retaining more or fewer tokens but by the criterion used to rank them, with the privacy-aware ranking proving most effective for medical VQA performance.