Paper deep dive

Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models

Gautam Rajendrakumar Gare, Siyi Li, Hewei Wang, Cesar Daniel Hernandez, Wei Zhao, Wolfgang M. Pauli, John Galeotti, Deva Ramanan

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 120

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 9/11/2026, 5:03:37 AM

Summary

The paper introduces SoftPrompt, a parameter-efficient adaptation method for Vision-Language Models (VLMs) in few-shot settings. By optimizing a small number of continuous soft tokens placed at the cross-modal boundary between visual and text inputs, SoftPrompt achieves performance comparable to LoRA fine-tuning on object detection benchmarks (e.g., Roboflow20-VL) while using significantly fewer trainable parameters (7,168 vs. millions). Crucially, SoftPrompt avoids catastrophic forgetting of general-domain capabilities (measured on NaturalBench VQA) because the pretrained backbone remains frozen. The method also supports transfer across model versions and distillation into human-readable hard prompts.

Entities (12)

SoftPrompt · method · 98%LoRA · method · 95%Qwen3-VL-8B · model · 93%Catastrophic Forgetting · concept · 92%Roboflow20-VL · dataset · 92%Cross-modal boundary · concept · 90%NaturalBench · dataset · 90%Qwen3.5-9B · model · 90%DetPO · method · 88%GEPA · method · 88%RoboCasa · dataset · 85%π-0.5 · model · 85%

Relation Signals (9)

SoftPrompt → avoids → Catastrophic Forgetting

confidence 95% · SoftPrompt leaves pretrained performance unchanged, whereas LoRA reduces NaturalBench VQA accuracy by 35-56%.

SoftPrompt → evaluatedon → Roboflow20-VL

confidence 95% · We address few-shot object detection with vision-language models (VLMs) in out-of-domain settings... on Roboflow20-VL.

SoftPrompt → outperformsormatches → LoRA

confidence 95% · SoftPrompt matches the best LoRA configuration on Roboflow20-VL (14.2 mAP) while training over 20,000x fewer parameters.

SoftPrompt → appliedto → Qwen3-VL-8B

confidence 93% · All main detection results use frozen Qwen3-VL-8B-Instruct.

SoftPrompt → uses → Cross-modal boundary

confidence 92% · Placing prompt tokens at the cross-modal boundary between visual and text tokens outperforms other placements (10.0 vs. 8.4 mAP).

SoftPrompt → evaluatedon → NaturalBench

confidence 90% · The LoRA rank matching our accuracy reduces NaturalBench VQA accuracy by 35% relative... whereas soft prompting leaves pretrained performance unchanged.

SoftPrompt → transfersto → Qwen3.5-9B

confidence 88% · They transfer to a newer model without retraining (+0.8 mAP on Qwen3.5-9B)

SoftPrompt → appliedto →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We address few-shot object detection with vision-language models (VLMs) in out-of-domain settings such as aerial, industrial, and medical imagery, using only ten annotated images for supervision. Existing adaptation methods are discrete prompt optimization and LoRA fine-tuning. We revisit a third option: soft prompting, where a small number of continuous prompt tokens are optimized while the pretrained backbone remains frozen. We identify two key design choices. First, placing prompt tokens at the cross-modal boundary between visual and text tokens outperforms other placements (10.0 vs. 8.4 mAP). Second, initializing prompts from the empty space token outperforms semantic and random initialization. With these choices, one to three learned tokens (7,168 parameters on average) match the best LoRA configuration on Roboflow20-VL (14.2 mAP, 10-shot) while training over 20,000x fewer parameters. Soft prompting remains harder to optimize, exhibiting higher variance across random seeds. Unlike LoRA, however, it causes no forgetting: the LoRA rank matching our accuracy reduces NaturalBench VQA accuracy by 35% relative, rising to 56% at the largest rank, whereas soft prompting leaves pretrained performance unchanged. The learned tokens behave like prompts rather than weights. They transfer to a newer model without retraining (+0.8 mAP on Qwen3.5-9B) and can be verbalized into readable prompts competitive with prompt-search methods (matching DetPO and outperforming GEPA). The approach also extends beyond detection. On RoboCasa manipulation tasks, the frozen $\pi_{0.5}$ vision-language-action policy benefits from soft prompting, matching the LoRA baseline on two of three tasks when tokens are placed at the gradient bottleneck. These results suggest modern VLMs already encode much of what is needed for specialized domains; the challenge is learning how to ask.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

117,110 characters extracted from source content.

Expand or collapse full text

Your Model Already Knows Don’t Teach It, Learn to Ask It Soft Prompting for Few-Shot Adaptation of Vision-Language Models Gautam Rajendrakumar Gare Siyi Li Hewei Wang Cesar Daniel Hernandez Wei Zhao Wolfgang M. Pauli John Galeotti Deva Ramanan Carnegie Mellon University Apple Abstract We address few-shot object detection with vision-language models (VLMs) in out-of-domain settings such as aerial, industrial, and medical imagery, where only ten annotated images are available for supervision. The dominant adaptation strategies are discrete prompt optimization, which is gradient-free, and LoRA fine-tuning, which augments frozen pretrained weights with learnt low-rank update matrices. We revisit a third option: soft prompting, where a few continuous prompt tokens are optimized by gradient descent against a fully frozen backbone. We show that two design choices strongly influence performance. First, injecting the tokens at the cross-modal boundary, between the visual and text tokens, beats every other ordering of image, prompt, and text, including the prefix placement inherited from NLP (10.010.0 vs. 8.48.4 average mAP). Second, initializing the prompt from a semantically empty space token, so that the prompt’s meaning is unchanged at initialisation, beats semantic and random starts. With these choices, a prompt of one to three learnt tokens, sized per domain by a short sweep (7,1687,168 parameters on average), matches the best LoRA configuration from a rank sweep on the Roboflow20-VL benchmark (14.214.2 vs. 14.214.2 mAP, 10-shot) while training over 20,000×20,000× fewer parameters. However, optimization remains challenging in the few-shot regime: soft prompting exhibits higher variance across random seeds than LoRA, although LoRA is itself not immune to instability. In contrast to LoRA, SoftPrompt exhibits no forgetting: the LoRA rank that matches our accuracy loses 35%35\% relative VQA accuracy on NaturalBench, growing to 56%56\% at the largest rank, while SoftPrompt provably loses none. We attribute this behavior to the fact that SoftPrompt leaves the pretrained model weights unchanged. The learnt tokens behave like prompts, not weights: they transfer to a newer model version without retraining (+0.8+0.8 mAP on Qwen3.5-9B) and can be verbalised into human-readable prompts competitive with dedicated prompt search (matching DetPO, outperforming GEPA). The recipe extends beyond detection: on RoboCasa manipulation tasks the frozen π0.5 _0.5 vision-language-action policy struggles; soft prompts again deliver substantial gains, matching the coverage-matched LoRA baseline on two of three tasks, provided the tokens are again placed where the gradient needs them; Our results suggest that modern VLMs already encode much of what specialised domains require; the task is not to teach them, but to learn how to ask. 1 Introduction Figure 1: SoftPrompt matches LoRA with zero forgetting at over 20,000×20,000× fewer trainable parameters. All methods use the default class-name evaluation protocol (§4.1). Top: all-domain mAP@50:95 on Roboflow20-VL (10-shot) vs. trainable parameters (log scale). Sweeping LoRA rank →1284\!→\!128 peaks at 14.214.2 mAP at r=64r=64 (174.6174.6M parameters) and then declines; SoftPrompt (IST, per-domain L, 7,1687,168 parameters on average, backbone frozen) reaches the same 14.214.2. Bottom: the hidden cost. Relative drop in NaturalBench VQA group accuracy vs. the unadapted backbone: LoRA’s forgetting grows with rank, 35%35\% at the accuracy-matching r=64r=64 and 56%56\% at r=128r=128, while SoftPrompt and the training-free prompt baselines sit at exactly 0%0\% because no backbone weight changes. The two panels move in opposite directions: the ranks that help detection most damage the generalist most. The instruction-augmented LoRA variant is reported in Appendix F (see §4.2). Beyond efficiency, the learnt soft prompts keep the virtues of text prompts: they transfer across models (§6.3) and can be distilled into editable natural language (§6.2). Figure 2: SoftPrompt combines the strengths of the two dominant adaptation approaches. Hard prompt optimisation (left) keeps the backbone frozen but searches a discrete vocabulary, which limits expressiveness on specialised domains and precludes gradient-based optimisation. LoRA fine-tuning (centre) enables gradient-based adaptation and high accuracy, but modifies millions of backbone weights and risks catastrophic forgetting. SoftPrompt (right) optimises L continuous tokens at the cross-modal boundary of the multimodal sequence, reaching LoRA-level accuracy while keeping the backbone fully frozen and training only L×dL×d parameters (8,1928,192 at L=2L=2; L is sized per domain, §5.2). Like hard prompts and unlike LoRA, the learnt tokens transfer across models (§6.3) and can be verbalised into editable text (§6.2). Vision-language models (VLMs) trained on web-scale data detect common objects well, but their accuracy collapses on specialised domains: aerial imagery, industrial inspection, medical scans (Radford et al., 2021; Bai et al., 2023; Robicheaux et al., 2025). In practice, domain experts are expensive and annotation budgets are small, often consisting of 10 labeled images. The question is how to spend them. Existing answers fall into two families: Discrete (hard) prompt optimisation and parameter-efficient fine-tuning (PEFT). Similar to our approach, discrete (hard) prompt optimisation (Agrawal et al., ; Gare et al., 2026) also adapts a fully frozen model through prompts rather than weight updates. However, it optimises discrete token sequences using gradient-free search, whereas SoftPrompt learns continuous prompt embeddings with gradient descent. In our experiments these approaches improve performance at most 1.21.2 mAP over a hand-written prompt (12.512.5 vs. 11.311.3 mAP). Parameter-efficient fine-tuning, chiefly LoRA (Hu et al., 2021), learns low-rank adaptations to the model parameters. Its rank must be selected per task, and the resulting specialization comes at the expense of general-domain performance. Figure 1 shows both facts on Roboflow20-VL: the LoRA rank sweep peaks at 14.214.2 mAP and then declines, and the same adapters degrade the backbone’s VQA accuracy on NaturalBench (Li et al., 2024a) by up to 56%56\% relative, a cost invisible to detection-only evaluation. This dichotomy raises a question: can we have gradient-based adaptation while keeping the pretrained backbone unchanged? Soft prompting (Lester et al., 2021) does exactly this: optimising a few continuous token embeddings at the input of a frozen model. Yet, it is rarely the method of record for multimodal detection. We find that its weak reputation is largely an artifact of two defaults inherited from NLP: soft tokens are prepended to the sequence, and initialised with task semantics. Our experiments show that both defaults are suboptimal in a multimodal sequence. Injecting the tokens instead at the cross-modal boundary, between the visual patches and the text query, wins our six-way ordering ablation: 10.010.0 average mAP vs. 8.48.4 for the NLP-standard prefix transplant and 6.76.7 for the worst ordering, up to 6×6× on a single dataset (§5.1). Initialising them with the token corresponding to a single space character, so that the prompt’s meaning is unchanged at the start, beats every semantic and random initialisation we tried (§5.3); the principle is the one behind residual learning (He et al., 2016), where the safest default for a new module is to change nothing. With these two choices, the resulting method, which we call SoftPrompt, is simple: we train L=1L=1–33 learnt tokens, sized per domain (7,1687,168 parameters on average for an 8B-parameter backbone) with a handful of single-GPU runs. On Roboflow20-VL (10-shot), our result equals the best configuration of a six-rank LoRA sweep (14.214.2 vs. 14.214.2 mAP at r=64r=64), outperforms PEFT across all evaluated ranks, surpasses gradient-free prompt optimisation, and does so with zero forgetting by construction (Figure 1). The learnt tokens also keep the practical virtues of text prompts that weight updates lose: they are transferable and interpretable. If loaded as-is into the newer Qwen3.5-9B, they still improve over that model’s zero-shot baseline (+0.8+0.8 mAP, §6.3). They are interpretable in the sense that asking the model to verbalise its own soft tokens yields a human-readable prompt that matches the strongest dedicated prompt search (DetPO) and outperforms GEPA on the same benchmark (§6.2). Finally, we test whether the recipe is specific to detection. On three RoboCasa (Nasiriany et al., 2024) manipulation tasks (10 demonstrations each), we adapt a frozen π0.5 _0.5 vision-language-action policy (Physical Intelligence et al., 2025) with SoftPrompt (§7). The position lesson reappears in a sharper form: a prefix-only prompt fails (1.7%1.7\% success), but a prompt that also reaches the action expert, the module that produces the output, matches or beats coverage-matched LoRA-based PEFT on the two tasks the base policy can barely do. The third task, one the base already performs, marks the boundary: the prompt overrides the competent policy while LoRA improves it. Where the prompt lives, and whether the frozen model needs to be asked or corrected, decide the outcome. Contributions. First, we identify the two levers that make soft prompting competitive for multimodal detection: injection at the cross-modal boundary and a meaning-preserving initialisation, each validated by a dedicated ablation (§5). Second, we compare SoftPrompt against hard prompts, discrete prompt optimisation, a full LoRA rank sweep, and a fine-tuned specialist detector on Roboflow20-VL under strict 10-shot supervision, and measure what each method costs the generalist backbone on NaturalBench VQA and RefCOCO grounding (§4). Third, we show that the learnt tokens behave like language: we analyse what they encode, distill them into portable hard prompts that beat prompt-search baselines, and quantify their transfer across model versions (§6). Finally, we extend the recipe to a vision-language-action policy across three manipulation tasks and show that the placement principle, not the detection task, is what carries it, and that the frozen model’s own competence bounds it (§7). 2 Related Work VLMs and open-vocabulary detection. Modern VLMs such as Qwen3-VL (Bai et al., 2023), LLaVA (Liu et al., 2023), and InstructBLIP (Dai et al., 2023) emit detections as structured text. Specialist open-vocabulary detectors, Grounding DINO (Liu et al., 2024) and OWL-ViT (Minderer et al., 2022), align region features with text embeddings and remain far stronger on raw mAP (Table 1); they are single-task models and serve as our specialist reference point. Prompt tuning and PEFT. Soft prompting originates with prompt tuning (Lester et al., 2021) and prefix tuning (Li and Liang, 2021), extended to deep layers by P-Tuning v2 (Liu et al., 2022), to vision encoders by VPT (Jia et al., 2022), and to CLIP-style alignment by CoOp/CoCoOp (Zhou et al., 2022b; Zhou et al., 2022a), ProDA (Lu et al., 2022), and MaPLe (Khattak et al., 2023). Like these methods, we train only continuous input tokens, but we are shallow, not deep, prompt tuning: unlike VPT-Deep and P-Tuning v2, which re-inject a fresh learnable prompt at every transformer layer and so touch the model’s internal representations directly, our soft tokens enter the sequence once, at the input, exactly as VPT-Shallow’s do, and never again; every other layer’s weights and activations remain frozen and untouched by anything trainable. We also study generative VLM detection, where the prompt sits inside a single autoregressive multimodal sequence and its position relative to the two modalities becomes a first-class variable, which we show is decisive (§5.1). LoRA (Hu et al., 2021) and adapters (Houlsby et al., 2019) are the weight-space alternatives we compare against. Prompt transfer across models was studied for text-only LMs by SPoT (Vu et al., 2022); §6.3 gives the multimodal counterpart. Alazraki et al. (2025) inject learnable textual preambles into a frozen LLM via reinforcement learning; their preambles live in discrete token space and need a reward signal, whereas our tokens are continuous and trained by plain gradient descent. Hard prompt optimisation. GEPA (Agrawal et al., ) evolves discrete prompt candidates; TextGrad (Yuksekgonul et al., 2024) propagates natural-language critiques as gradient-like optimisation signals, but, unlike SoftPrompt, does not perform numerical gradient descent; AutoPrompt (Shin et al., 2020) performs gradient-guided discrete search; DetPO (Gare et al., 2026) specialises discrete search to detection. These methods keep the model frozen, as we do, but search a discrete space without gradient access. We quantify the resulting gap on identical benchmarks (Table 1) and show that a gradient-trained continuous prompt can be distilled back into their discrete space at a profit (§6.2). Prompt distillation and forgetting. PromptKD (Li et al., 2024b) distills knowledge between prompts in a teacher-student setup; our Soft2Hard instead asks the model itself to verbalise its own soft tokens, requiring no student training. Catastrophic forgetting (McCloskey and Cohen, 1989) motivates our cost analysis; our contribution there is measurement (NaturalBench and RefCOCO probes for detection-adapted VLMs), not a new mitigation. 3 Method Figure 2 positions SoftPrompt between the two existing adaptation families; Figure 3 below details the full pipeline and, in panel (E), every injection ordering ablated in §5.1. The method has three ingredients: where the tokens go, how they are trained, and how they are initialised. Each is simple; each is validated by an ablation in §5. 3.1 Soft tokens at the cross-modal boundary A VLM fθf_θ encodes an image into NIN_I patch embeddings ∈ℝNI×dI ^N_I× d and a text query into NTN_T token embeddings ∈ℝNT×dT ^N_T× d, then decodes bounding boxes autoregressively as structured text. We introduce L learnable soft tokens ∈ℝL×dS ^L× d and must choose where in the sequence they go. The six orderings of image (I), soft tokens (S), and text (T) define the candidate positions: the soft-first prefixes (SIT, the NLP-standard transplant, and STI), the soft-last suffixes (ITS and TIS), and the two cross-modal boundaries (IST and TSI). We place them between the modalities: ~=[;;]∈ℝ(NI+L+NT)×d, X=[I;\;S;\;T] ^(N_I+L+N_T)× d, (1) the IST ordering; each ordering acronym is literally the sequence order of I, S, and T. The intuition is that tokens at this position attend to every visual patch and are attended by every text token, making them a natural bottleneck for a domain-specific visual prior; a prefix token, by contrast, precedes all context and has nothing to condition on. §5.1 confirms this intuition: IST beats every prefix and suffix ordering, and neither prefix ever wins an above-floor dataset (one of the ablation pool’s eight where scores are not pinned near zero regardless of configuration; the ninth, dentalai, is a floor control, §4.1). The embedding matrix itself is never modified; it only gains L new rows and loses nothing. Image“detect aircraft”QueryI (NIN_I image tokens)T (NTN_T text tokens)S (soft tokens)only S trainedL×d=8,192L×d=8,192 params[;;][\,I;\;S;\;T\,]IST orderingFrozen VLM8B params (frozen) "class": "aircraft", "bbox":[x,y,w,h] Detection output(A) Inputs(B) Multimodal sequence(C) Frozen VLM(D) Output(E) All six orderings (§5.1)ISTdefault, shown aboveTSIITSTISSITSTI Figure 3: Overview of SoftPrompt: L continuous tokens at the cross-modal boundary of a frozen VLM. (A) An image and a text query are encoded into patch and token embeddings. (B) L learnable soft tokens S (blue) are inserted between image tokens I (purple) and text tokens T (teal), forming [;;][I;S;T], the IST ordering. (C) The frozen backbone processes the sequence; only S (L×d=8,192L×d=8,192 parameters for Qwen3-VL-8B at L=2L=2) is trained on the 10-shot support set. (D) Detections are generated as structured text. (E) Simplified token layout for all six orderings ablated in §5.1 (Table 4): the two cross-modal boundaries (IST, TSI), the two soft-last suffixes (ITS, TIS), and the two soft-first prefixes (SIT, STI); IST (outlined) is the default used in panel (B). Unlike prompt-tuning variants that fine-tune the embedding layer, the embedding matrix itself is untouched: the model gains L new rows and loses nothing. 3.2 Training and what the gradient reaches Training minimises the backbone’s standard next-token detection loss over the 10-shot support set D, with θ frozen: min⁡ℒdet(,,fθ). _S\;L_det(S;\;D,f_θ). (2) The length L is selected per domain by the short sweep of §5.2; on Roboflow20-VL the selected values are L∈1,2,3L∈\1,2,3\ with mean 1.751.75 (per-dataset configurations in Appendix C). For Qwen3-VL-8B (d=4096d=4096) this is 4,0964,096–12,28812,288 trainable parameters, 7,1687,168 on average, roughly 1,500×1,500× fewer than the smallest LoRA we evaluate (r=4r=4) and 24,000×24,000× fewer than the rank that matches its accuracy (r=64r=64). This frugality comes with a distinct optimisation signature, detailed in Appendix A (Figure 7). The backward pass is the same computation for both methods: the gradient flows from the loss back through every frozen block. The difference is where the trainable sinks sit. LoRA places one at every depth, so the nearest adapter is one block from the loss and an update lands locally at every layer along the chain; SoftPrompt’s single sink is the input-side soft tokens, at the far end of the longest backward path, with no update anywhere before it. The soft tokens do sit in the stream that produces the loss, so the gradient survives and prompt tuning is viable; but this single far-end lever is exactly why where the tokens are injected matters so much (§5.1), a sensitivity a per-layer weight edit does not share. The same principle returns, in a sharper form, when we prompt an action policy in §7. 3.3 Initialisation: change nothing at the start The third lever is where optimisation starts. Residual learning (He et al., 2016) made deep networks trainable by making the default behaviour of every layer the identity; we apply the same principle to the prompt. We initialise all soft tokens with the embedding of a single space character, a token that leaves the prompt’s meaning unchanged. The model’s behaviour therefore starts exactly at the (already sensible) frozen baseline, and optimisation moves away only where the loss demands it. A semantic initialisation, such as class names or a task description, instead starts by changing what the prompt says, and the optimiser must first undo any damage before it can help. §5.3 shows the space token wins 5 of 8 above-floor pool datasets and beats the frozen baseline on all 8, while the semantically richest initialisations are the least reliable. 3.4 Soft-to-Hard distillation (Soft2Hard) The soft tokens live in the backbone’s own representation space, so the model itself should be able to say what they mean. Soft2Hard converts a trained soft prompt into a reusable hard (discrete) prompt in three stages, all served by a single extended-vocabulary engine (the base model plus the trained soft-embedding rows): 1. Generate. For every training image, query the model in its training format and ask it to describe the objects being detected; keep the natural-language responses. 2. Summarise. A text-only pass condenses the per-image descriptions into one global detection instruction. 3. Evaluate. Insert the distilled instruction into the standard detection template and score the base model, with no soft tokens anywhere, on the test split. To isolate the soft prompt’s contribution, we run the identical recipe with descriptions generated without the soft tokens (the base source); the soft−-base gap is what the soft prompt adds (§6.2). 4 Experiments 4.1 Setup Benchmark. We evaluate on Roboflow20-VL, a 20-domain subset of Roboflow-100-VL (Robicheaux et al., 2025) covering all seven of its super-categories (aerial, documents, flora and fauna, industrial, medical, sports, other). Each domain provides a 10-shot support set; evaluation uses the benchmark’s full held-out test split. We additionally use the LVIS Rare 50 protocol of DetPO (Gare et al., 2026) as a long-tail cross-benchmark check (§4.4). Models. All main detection results use frozen Qwen3-VL-8B-Instruct (Bai et al., 2023). Qwen3.5-9B appears only as the target of the cross-model transfer study (§6.3), Gemma-4-12B-it (Gemma Team et al., 2024) as an unrelated second backbone trained from scratch (§4.5), and π0.5 _0.5 (Physical Intelligence et al., 2025) in the robotics study (§7). Baselines. HardPrompt: the frozen backbone queried with class names, and, as a training-free reference, with hand-written per-class instructions. GEPA (Agrawal et al., ) and DetPO (Gare et al., 2026): discrete prompt optimisation on the same frozen backbone. LoRA (Hu et al., 2021): ranks 44–128128; Table 1 shows the best rank and Appendix F the full sweep, including the instruction-augmented variant discussed in §4.2. Grounding DINO (Liu et al., 2024): zero-shot and 10-shot fine-tuned specialist reference. All generalist methods share the same backbone and the same multi-class evaluation protocol (all classes queried in one call). Class-name queries are the default protocol throughout; appending hand-written per-class instructions is the variant reported for HardPrompt (Table 1) and LoRA (Appendix F), and GEPA and DetPO supply their optimised prompt in place of a hand-written one, which is the method itself. DetPO’s single-class protocol (one class per call, more calls per image) is not cost-comparable and is reported in Appendix G. Training. SoftPrompt: AdamW, lr 5×10−35×10^-3, cosine schedule, batch size 4, up to 5 epochs; space-token initialisation; metrics are COCO-style mAP@50:95 on a 0–100 scale. Full implementation details, including the LoRA recipe, are in Appendix B. Ablation pool. All ablations (§5, Appendix K) run on a fixed pool of 9 of the 20 datasets, chosen so that every RF20 super-category is represented: aerial-airport (Aerial), actions and lacrosse (Sports), all-elements (Documents), defect-detection (Industrial), dentalai (Medical), flir-camera-objects and new-defects-in-wood (Other), and wb-prova (Flora and Fauna). The same nine datasets appear in every ablation table and figure. dentalai, where every method scores ∼0 0 (the medical failure discussed in §4.2), is kept as the pool’s known floor control: no manipulation in any ablation moves it, so it is excluded from ablation win-counts and averages but reported in every table. Each table carries the frozen-baseline row (class-name query, no soft tokens), taken from the same evaluation that produces Table 1; the identical values serve as the L=0L=0 points of the length sweep. Disclosures. (i) Numbers are single runs; we observe a 2020–28%28\% spread across seeds in this few-shot regime (measured directly: two additional seeds per pool dataset, Appendix M), so per-dataset deltas below ∼1 1 mAP should be read as ties, and the headline claim (14.214.2 vs. 14.214.2) as parity. (i) All main-table SoftPrompt numbers use the fixed IST position. A per-dataset best-of-IST/TSI selection reaches 14.714.7 but selects the position on the test split; it appears only as the distillation ceiling and transfer origin (§6), flagged there, and in Appendix F. 4.2 Main results on Roboflow20-VL Table 1: Two learnt tokens match the best LoRA fine-tune on Roboflow20-VL 10-shot (mAP@50:95). SoftPrompt reaches 14.214.2 all-domain mAP while training 24,000×24,000× fewer parameters than LoRA r=64r=64, beats every training-free prompt baseline by 1.71.7–3.43.4 mAP, and wins the Sports and Other categories outright among generalist methods; only the fine-tuned specialist (bottom block) remains ahead, at 2.5×2.5× the mAP. Domains aggregated into the seven Roboflow-100-VL super-categories; the last column averages all 20 datasets. Params is the trainable footprint. The LoRA row shows the best rank of the class-names sweep; the full r=4r=4–128128 sweep and the instruction-augmented variant (see §4.2) are in Appendix F. SoftPrompt’s prompt length is chosen per domain (L∈1,2,3L∈\1,2,3\, mean 1.751.75, hence the 7.27.2K average parameter count; per-dataset configurations in Appendix C). Bold: best per column within the generalist comparison. DetPO multi-class numbers are its re-evaluation under the shared evaluation template used by all rows; single-class DetPO variants (not cost-comparable) are in Appendix G. The specialist block bounds the claim: a fine-tuned single-task detector remains far stronger on raw mAP (see the discussion in §4.2). Method Params Aerial Doc. F. & F. Ind. Med. Sports Other All Frozen backbone, prompt-space (training-free) HardPrompt (class names) 0 6.4 7.5 25.0 8.2 0.0 8.6 9.1 10.8 HardPrompt + instructions 0 7.4 8.5 25.0 9.1 0.2 9.7 9.6 11.3 GEPA (Agrawal et al., ) 0 6.3 10.8 22.4 8.5 0.1 12.8 11.5 11.6 DetPO (Gare et al., 2026) 0 8.6 10.1 26.7 9.9 0.1 11.0 10.5 12.5 Frozen backbone, prompt-space (gradient-trained, ours) SoftPrompt (IST, per-domain L) 7.2K 9.0 14.6 27.7 8.7 0.0 13.8 14.3 14.2 Weight-space fine-tuning (best rank of the class-names sweep; full sweep in Appendix F) LoRA (r=64r=64) 174.6M 20.2 19.9 28.6 7.1 0.7 9.6 9.4 14.2 Specialist detector (single-task reference) Grounding DINO (Liu et al., 2024) (zero-shot) 0 31.2 5.0 34.3 13.0 0.4 5.7 16.9 17.3 Grounding DINO (10-shot fine-tune) 172M 40.6 32.0 44.9 41.0 35.3 30.8 26.8 35.7 Table 1 compares all methods under 10-shot supervision. Three observations follow. First, at any comparable parameter budget, SoftPrompt dominates. With 7,1687,168 parameters on average (per-domain L∈1,2,3L∈\1,2,3\) it reaches 14.214.2 all-domain mAP, 1.71.7–3.43.4 mAP above every training-free prompt baseline (closest: DetPO at 12.512.5) and above every LoRA rank except r=64r=64, which it equals while training 24,000×24,000× fewer parameters. Second, the LoRA rank sweep is itself informative: accuracy increases up to r=64r=64 before declining at r=128r=128 (13.613.6; Appendix F), revealing diminishing returns from increasing adaptation capacity. Yet even before this decline, the additional accuracy comes at the hidden cost of forgetting. As shown in Figure 1, the relative drop in NaturalBench VQA group accuracy compared to the unadapted backbone rises with LoRA rank, reaching 35%35\% at the accuracy-matching r=64r=64 and 56%56\% at r=128r=128. In contrast, SoftPrompt incurs no forgetting, as the soft prompt leaves the backbone weights entirely unchanged. Third, discrete prompt optimisation under the same multi-class protocol adds at most 1.21.2 mAP over a hand-written prompt: GEPA reaches 11.611.6 and DetPO 12.512.5, versus 11.311.3 for class names with instructions. Gradient access to a continuous prompt, not prompt optimisation per se, is what closes the remaining gap to fine-tuning (12.5→14.212.5→ 14.2). The contrast in prompt size makes the same point from the other side: SoftPrompt uses one to three learnt tokens, while the optimised discrete prompts run to hundreds or thousands of tokens that are re-processed at every inference call. Why not a specialist detector? Table 1 includes the strongest specialist: Grounding DINO fine-tuned on the same 10 shots reaches 35.735.7 mAP, 2.5×2.5× any generalist method. If raw single-task mAP is the only goal, a specialist detector remains the right tool. Our question is different: practitioners increasingly deploy one generalist VLM for detection, VQA, grounding, and captioning at once, and the relevant cost is what adapting that generalist to a new domain does to the generalist. Figure 1 shows the weight-space answer (up to 56%56\% of its VQA ability) and the prompt-space answer (nothing). The specialist, which can do none of the other tasks, is the boundary of our claim, not a refutation of it. The medical failure. On the Medical category, every generalist method collapses (≤1.0≤ 1.0 mAP for all prompt- and weight-space adapters) while the fine-tuned specialist reaches 35.335.3. Ten shots recover nothing in either adaptation space, consistent with the knowledge being genuinely absent from the backbone rather than badly queried. Where the model does not already know, neither asking nor light teaching helps; we return to this boundary in §8. Finding. In the 10-shot regime, prompt-space and weight-space adaptation are accuracy-equivalent: 7,1687,168 prompt parameters on average reach 14.214.2 all-domain mAP, equal to the best of six LoRA ranks (r=64r=64, 174.6174.6M parameters), with every other rank strictly below the prompt. 4.3 What adaptation costs the generalist A generalist VLM is deployed for more than the domain it was just adapted to. We measure what each adaptation does to two capabilities the detection task never touches: general VQA and referring-expression grounding. SoftPrompt cannot move either number: the backbone is frozen and the domain prompt is simply not supplied off-domain, so its off-domain behaviour is identical to the base model by construction. The question is what LoRA does. Cross-task VQA (NaturalBench). Figure 1 (bottom) plots the relative drop in NaturalBench group accuracy (base model: 37.137.1 G-Acc) for every rank of the class-names LoRA sweep. The drop grows with rank, 35%35\% relative at r=64r=64 and 56%56\% at r=128r=128, and it grows through exactly the rank where detection accuracy peaks: the one LoRA configuration that matches SoftPrompt’s accuracy pays a third of the backbone’s VQA ability for it. Detection-only evaluation would never see this cost. Cross-task grounding (RefCOCO). Forgetting is not uniform, and honesty requires the other half of the picture. A low-rank adapter (r=4r=4, single source domain) evaluated on the eight standard RefCOCO splits (Yu et al., 2016; Mao et al., 2016) drifts by −0.01-0.01 points on average, within run-to-run noise (full table in Appendix H). A gentle LoRA does not measurably forget grounding. The correct claim is therefore structural, not absolute: LoRA’s forgetting is a rank- and scale-dependent risk that grows with exactly the configurations practitioners scale to when low ranks underperform, whereas the frozen-backbone prompt carries zero risk at any scale, guaranteed rather than measured. The deployment consequence is architectural: a single frozen backbone serves all domains simultaneously, with each domain requiring only an independent L×dL×d soft-prompt parameter block. Because the backbone remains unchanged, introducing a new domain cannot degrade performance on any existing domain or task. In contrast, weight-space adaptation requires either a separate checkpoint per domain or a continual-learning strategy to offer the same guarantee. Finding. LoRA’s detection gains carry a rank-dependent tax: 35%35\% relative VQA degradation on NaturalBench at the accuracy-matching r=64r=64, and 56%56\% at r=128r=128; at r=4r=4 the RefCOCO drift is −0.01-0.01 points, so the tax is scale-dependent rather than universal. SoftPrompt pays zero at any scale, by construction. 4.4 A higher-baseline check: LVIS Rare 50 Roboflow20-VL domains are chosen to be far from web pretraining; a fair worry is that soft prompts only help when the backbone is weak. LVIS Rare 50 (Gare et al., 2026; Gupta et al., 2019) tests the opposite regime: 50 long-tail LVIS classes whose names are ordinary English nouns, where the frozen baseline is already 2×2× higher (22.422.4 mAP, vs. 10.810.8 on Roboflow20-VL) despite still being far from saturated. Table 2: Gains survive a higher baseline. SoftPrompt adds +3.3+3.3 mAP over the 22.422.4 frozen baseline (2×2× Roboflow20-VL’s) at its best length (L=1L=1); a mis-sized prompt reverses the gain (L=3L=3 falls to 20.720.7, below baseline), the same length-sensitivity §5.2 finds on Roboflow20-VL. LVIS Rare 50 (Gare et al., 2026), mAP@50:95. Method mAP@50:95 Δ HardPrompt (frozen baseline) 22.4 – SoftPrompt (IST, L=1L=1) 25.7 +3.3+3.3 SoftPrompt adds +3.3+3.3 mAP on top of that higher baseline (Table 2), comparable to its margin over hard prompts on Roboflow20-VL (+3.4+3.4), rather than overfitting its 10 shots per class. The gain is length-sensitive in the direction §5.2 predicts: L=1L=1 wins, while L=3L=3 falls to 20.720.7, below the baseline. A mis-sized prompt does not merely plateau, it hurts, which is why the per-domain L sweep (with the L=0L=0 control) is part of the method rather than an optional tuning step. 4.5 Generalising to a different VLM family: Gemma-4-12B Every result so far trains and evaluates on a single VLM family (Qwen3-VL); the cross-model study of §6.3 moves a prompt trained on Qwen3-VL-8B to a newer Qwen3-VL release, still within-family. A stronger generality claim requires training the method fresh, from scratch, on a genuinely unrelated backbone. We repeat the recipe – cross-modal-boundary placement, space-token initialisation, a per-domain hyperparameter search – on Gemma-4-12B-it (Gemma Team et al., 2024), on the 9-dataset ablation pool (§4.1). Because Gemma-4’s optimisation landscape turned out to be noisier than Qwen3-VL’s (Appendix D), the per-domain search here also sweeps format, learning rate, gradient-accumulation, and initialisation, not length alone, all selected on a held-out split of the 10-shot support set, never on test. Table 3: The recipe generalises to a second, unrelated VLM family, but the search’s holdout margin does not reliably predict the test-time margin. Gemma-4-12B-it, 9-dataset ablation pool, mAP@50:95. Test Δ : the search-selected config’s held-out-set-selected hyperparameters, evaluated on test, minus the frozen baseline – the number a practitioner would actually get by following the search. Bold: dataset where the search-selected config wins on test. dentalai is the pool’s floor control (§4.1), reported but excluded from the win-count and the mean. Full per-dataset holdout-vs-test comparison, search methodology, and every config in Appendix D. Dataset Baseline SoftPrompt (search-selected) Test Δ wb-prova 18.5 24.3 +5.8+5.8 all-elements 13.5 17.4 +3.9+3.9 new-defects-in-wood 3.7 3.9 +0.2+0.2 aerial-airport 11.3 11.4 +0.1+0.1 actions 0.8 1.0 +0.2+0.2 dentalai (floor) 0.5 0.6 +0.1+0.1 flir-camera-objects 14.0 13.9 −0.1-0.1 lacrosse-object-detection 11.3 10.5 −0.8-0.8 defect-detection 6.6 3.7 −2.9-2.9 Mean (above-floor) 10.0 10.8 +0.8+0.8 The recipe transfers: the search-selected soft prompt beats the frozen baseline on 5 of 8 above-floor datasets (Table 3), including a +5.8+5.8 mAP gain on wb-prova and +3.9+3.9 on all-elements. But the search’s own holdout margin is a poor predictor of that test gain. Averaged over the same 8 datasets, the selected config’s holdout margin over its own control is +11.0+11.0 mAP; the realised test margin is +0.8+0.8, roughly an order of magnitude smaller, and on 3 datasets (defect-detection, flir-camera-objects, lacrosse-object-detection) the sign flips outright: a holdout margin of +4.9+4.9 to +10.9+10.9 mAP becomes a test loss. defect-detection is the sharpest case: the selected config wins its holdout comparison by 4.9×4.9× the control, then loses to baseline by 2.92.9 mAP on test, worse than the simplest untuned default we tried for that dataset (Appendix D). This is not the behaviour §5’s screening regime shows on Qwen3-VL, where a short holdout sweep reliably separates configs by a wide, test-consistent margin; on Gemma-4 it does not, at least not with the training budget used here. Finding. The soft-prompt recipe generalises to a second, architecturally unrelated VLM family (Gemma-4-12B), beating the frozen baseline on 5 of 8 above-floor datasets by a mean of +0.8+0.8 mAP. But holdout-based hyperparameter selection, which reliably predicts test performance for Qwen3-VL (§5), does not transfer as a selection procedure to Gemma-4: the mean holdout margin (+11.0+11.0 mAP) overstates the mean realised test margin by roughly 14×14×, and on 3 of 8 datasets the selected config’s holdout win becomes a test loss. Backbone generality and search-transfer reliability are separate claims, and only the first holds here without qualification. 5 Ablation Study The method rests on two claims: position matters and initialisation matters. This section tests each in isolation, plus the prompt-length choice both interact with. All ablations run on the fixed 9-dataset pool of §4.1, which covers every RF20 super-category, in the 5-epoch fast-training regime with each dataset’s canonical L; mAP@50:95, single runs. Every table carries the frozen baseline as a reference row. 5.1 Injection position We ablate all six orderings of image (I), soft tokens (S), and text (T) across the pool (Table 4): the two cross-modal bridges, the two soft-last suffixes, and the two soft-first prefixes. Table 4: Where the tokens are injected is worth up to 6×6× mAP. The cross-modal boundary (IST) wins 6 of 8 above-floor datasets, beats the frozen baseline on all 8, and is never worst; neither soft-first prefix (SIT, STI) ever wins one. Bold: best position per dataset. dentalai is the floor control: five of six orderings score exactly 0.00.0 and STI alone reaches 0.20.2, consistent with the domain-knowledge gap discussed in §4.2 rather than any one position choice. †wb-prova’s TIS run collapsed to 0.00.0. Position Description aerial actions all- elements defect- detection dentalai flir lacrosse new- defects wb- prova† Avg Frozen baseline (L=0L=0) 4.4 2.9 10.6 4.9 0.0 9.3 14.2 5.9 27.6 8.9 IST Cross-modal bridge 8.0 4.0 11.7 8.0 0.0 16.6 24.0 11.0 28.8 12.4 TSI Text-vision bridge 11.4 1.9 14.4 6.7 0.0 5.1 4.1 8.2 26.7 8.7 ITS Suffix 6.3 3.3 9.1 5.6 0.0 6.8 17.1 9.1 25.9 9.2 TIS Suffix, modalities swapped 7.2 0.8 13.1 6.7 0.0 13.2 18.7 8.1 0.0 7.5 SIT Prefix (NLP transplant) 6.3 0.8 10.4 5.6 0.0 11.8 18.3 10.0 22.8 9.6 STI Prefix, modalities swapped 6.7 2.0 6.7 2.8 0.2 10.2 6.2 5.2 22.3 6.9 IST wins 6 of the 8 above-floor pool datasets, beats the frozen baseline on all 8, and is never the worst ordering. IST leads clearly (12.412.4 average); the prefix SIT, the plain suffix ITS, and the other boundary ordering TSI cluster in a middle tier (9.69.6, 9.29.2, 8.78.7) closer to each other than to either extreme; the mismatched-modality suffix and prefix trail (TIS 7.57.5, STI 6.96.9). SIT’s respectable average does not translate into ever winning: despite the middle-tier score, neither prefix ordering wins a single above-floor dataset, IST does; on lacrosse the spread between the best and worst ordering is ∼6× 6× (24.024.0 vs. 4.14.1). A prompt that precedes all context has nothing to condition on; at the cross-modal boundary, the same tokens see every visual patch and are seen by every text token. This is the concrete sense in which multimodal soft prompting is not NLP prompt tuning transplanted: the transplant’s default placement, SIT, gives up 2.82.8 average mAP to the boundary despite its middling average and never wins outright, and reversing the modalities on top of it (STI) gives up nearly a third more (9.6→6.99.6→ 6.9). The exceptions are systematic rather than noise, and they polarise. TSI (text before image) wins exactly the datasets where the text query alone is most informative, single-class aerial (+58%+58\% over IST) and the document-heavy all-elements, and it collapses on lacrosse, a multi-class sports scene (4.14.1 vs. IST’s 24.024.0). When the query names one known concept, priming the model with text first appears to help; when the relevant class depends on what is in the image, it hurts. We use IST as the default and treat TSI as the alternative worth testing on single-class or text-dominant domains. Finding. Injection position spans 6.96.9 to 12.412.4 average mAP across the six orderings of image, soft tokens, and text, up to 6×6× on a single dataset; the cross-modal boundary placement (IST) achieves the highest performance on 6 of the 8 above-floor pool datasets, while neither soft-first prefix ordering (SIT or STI) wins on any of them. Interestingly, STI is the only ordering to achieve non-zero performance on the challenging dentalai dataset (0.2 vs. 0.0 mAP), suggesting that placing soft prompts before the image can occasionally aid adaptation in particularly difficult domains. 5.2 Prompt length Figure 4 sweeps L∈0,1,2,3,4,8L∈\0,1,2,3,4,8\ on all nine pool datasets (IST), plus the aerial-airport TSI curve; the L=0L=0 points are the frozen baselines of Table 4. Figure 4: Optimal prompt length is small and dataset-dependent, and a mis-sized prompt can lose to L=0L=0. mAP@50:95 vs. L (10-shot, 5-epoch training) on the nine ablation-pool datasets under IST, plus the aerial-airport TSI curve (dashed); stars mark each curve’s best L, and L=0L=0 is the frozen baseline (Table 1 source). On aerial-airport, IST decays with L while TSI grows, reaching 12.112.1 at L=8L=8: length and position interact. dentalai is the flat ∼0 0 floor control. No universal L exists, and no clean class-count rule survives the data: lacrosse peaks at L=1L=1 (24.324.3) and loses more than half of that by L=8L=8; flir peaks at L=3L=3; new-defects drifts upward to L=3L=3; aerial is flat until L=3L=3 and then decays. Two results discipline any tuning recipe. First, on wb-prova a mis-sized prompt is catastrophic (5.95.9 at L=8L=8, against a 27.627.6 baseline): a soft prompt can hurt, and L=0L=0 belongs in every sweep as the sanity control. Second, length interacts with position: on aerial, IST decays with L while TSI (dashed in Figure 4) improves monotonically to L=8L=8 (12.112.1, the best aerial result in our sweeps). The practical recipe is a short sweep over 0,1,2,3,4,8\0,1,2,3,4,8\ per domain; each configuration is a single-GPU run whose training loop takes minutes on low-resolution domains and a median of ∼2 2 hours across our ablation runs (Appendix B). 5.3 Initialisation Table 5 compares five initialisations across the pool. Table 5: What should soft prompts be initialised with? The token that means nothing. A single space token wins 5 of 8 above-floor datasets and beats the frozen baseline on all 8; the semantically richer initialisations are the least reliable ones (task-text collapses on lacrosse, 6.66.6 vs. space’s 24.124.1; class-seeded collapses on all-elements, 5.35.3 vs. 11.611.6). Bold: best initialisation per dataset. dentalai is the floor control: every initialisation sits at the baseline, consistent with the domain-knowledge gap discussed in §4.2 rather than any one initialisation choice. Init strategy Description aerial actions all- elements defect- detection dentalai flir lacrosse new- defects wb- prova† Avg Frozen baseline (L=0L=0) 4.4 2.9 10.6 4.9 0.0 9.3 14.2 5.9 27.6 8.9 Space (" ") Single whitespace token 8.0 3.9 11.6 8.0 0.0 16.6 24.1 11.0 28.8 12.4 Mean Mean vocab embedding 9.2 3.9 6.7 4.9 0.0 7.5 14.4 7.4 13.3 7.5 Xavier Xavier uniform 6.7 3.0 8.4 5.0 0.0 12.1 21.2 13.1 19.5 9.9 Class-seeded Class name tokens 7.5 2.2 5.3 4.7 0.0 13.3 13.6 12.2 22.6 9.0 Task-text ("detect") Task description tokens 7.6 3.6 7.3 6.5 0.0 13.3 6.6 10.8 30.3 9.6 The single space token wins 5 of the 8 above-floor pool datasets, has the best average (12.412.4), and beats the frozen baseline on all 8. The semantically richer initialisations are the less reliable ones: task-text (“detect”) collapses on lacrosse (6.66.6 vs. space’s 24.124.1), and class-seeded starts collapse on all-elements (5.35.3 vs. 11.611.6). This is the identity principle of §3.3 in the data: an initialisation that preserves the prompt’s meaning starts from the frozen baseline’s behaviour and can only improve, while a semantic start first has to undo its own interference. The one dataset where space is not the best initialisation is wb-prova†: task-text wins by a narrow margin (30.330.3 vs. space’s 28.828.8), consistent with a dataset whose query, not its visuals, is the binding constraint. Mean-embedding initialisation offers a cautionary aside: it achieves the lowest validation loss on every dataset while winning or tying on mAP for only two (aerial outright, actions tied with space). Across all three ablations, validation loss does not track detection mAP; practitioners should select models on a task metric, not the language-model loss. Finding. A single space token is the best initialisation, winning 5 of 8 above-floor pool datasets (12.412.4 avg vs. 7.57.5 for mean-embedding and 9.69.6 for task-text): initialisations that preserve the prompt’s meaning dominate initialisations that encode task semantics. 6 Prompt Distillation: What Do the Tokens Encode? The ablations say success rides on prompt-like choices: position, brevity, meaning preservation. If that is right, the learnt tokens should behave like language in other respects too. This section reads them out directly, verbalises them, and transfers them. 6.1 Two regimes: class surrogates vs. general instructions To read the tokens, we retrieve their nearest neighbours in the model’s own representation spaces: the text-vocabulary embeddings and a cache of visual patch embeddings from the frozen encoder (Figure 5). The answer depends on what the prompt was forced to learn. In a probe configuration where the class name is removed from the text query and the soft tokens must stand in for it, the trained tokens retrieve the object itself: their nearest visual patches are patches of the detected class. In the high-performing IST/TSI configuration, where the class names remain in the query, the trained tokens do not match object patches; describing the class again would be redundant, and the tokens instead act as general task instructions. Figure 5: What the tokens learn depends on what the prompt lacks. Nearest text tokens and image patches for trained soft prompts. When the class name is withheld from the query (probe configuration), the soft tokens’ nearest neighbours become image patches of the target class, effectively acting as a visual surrogate for the missing class name. When class names are present (IST/TSI), the tokens no longer align with object patches and instead seem to encode general detection instructions. In both settings, the nearest text tokens remain close to those at initialization, indicating that the learnt embeddings stay centered around their initialization despite adaptation. Remarkably, when the model must represent a missing class concept, it does so more naturally in the visual embedding space than in the linguistic one, suggesting that visual features provide a more effective representation of object identity than nearby words. The nearest neighbours are not just interpretable, they are functional: substituting each trained token with its nearest text token (or visual patch) at evaluation retains most of the adaptation on the ablation pool (Appendix K). 6.2 Verbalising the prompt: Soft2Hard With the learnt soft-tokens living in the model’s own representational space, a VLM should be able to verbalize the meaning of soft tokens. We find that Naively asking the model to describe the learnt soft tokens leads to the model regarding the soft-token to its closest text tokens which is usually the initialized (space) token. We find that the soft-tokens carry meaning in the context that they where learnt with i.e. with the Image and Prompt Query, under this setting attempts to reexplain the soft-tokens fails as the soft-tokens focus the model to predict detections. Thus our solution is to ask the model to describe the image first before predicting bounding boxes thus we can compare the descriptions generated by this pipeline with the control where no soft-tokens are presented, this should show the qualitative deference in object descriptions which we can use as a prompt for detection to get a quantitative measure. If the learned token embeddings capture prompt-like semantics, those semantics should be recoverable as text, and the recovered text should retain predictive utility. Table 6 evaluates the generate-summarise-evaluate recipe of §3.4 and contains two honest decompositions. First, the distilled prompt is competitive with purpose-built discrete prompt search: 12.512.5 all-domain mAP, a tie with DetPO (12.512.5) and above GEPA (11.611.6), despite being obtained indirectly (by verbalising a gradient-trained object) rather than by searching prompt space. Second, most of that margin comes from the describe-then-summarise recipe itself: the base-source variant already reaches 12.212.2, and the soft prompt adds +0.3+0.3 on average. The soft prompt’s contribution is real but concentrated where the frozen backbone is weakest, Aerial (+4.1+4.1) and Documents (+2.6+2.6), and neutral-to-negative on categories the base model already describes well. Qualitatively (Figure 6), descriptions generated under the soft prompt highlight finer distinguishing features of the target objects, which is the plausible carrier of the gain. The distilled prompt recovers 12.512.5 of the 14.714.7 ceiling set by the per-dataset-best soft prompt (test-split position selection; §4.1, disclosure i) while being editable, auditable, and usable with any model that accepts text, including closed-source APIs that expose no embedding interface. Table 6: Verbalised soft prompts match dedicated prompt search, and most of the recipe’s value is the recipe – but the marginal +0.3+0.3 average hides a real, dataset-limited gain. Distilled global hard-prompt test mAP@50:95 per super-category; 12.212.2 of Soft2Hard’s 12.512.5 comes from the describe-summarise recipe alone, and only +0.3+0.3 from the soft prompt on average, concentrated where the backbone is weakest (Aerial +4.1+4.1, Documents +2.6+2.6). Soft2Hard (soft): descriptions generated with the soft prompt active; base recipe: the identical pipeline without soft tokens; Δ isolates the soft prompt’s contribution. SoftPrompt row: the trained soft prompt’s own test mAP, the ceiling distillation aims to recover. Bold: best method using no soft tokens at inference (a tie between Soft2Hard and DetPO). Gain/Loss: mean per-dataset Δ (soft −- base), among the 20 Roboflow20-VL datasets where the soft prompt helped/hurt the recipe, with the dataset count in parentheses. Method Aerial Doc. F. & F. Ind. Med. Sports Other All Gain (n) Loss (n) SoftPrompt (soft prompt, ceiling) 12.2 15.1 28.5 8.8 0.0 13.8 14.3 14.7 Soft2Hard (soft source, ours) 9.6 10.8 26.3 9.2 0.1 10.9 10.9 12.5 Base recipe (no soft tokens) 5.5 8.2 27.0 8.5 0.0 11.4 12.1 12.2 Δ (soft −- base) +4.1+4.1 +2.6+2.6 −0.7-0.7 +0.7+0.7 +0.1+0.1 −0.5-0.5 −1.2-1.2 +0.3+0.3 +1.9+1.9 (10) −1.4-1.4 (10) GEPA (Agrawal et al., ) 6.3 10.8 22.4 8.5 0.1 12.8 11.5 11.6 DetPO (Gare et al., 2026) 8.6 10.1 26.7 9.9 0.1 11.0 10.5 12.5 Figure 6: Descriptions generated under the soft prompt notice finer object detail. Asked to describe the objects being detected, the model with soft tokens active names distinguishing features that the base model’s description omits; these richer descriptions are what stage 2 of Soft2Hard summarises into the distilled prompt. Finding. A soft prompt can be verbalised: the distilled natural-language prompt reaches 12.512.5 all-domain mAP, a tie with the strongest dedicated prompt search (DetPO, 12.512.5) and above GEPA (11.611.6); 12.212.2 of that comes from the describe-summarise recipe and +0.3+0.3 from the soft prompt, concentrated where the backbone is weak (Aerial +4.1+4.1). That +0.3+0.3 average is a blend, not a uniform effect: the soft prompt helps the recipe on 10 of 20 Roboflow20-VL datasets (mean +1.9+1.9 mAP) and hurts it on the other 10 (mean −1.4-1.4), so the gain is real but dataset-limited, not a broad, small lift everywhere. Appendix L isolates this same soft-to-hard behaviour from the vision pipeline entirely, in a minimal synthetic-arithmetic setting: a trained soft prompt is uninterpretable when queried alone, but becomes reliably, often exactly, describable once the query preserves the context it was trained in. 6.3 Transfer across model versions Unlike weight updates, a trained soft prompt is an input, so it can at least be offered to a different model. We test the simplest protocol: prompts trained on Qwen3-VL-8B are loaded, with no projection or retraining, into Qwen3.5-9B (Table 7). Table 7: Naive cross-model transfer recovers about 17%17\% of the adaptation gap on average – but that average blends a real gain on most datasets against a real loss on the rest. The transferred prompt nets +0.8+0.8 mAP over the 9B zero-shot baseline, only 0.80.8 of the full +4.8+4.8 mAP gap to the 8B origin, and sits 4.04.0 mAP below it; per-category, it helps most on Sports (+5.3+5.3) and hurts on Industrial (−2.9-2.9, Appendix I). Qwen3-VL-8B → Qwen3.5-9B, all-domain mAP@50:95 over the 20 Roboflow20-VL datasets. 9B baseline: target model, no prompt; 9B + transfer: the 8B-trained prompt loaded as-is; 8B origin: the same prompt on its training model (upper bound). Gain/Loss: mean per-dataset Δ vs. the 9B baseline, among the datasets where transfer helped/hurt, with the dataset count in parentheses. Method All-domain mAP Δ Gain (n) Loss (n) Qwen3.5-9B baseline 9.9 – 9B + transferred prompt 10.7 +0.8+0.8 +2.9+2.9 (13) −3.1-3.1 (7) 8B origin (upper bound) 14.7 +4.8+4.8 The transferred prompt nets +0.8+0.8 all-domain mAP over the 9B zero-shot baseline, helping most on the Sports (+5.3+5.3) and Documents (+3.3+3.3) categories and hurting on Industrial (−2.9-2.9); per-category numbers are in Appendix I. That +0.8+0.8 average is again a blend, not a uniform effect: transfer beats the 9B baseline on 13 of the 20 Roboflow20-VL datasets, by a mean of +2.9+2.9 mAP, and loses on the other 7, by a mean of −3.1-3.1; the two groups are similar in size and opposite in sign, so the small headline average is the arithmetic of a real split, not a small effect everywhere. It recovers only part of the adaptation, sitting 4.04.0 mAP below its 8B-origin performance. So the tokens carry some model-independent domain signal, as a text prompt would, but most of their value is expressed in the origin model’s embedding geometry; a learnt projection between embedding spaces is the natural next step, as in text-only prompt transfer (Vu et al., 2022). Finding. Cross-model transfer’s +0.8+0.8 mAP average is a blend of a real gain and a real loss, not a small uniform effect: it beats the 9B zero-shot baseline on 13 of 20 Roboflow20-VL datasets (mean +2.9+2.9 mAP) and loses on the other 7 (mean −3.1-3.1). The gain is genuine where it appears, but is not a property of the transfer protocol uniformly across domains. The embeddings transfer better than their own description. §6.2 shows a trained prompt can be verbalised into a natural-language instruction on its own model. A natural question is whether that verbalised instruction is a better cross-model transfer vehicle than the raw embeddings above, since it is plain text and never touches the target model’s embedding space at all: we distil each dataset’s canonical Qwen3-VL-8B soft prompt into a hard prompt (the §6.2 describe-summarise recipe, best of its global/per-class variant) and evaluate it directly on Qwen3.5-9B with no soft tokens, against the same raw-embedding transfer as Table 7 (Table 8; per-dataset results in Appendix J). Table 8: Raw-embedding transfer beats its own distilled description on 18 of 20 datasets. Two cross-model transfer strategies for the same Qwen3-VL-8B-trained soft prompt, evaluated on Qwen3.5-9B: loading the trained embeddings directly (as in Table 7) vs. evaluating the §6.2 hard prompt distilled from that same prompt, with no soft tokens at all. All-dataset mean mAP@50:95 over the 20 Roboflow20-VL datasets (per-dataset results in Appendix J);“distilled” takes the better of the global/per-class variant per dataset. Transfer strategy All-dataset mAP Datasets won Median mAP Distilled hard prompt (best variant) 9.3 2/20 5.8 Raw embeddings (direct) 11.6 18/20 9.4 Direct embedding transfer wins on 18 of the 20 datasets, by a mean of +2.3+2.3 mAP and up to +9.9+9.9 (lacrosse-object-detection, 7.1→17.17.1→ 17.1). The two exceptions are both near-floor swaps below 44 mAP either way (defect-detection, 3.2→3.63.2→ 3.6; gwhd2021, 0.9→1.80.9→ 1.8), where any ordering is noise, not signal. Verbalising a soft prompt is useful for interpretability and for search-space competitiveness (§6.2), but it is lossy as a transfer channel: the continuous vector, even off its training model, carries more of the original prompt’s signal than the paragraph the model itself would write to describe it, which matches the intuition that natural language is a lower-bandwidth bottleneck than the embedding it was distilled from. Finding. The trained embeddings transfer across model versions better than their own natural-language description: raw-embedding transfer beats the distilled hard prompt on 18 of 20 Roboflow20-VL datasets (mean +2.3+2.3 mAP, up to +9.9+9.9), and both exceptions are near-floor swaps below 44 mAP. A soft prompt is stronger transferable evidence than the text it can be turned into. 7 Beyond Detection: Prompting a Robot Policy Is the recipe specific to detection, or does it adapt any frozen generative model with a multimodal input stream? We attempt to answer this question by applying our approach to optimizing a vision-language-action (VLA) policy: π0.5 _0.5 (Black et al., 2025; Physical Intelligence et al., 2025), a flow-matching policy built on a PaliGemma VLM backbone with a separate action-expert module that produces continuous robot actions. Setup. We use three atomic tasks from the RoboCasa kitchen suite (Nasiriany et al., 2024) on the PandaOmron mobile manipulator, an embodiment of the base policy was pretrained on with RoboCasa data (Mandlekar et al., 2023). The frozen base policy performs poorly on two of the tasks (PickPlaceCounterToCabinet and TurnOnSinkFaucet, 5.0%5.0\% zero-shot success each); the third it already partly solves (OpenDrawer, 30.0%30.0\%). Adaptation uses K=10K=10 demonstration episodes per task, the manipulation analogue of our 10-shot detection protocol. Evaluation is closed-loop success rate over 20 freshly sampled scene instantiations (held-out object placements and layouts) per seed, 3 seeds per arm; every arm sees byte-identical scenes via per-episode environment seeding. The soft prompt follows the detection recipe exactly: IST format, space-token initialisation, trained with the policy frozen. The LoRA baselines follow the openpi reference configuration (rank 16); details in Appendix B. Table 9: The placement lever generalises to VLAs, and base competence marks its boundary. A dual prompt reaching the action expert lifts the two near-floor tasks from 5.0%5.0\% to 23.3%23.3\% and 31.7%31.7\% success, matching or beating coverage-matched LoRA at ∼400× 400× fewer parameters; the same prompt drops the task the base already partly solves, OpenDrawer, from 30.0%30.0\% to 21.7%21.7\% unless early-stopped (45.0%45.0\%), while every LoRA variant improves it. Prefix-only prompts condition only the PaliGemma backbone; the dual prompt adds M=16M=16 tokens on the action-expert input, in the gradient path of the flow-matching loss; the early-stopped variant halts training at 750 of 2,000 steps. Trainable-parameter counts are exact, not estimated: N=8N=8 prefix tokens at PaliGemma’s width 20482048 gives 16,38416,384; the dual prompt adds M=16M=16 action-expert tokens at width 10241024, giving 16⋅2048+16⋅1024=49,15216·2048+16·1024=49,152 (verified directly against the saved adapter checkpoints’ tensor shapes). The action-expert tokens are randomly initialised, not space-token initialised like the PaliGemma-side tokens: the action expert has no token vocabulary to seed from. Bold: best prompt-based and best weight-based adapter per task; “–”: arm not run. Weak base Competent base Adapter Trainable PickPlace Faucet OpenDrawer π0.5 _0.5 base (zero-shot) – 5.0% 5.0% 30.0% Prompt-space (frozen policy, ours) Soft prompt, prefix-only (N=8N=8) 16K 1.7% – – Soft prompt, dual (N=16+M=16N=16+M=16) 49K 23.3% 31.7% 21.7% Soft prompt, dual, early-stopped 49K – – 45.0% Weight-space (LoRA, rank 16) LoRA, PaliGemma-only (coverage-matched) 19.6M 23.3% 21.7% 66.7% LoRA, + action expert 26.5M 26.7% 26.7% 78.3% Placement decides whether the prompt can act. On PickPlaceCounterToCabinet, a prefix-only soft prompt, the direct transplant of the detection recipe onto the PaliGemma prefix, fails: 1.7%1.7\% success, at the zero-shot floor, and no sweep of token count, learning rate, or training steps lifts it reliably. The failure is one of placement, not capacity. In π0.5 _0.5, the training loss is a flow-matching regression on the action-expert output; the PaliGemma prefix output is discarded, so a prefix prompt influences the action only indirectly, through the action expert’s attention back into the prefix. This is our gradient-path argument (§3.2) taken to its limit: the prompt is no longer merely a weak lever, it is outside the output’s direct path. Moving the prompt fixes it. A dual prompt, N=16N=16 tokens on the PaliGemma prefix plus M=16M=16 tokens on the action-expert input, sits directly in the gradient path of the action loss and reaches 23.3%23.3\% success, matching the coverage-matched PaliGemma-only LoRA at 400×400× fewer trainable parameters. The prefix tokens keep the space-token initialisation of §3.3; the action-expert tokens are initialised randomly instead, since the action expert has no token vocabulary to seed an identity-preserving start from. One optimisation detail matters: the flow-matching gradient is high-variance (fresh time step and noise per sample), and a small prompt starves at small batch; gradient accumulation to an effective batch of 128 with a high learning rate is what makes the dual prompt trainable. Where the base is weak, the prompt keeps pace with the weight update. The same dual recipe, hyperparameters unchanged, transfers to TurnOnSinkFaucet: from the same 5.0%5.0\% floor it reaches 31.7%31.7\%, ahead of both the coverage-matched LoRA (21.7%21.7\%) and the rank-16 LoRA that also adapts the action expert (26.7%26.7\%). If the base policy is competent, the prompt harms perfomance. OpenDrawer inverts the picture: the frozen base already succeeds at 30.0%30.0\%, the same full-strength dual prompt drops reduces performance to 21.7%21.7\%, and every LoRA variant improves it (66.766.7 to 78.3%78.3\%). The prompt is not failing to learn; it fits the ten demonstrations as well as on the other tasks (train loss 0.020.02–0.050.05). It is a learned, fixed edit of the input that shifts the frozen policy globally, and on a task the base already solves, that shift replaces a competent pretrained behaviour with an overfit ten-demonstration one. A weight update does not share this failure mode: its per-layer corrections can leave the base computation in place except where the demonstrations demand change. Restraining the prompt recovers the sign: stopping the same configuration at 750 of 2,000 steps yields 45.0%45.0\%, above the base, though still far below rank-16 LoRA (78.3%78.3\%). Read together, the three tasks say that a soft prompt acts as a few-shot override of a frozen policy: decisive where the base is near the floor, harmful at full strength where the base is already competent. This is the action-space analogue of the regime our detection results live in, where the frozen baseline is weak on the adapted domains. Caveats. These are three atomic tasks with 20 trials per seed (±5%±5\% per point); the dual prompt’s seed spread on PickPlace is 1010–30%30\%; the early-stop point was selected on a seed-0 sweep before its 3-seed re-run. We read this as a mechanism result, not a benchmark claim. Finding. The placement principle carries beyond detection, and base competence marks its boundary: a dual prompt reaching the action expert lifts two near-floor RoboCasa tasks from 5.0%5.0\% to 23.3%23.3\% and 31.7%31.7\% success, matching or beating coverage-matched LoRA at ∼400× 400× fewer parameters, yet drops an already-competent task from 30.0%30.0\% to 21.7%21.7\% unless early-stopped (45.0%45.0\%). A soft prompt is a few-shot override: it must sit in the gradient path of the output it steers, and it pays off where the frozen model is weak. 8 Discussion Why does asking match teaching at 10 shots? Ten images provide on the order of tens of labelled boxes: enough to identify what to attend to, not enough to estimate millions of weight updates. LoRA’s own rank sweep is the cleanest evidence: accuracy peaks at r=64r=64 and falls at r=128r=128 (Appendix F), classic capacity overshoot, while the forgetting cost keeps rising. The soft prompt puts its ∼7 7K parameters entirely inside the frozen model’s input distribution, where the backbone’s own inductive biases regularise them. We state this as an interpretation of the observed regime, not a law: with hundreds of shots the balance may move toward weight space, and we have not measured where. The robotics results draw the same boundary from the other side: when the target behaviour is genuinely new to the module that produces the output, a prompt succeeds only if it reaches that module, and a full fine-tune still leads. They also expose the boundary’s second face: on a task the frozen policy already performs, the prompt’s global input edit overwrites competence that a weight update’s targeted corrections preserve, and only a restrained (early-stopped) prompt stays above the base. Adaptation is insufficient if the backbone lacks the required knowledge. The Medical category is the thesis’s own counterexample, and we keep it in the headline table: every prompt- and weight-space adapter on the generalist backbone scores ≤1.0≤ 1.0 mAP while a fine-tuned specialist reaches 35.335.3. Ten-shot adaptation, in any space, cannot conjure absent knowledge. The recommendation that follows is diagnostic: if a short soft-prompt sweep (single-GPU runs; Appendix B timings) does not move a domain off the floor, the failure is knowledge, not phrasing, and a specialist or heavier adaptation is the right tool. Practical recipe. For a new domain: IST position; L swept over 0,1,2,3,4,8\0,1,2,3,4,8\ (TSI worth testing for single-class or text-dominant domains); space-token initialisation; the minimal class-name query the prompt was trained with; model selection by task metric rather than loss. For a VLA policy: prompt both the VLM prefix and the action-expert input, and train with a large effective batch. Total cost is a handful of single-GPU runs per domain (median ∼2 2 GPU-hours each in our 5-epoch regime; Appendix B). Limitations. (1) Statistical support: all numbers are single runs in a regime with 2020–28%28\% seed spread; per-dataset deltas under ∼1 1 mAP are ties, and the headline claim is parity with the best LoRA, not victory over it. (2) Scope: the main detection results, the ablations, and the cross-model transfer study are all within the Qwen3-VL family; a from-scratch check on a second, unrelated backbone (Gemma-4-12B, §4.5) shows the recipe itself generalises, but also that its per-domain hyperparameter search does not carry Qwen3-VL’s holdout-to-test reliability to that backbone (Appendix D) – one shot count (10) and three atomic VLA tasks for robotics remain untested outside those settings, and claims are scoped accordingly. (3) Selection: the per-dataset best-of-IST/TSI checkpoints used as the distillation ceiling and transfer origin were selected on the test split, disclosed in §4.1; the main comparison uses the fixed IST position only. (4) Training memory: the trainable-parameter count does not translate into proportional training-memory savings; peak memory is dominated by activations of the frozen backbone, and in our measurements soft-prompt training at batch size 4 has peak memory comparable to a LoRA r=16r=16 run at batch size 1. Gradient checkpointing (Chen et al., 2016) is the practical mitigation; storage and deployment efficiency (L×dL×d floats per domain) are unaffected: a trained prompt is 17.617.6–49.649.6 KB on disk depending on L, against LoRA r=64r=64’s 666666 MB, a ∼23,000× 23,000× reduction measured directly from the checkpoint files (Appendix E). (5) Scope of the query: all SoftPrompt results use the minimal class-name query; combining trained soft tokens with descriptive per-class instructions, the setting in which LoRA reaches 14.814.8 (Appendix F), is left to future work. 9 Conclusion We revisited soft prompting for few-shot adaptation of vision-language models and found that two choices decide whether it works: injecting the tokens at the cross-modal boundary, not as a NLP-standard prefix, and and initializing them so that the prompt’s semantic meaning remains unchanged at the start of training. With those choices, one to three learnt tokens per domain equal the best of a LoRA rank sweep on Roboflow20-VL, at over 20,000×20,000× fewer trainable parameters and zero forgetting, while the accuracy-matching LoRA rank forgets 35%35\% of the backbone’s NaturalBench VQA ability. The learnt tokens keep the virtues of language: they transfer across model versions, and they can be verbalised into editable prompts that match dedicated prompt search. The same placement principle carries to a frozen robot policy, where a prompt succeeds exactly when it reaches the module that produces the action. We take these results as evidence that modern VLMs already encode much of what specialised domains require. Before teaching a model with weight updates, and paying what that costs the generalist, it is worth learning to ask. Ethics Statement This work aims to make few-shot adaptation of large vision-language models cheaper and safer: prompt-space adaptation lowers the annotation and compute barrier for specialised domains (environmental monitoring, industrial inspection, medical triage) and, unlike weight updates, cannot silently degrade the deployed model’s other capabilities. Distilled prompts are readable text, a modest transparency benefit over opaque weight deltas. Risks mirror those of object detection generally, including surveillance applications; we do not identify consequences specific to this method beyond them. Reproducibility Statement All benchmarks are public (Roboflow-100-VL, LVIS, NaturalBench, RefCOCO, RoboCasa). Training and evaluation configurations for every arm, including the LoRA recipes and the robotics setup, are in Appendix B; disclosure notes on seeds, selection, and missing cells are in §4.1. Code and trained prompt checkpoints will be released, including a minimal, runnable notebook (code/soft2hard_synthetic_arithmetic.ipynb) reproducing the synthetic-arithmetic soft-to-hard illustration of Appendix L end to end outside the full pipeline. References [1] L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. G. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab GEPA: REFLECTIVE PROMPT EVOLUTION CAN OUTPERFORM REINFORCEMENT LEARNING. Cited by: §1, §2, §4.1, Table 1, Table 6. Alazraki et al. (2025) L. Alazraki, Y. Tan, J. A. Campos, M. Mozes, M. Rei, and M. Bartolo Reverse engineering human preferences with reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Bai et al. (2023) J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. External Links: Link Cited by: §1, §2, §4.1. Black et al. (2025) K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. π0 _0: A vision-language-action flow model for general robot control. In Robotics: Science and Systems (RSS), Cited by: §7. Chen et al. (2016) T. Chen, B. Xu, C. Zhang, and C. Guestrin Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174. Cited by: §8. Dai et al. (2023) W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi InstructBLIP: towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems (NeurIPS). Cited by: §2. Gare et al. (2026) G. R. Gare, N. Peri, M. Popov, S. Jain, J. Galeotti, and D. Ramanan DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection. External Links: Link Cited by: Table 14, §1, §2, §4.1, §4.1, §4.4, Table 1, Table 2, Table 6. Gemma Team et al. (2024) Gemma Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, et al. Gemma: open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. Cited by: §4.1, §4.5. Gupta et al. (2019) A. Gupta, P. Dollár, and R. Girshick LVIS: a dataset for large vocabulary instance segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §4.4. He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §3.3. Houlsby et al. (2019) N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning (ICML), Cited by: §2. Hu et al. (2021) E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022 - 10th International Conference on Learning Representations. External Links: Link, ISBN 2106.09685v2 Cited by: §1, §2, §4.1. Jia et al. (2022) M. Jia, L. Tang, B. Chen, C. Cardie, S. Belongie, B. Hariharan, and S. Lim Visual prompt tuning. In European Conference on Computer Vision (ECCV), Cited by: §2. Khattak et al. (2023) M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan MaPLe: multi-modal prompt learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. Lester et al. (2021) B. Lester, R. Al-Rfou, and N. Constant The Power of Scale for Parameter-Efficient Prompt Tuning. Conference on Empirical Methods in Natural Language Processing (EMNLP). Cited by: §1, §2. Li et al. (2024a) B. Li, Z. Lin, W. Peng, J. De Dieu Nyandwi, D. Jiang, Z. Ma, S. Khanuja, R. Krishna, G. Neubig, and D. Ramanan NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples. Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track. External Links: Link Cited by: §1. Li and Liang (2021) X. L. Li and P. Liang Prefix-tuning: optimizing continuous prompts for generation. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §2. Li et al. (2024b) Z. Li, X. Li, X. Fu, X. Zhang, W. Wang, S. Chen, and J. Yang PromptKD: unsupervised prompt distillation for vision-language models. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Cited by: §2. Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual Instruction Tuning. Advances in Neural Information Processing Systems 36. External Links: ISBN 9781713899921, ISSN 10495258 Cited by: §2. Liu et al. (2024) S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 15105 LNCS, p. 38–55. External Links: Link, ISBN 9783031729690, Document, ISSN 16113349 Cited by: §2, §4.1, Table 1. Liu et al. (2022) X. Liu, K. Ji, Y. Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang P-tuning v2: prompt tuning can be comparable to fine-tuning universally across scales and tasks. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §2. Lu et al. (2022) Y. Lu, J. Liu, Y. Zhang, Y. Liu, and X. Tian ProDA: prompt distribution learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. Mandlekar et al. (2023) A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox MimicGen: a data generation system for scalable robot learning using human demonstrations. In Conference on Robot Learning (CoRL), Cited by: §7. Mao et al. (2016) J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy Generation and comprehension of unambiguous object descriptions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §4.3. McCloskey and Cohen (1989) M. McCloskey and N. J. Cohen Catastrophic interference in connectionist networks: the sequential learning problem. Psychology of Learning and Motivation 24, p. 109–165. Cited by: §2. Minderer et al. (2022) M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Mahendran, A. Arnab, M. Dehghani, Z. Shen, et al. Simple open-vocabulary object detection. In European Conference on Computer Vision (ECCV), Cited by: §2. Nasiriany et al. (2024) S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu RoboCasa: large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems (RSS), Cited by: §1, §7. Physical Intelligence et al. (2025) Physical Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, et al. π0.5 _0.5: A vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054. Cited by: §1, §4.1, §7. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning Transferable Visual Models From Natural Language Supervision. External Links: Link, Document Cited by: §1. Robicheaux et al. (2025) P. Robicheaux, M. Popov, A. Madan, I. Robinson, J. Nelson, D. Ramanan, and N. Peri Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models. External Links: Link Cited by: §1, §4.1. Shin et al. (2020) T. Shin, Y. Razeghi, R. L. Logan, E. Wallace, and S. Singh AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, p. 4222–4235. External Links: Link, ISBN 9781952148606, Document Cited by: §2. Vu et al. (2022) T. Vu, B. Lester, N. Constant, R. Al-Rfou, and D. Cer SPoT: better frozen model adaptation through soft prompt transfer. In Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §2, §6.3. Yu et al. (2016) L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg Modeling context in referring expressions. In European Conference on Computer Vision (ECCV), Cited by: §4.3. Yuksekgonul et al. (2024) M. Yuksekgonul, F. Bianchi, J. Boen, S. Liu, Z. Huang, C. Guestrin, and J. Zou TextGrad: automatic “differentiation” via text. arXiv preprint arXiv:2406.07496. Cited by: §2. Zheng et al. (2024) Y. Zheng, R. Zhang, J. Zhang, Y. Ye, Z. Luo, Z. Feng, and Y. Ma LlamaFactory: unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Cited by: Appendix F. Zhou et al. (2022a) K. Zhou, J. Yang, C. C. Loy, and Z. Liu Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. Zhou et al. (2022b) K. Zhou, J. Yang, C. C. Loy, and Z. Liu Learning to Prompt for Vision-Language Models. International Journal of Computer Vision (IJCV). External Links: Link Cited by: §2. Appendix A Gradient Flow Comparison Figure 7 contrasts the backward-pass optimisation signature of SoftPrompt against LoRA (§3.2): the same backward chain through every frozen block, but a single trainable sink at its far end rather than one at every depth. Figure 7: A soft prompt is a weaker optimisation lever than LoRA: the same backward chain, but a single trainable sink at its far end. Gradient flow for the two adaptation methods on a frozen decoder-only VLM trained with the next-token detection loss. Gray boxes are frozen modules and blue boxes frozen input embeddings; a flame marks a trainable parameter (the amber soft-prompt tokens on the left, the teal LoRA adapters on the right). Gray arrows are the forward pass; crimson arrows the backward chain from ∂ , which traverses the same frozen blocks in both methods. (Left, SoftPrompt) the chain crosses all L frozen layers before reaching the only trainable sink (dashed: no update until the input), the soft tokens injected at the cross-modal boundary. (Right, LoRA) the same chain feeds a trainable adapter at every depth (solid taps); the nearest sink is one block from the loss, and updates land at every layer. SoftPrompt trades this optimisation disadvantage for ∼103 10^3–104×10^4× fewer trainable parameters and a backbone left fully intact. Appendix B Implementation Details SoftPrompt (detection). All experiments use PyTorch 2.x with bfloat16 on NVIDIA A6000 GPUs. AdamW, lr 5×10−35×10^-3, cosine schedule, batch size 4, up to 5 epochs, space-token initialisation. Measured over all 226 ablation runs, the training loop’s single-GPU wall-clock in the 5-epoch regime (excluding vLLM export and test evaluation) has median 2.02.0 hours (p10 77 minutes, p90 5.55.5 hours): the fastest domain (aerial-airport) trains in ∼4 4 minutes, while high-resolution domains (paper-parts) take up to 1515 hours. Evaluation uses a vLLM engine with the trained soft-prompt rows appended to the embedding table as extended vocabulary. LoRA SFT baseline (detection). Adapters on the language-model decoder’s q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj via peft; α=2rα=2r, no dropout; lr 5×10−55×10^-5, cosine schedule, warmup ratio 0.20.2, effective batch size 32, 30 epochs on 8 A6000s; best checkpoint by validation mAP; image_max_pixels =1048576=1048576. Robotics (π0.5 _0.5 / RoboCasa). Base policy: π0.5 _0.5 pretrained with RoboCasa data (PandaOmron embodiment; three camera views, 16-dim state, 12-dim action). Tasks: PickPlaceCounterToCabinet, TurnOnSinkFaucet, OpenDrawer, each with its own K=10K=10 per-seed demonstration subsets and normalisation statistics. Soft prompt: learnable tokens on the PaliGemma prefix (prefix-only, N=8N=8) or on both the prefix and the action-expert input (dual, N=16N=16, M=16M=16); IST format; the PaliGemma-side tokens use space-token initialisation as in §3.3, but the action-expert tokens are initialised randomly, since the action expert has no token vocabulary to seed from; trained with gradient accumulation to an effective batch of 128 and lr 2×10−12×10^-1 for the dual configuration, 2,000 steps. The early-stopped OpenDrawer variant halts the same dual configuration at 750 steps; the stopping point was selected on a seed-0 sweep over steps and learning rate, then re-run on 3 seeds. LoRA: openpi reference configuration, rank 16 on attention and FFN modules; the “+ action expert” variant restricts to the rank-16 adapters (26.5M), and the PaliGemma-only variant matches the prefix prompt’s coverage (19.6M). Both arms train on the same K=10K=10 demonstration episodes per seed. Evaluation: 20 closed-loop episodes per seed on freshly sampled held-out scene instantiations; the environment’s random generator is re-seeded per episode index, so all arms face byte-identical scenes. Appendix C Per-Dataset Configurations Table 10 lists, for every Roboflow20-VL dataset, the frozen baseline, the selected SoftPrompt configuration behind Table 1’s IST row (prompt length L, the epoch actually reached in training, and test mAP), and the best-of-position configuration (winning ordering, its L, and test mAP). The selected lengths are L∈1,2,3L∈\1,2,3\ (mean 1.751.75, i.e. 7,1687,168 trainable parameters on average, range 4,0964,096–12,28812,288); TSI wins 6 of 20 datasets, concentrated on single-class and text-dominant domains (§5.1). All other hyperparameters are shared across datasets (Appendix B). The IST row uses dataset-specific budgets: 13 of 20 datasets train for 5 epochs (matching the §5 ablation setting), and 7 train for 50 epochs with early stopping (patience 15). Table 10: IST wins 14 of 20 datasets; TSI takes the other 6, concentrated on single-class and text-dominant domains. Per-dataset SoftPrompt configurations on Roboflow20-VL (10-shot, mAP@50:95). IST: the fixed-position row of Table 1 (per-domain L). Ep: epoch actually reached by the IST checkpoint. Best of IST/TSI: the per-dataset best ordering (test-split selection; §4.1, disclosure i). IST Best of IST/TSI Dataset Category Baseline Ep L mAP Format L mAP actions Sports 2.9 5 3 3.2 IST 3 3.2 aerial-airport Aerial 4.4 41 1 9.1 TSI 8 12.1 all-elements Documents 10.6 15 1 13.4 TSI 1 14.4 aquarium-combined F. & F. 29.2 7 1 35.7 IST 1 35.7 defect-detection Industrial 4.9 5 1 6.3 TSI 1 6.7 dentalai Medical 0.0 3 2 0.1 IST 2 0.1 flir-camera-objects Other 9.3 5 2 16.6 IST 2 16.6 gwhd2021 F. & F. 1.3 21 1 4.2 TSI 1 5.8 lacrosse-object-detection Sports 14.2 5 1 24.3 IST 1 24.3 new-defects-in-wood Other 5.9 5 3 12.0 IST 3 12.0 orionproducts Other 3.1 5 3 6.0 IST 3 6.0 paper-parts Documents 4.4 5 2 15.9 IST 2 15.9 recode-waste Industrial 17.0 5 1 16.7 IST 1 16.7 soda-bottles Other 15.3 23 1 14.2 IST 1 14.2 the-dreidel-project Other 12.1 1 3 22.9 IST 3 22.9 trail-camera F. & F. 41.9 5 2 41.9 TSI 2 43.5 water-meter Industrial 2.6 5 1 2.9 IST 1 2.9 wb-prova F. & F. 27.6 5 2 28.8 IST 2 28.8 wildfire-smoke Aerial 8.3 41 1 8.9 TSI 1 12.3 x-ray-id Medical 0.1 5 3 0.0 IST 3 0.0 Mean 10.8 1.75 14.2 14.7 Appendix D Gemma-4 Generalisation: Search Methodology and Full Results §4.5 trains the method fresh on Gemma-4-12B-it, a VLM family unrelated to Qwen3-VL, and finds that while the recipe generalises (beats baseline on 5 of 8 above-floor pool datasets), the per-domain hyperparameter search used to pick each dataset’s configuration does not transfer its own holdout signal to test as reliably as the equivalent Qwen3-VL search does. This appendix gives the search methodology and the complete per-dataset accounting behind that finding. Search methodology. For each dataset, using only its 10-shot support set (never test), we hold out a fixed fraction as an internal validation split and run support-holdout successive halving over five stages: (S1) a format × length screen (\IST, TSI\ × L∈1,2,3L∈\1,2,3\, plus an L=0L=0 control, space initialisation, lr 5×10−35×10^-3, no gradient accumulation); (S2) a gradient-accumulation sweep (accumulation over 44 steps or the full support set) on the top-2 S1 configurations; (S3) a learning-rate refinement (1,3,10×10−3\1,3,10\×10^-3) of the best config so far; (S4) an initialisation check (xavier, mean) against space initialisation; and, added after the first pass on Qwen3-VL showed a wider net was needed for this backbone, (S5) an epoch-budget refinement (1010, 1515 epochs against the default 55) and an extended length screen up to L=5L=5 for two datasets (defect-detection, flir-camera-objects) whose initial holdout margin over control was thin or, on a first attempt, entirely lost to transient GPU contention on a shared cluster node (confirmed by re-running in isolation). Every stage uses the same holdout-mAP selection rule as the Qwen3-VL search (Appendix B); the only difference from that search is the wider hyperparameter net (S1–S5 here vs. length alone there), made necessary because Gemma-4’s holdout margins were consistently thinner at the default setting. The selected config is then trained once more at full budget and evaluated on the true test split – the number in Table 3 and below. Table 11: The search’s holdout margin does not predict its test margin. Per-dataset search-selected configuration and the full holdout-vs-test accounting behind Table 3 and Table 3’s finding box. Test (untuned): an L∈1,2,3L∈\1,2,3\ IST, space-init, lr 5×10−35×10^-3, no-accumulation config with no search at all, for reference. Accum: gradient accumulation steps (f = full support set per step). All mAP@50:95, test-split numbers are best of two independent evaluations of the selected config (retrained from scratch; and the search-time checkpoint re-exported and re-evaluated), matching §4.1’s disclosure that transfer-study checkpoints are test-selected for the ceiling but never for the headline claim. Dataset L Fmt Init LR Accum Hold. ctrl Hold. sel. Test base Test sel. Test (untuned) actions 1 IST space 5e-3 1 6.5 13.8 0.8 1.0 0.5 aerial-airport 1 IST mean 1e-3 1 2.9 3.4 11.3 11.4 9.8 all-elements 3 TSI space 5e-3 1 9.9 14.7 13.5 17.4 10.6 defect-detection 5 TSI space 1e-3 4 2.8 13.7 6.6 3.7 5.6 dentalai (floor) 1 TSI space 1e-3 f 0.9 9.9 0.5 0.6 0.4 flir-camera-objects 5 TSI space 5e-3 4 11.7 16.5 14.0 13.9 13.7 lacrosse-object-detection 2 TSI space 5e-3 f 9.0 18.5 11.3 10.5 13.1 new-defects-in-wood 2 IST xavier 1e-3 f 8.9 13.2 3.7 3.9 4.2 wb-prova 3 TSI space 1e-2 4 2.5 48.0 18.5 24.3 19.0 Two things follow from Table 11 that are not visible from the win/loss tally alone. First, on defect-detection, lacrosse-object-detection, and new-defects-in-wood, the search-selected configuration is not merely a weak win, it is beaten on test by the untuned default we never selected on anything – the search actively picks the wrong configuration for these three datasets, not just an insufficiently good one. Second, the size of the holdout margin carries no information about test reliability: wb-prova has the largest holdout margin in the pool (48.048.0 vs. 2.52.5 control, 19×19×) and the largest test gain (+5.8+5.8); defect-detection has the second-largest holdout margin (13.713.7 vs. 2.82.8, 4.9×4.9×) and the worst test result in the pool. A held-out mAP sweep over 1010 support images per class is evidently enough signal to separate configurations that generalise to the model’s own training distribution, but not always enough to predict which configuration the frozen backbone will actually prefer on unseen test images – a distinction the Qwen3-VL ablations (§5) do not surface, because there the two happen to agree. We report this as an open reliability gap in the search procedure itself, not as evidence against the underlying method: the same 1010-shot budget that lets §5’s Qwen3-VL search reliably rank configurations does not, on this second backbone, carry enough signal to always rank them correctly. Appendix E Serving Soft Prompts with Standard Inference Engines A practical objection to soft prompts is that they seem to require a custom forward pass (an embedding hook at the injection position), locking inference to a research codebase. Our evaluation pipeline shows this is not the case: a trained soft prompt can be served by an unmodified high-throughput engine, vLLM in our case, through the model’s own vocabulary. The export works as follows. After training, the L learnt vectors are appended as L new rows of the backbone’s input-embedding matrix, and L new special tokens (<SP_0>, <SP_1>, …) are registered in the tokenizer, one per row; the result is written as a standard checkpoint. The base weights and the original embedding rows are untouched: the model gains L rows and loses nothing. At inference, the detection query simply contains the placeholder string <SP_0>…at the chosen injection position (e.g. between the image and the text for IST), which tokenises to exactly the L new ids (verified at load time); the engine embeds them like any other tokens, which is the soft prompt. No engine modification, no custom hooks, and all of the engine’s optimisations (paged attention, tensor parallelism, batched generation) apply unchanged. Three consequences are useful in practice. First, one exported engine serves both arms of any soft-vs-base comparison: a query that omits the placeholder tokens is processed exactly as by the base model, which is how the distillation baseline of §6.2 shares a single engine with the soft-prompted arm. Second, the marginal cost of a domain is L embedding rows, so shipping an adapted domain is a vocabulary patch rather than a model copy: Table 12 measures this directly from the checkpoint files rather than from parameter counts alone, since on-disk size also reflects serialization overhead that a raw parameter count does not. Third, switching domains is cheaper in kind, not just in size. Injection is a lookup into the frozen embedding matrix, so activating a different prompt means addressing a different span of embedding rows, not attaching, merging, or hot-swapping a weight delta inside every target layer the way LoRA serving must; our reference implementation (code/softprompt_pkg) demonstrates the minimal case, detaching one trained prompt’s hook and re-attaching another’s in a single call, with no change to the frozen backbone’s weights. The same property extends to the exported checkpoint above: because each domain occupies a disjoint, fixed span of token ids, multiple domains’ rows can coexist in one resident vocabulary, and a request activates whichever domain’s tokens it includes. All main-paper detection evaluations use this path. Code, including the export and evaluation pipeline, will be released. Table 12: The trained checkpoint is proportionally as small as the parameter count suggests, measured, not estimated. On-disk size of the trainable adapter only (no base-model weights): SoftPrompt’s torch.save tensor vs. LoRA’s adapter_model.safetensors, both at their native training precision (fp32). SoftPrompt sizes are exact measurements at each L; the mean-L=1.75L=1.75 row interpolates linearly (each additional token costs a fixed 1616 KB, =d×4=d×4 bytes for d=4096d=4096), matching the per-dataset L mix used in Table 1. Ratio: LoRA size / SoftPrompt mean-L size. Bold: r=64r=64, the rank that matches SoftPrompt’s accuracy (Table 1). Checkpoint Size Ratio vs. SoftPrompt (mean L) SoftPrompt (L=1L=1) 17.6 KB – SoftPrompt (L=2L=2) 33.6 KB – SoftPrompt (L=3L=3) 49.6 KB – SoftPrompt (mean L=1.75L=1.75) 29.6 KB 1×1× LoRA (r=4r=4) 41.7 MB 1,442×1,442× LoRA (r=8r=8) 83.3 MB 2,882×2,882× LoRA (r=16r=16) 166.6 MB 5,762×5,762× LoRA (r=32r=32) 333.1 MB 11,522×11,522× LoRA (r=64r=64) 666.1 MB 23,041×23,041× LoRA (r=128r=128) 1.3 GB 46,080×46,080× The storage ratio at r=64r=64 (23,041×23,041×) is close to but slightly below the raw parameter-count ratio (174.6M/7,168≈24,357×174.6M/7,168≈ 24,357×, §3): both checkpoints are fp32, so the gap is entirely serialization overhead, a fixed torch.save pickle header that is negligible against LoRA’s multi-megabyte files but not against SoftPrompt’s few-KB ones. We report the measured figure rather than the parameter estimate because it is what a deployment actually pays. Appendix F Full LoRA Rank Sweep Table 13 reports both LoRA variants at every rank. The + instructions rows are the instruction-augmented protocol variant (§4.2); SoftPrompt has no counterpart in that setting yet (an instruction-augmented soft prompt is future work), so the main comparison (Table 1, Figure 1) uses the default class-name protocol and the variant is shown here. For reference, SoftPrompt with per-dataset choice of IST/TSI (position selected on the test split; §4.1, disclosure i) reaches 14.714.7. LoRA training configuration. Every rank is trained per-dataset with LLaMA-Factory (Zheng et al., 2024) on Qwen3-VL-8B, target modules q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (attention and MLP), α=2rα=2r (e.g. α=128α=128 at r=64r=64), dropout 00, AdamW, learning rate 5e-55e-5, cosine schedule with 0.20.2 warmup ratio, effective batch size 22 (per-device batch 11, gradient accumulation 22), 7070 optimizer steps, bf16, seed 4242 — matching this paper’s own default training seed (§4.1). Verified directly against the saved checkpoints’ own adapter_config.json and training_args.bin (not re-derived or estimated), confirmed identical across every sampled rank and dataset. Table 13: LoRA rank sweep on Roboflow20-VL 10-shot (mAP@50:95): accuracy peaks at r=32r=32–6464 and declines at r=128r=128. Both variants (class names only; with per-class instructions). Figure 1 plots the class-names sweep against its NaturalBench forgetting cost. Bold: best configuration per variant. Method Params Aerial Doc. F. & F. Ind. Med. Sports Other All LoRA (r=4r=4) 10.9M 8.1 8.1 27.0 8.8 0.0 11.9 10.5 12.1 LoRA (r=8r=8) 21.8M 11.3 7.3 28.7 8.3 0.2 10.3 11.1 12.7 LoRA (r=16r=16) 43.6M 12.3 13.8 28.1 6.6 0.3 8.6 10.9 12.8 LoRA (r=32r=32) 87.3M 17.1 18.7 28.6 7.6 0.4 10.4 8.7 13.7 LoRA (r=64r=64) 174.6M 20.2 19.9 28.6 7.1 0.7 9.6 9.4 14.2 LoRA (r=128r=128) 349.2M 15.6 19.7 25.5 8.8 0.6 11.1 10.0 13.6 LoRA + instr. (r=4r=4) 10.9M 8.5 9.4 26.9 10.8 0.3 12.2 12.1 13.1 LoRA + instr. (r=8r=8) 21.8M 10.8 9.7 26.8 9.6 0.9 14.8 12.4 13.5 LoRA + instr. (r=16r=16) 43.6M 12.7 13.0 29.4 11.3 0.3 10.3 10.6 13.9 LoRA + instr. (r=32r=32) 87.3M 15.0 17.8 27.9 12.8 0.5 10.8 11.8 14.8 LoRA + instr. (r=64r=64) 174.6M 15.6 20.8 26.7 9.2 1.0 10.3 12.1 14.5 LoRA + instr. (r=128r=128) 349.2M 14.7 19.0 27.7 7.4 0.7 12.4 10.9 14.0 Appendix G Single-Class Evaluation Baselines DetPO’s strongest numbers use a single-class protocol: the VLM is queried once per class per image, which multiplies inference cost by the class count and is not comparable to the single-call multi-class protocol used by every method in Table 1. For reference, DetPO reaches 15.315.3 all-domain mAP under single-class evaluation and 17.517.5 with VQA-score reranking. VQA-score reranking is orthogonal to the prompt and applies equally to SoftPrompt; combining them is left for future work. Table 14: DetPO’s single-class protocol reaches 17.517.5 mAP with VQA-score reranking, above SoftPrompt’s 14.214.2 multi-class, at class-count× the inference cost. Single-class evaluation (one class per VLM call; reference only, not cost-comparable to Table 1). Method Params Aerial Doc. F. & F. Ind. Med. Sports Other All DetPO (Gare et al., 2026) 0 8.3 19.1 30.3 13.7 0.1 14.2 12.2 15.3 DetPO + VQA Score 0 12.3 24.2 32.3 13.5 0.2 17.2 14.3 17.5 Appendix H Cross-Task Grounding: Full RefCOCO Results Table 15: Low-rank LoRA does not measurably forget grounding; the advantage of SoftPrompt here is the guarantee, not the gap. Cross-task forgetting on the RefCOCO family, acc@0.50 (%). Δ is LoRA (fine-tuned on the-dreidel-project, r=4r=4) minus the frozen backbone. SoftPrompt is identical to the frozen backbone by construction (no prompt applied off-domain). Method RefCOCO val RefCOCO testA RefCOCO testB RefCOCO+ val RefCOCO+ testA RefCOCO+ testB RefCOCOg val RefCOCOg test Mean Frozen backbone 91.3 93.1 87.6 85.9 90.1 80.7 88.2 88.0 88.1 LoRA (dreidel, r=4r=4) 91.3 92.9 87.5 85.8 89.9 80.8 88.3 88.1 88.1 Δ (LoRA −- base) +0.00+0.00 −0.10-0.10 −0.10-0.10 −0.06-0.06 −0.16-0.16 +0.10+0.10 +0.10+0.10 +0.18+0.18 −0.01-0.01 SoftPrompt ≡ frozen backbone (identical by construction) Appendix I Cross-Model Transfer: Per-Category Results Table 16: Transfer helps most on Sports (+5.3+5.3) and Documents (+3.3+3.3), and hurts on Industrial (−2.9-2.9). Cross-model transfer of trained soft prompts per Roboflow20-VL super-category (Qwen3-VL-8B → Qwen3.5-9B, 10-shot, mAP@50:95). The 8B-trained prompt is loaded into Qwen3.5-9B with no retraining or projection. Method Aerial Doc. F. & F. Ind. Med. Sports Other All Qwen3.5-9B baseline 8.8 7.4 26.0 4.3 0.1 4.5 8.1 9.9 9B + transferred prompt 9.0 10.7 25.6 1.3 0.1 9.8 9.9 10.7 8B origin (upper bound) 12.2 15.1 28.5 8.8 0.0 13.8 14.3 14.7 Δ vs. 9B baseline +0.2+0.2 +3.3+3.3 −0.4-0.4 −2.9-2.9 +0.1+0.1 +5.3+5.3 +1.8+1.8 +0.8+0.8 Δ vs. 8B origin −3.2-3.2 −4.4-4.4 −2.8-2.8 −7.5-7.5 +0.1+0.1 −4.0-4.0 −4.5-4.5 −4.0-4.0 Appendix J Distillation vs. Direct Transfer: Per-Dataset Results Table 17 gives the per-dataset numbers behind Table 8: the same Qwen3-VL-8B-trained soft prompt, transferred to Qwen3.5-9B either as raw embeddings (direct, no retraining) or as the §6.2 hard prompt distilled from it (best of the global/per-class variant, no soft tokens at all). Table 17: Direct embedding transfer beats its own distilled description on 18 of 20 datasets. Qwen3-VL-8B → Qwen3.5-9B, test mAP@50:95, 10-shot. ”Distilled” is the better of the §6.2 global/per-class hard prompt, evaluated with no soft tokens; ”direct” loads the trained embeddings as-is (as in Appendix I). Δ is direct −- distilled. Dataset Distilled Direct Δ actions 1.6 2.5 +1.0+1.0 aerial-airport 7.8 10.7 +3.0+3.0 all-elements 9.8 11.2 +1.4+1.4 aquarium-combined 21.2 27.4 +6.2+6.2 defect-detection 3.6 3.2 −0.4-0.4 dentalai 0.1 0.2 +0.1+0.1 flir-camera-objects 14.7 17.6 +2.9+2.9 gwhd2021 1.8 0.9 −0.9-0.9 lacrosse-object-detection 7.1 17.1 +9.9+9.9 new-defects-in-wood 3.7 7.4 +3.8+3.8 orionproducts 3.5 4.6 +1.1+1.1 paper-parts 6.1 10.2 +4.1+4.1 recode-waste 7.5 9.8 +2.3+2.3 soda-bottles 3.6 9.0 +5.4+5.4 the-dreidel-project 15.9 17.0 +1.0+1.0 trail-camera 46.3 46.4 +0.1+0.1 water-meter 0.4 0.7 +0.4+0.4 wb-prova 26.2 27.8 +1.6+1.6 wildfire-smoke 5.5 7.6 +2.1+2.1 x-ray-id 0.0 0.1 +0.0+0.0 Mean 9.3 11.6 +2.3+2.3 The two datasets where distillation wins (defect-detection, gwhd2021) are both near-floor swaps below 44 mAP either way; every dataset with a meaningful gap favours direct transfer, up to +9.9+9.9 on lacrosse-object-detection. Appendix K Token-Substitution Probe: Discretising the Trained Prompt If the trained soft tokens have meaningful discrete neighbours (§6.1), those neighbours should work as a prompt. This probe tests that directly: at evaluation time, each trained soft token is replaced by its nearest neighbour (cosine, top-1) in the model’s own representation spaces, either the text-vocabulary embeddings (text) or the cached frozen-encoder patch embeddings (patch), and the discretised prompt is evaluated in place of the trained one. No retraining is involved; each dataset uses its canonical checkpoint. Table 18: The trained prompt’s discrete shadow carries most of its signal, more reliably in text than in image-patch space. Substituting every trained token with its nearest text token keeps the prompt above the frozen baseline on 6 of 8 above-floor datasets and wins on every one of them; nearest-patch substitution is consistently weaker and collapses generation entirely (zero parseable detections) on three datasets. Bold: better substitution mode per dataset. dentalai is the floor control: neither the trained prompt nor its discrete shadow moves it. †wb-prova’s nearest-patch substitution collapses generation entirely. Dataset Config Baseline Trained soft Nearest text Nearest patch aerial-airport 1T TSI 4.4 11.4 6.9 6.2 actions 3T IST 2.9 4.0 2.6 0.0 all-elements 1T TSI 10.6 14.4 11.1 11.0 defect-detection 1T IST 4.9 8.0 5.8 5.7 dentalai 1T IST 0.0 0.0 0.0 0.0 flir-camera-objects 2T IST 9.3 16.6 11.3 0.0 lacrosse-object-detection 2T IST 14.2 24.0 17.6 17.0 new-defects-in-wood 2T IST 5.9 11.0 8.7 8.1 wb-prova† 2T IST 27.6 28.8 25.1 0.0 Mean 8.9 13.1 9.9 5.3 Two observations (Table 18). First, text substitution stays above the frozen baseline on 6 of 8 above-floor pool datasets, all but actions and wb-prova†, which happen to be the two datasets where the trained prompt’s own gain over baseline is smallest to begin with (+1.1+1.1 and +1.2+1.2 mAP, versus +3.1+3.1 to +9.8+9.8 elsewhere), leaving the least room for a discretised approximation to still clear the baseline. Where it does clear the baseline, text substitution retains roughly 33%33\% of the trained prompt’s gain in aggregate across those six datasets. The learned directions therefore point at genuinely useful discrete tokens, the functional counterpart of the retrieval reading in §6.1 and of the distillation result in §6.2, though the effect is a partial recovery rather than a free one. Second, text substitution consistently outperforms patch substitution, winning on every above-floor dataset (pool means 9.99.9 vs. 5.35.3); on three datasets (actions, flir-camera-objects, wb-prova†), patch substitution collapses generation entirely to 0.00.0 mAP, zero parseable detections, while text substitution on the same three datasets still produces valid, if below-baseline on two of them, output. The useful discrete signal lives more reliably in the text-vocabulary space than in the visual-patch space. Appendix L A Minimal Soft2Hard Illustration: Synthetic Arithmetic The distillation result in §6.2 and the substitution probe above both study soft-to-hard behaviour inside the full RF20 vision pipeline, where the soft tokens, the image, and a class-name query are all present together. To isolate the phenomenon from that complexity, we built a minimal, non-visual companion: freeze a small causal LM (Qwen3-VL-2B-Instruct, used purely as a text model here) and train L=10L=10 continuous soft tokens, in the same cross-modal-boundary-style placement (tokens immediately before the payload, §3.1), to solve a synthetic arithmetic task – given a short list of integers with no instruction text, predict their sum or product – then ask the trained model, in plain English, what the tokens mean. Full code and a runnable notebook reproducing every number below are released with the paper (code/soft2hard_synthetic_arithmetic.py / .ipynb). Table 19: The soft prompt learns the synthetic task reliably across all four conditions. Exact-match accuracy, 100 held-out examples, greedy decoding, 20 epochs, L=10L=10. Task Exact match Addition, 2-digit operands (0–99) 1.00 Addition, 1-digit operands (0–9) 0.96 Multiplication, 2-digit operands (0–99) 0.95 Multiplication, 1-digit operands (0–9) 0.86 Asked about the soft tokens in isolation ("Elaborate: ‘<SP>’.", no numbers), the model produces incoherent output in every condition: fragments of Hebrew, Greek, or other scripts unrelated to arithmetic. A nearest-vocabulary-token check, the same probe as Appendix K applied here, explains why: across all 4 conditions (4040 soft-token positions checked in total), the single nearest vocabulary neighbour by cosine similarity is, without exception, the same rare musical-notation glyph (cosine sim.≈0.80cosine sim.≈ 0.80–0.820.82), regardless of task. This is not operation-specific signal, just where a lightly-trained 1010-token prompt happens to sit in embedding space; the isolated query’s incoherence is consistent with the model echoing something near that generic neighbour rather than reasoning about the span at all. Asking about the tokens in the context they were trained in – immediately followed by a real number list, the exact shape seen during training – changes this completely. "What is the task prompt of: ‘<SP>: 53, 93’?" answers ”What is the sum of the two numbers: 53 and 93?” Answer: 53+93=53+93=146 (correct, on the 2-digit addition prompt), and ”Multiply 6 by 0.” … 6×0=06× 0=0. Answer: 0 (correct, 1-digit multiplication). "Rewrite in your own words: ..." reliably names the operation (”Multiply 53 by 93.”) without always attempting to solve it; "Describe the task prompt of: ..." is good but the least stable of the in-context templates, occasionally hallucinating an unrelated frame entirely. Templates that invite ad-hoc computation rather than description ("Elaborate", bare "Rewrite") more often get the arithmetic itself wrong on 2-digit multiplication (53×9353× 93 answered 49494949 or 49894989, not 49294929) even while still correctly implying the operation – the exact trained template reaches 0.860.86–1.001.00 exact match (Table 19), but placing the same tokens inside different surrounding phrasing does not inherit that reliability. One further asymmetry: self-description accuracy does not track task-solving accuracy – the 1-digit addition prompt solves the task at 0.960.96 exact match but is the one condition where "What is the task prompt of" most often misreads it as a sequence-completion task rather than addition. Finding. In a minimal setting stripped of vision entirely, a trained soft prompt is not self-describing in isolation – an isolated query only recovers the nearest generic vocabulary token, not task content – but it is soft-to-hard distillable once the query preserves the context it was trained in, at which point natural language recovers a real, often exactly correct, description of the task the tokens encode. This mirrors, in miniature, the RF20 result that verbalisation requires the tokens to be queried inside a working prompt, not in a vacuum. Appendix M Multi-Seed Variance Check Every ablation-table number in the main paper is a single run; §4.1’s disclosure (i) estimates a 2020–28%28\% seed-to-seed spread from this check. Each pool dataset’s canonical Space/IST checkpoint (the same configuration reported in §5) is retrained from two additional seeds (123, 2024) alongside the original (seed 42), with no other change. Table 20: Seed-to-seed spread averages 28%28\% of the per-dataset mean, matching the 2020–28%28\% estimate the main paper’s ablation tables are built on. Test mAP@50:95 across three training seeds per pool dataset, canonical Space/IST configuration. CV: coefficient of variation (std / mean), the seed-noise scale disclosure (i) reports. dentalai is the floor control (excluded from the summary, as in every other ablation table); Dataset seed 42 seed 123 seed 2024 Mean ± Std CV aerial-airport 7.98 6.45 8.10 7.51±0.927.51± 0.92 12%12\% actions 3.94 5.40 3.93 4.42±0.844.42± 0.84 19%19\% all-elements 11.59 6.81 14.97 11.12±4.1011.12± 4.10 37%37\% defect-detection 7.97 5.52 5.45 6.31±1.436.31± 1.43 23%23\% flir-camera-objects 16.55 16.96 7.00 13.50±5.6413.50± 5.64 42%42\% lacrosse-object-detection 24.10 11.02 13.24 16.12±7.0016.12± 7.00 43%43\% new-defects-in-wood 10.97 12.37 9.42 10.92±1.4710.92± 1.47 14%14\% dentalai (floor control) 0.00 0.00 0.01 excluded wb-prova† 28.80 13.74 19.64 20.73±7.5920.73± 7.59 37%37\% The mean coefficient of variation across the eight fully-collected, above-floor datasets is 28%28\% (range 1212–43%43\%), confirming the 2020–28%28\% disclosed in §4.1: single-run ablation numbers carry real seed noise, and per-dataset deltas below ∼1 1 mAP should be read as ties rather than genuine effects. The spread is not uniform: aerial-airport and new-defects-in-wood are comparatively stable (1212–14%14\%), while all-elements, flir-camera-objects, lacrosse-object-detection, and wb-prova swing by 3737–43%43\% of their own mean. This does not track test-set size: flir-camera-objects has the pool’s largest test split (500 images) and one of the highest CVs, while aerial-airport has the smallest (34 images) and the lowest, so whatever drives a given dataset’s seed sensitivity is not simply ”fewer test images.” M.1 The spread is training noise, not evaluation noise Table 20 retrains from three seeds, so its spread conflates two sources: different random initializations reaching different local optima during training, and stochasticity in the vLLM generation step at test time. To isolate them, we take each pool dataset’s seed-42 checkpoint (the one already trained above) and re-run test-set evaluation twice more, varying only the generation seed (--skip_training, no retraining). Table 21: Eval-time variance is near zero; the seed-to-seed spread in Table 20 comes from training, not evaluation. Same checkpoint, three evaluation seeds, test mAP@50:95. CV here is ≤5%≤\!5\% on every dataset, an order of magnitude below the 1212–43%43\% training-seed CVs. Dataset eval-seed A eval-seed B eval-seed C Std aerial-airport 7.98 7.39 7.36 0.35 actions 3.94 3.94 3.96 0.01 all-elements 11.59 11.76 11.86 0.13 defect-detection 7.97 7.97 7.97 0.00 flir-camera-objects 16.55 16.13 16.56 0.24 lacrosse-object-detection 24.10 23.99 23.99 0.06 new-defects-in-wood 10.97 11.00 11.00 0.02 wb-prova† 12.05 12.93 12.95 0.51 dentalai (floor control) 0.00 0.00 0.00 excluded †wb-prova’s checkpoint is re-evaluated at max_pixels=3=3M (matching Table 20’s canonical setting) for all three eval seeds; its own init_ablation run used a different setting, so it is not the source of the value quoted elsewhere in the paper. Finding. Re-evaluating a fixed checkpoint under three generation seeds moves test mAP by at most 0.510.51 points on any pool dataset (mean std 0.170.17), against 1212–43%43\% training-seed CVs on the same datasets (Table 20). The seed-to-seed spread this paper discloses is a training-time phenomenon: different soft-prompt initializations land in different local optima, not measurement noise at test time. M.2 Is the spread specific to SoftPrompt? The 10-shot regime itself could simply be noisy, in which case any adapter would show comparable run-to-run spread. To test this, we retrain the accuracy-matching LoRA configuration (r=64r=64, §4.2) from the same three seeds, same nine pool datasets, identical data and hyperparameters (Appendix F) apart from seed, and evaluate test mAP with the same protocol used to produce every LoRA number in this paper. Table 22: SoftPrompt’s training-seed spread is not a general property of 10-shot Roboflow20-VL training: it averages 7.9×7.9× larger than LoRA’s at the same accuracy-matching rank, on 6 of 8 above-floor pool datasets. Test mAP@50:95, three training seeds (4242, 123123, 20242024), r=64r=64. Std (mAP pts) Dataset SoftPrompt LoRA Ratio SoftPrompt mean LoRA mean aerial-airport 0.92 1.84 0.5 7.51 11.19 actions‡ 0.84 0.40 2.1 4.42 8.08 all-elements 4.10 0.37 11.0 11.12 13.44 defect-detection 1.43 0.48 3.0 6.31 10.12 flir-camera-objects 5.64 0.15 36.9 13.50 4.99 lacrosse-object-detection 7.00 1.42 4.9 16.12 13.56 new-defects-in-wood 1.47 2.12 0.7 10.92 9.51 wb-prova 7.59 1.80 4.2 20.73 27.75 dentalai (floor control) 0.01 0.04 – 0.00 0.38 ‡actions’ seed-42 LoRA checkpoint (qwen3vl-8b-rank64/actions/multi_class) is corrupted at the source (a 13.713.7MB adapter_model.safetensors, versus ∼698 \!698MB for every other dataset and seed; safetensors fails to deserialize it), so only seeds 123123 and 20242024 are available for this row. Finding. LoRA at the accuracy-matching rank is markedly more stable across training seeds than SoftPrompt: on 6 of 8 above-floor pool datasets its std is 2.12.1–36.9×36.9× smaller, averaging 7.9×7.9× across all 8 (excluding dentalai’s floor effect on both), and its eval-only variance is likewise negligible (Table 21), ruling out measurement noise as the explanation. Two datasets (aerial-airport, new-defects-in-wood) are exceptions where LoRA’s spread instead exceeds SoftPrompt’s. The dominant pattern nonetheless says the seed-to-seed spread disclosed throughout this paper is largely an artifact of SoftPrompt’s tiny parameter count, not a property of 10-shot Roboflow20-VL training in general: a few thousand trainable prompt parameters land in more different local optima across random seeds than LoRA’s 174.6174.6M do. Appendix N Qualitative Detections Figure 8: Qualitative detections on one held-out test image per super-category (7 of Roboflow20-VL’s domains), comparing the frozen zero-shot baseline, DetPO, LoRA (rank 64), and SoftPrompt against groundtruth. The frozen baseline is the weakest column throughout, often over-generating or repeating a single label (dentalai, defect-detection); SoftPrompt localizes and labels cleanly on most domains but inherits the baseline’s failure on dentalai, where no adapter recovers domain knowledge the frozen backbone never had.

Instant research discovery