Paper deep dive
In-Context Collapse in Vision-Language Models and How to Mitigate it?
Mohammad Rostami
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/5/2026, 4:33:49 AM
Summary
This paper identifies and analyzes 'in-context collapse' in Vision-Language Models (VLMs), a phenomenon where accuracy drops sharply as the number of in-context learning (ICL) demonstrations increases. The authors demonstrate that robustness to accumulating demonstrations and the ability to learn novel rules are dissociable capabilities. Through lesion-and-rescue experiments, they localize the collapse to the vision-language integration pathway (connector and early/mid layers), distinct from the late readout layers used for weight-based consolidation. They propose CircA, a lightweight adapter trained on a synthetic task that acts as an 'integration vaccine,' successfully mitigating collapse and transferring resistance to unseen tasks.
Entities (9)
Relation Signals (6)
In-Context Collapse → affects → Vision-Language Models
confidence 95% · as demonstrations accumulate, a subset of VLMs undergo an in-context collapse
CircA → mitigates → In-Context Collapse
confidence 94% · CircA ... confers collapse-resistance that transfers to unseen task families
In-Context Collapse → localizedto → Vision-Language Integration Pathway
confidence 93% · causally localizes the collapse to the vision-language integration pathway: an adapter on the connector and early/mid layers
Robustness to Demonstrations → dissociablefrom → Ability to Learn Novel Rule
confidence 92% · two capabilities turn out to be dissociable: robustness to accumulating demonstrations and the ability to learn a novel rule in context
Vision-Language Integration Pathway → distinctfrom → Late Readout Layers
confidence 91% · the layers best for in-context integration are not the layers best for weight-based consolidation
CircA → implementedas → Adapter
confidence 90% · a single adapter, trained once on one synthetic task, that confers collapse-resistance
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Many-shot in-context learning (ICL) lets vision-language models (VLMs) adapt from image--label demonstrations without weight updates, and is widely assumed to improve as more demonstrations are supplied. We show the opposite: as demonstrations accumulate, a subset of VLMs undergo an \emph{in-context collapse}, a sharp, sometimes catastrophic accuracy drop spanning synthetic classification, natural-image classification, and VQA benchmarks, in some models falling below chance while outputs remain well-formed. Across an open VLM panel ($0.5$B--$11$B) and a frontier model (Claude Sonnet 4.5), the collapse is graded. Two capabilities turn out to be dissociable: robustness to accumulating demonstrations and the ability to learn a novel rule in context, their combinations yield three reproducible regimes. A parameter-matched lesion-and-rescue causally localizes the collapse to the vision-language integration pathway: an adapter on the connector and early/mid layers restores genuine learning (remap accuracy $0.39!\rightarrow!0.91$ at 16 shots), while an equal-capacity adapter on the late readout does not. We propose \textsc{CircA}, whose core is a one-time integration vaccine: trained once on one synthetic task, it transfers collapse-resistance to unseen task families (chance$\rightarrow$$0.71$/$0.60$ on CIFAR/Fashion). The layers best for in-context integration are not the layers best for weight-based consolidation, the late readout achieves higher accuracy and less forgetting at fewer parameters. The collapse is an integration failure at the vision--language interface, correctable by a lightweight, transferable intervention.
Tags
Links
- Source: https://arxiv.org/abs/2608.02830v1
- Canonical: https://arxiv.org/abs/2608.02830v1
Trouble viewing inline? Open PDF directly →
Full Text
122,954 characters extracted from source content.
Expand or collapse full text
1] Generative AI Innovation Center111This work is independent of the position at Amazon, In-Context Collapse in Vision-Language Models and How to Mitigate it? mrostami@seas.upenn.edu [ Abstract Many-shot in-context learning (ICL) lets vision-language models (VLMs) adapt from image–label demonstrations without weight updates, and is widely assumed to improve as more demonstrations are supplied. We show the opposite is possible: as demonstrations accumulate, a subset of VLMs undergo an in-context collapse, a sharp, sometimes catastrophic drop in accuracy, in some models to far below chance, spanning synthetic classification, natural-image classification, and standard visual-question-answering benchmarks. The collapse is a genuine failure of judgment, not of output formatting: it occurs in-distribution and while the model still emits well-formed labels. Across an open VLM panel (0.50.5B–1111B parameters, several connector designs) and, strikingly, a frontier model (Claude Sonnet 4.5), the collapse is graded rather than universal. Central to this work, two capabilities usually conflated turn out to be dissociable: robustness to accumulating demonstrations and the ability to learn a novel rule in context. Their combinations yield three reproducible regimes, which we separate with contamination-free synthetic concepts that distinguish genuine in-context learning from retrieval of pretraining priors. We then move from diagnosis to mechanism and remedy. In MLP-projector VLMs, a parameter-matched lesion-and-rescue causally localizes the collapse to the vision–language integration pathway: a small adapter on the connector and early/mid layers removes it and restores genuine learning (remap accuracy 0.39→0.910.39\!→\!0.91 at 1616 shots on a contamination-free task), whereas an equal-capacity adapter on the late readout layers does not, and can drive learning below chance; the same integration-not-readout ordering, and the repair built on it, reproduce across three architectures spanning two connector families. Building on this locus we propose CircA, an integration-circuit adaptation framework whose core is a one-time integration vaccine: a single adapter, trained once on one synthetic task, confers collapse-resistance that transfers to unseen task families (chance → 0.710.71/0.600.60 on held-out CIFAR/Fashion at 1616 shots), while an equal-capacity late-locus variant fails. A final dissociation carries a continual-learning consequence: the layers best for in-context integration (early/mid) are not the layers best for durable weight-based consolidation across a task stream (late), which achieve higher accuracy and less forgetting at a fraction of the parameters. The in-context collapse is thus an integration failure at the vision–language interface, mechanistically distinct from both the readout and the consolidation locus, and correctable by a lightweight, transferable intervention. Code and data to reproduce all results are available at https://github.com/rostami-m/incontext-collapse. keywords: Vision-Language Models, In-Context Learning, Mechanistic Interpretability, Continual Learning, Catastrophic Forgetting 1 Introduction In-context learning (ICL) has become the default way to adapt large models to new tasks. Instead of collecting a dataset and running gradient descent, a practitioner places a few labeled examples in the prompt and the model generalizes to a new query, no weight updates, no optimizer, nothing beyond a forward pass [4]. For text the capability is now foundational, and as context windows have grown, few-shot prompting has become many-shot prompting, with hundreds or thousands of demonstrations rivaling fine-tuning [1]. Vision-language models (VLMs) inherit this promise and raise the stakes: image labels are expensive, visual domains drift after deployment, and many-shot multimodal ICL is the natural weight-free way to adapt a model to a new domain, label set, or visual style, attractive precisely because it updates no weights and therefore cannot, by construction, cause catastrophic forgetting [2, 22]. If accuracy grew with the number of demonstrations, as it largely does for text, then showing a VLM more examples would be close to a free lunch. It is not. We find that supplying more demonstrations frequently makes VLMs worse, and often abruptly. Consider Qwen2-VL-2B classifying CIFAR images into four categories: with no demonstrations it is correct 94%94\% of the time, yet as correctly labeled examples accumulate its accuracy falls to 38%38\% by K=64K=64 shots, and it does so while still emitting well-formed labels, so the drop is a failure of judgement, not of output formatting. Other models fail harder still: LLaVA-OneVision-0.5B slides from 72%72\% to below the 25%25\% of random guessing. The demonstrations did not merely fail to help; they destroyed a capability the model already had. We call this phenomenon the in-context collapse: as in-context demonstrations accumulate, accuracy falls, sometimes catastrophically and sometimes below chance. It is easy to miss, because it hides behind reported averages: it emerges only past a few shots, varies sharply from model to model, and coexists with a second, quieter failure: even VLMs that resist collapse often cannot learn a genuinely novel rule from the demonstrations, silently falling back on pretraining priors instead of the in-prompt mapping. We observe the collapse across procedurally generated concepts, natural-image classification, and standard visual-question-answering benchmarks, in open models from 0.50.5B to 1111B parameters and, strikingly, in a frontier model. Three observations in the recent literature each touch part of this picture but have remained disconnected. First, several empirical studies of many-shot multimodal ICL report that it can be unreliable or actively harmful, accuracy plateaus and then declines, models imitate the surface form of the demonstrations rather than reasoning from their content, and VLMs “struggle to make use of a larger number of ICL examples” [20, 30, 60]. These report the degradation behaviorally, as imitation, copying, or insensitivity to shot count; what has gone unremarked is that the drop can fall below random chance while the model still emits well-formed labels, a destruction of existing capability rather than a failure to improve, and that it is only one of two separable failure axes (below). Second, mechanistic studies of VLMs find that visual information becomes linearly readable only in the deep layers of the language model [35, 12]. Third, VLMs forget catastrophically when their weights are fine-tuned on a sequence of tasks [57, 10]. These findings live in three separate communities (a behavioral anomaly, an interpretability result, and a continual-learning problem), and the links among them have gone largely unexamined; in particular, the unreliability of many-shot adaptation has not been tied to a specific, editable mechanism, much less to a remedy. Where prior work asks whether many-shot multimodal ICL works, we ask why it fails, where in the network, and what to do about it. This paper both reports a phenomenon and proposes an algorithm that addresses it. We proceed in three steps. (i) Characterize. We show that robustness to accumulating demonstrations and the ability to learn a novel rule in context are dissociable axes, using contamination-free synthetic concepts whose answers cannot be retrieved from pretraining and must be learned in the prompt. (i) Localize. A parameter-matched lesion-and-rescue causally places the collapse at the vision–language integration pathway: a small adapter on the connector and early/mid layers removes it and restores genuine learning, whereas an equal-capacity adapter on the late readout layers does not. (i) Repair. We turn the locus into a method, CircA (integration-circuit adaptation), whose core is a one-time integration vaccine, a single adapter, trained once on one synthetic task, that confers collapse-resistance transferring to unseen task families. Because ICL is weight-free, the collapse is a capacity bound on the fast adaptation path; we further show that the integration locus that repairs it is not the locus where weight-based consolidation across a task stream is most durable, yielding a concrete rule for when to stop prompting and start consolidating. Our specific contributions include: 1. The in-context collapse and its graded regimes (Sec. 4.2.1). Prior work has noted that many-shot multimodal ICL can degrade or that models copy rather than reason [20, 60]; we sharpen this into a measured phenomenon, accuracy falling below chance while outputs remain well-formed, graded across a panel from 0.50.5B to 1111B and at frontier scale, and we separate it from mere output degeneration. Takeaway: many-shot adaptation is not free; whether it helps or harms is set by the alignment recipe, not by parameter count. 2. A robustness–learning dissociation (Sec. 4.2.2). With contamination-free concepts and a label-shuffle control we separate surviving demonstrations from learning from them. Takeaway: a model can be perfectly robust to added demonstrations yet unable to learn a novel rule, “does ICL work?” is two questions, not one. 3. A causal integration locus (Sec. 4.2.4). A parameter-matched lesion-and-rescue makes the collapse appear and disappear on demand. Takeaway: the collapse is not diffuse but editable, it lives at the connector and early/mid layers, not at the readout. 4. CircA, a transferable repair (Sec. 4.3.1). A one-time integration vaccine immunizes a collapse-prone model, and the immunity transfers to unseen tasks (chance → 0.710.71/0.600.60 on held-out CIFAR/Fashion), while an equal-capacity late-locus variant fails. Takeaway: knowing where the collapse lives is enough to repair it cheaply and off-line. 5. Integration ≠ consolidation (Sec. 4.4.3). The integration locus differs from where durable weight-based consolidation across a stream is most effective. Takeaway: the place to repair fast in-context adaptation is not the place to write slow, durable memories. 2 Related Work Our study sits at the intersection of three literatures that have largely developed in isolation: the empirical study of multimodal in-context learning, the mechanistic interpretability of vision-language models, and continual learning for VLMs. We review each in turn, emphasizing both what is established and what is missing, and we close by stating precisely where this work sits among them. Multimodal in-context learning. The ability to adapt a VLM from a handful of image–text demonstrations, with no weight update, was established by Frozen [47], which showed that a frozen language model prompted with a learned visual prefix can perform few-shot visual question answering, and by Flamingo [2], whose gated cross-attention layers interleave visual features into a frozen language model and enabled strong few-shot transfer from in-context examples. Otter [27] then made the capability an explicit training target, instruction-tuning on a corpus of interleaved in-context image–text examples (MIMIC-IT) so that following demonstrations becomes a trained behavior rather than an emergent side effect. As context windows grew, attention turned to the many-shot regime, where the literature splits. On closed frontier models, Jiang et al. [22] report that accuracy improves roughly log-linearly out to hundreds or thousands of demonstrations, suggesting that “more is better.” On open models a contrary picture has emerged: adding demonstrations is frequently unreliable or harmful, accuracy plateaus and then declines, and models appear to imitate the surface form of the demonstrations rather than reason from them [20, 30]. Two lines of work try to repair many-shot MICL, tacitly conceding that the naive form often fails. Multimodal task vectors [19] distill a block of demonstrations into a compact set of activation edits patched into the forward pass at inference, recovering much of the many-shot benefit while spending almost no context, evidence that the useful content of the demonstrations is low-dimensional and relocatable. Link-context learning [45] instead restructures the demonstrations, pairing positively and negatively “linked” image–label examples with shared causal structure so a model can bind a genuinely novel word to a novel visual concept from the prompt alone. Both treat the degradation as given and engineer around it; we instead ask what makes the naive form fail, and find that the integration locus they implicitly target can be repaired directly. A complementary benchmark line, VL-ICL Bench [60], reports that VLMs “struggle to make use of a larger number of ICL examples” and that zero-shot ability does not predict ICL ability, the closest prior statement to our robustness–learning split; it establishes the symptom across tasks but does not separate the two axes as distinct regimes, nor localize a cause. Our closest point of contact is [20], which argues that VLMs “mimic rather than reason” and that accuracy can degrade with more shots. We share that starting observation but depart from it in three ways. First, we show the degradation is only one of two separable axes formalized as three regimes (Sec. 4.2.2), a model can be perfectly robust to added demonstrations and yet wholly unable to learn a novel rule from them, a distinction neither [20] nor [60] draws. Second, rather than characterizing the failure behaviorally, we trace it to a specific, editable circuit at the vision–language interface. Third, we connect that circuit to the downstream question of where adaptation should be written into weights. Where prior work asks whether many-shot MICL works, we ask why it fails, where in the network, and what follows for adaptation. In-context learning over images. A separate strand performs ICL in the pixel space rather than over image–label pairs. Visual prompting defines a task by inpainting a grid that already contains a worked input–output example [3]; generalist in-context models such as Painter [50] and SegGPT [52] cast dense prediction as image-to-image translation conditioned on an example pair; and other work studies how to select good visual exemplars, finding that the choice of demonstration strongly affects performance [58]. These study a fundamentally different mechanism, completing an image given an example, whereas we study classification ICL, in which the demonstrations encode a discrete image→ rule. That difference is what lets us make the task contamination-free and thereby separate genuine in-context rule learning from retrieval of pretraining priors, a separation the pixel-space setting does not afford. Mechanistic interpretability of vision-language models. A young but fast-growing literature opens up how VLMs process images internally, and it supplies the machinery our causal analysis builds on. Palit et al. [38] adapted causal tracing to BLIP, the first such tool for a VLM. Neo et al. [35] applied the logit-lens to visual tokens and argued for a “process-then-retrieve” account in which visual information is refined locally and becomes decision-relevant only in deeper layers; Kaduri et al. [23] reach a related conclusion in a broad dissection of where and how VLMs use visual content, reporting that image tokens are queried selectively and predominantly in the middle-to-late layers, and that much of the visual signal is summarized into a few register-like positions before the answer is formed. Cohen et al. [12] show that image-token information reaches the answer only in deep layers, producing a cross-modal gap in entity-knowledge extraction relative to text, and Golovanevsky et al. [15] introduce a semantic (rather than Gaussian-noise) image-corruption pipeline to localize the heads and tokens a VLM depends on. Collectively this work establishes where visual information becomes readable, but it stops at correlational localization: it does not turn a localization into an intervention that adds or removes a behavioral failure. Supplying exactly that step, a parameter-matched, bidirectional manipulation tied to a specific behavior, is the methodological core of our paper, and our result refines the readability picture by distinguishing where the decision is read out from where the demonstrations are integrated. In-context learning mechanisms in language models. The text-only community has gone further toward mechanism, and we borrow its hypotheses. Induction heads were identified as a circuit that copies and completes repeated patterns and whose formation coincides with the emergence of ICL [36]. Hendel et al. [16] showed that ICL compresses a task into a single “task vector” in the residual stream, and Todd et al. [46] found transplantable “function vectors” that causally transport a task between contexts. These results make it natural to expect that multimodal ICL is likewise computed by an identifiable, editable circuit, but they operate entirely within the language stream. Our intervention is the multimodal analogue located at the modality boundary: we localize and edit the integration of visual demonstrations at the connector and early/mid layers, a locus with no counterpart in text-only studies and the one place where the visual and linguistic streams must be bound. Continual learning for vision-language models. Because in-context adaptation is transient, durable adaptation requires writing to weights, which re-introduces forgetting. Zhai et al. [57] document severe catastrophic forgetting when multimodal LLMs are fine-tuned sequentially; CoIN [10] establishes a standard continual-instruction-tuning benchmark and shows that even strong VLMs degrade across a task stream; and SMoLoRA [53] dissects a dual forgetting, of both instruction-following and visual understanding, mitigated by separable low-rank adapters. The CLiMB benchmark [43] specifically studies continual learning across vision-and-language tasks, finding that standard CL algorithms reduce forgetting but do not enable cross-task knowledge transfer; transformer architectures that dynamically expand capacity while distilling prior knowledge address both challenges [8, 7]. Other mitigations are tailored to the multimodal setting: ModalPrompt [55] maintains a pool of task-specific prompts and routes between them by image–text similarity, avoiding weight overwrites altogether; CluMo [6] pairs visual and textual prompt keys via clustering for continual VQA; while ModelTailor [59] identifies and surgically preserves the small “forgetting-sensitive” subset of parameters most responsible for degradation. These build on the classical continual-learning toolkit: weight-importance regularization that slows updates to parameters deemed important for past tasks (EWC [24], SI [56]) and function-space distillation from the previous model (LwF [31]); episodic and dark-experience rehearsal that replays stored exemplars or their logits (GEM/A-GEM [33, 9], iCaRL [39], DER [5]); generative replay that synthesizes past data with a learned generator [42]; and parameter-efficient or orthogonal-subspace adaptation [17, 51, 44], surveyed in [26, 48]. This entire literature asks how to consolidate without forgetting; we contribute an orthogonal and, to our knowledge, previously unremarked observation about where: the layers at which weight-based consolidation across a stream is most effective are not the layers at which in-context demonstrations are integrated, which constrains how parameter-efficient continual methods ought to be targeted. The broader fast/slow division between weight-free and weight-based adaptation echoes complementary learning systems in the brain [34, 25, 37] and the in-context vs. in-weight interplay studied in humans and networks [41]; cognitively inspired models of incremental concept learning draw on the same CLS theory to enable networks to expand knowledge without cross-task interference [40]; we use this only as a light organizing lens, not as a claim. Three facts are, by now, each established in isolation. (i) Many-shot multimodal ICL can degrade, and degrade in a way that looks like imitation rather than reasoning [20, 30]. (i) Visual information becomes linearly readable only in the deep layers of a VLM [35, 12]. (i) VLMs forget catastrophically under sequential fine-tuning [57, 10]. Each has been pursued in a separate community (a behavioral anomaly, an interpretability finding, and a continual-learning problem), and the connections among them have gone largely unremarked. The thesis of this paper is that they are three views of one underlying structure. The behavioral degradation (i) is only one half of a robustness–learning dissociation; we causally localize it to the integration interface with a parameter-matched lesion-and-rescue, and in doing so we refine the readability picture (i) by showing that the late layers read out the decision but do not integrate the demonstrations, readability and manipulability come apart. The same intervention reveals that the integration locus is distinct from where weight-based consolidation (i) is most durable, so the remedy for the collapse and the remedy for forgetting live in different parts of the network. Our contribution is therefore not a new model, benchmark, or training method, but a unifying causal account that converts a scattered set of observations into a single, editable mechanism at the vision–language boundary. 3 The In-Context Collapse and How to Address It We first establish that many-shot multimodal ICL can collapse, then argue that the collapse is a localized failure of a specific computation, and finally use that diagnosis to design a fix. This section develops the phenomenon and the algorithm; the empirical evidence for every claim made here is given in Sec. 4. We first fix notation for a VLM and for many-shot multimodal ICL (Sec. 3.1), use it to define the collapse and the robustness–learning dissociation precisely (Sec. 3.2), formalize the integration/readout partition and the lesion-and-rescue intervention that localizes the failure (Sec. 3.3), and finally state the three components of CircA as operations on that partition (Sec. 3.4). 3.1 Preliminaries and notation A vision–language model as a composition of maps. We consider an autoregressive VLM that consumes an interleaved sequence of images and text and emits text. It factorizes into four stages. A vision encoder gϕ:→ℝnv×dvg_φ:X ^n_v× d_v maps an image x∈x to nvn_v visual feature vectors. A connector cψ:ℝnv×dv→ℝm×dc_ψ:R^n_v× d_v ^m× d projects those features into the d-dimensional token-embedding space of the language model, producing m visual tokens (for an MLP-projector model m=nvm=n_v and cψc_ψ is a small feed-forward network; for a resampler/perceiver it is a fixed set of learned queries). A language model is a stack of L pre-norm transformer blocks hθℓ(ℓ):ℝT×d→ℝT×dh^( )_ _ :R^T× d ^T× d, ℓ=1,…,L =1,…,L, each acting on a length-T sequence of d-dimensional hidden states by the residual updates H~(ℓ)=H(ℓ−1)+Attn(ℓ)(H(ℓ−1)),H(ℓ)=H~(ℓ)+MLP(ℓ)(H~(ℓ)), H^( )=H^( -1)+Attn^( )\! (H^( -1) ), H^( )= H^( )+MLP^( )\! ( H^( ) ), (1) where Attn(ℓ)Attn^( ) and MLP(ℓ)MLP^( ) are the attention and feed-forward sublayers, each a set of linear projections with frozen weights. Finally an unembedding WU∈ℝ||×dW_U ^|V|× d maps the last hidden state of the final layer to logits over the vocabulary V. Writing the full network as fθf_θ with parameters θ=(ϕ,ψ,θ1:L,WU)θ=(φ,ψ, _1:L,W_U), the next-token distribution given a token/visual-token sequence s is pθ(⋅∣s)=softmax(WUHlast(L)(s))p_θ(· s)=softmax (W_U\,H^(L)_last(s) ). Tasks, verbalizers, and the standard/remap conditions. A concept task is a distribution D over pairs (x,y)(x,y) with x∈x an image and y∈=1,…,Cy =\1,…,C\ a class. A verbalizer v:→∗v:C ^* assigns each class a label string. The verbalizer is the lever that separates retrieval from learning, and it defines our two conditions: • standard: v=vnatv=v_nat maps each class to its natural name (e.g. v(cat class)="cat"v(cat class)= "cat"), so a correct answer may come either from the pretraining prior or from the demonstrations. • remap: v=ρv=ρ is a fixed bijection onto arbitrary tokens disjoint from the natural names (e.g. class ↦ "blorp"). Because ρ is independent of pretraining, above-chance accuracy under remap can only arise from binding the in-prompt mapping, never from retrieval. This is what we mean by contamination-free. Many-shot multimodal ICL. A K-shot episode draws a support set of demonstrations SK=((x1,y1),…,(xK,yK))S_K= ((x_1,y_1),…,(x_K,y_K) ) and a query xqx_q i.i.d. from D, class-balanced, and forms the interleaved prompt PK=(x1,v(y1),x2,v(y2),…,xK,v(yK),xq),P_K\;=\; (\,x_1,\,v(y_1),\;x_2,\,v(y_2),\;…,\;x_K,\,v(y_K),\;x_q\, ), (2) each image first mapped to visual tokens by gϕg_φ and cψc_ψ. The model predicts the class whose label string is most probable, y^q=argmaxc∈pθ(v(c)∣PK) y_q= _c p_θ\! (v(c) P_K ), and we report the balanced accuracy Av(K)=PrSK,xq∼[y^q=yq],A^v(K)\;=\; _S_K,x_q [ y_q=y_q ], (3) as a function of the shot count K, separately for v=vnatv=v_nat (written AstdA^std) and v=ρv=ρ (written ArmpA^rmp). Chance accuracy is 1/C1/C. The K=0K=0 point is the zero-shot prior: Astd(0)A^std(0) measures pretrained competence on the task, while Armp(0)≈1/CA^rmp(0)≈ 1/C (chance) by construction, since the arbitrary labels carry no pretraining signal and a model with no demonstrations can do no better than guess among them. 3.2 The phenomenon By the in-context collapse we mean a simple but counter-intuitive behavior: for a subset of VLMs, accuracy on a task decreases as in-context demonstrations are added, sometimes gradually, often abruptly, and in the worst cases to below chance (Fig. 1, red). With the notation of Sec. 3.1 in hand, we can state this more accurately. Whereas the canonical few-shot intuition is that accuracy is non-decreasing in K, the collapse is the opposite monotonicity. Figure 1: The in-context collapse (schematic). As demonstrations accumulate, a collapse-prone model’s accuracy falls, often below chance, while a collapse-resistant model is flat and a genuine learner improves. Robustness (does accuracy survive added demonstrations?) and learning (does accuracy rise on a novel rule?) are separable axes whose combinations define three regimes, established empirically in Sec. 4. Definition 1 (In-context collapse). A model exhibits an in-context collapse on a task under verbalizer v if its accuracy curve Av(K)A^v(K) from Eq. (3) is decreasing past some shot count: there exist K1<K2K_1<K_2 with Av(K2)<Av(K1)A^v(K_2)<A^v(K_1). We call the collapse catastrophic when Av(K2)≤Av(0)A^v(K_2)≤ A^v(0) (the demonstrations leave the model worse than seeing none) and sub-chance when Av(K2)<1/CA^v(K_2)<1/C. We summarize its magnitude by the drop δv=maxKAv(K)−Av(Kmax)δ^v= _KA^v(K)-A^v(K_ ). The collapse is not universal: some VLMs hold a flat curve as demonstrations accumulate (blue) or even improve (green). That gradedness is precisely what makes it dangerous in practice: a practitioner who validates many-shot prompting on one model and deploys it on another can silently lose most of the model’s accuracy. Crucially, two capabilities that are usually conflated come apart, and the contrast between the two verbalizers makes them separately measurable. Definition 2 (Robustness and learning). For a task and model we define two scalar axes from the accuracy curves of Eq. (3): robustness Rob=Astd(Kmax)−Astd(0)Rob=A^std(K_ )-A^std(0), the change in standard accuracy as demonstrations accumulate (negative under collapse); and learning Lrn=Armp(Kmax)−Armp(0)Lrn=A^rmp(K_ )-A^rmp(0), the gain in remap accuracy over its at-chance starting point, which by contamination-freeness measures the ability to acquire a novel in-prompt rule (any above-chance remap accuracy can come only from the demonstrations). Robustness and learning are logically independent: a model may be perfectly robust (Rob≈0Rob≈ 0) yet unable to learn (Lrn≈0Lrn≈ 0), answering from its prior rather than the demonstrations, or it may learn while collapsing on the natural labels. Thresholding the two axes partitions models into the regimes of Fig. 1 (made empirical in Sec. 4.2.1–4.2.2): collapse-prone (Rob≪0Rob\! \!0), collapse-resistant retrieval (Rob≈0,Lrn≈0Rob\!≈\!0,\ Lrn\!≈\!0), and genuine learner (Rob≳0,Lrn≫0Rob\! \!0,\ Lrn\! \!0). The collapse is thus graded and recipe-dependent rather than a function of scale, and the remap verbalizer ρ is what lets above-chance accuracy be attributed to learning the in-prompt mapping, never to retrieval. 3.3 Why it happens, and where to intervene A VLM routes an image through a vision encoder and a connector into a language model whose early/mid layers must integrate the interleaved demonstrations and whose late layers read out the answer (Fig. 2). We make “where” precise by partitioning the model’s parameters into disjoint loci by depth. Figure 2: The in-context collapse is an integration failure at the vision–language interface. A VLM maps an image through a vision encoder and a connector into a language model whose early/mid layers integrate the in-context demonstrations and whose late layers read out the decision. We show the collapse is causally removable by a small adapter on the connector and early/mid layers (green) but not by an equal-capacity adapter on the late readout (red); conversely, durable weight-based consolidation across a task stream is most effective at that same late readout, a double dissociation between the integration and consolidation loci. The integration/readout partition. Index the connector together with the L language-model blocks, and split the block indices into contiguous thirds, ℰ=1,…,⌊L/3⌋E=\1,…, L/3 \ (early), ℳ=⌊L/3⌋+1,…,⌊2L/3⌋M=\ L/3 +1,…, 2L/3 \ (mid), and ℒ=⌊2L/3⌋+1,…,LL=\ 2L/3 +1,…,L\ (late). We group these into an integration locus and a readout locus, ℐ=cψ∪ℰ∪ℳ,ℛ=ℒ,I\;=\;\c_ψ\ , \;=\;L, (4) where ℐI is the pathway that binds visual demonstrations to their in-prompt labels at the modality boundary, and ℛR is the pathway that maps an already-formed decision to output tokens. Our central hypothesis is mechanistic and falsifiable. Definition 3 (Integration hypothesis). The in-context collapse is a deficit of the integration computation localized to ℐI, not of the readout ℛR: as demonstrations accumulate, the vision–language integration pathway is overwhelmed and the model reverts to its prior instead of binding the in-prompt mapping. Lesion-and-rescue as a causal test. To test Definition 3 we ask whether adding a small amount of trainable capacity at a chosen locus can restore learning that the frozen model lacks. Concretely, for a target region ⊆ℐ∪ℛG we attach a low-rank adapter [17] to the linear projections in G: each such weight W∈ℝdout×dinW ^d_out× d_in is replaced by W+ΔW+ _G with Δ=αrBA _G= αrBA, B∈ℝdout×rB ^d_out× r, A∈ℝr×dinA ^r× d_in, rank r≪min(din,dout)r (d_in,d_out), while all original parameters θ stay frozen. We then train only Δ _G on remap episodes, minimizing the query cross-entropy Δ⋆=argminΔSK,xq∼[−logpθ⊕Δ(ρ(yq)∣PK)], _G\;=\; _ _G\;E_S_K,x_q [- p_θ _G (ρ(y_q) P_K ) ], (5) and re-measure the rescued learning curve Armp(K)A^rmp_G(K). A region rescues the collapse if it lifts learning well above the unintervened baseline, Armp(Kmax)≫Armp(Kmax)A^rmp_G(K_ ) A^rmp(K_ ). The hypothesis predicts an asymmetry: rescue should succeed when ⊆ℐG and fail when =ℛG=R. Controlling for capacity. Because a larger adapter could rescue trivially by sheer added capacity, the test is only meaningful at matched budget. We therefore compare an integration region against the readout at an equal module count |ℐ|≈|ℛ||G_I|≈|G_R| and equal rank r (e.g. on Qwen2-VL-2B the early and late regions differ by under 10%10\% of adapter modules; see Sec. 4.1). Under this control, any difference between rescuing at ℐI versus ℛR reflects location, not parameter count, so a confirmed asymmetry, Aℐrmp(Kmax)≫Armp(Kmax)≈Aℛrmp(Kmax),A^rmp_I(K_ )\; \;A^rmp(K_ )\;≈\;A^rmp_R(K_ ), (6) is a causal localization of the collapse to the integration pathway. Sec. 4.2.4 confirms Eq. (6) with a parameter-matched lesion-and-rescue, turning the hypothesis about “where” into a causal handle we build on. 3.4 CircA: repairing the integration circuit The diagnosis in Sec. 3.3 is constructive: it does not merely say that the collapse exists, it says where the fault lies (the integration locus ℐI) and what would fix it (more capacity placed exactly there). CircA (integration-circuit adaptation) is the operationalization of that diagnosis. A collapse-prone model has the right perceptual machinery but a weak integration step: when many demonstrations arrive, it fails to bind them to the query and falls back on copying the most recent label or answering from its pretraining prior. CircA addresses this in two complementary ways. First, if we are allowed a small one-time offline investment, we strengthen the integration step directly: we train a tiny adapter at ℐI on a single synthetic task and obtain a model that now integrates demonstrations in general, a procedure we call a vaccine because it is administered once, ahead of time, and confers lasting resistance. Second, if we must take the model as given at serving time, we avoid the regime where integration breaks: a gate stops adding demonstrations once a cheap diagnostic shows the model has started copying, and an inject path folds extra demonstrations into a single activation edit instead of spending context on them. The three components share one locus, ℐI, and split cleanly by what they assume: the vaccine creates integration capacity where it is missing, whereas the gate and inject exploit integration capacity that already exists (Fig. 3). We now define each as an explicit operation on the model. Figure 3: CircA: integration-circuit adaptation. A VLM maps an image through a vision encoder and connector into a language model that first integrates demonstrations (early/mid, green) and then reads out the answer (late, red). CircA targets the integration locus with three complementary components: a one-time offline vaccine (adapter on the connector + early/mid layers), an inference-time gate that caps the demonstration budget at saturation, and an inject path that adds a demonstration-derived task vector at the same locus. 3.4.1 The vaccine: a transferable integration adapter. The vaccine is the load-bearing component and is the same intervention as the diagnostic lesion-and-rescue of Sec. 3.3 (Eq. (5)), used now as a remedy rather than a probe. Recall that in the diagnosis we showed a small low-rank adapter placed at the integration locus ℐI can restore a model’s ability to learn an arbitrary in-context rule. The vaccine asks the natural follow-up question: is the restored ability specific to the task we trained on, or have we installed a general “read the demonstrations and use them” skill that carries over to new tasks? If the latter, the fix becomes practical: train once, deploy everywhere. Concretely, we attach a low-rank adapter Δℐ _I to the integration locus, freeze every original weight θ, and train only Δℐ _I on a single contamination-free task 0T_0 in the remap condition (arbitrary labels ρ, so the only way to be right is to use the demonstrations). The training objective is ordinary supervised fine-tuning, minimizing the negative log-likelihood of the query’s correct (remapped) label given the K-shot prompt, Δ⋆=argminΔℐSK,xq∼0[−logpθ⊕Δℐ(ρ(yq)∣PK)],fvac:=fθ⊕Δ⋆. \;=\; _ _I\;E_S_K,x_q _0 [- p_θ _I (ρ(y_q) P_K ) ], f^\,vac:=f_θ . (7) In other words, across many sampled demonstration sets SKS_K and queries xqx_q from the one training task, push the model to predict the query’s true label from the in-prompt mapping alone. The result is a single immunized model fvacf^\,vac, which is then frozen and reused on every downstream task with no further training, no per-task tuning, and no labels at deployment. Three design choices are deliberate and worth making explicit, because they are what make the vaccine a one-time, transferable fix rather than ordinary task-specific fine-tuning. (i) Locus. The adapter is confined to ℐI (connector and early/mid layers), the site the diagnosis identified; placing the same capacity at the readout does not work (the control of Eq. (6)). (i) A contamination-free training task. Training on remap arbitrary labels prevents the adapter from memorizing a label→ shortcut and forces it to learn the demonstration-using procedure, which is what transfers. (i) Parameter efficiency. Δℐ _I is a rank-r update on a small subset of modules (∼1% \!1\% of parameters), so the vaccine is cheap to train and adds negligible inference cost. The claim, tested in Sec. 4.3.1, is that Δ⋆ installs a general capability rather than a 0T_0-specific lookup, so that for unseen task families ≠0T _0 the learning curve satisfies Afvacrmp(Kmax;)≫Afθrmp(Kmax;)A^rmp_f^\,vac(K_ ;T) A^rmp_f_θ(K_ ;T), i.e. a model that could not learn novel rules in context now can, on tasks it was never trained on.222We use vaccine in the sense of a one-time offline intervention conferring transferable resistance. The term has been used independently for harmful-fine-tuning-robust alignment [21] and, as inoculation, for prompt-based suppression of undesired behaviors [54]; our usage and mechanism (an integration-locus adapter against in-context collapse) are unrelated to both. 3.4.2 The gate: stop before saturation. The gate addresses a different situation: we cannot modify the model (no offline access, a closed API), so we cannot vaccinate it. The only lever left is how many demonstrations we feed it. Since accuracy falls as K grows past a model-specific point, the safe policy is to stop adding demonstrations just before that point, getting the benefit of a few shots without paying the cost of collapse. The difficulty is detecting the onset of collapse without ground-truth labels at serving time. We use a cheap, label-free signal, the copy rate, defined as the fraction of queries whose prediction equals the label attached to the most recent demonstration: r^(K)=PrSK,xq[y^q=yK]. r(K)\;=\; _S_K,x_q [ y_q=y_K ]. (8) Intuitively, a healthy model’s answer should depend on the query image, not on which demonstration happened to come last; if the prediction tracks the last label instead, the model has stopped reasoning and started copying. A rising r^(K) r(K) is therefore the behavioral fingerprint of the collapse, and, crucially, it needs no labels to compute, only the model’s own outputs. Given a tolerance τ, the gate serves with the largest shot count whose copy rate is still acceptable, K⋆=maxK:r^(K)≤τ,K \;=\; \\,K: r(K)≤τ\,\, (9) truncating the demonstration budget at the onset of saturation. The copy rate is estimated once on a small held-out probe, so the gate is essentially free. Unlike the vaccine, it adds no parameters and changes nothing inside the model; it simply avoids operating the model in the regime where its integration step is known to fail. 3.4.3 The inject path: amortize context at the integration locus. The third component targets cost rather than accuracy. Demonstrations are expensive: each image consumes hundreds of tokens, so a many-shot prompt can be slow and may not even fit in the context window. The inject path lets a model that can already integrate pay that cost once and reuse the result. The idea, the multimodal analogue of a task/function vector in language models [16, 46, 19], is that the useful effect of a block of demonstrations on the model’s internal state is low-dimensional and can be summarized by a single vector. We compute that vector by measuring how a demonstration block D shifts the model’s hidden state at the integration locus, relative to seeing no demonstrations at all: vℐ(D)=1|ℐ⋆|∑ℓ∈ℐ⋆(H¯(ℓ)(D)−H¯(ℓ)(∅)),v_I(D)\;=\; 1|I_ |\!\! _ _ \!\! ( H^( )(D)- H^( )( ) ), (10) Reading the equation: for each chosen integration layer ℓ∈ℐ⋆ _ , H¯(ℓ)(D) H^( )(D) is the hidden state (at a fixed reference position) when the prompt contains the demonstrations and H¯(ℓ)(∅) H^( )( ) is the same quantity with no demonstrations; their difference is the “demonstration effect” at that layer, and we average it over layers to obtain one vector vℐ(D)v_I(D). At inference we then answer the query with vℐ(D)v_I(D) added back into the residual stream at those layers, reproducing the effect of the demonstrations while spending no demonstration tokens in the prompt. The key conceptual point, which also delimits the framework, is that injection can only relocate a computation the model already performs: if the model cannot integrate demonstrations in the first place, the extracted vector encodes nothing useful. This is exactly why inject amortizes context for a vaccinated or collapse-resistant model but cannot, by itself, repair a collapse-prone one, a prediction we confirm in Sec. 4.3.1. Our experiments load-test this division of labor (Sec. 4.3.1): the vaccine of Eq. (7) confers collapse-resistance that transfers to unseen task families; a locus ablation shows the repair must live at ℐI, since an equal-capacity adapter at the readout ℛR fails entirely (the asymmetry of Eq. (6)); and the training-free injection of Eq. (10), which can only relocate a computation the model already performs, does not by itself repair a collapse-prone model, marking the boundary of the framework. Which component to use when? The three components are not alternatives to choose among but a small decision tree keyed to deployment constraints. If one can touch the model offline even once, apply the vaccine: it is the only component that raises the accuracy ceiling on a collapse-prone model, and it is paid for once. If the model is fixed at serving time (e.g. a closed API), use the gate to stay out of the collapsed regime. If, in addition, prompt length or latency is a binding constraint and the model can already integrate (natively or post-vaccine), use inject to amortize the demonstrations into a single edit. Vaccine and gate/inject compose: a vaccinated model can still be gated and injected at serving time. Algorithm 1 states the framework in full and mirrors this split. Lines 1–4 are the one-time offline vaccine that produces the immunized model fvacf^\,vac; lines 5–11 are the per-deployment serving policy, in which the gate caps the shot budget at saturation (line 6) and the inject path optionally amortizes a block of demonstrations into a single task-vector edit (lines 7–9) when context budget is tight. Algorithm 1 CircA: integration-circuit adaptation 1:VLM fθf_θ; integration/readout partition ℐ,ℛI,R (Eq. (4)); contamination-free synthetic task 0T_0 with remap verbalizer ρ; copy-rate threshold τ. 2:// One-time offline vaccine (Eq. (7)) 3:Build remap prompts PKP_K from 0T_0 (arbitrary labels ρ; query supervised by ρ(yq)ρ(y_q)) 4:Attach low-rank adapter Δℐ _I to the integration locus ℐI; freeze all of θ 5:Δ⋆←argminΔℐ[−logpθ⊕Δℐ(ρ(yq)∣PK)] ← _ _IE[- p_θ _I(ρ(y_q) P_K)] 6:return immunized model fvac=fθ⊕Δ⋆f^\,vac=f_θ ⊳ reused across all downstream tasks 7:// Per-deployment serving with demonstration block D=(xi,yi)D=\(x_i,y_i)\ 8:Estimate copy rate r^(K) r(K) on a small held-out probe (Eq. (8)) 9:K⋆←maxK:r^(K)≤τK ← \K: r(K)≤τ\ ⊳ gate: stop before saturation, Eq. (9) 10:if context budget is tight then 11: v←vℐ(D)v← v_I(D) task vector from fvacf^\,vac at locus ℐI (Eq. (10)) 12: answer query with v injected, spending no demonstration context ⊳ inject 13:else 14: answer query with the first K⋆K demonstrations in context 15:end if 4 Empirical Validation We now validate empirically every claim previewed in Sec. 3. We first describe the experimental setup, then report (a) the in-context collapse and where it lives, (b) its repair by CircA, and (c) deeper analyses and ablations. All code, the synthetic-concept generator, the intervention harness, and the result files behind every figure and table are publicly available at https://github.com/rostami-m/incontext-collapse. 4.1 Experimental setup 4.1.1 Models and access Our open-weight panel comprises 1111 VLMs spanning three connector/backbone families and a 0.50.5–1111B (22×22×) scale range: MLP-projector VLMs (Qwen2-VL-2B/7B [49], Qwen2.5-VL-3B/7B, LLaVA-1.5-7B [32], LLaVA-1.6-Mistral-7B, LLaVA-OneVision-0.5B/7B [28]), pixel-shuffle VLMs (SmolVLM-2B and SmolVLM-500M), and a cross-attention VLM (Llama-3.2-11B-Vision [14]), chosen to decorrelate scale from connector design. The lesion-and-rescue cross-architecture replication additionally uses InternVL3-2B [11]. Behavioural sweeps were run on self-hosted checkpoints in bfloat16 with each model’s native chat template applied explicitly, so that inference-time prompts match the format the model was aligned on. The white-box analyses, the lesion-and-rescue, its cross-architecture replication, and the continual-learning study, require access to hidden states and weights and were run on the open checkpoints on single 2424 GB GPUs, with 44-bit adapters where memory required and the quantization held fixed across the before/after comparison so it cannot affect a difference score. For frontier scale we probe the Amazon Nova (Lite, Pro) and Anthropic Claude (Haiku 4.5, Sonnet 4.5) inference APIs at default decoding. Q-Former models [29, 13] are single-image architectures that cannot represent interleaved multi-image prompts and are therefore out of scope for image–label ICL. 4.1.2 Tasks and the synthetic-concept generator The classification suite has three 44-way tasks chosen to vary the strength of pretraining priors: CIFAR-4 (strong natural-image priors), Fashion-4 (fine-grained), and a procedurally generated synthetic shapes task. Each shapes image contains one of four geometric primitives with randomized colour, size and position; the shape→ rule is assigned arbitrary tokens and cannot have been seen in pretraining, so above-chance accuracy can only be produced by learning the mapping in context. We note that synthetic data training is itself a source of degradation in iterative VLM training loops [18]; our synthetic task is used only for evaluation and adapter training, never for pretraining, so these failure modes are orthogonal. Demonstration pools and query sets are class-balanced and disjoint; chance accuracy is 0.250.25 throughout. We sweep the number of shots K∈0,1,2,4,8,16K∈\0,1,2,4,8,16\. Figure 4 shows representative stimuli from each task together with their standard and remap labels. Figure 4: The task suite and the two label spaces. One representative image per class for each of the three 44-way tasks: procedurally generated synthetic shapes (contamination-free), CIFAR-4 (strong natural-image priors), and Fashion-4 (fine-grained). Each image is annotated with its standard label (true class name, top) and its remap label (an arbitrary token such as “alpha” that cannot have appeared in pretraining, bottom). Above-chance remap accuracy can therefore come only from learning the in-prompt image→ rule, never from retrieval. 4.1.3 Conditions and the copy-rate diagnostic Each demonstration set is presented in three conditions. standard uses the true class names. shuffled permutes the demonstration labels: if shuffled≈standard shuffled\!≈\! standard the model is ignoring demonstration content and riding its priors. remap assigns arbitrary tokens to the classes, so only a model that learns the in-prompt mapping can exceed chance. As a mechanism probe we additionally record the copy rate: the fraction of predictions equal to the label of the most recent demonstration. A high copy rate together with shuffled≈standard shuffled\!≈\! standard identifies a model keying on demonstration format/recency rather than integrating demonstration content. Figure 5 renders a four-shot prompt under all three conditions, making concrete what the model is shown in each. Figure 5: An image–label ICL prompt under the three conditions. A K=4K=4 prompt on synthetic shapes: four (image, label) demonstrations followed by a held-out query (red). standard pairs each image with its true label; shuffled keeps the same images but permutes the labels (so a model riding its priors is unaffected, while a model integrating demonstration content is misled); remap replaces the labels with arbitrary tokens (so only genuine in-context learning can solve it). The model must answer the query (“?”) from the demonstrations alone. 4.1.4 Prompt construction and decoding For a task and shot count K, a prompt interleaves K class-balanced (image, label) demonstrations sampled without replacement with a single held-out query image, formatted with the model’s chat template; LLaVA-OneVision models use a multi-turn template, as their processor requires. The query is never among the demonstrations, and images are presented at 112×112112× 112. Generation is greedy and length-limited to the label span; outputs are mapped to the label set by constrained string matching, and an output matching no label is recorded as unparseable. We evaluate 4040 held-out queries per cell for the intervention study and 2525 for the panel, and report the unparseable fraction alongside accuracy so that a formatting failure is not mistaken for a reasoning collapse. A note on the shot range is in order. Each demonstration in a VLM prompt carries a full image, which a modern vision encoder expands into hundreds to thousands of tokens; a single K therefore costs far more context than the same K in a text-only prompt. At K=64K=64 the longest prompts already exceed 10510^5 tokens, SmolVLM cannot run that cell within its 1616k window at all, so our sweep up to K=64K=64 spans the same context-budget regime that “many-shot” denotes for text, even though the shot count is numerically smaller. The collapse, moreover, sets in well before this ceiling, typically by K=4K=4–88 (Fig. 8), so it is a property of the early accumulation of demonstrations rather than of an exotic long-context regime. 4.1.5 Defining the collapse and classifying regimes For each model–task pair we obtain an accuracy-versus-K curve under each condition. A pair is collapse-prone if standard accuracy at some K>0K>0 falls below the zero-shot value by a margin exceeding seed-to-seed variability, and collapse-resistant otherwise. Orthogonally, a pair is a learner if remap accuracy rises above chance with K, and retrieval-only otherwise. The two binary axes define the three observed regimes; the full grid is in Table 1. 4.1.6 Parameter-efficient lesion-and-rescue This is the experimental backbone of our causal claim. The intervention adds a small, region-restricted block of trainable capacity to an otherwise frozen model and asks whether that capacity, placed at a particular depth, restores in-context learning. “Lesion” refers to the fact that the unintervened model is functionally deficient at the integration step; “rescue” to adding capacity exactly where we hypothesize the deficit lives. Because the model is frozen everywhere else, any recovery is attributable to the adapter’s location. Adapter and optimization. The trainable block is a LoRA adapter [17]: for each targeted linear weight W∈ℝdout×dinW ^d_out× d_in we learn a low-rank update W↦W+αrBAW W+ αrBA with B∈ℝdout×rB ^d_out× r, A∈ℝr×dinA ^r× d_in, rank r=8r=8, scaling α=16α=16 (so α/r=2α/r=2), and LoRA dropout 0.050.05; A is Gaussian-initialized and B is zero-initialized, so the adapted model is identical to the base model at step 0 and the intervention can only add capability. We optimize the adapter with AdamW at learning rate 10−410^-4 for 150150 steps (batch size 11), with all base parameters θ frozen; gradient checkpointing and “use_cache==false” keep activation memory bounded so the whole study fits on a single 2424 GB GPU. Regions and the equal-capacity control. A region is a set of linear modules selected by depth. Following the partition of Eq. (4), we define five regions: the connector (the projection/resampler linear modules bridging the vision encoder to the language model), the early/mid/late thirds of the language-model blocks, and all blocks. Each language-model region is additionally split into attention-only (q,k,v,oq,k,v,o projections) and MLP-only (gate/up/down projections) variants, for the component cut of Sec. 4.1.7. The number of adapter modules per region is capped (at 6464 language-model modules, 88 connector modules) so that the compared regions carry matched parameter budgets: on Qwen2-VL-2B the early and late regions differ by under 10%10\% of modules (7070 vs. 6363). This equal-capacity control is essential, because a region could otherwise appear to rescue simply by having more parameters; with budget held fixed, a difference between rescuing at the integration locus and at the readout reflects location, not capacity. Targeting is verified at run time: after construction we enumerate the instantiated adapter modules and assert every one falls within the intended region, so a region label always denotes exactly the modules it names. What is trained, and why success is meaningful. Each task supplies a class-balanced training pool and a held-out evaluation set; we further split the evaluation items so the adapter is trained and measured on disjoint queries (no query seen in training is scored). At each step we sample a fresh K=8K=8-shot demonstration block and a query, format them with the same interleaved prompt used everywhere (Sec. 4.1.4), and apply the crucial supervision asymmetry: the in-context demonstrations carry the arbitrary remap labels ρ, while the loss supervises the query’s true class name, with the loss masked to the answer tokens only. The adapter therefore cannot succeed by memorizing any fixed label→ association, the remapped labels make that useless; it can only succeed by learning to read the demonstrations and apply their mapping to the query. After training we re-measure the full standard and remap K-curves by greedy generation and the same answer parser as the panel. Success, a remap curve that now rises with K where the frozen model’s was flat at chance, thus certifies that the adapter restored the ability to use demonstrations, the exact capability the collapse removes. The complementary induce arm inverts the supervision on a robust learner (demonstrations carry a conflicting shuffled mapping, supervision targets the prior) to test whether the collapse can be installed at the same locus. 4.1.7 K-specificity, component cut and cross-architecture replication K-specificity is read from the full rescue curve (Table 7): the integration–late gap is negligible at K≤2K≤2 and maximal at K=8,16K=8,16. The component cut restricts the within-region adapter to the attention projections or to the MLP projections only. Cross-architecture replication repeats the full region protocol on InternVL3-2B; because its integration capacity is higher, we use a larger adapter (rank 3232, 400400 steps), and we report it as a directional replication. 4.1.8 Frontier probe Frontier models are evaluated through the Converse API with interleaved image and text content blocks, on CIFAR-4 and synthetic shapes, sweeping K∈0,1,2,4,8K∈\0,1,2,4,8\ over 2424–4040 balanced queries. The remap learning curves are reported as the mean over three seeds (with 95%95\% CIs); the standard collapse curves are single-seed probes over the full balanced query set. 4.1.9 Continual learning protocol The continual-learning stream splits CIFAR-100 into five disjoint 44-class tasks presented sequentially. After each task we fit a LoRA restricted to one of three loci, the circuit (connector + early/mid), all layers, or late layers, and evaluate every task seen so far. Average accuracy (A) is the mean over tasks after the final task; backward transfer (BWT) is the mean change in each task’s accuracy from when it was learned to the end of the stream. We report mean ± s.d. over three seeds. 4.1.10 Statistics, compute and reproducibility Intervention and continual-learning results are reported as mean ± 95%95\% confidence interval (intervention) or ± s.d. (continual learning) over three seeds; panel sweeps use two seeds. No runs were excluded. All white-box cells complete in under an hour on a single GPU, so the full localization, component-cut, cross-architecture and continual-learning panels are inexpensive to reproduce. All prompt templates, the synthetic-concept generator, the intervention harness, the figure-generation script, and the per-seed result files are released. 4.2 The in-context collapse: prevalence, regimes, and where it lives We first establish that the collapse is real and graded, that it is dissociable from genuine in-context learning, that it reaches standard VQA benchmarks and frontier scale, and that it is causally localized to the vision–language integration pathway. 4.2.1 A graded in-context collapse across the panel We evaluate every model in the panel on all three classification tasks (synthetic shapes, CIFAR-4, Fashion-4) under both verbalizers, and summarize the complete grid in Table 1. The headline is that the collapse is real and graded: on standard classification many VLMs are strong zero-shot classifiers whose accuracy falls as demonstrations are added (Fig. 6a), but how far it falls varies sharply from model to model. We measure 1111 open VLMs spanning 0.50.5–1111B parameters and several connector designs (MLP projectors, pixel-shuffle, cross-attention, perceiver/resampler), and the effect ranges from negligible to catastrophic. The phenomenon is broad, not a quirk of one model or one task. On CIFAR-4 the collapse appears, to differing degrees, in eight of the eleven models. It is mild in some (Qwen2.5-VL-3B, 0.92→0.640.92\!→\!0.64 by K=16K=16), substantial in many (Qwen2-VL-2B 0.94→0.440.94\!→\!0.44; LLaVA-1.6-Mistral-7B 0.84→0.250.84\!→\!0.25; Llama-3.2-11B 0.84→0.240.84\!→\!0.24), and catastrophic in the smallest or least-aligned (LLaVA-OV-0.5B 0.72→0.040.72\!→\!0.04 by K=8K=8; LLaVA-1.5-7B and SmolVLM-500M fall to chance and then degenerate). A separate group is essentially flat (Qwen2-VL-7B 0.96→0.790.96\!→\!0.79; Qwen2.5-VL-7B 0.96→0.960.96\!→\!0.96; LLaVA-OV-7B holds the high 0.70.7s). The same pattern reproduces across all three tasks (Table 1): a given model’s regime is largely stable across shapes, CIFAR-4, and Fashion-4, with the contamination-free shapes task tending to give the highest zero-shot accuracy (no domain gap) and therefore the steepest visible collapse. Crucially, the collapse spans architecture families, it is present in MLP-projector (Qwen, LLaVA-1.5/1.6), cross-attention (Llama-3.2-11B), and high-token (LLaVA-OV) designs alike, so it is a property of how a VLM is aligned to follow many-shot prompts rather than of any single connector design. Susceptibility tracks the alignment recipe, not parameter count. Scale alone does not predict the regime. An 1111B cross-attention model (Llama-3.2-11B) and a 77B MLP-projector model (LLaVA-1.6-Mistral) both collapse substantially, while a 77B model from a different recipe (Qwen2-VL-7B) and even a 0.50.5–22B SmolVLM-2B do not; conversely within one model family the 22B Qwen collapses where the 77B does not. What separates the regimes is consistent: the collapse-resistant models are those whose post-training makes them treat demonstrations as evidence to be integrated, the collapse-prone models revert to a recency heuristic (quantified in Sec. 4.2.2). Takeaway: many-shot multimodal ICL is not uniformly safe; whether it helps or harms is set by the alignment recipe, and a practitioner cannot infer immunity from model size. Beyond prevalence, three properties of the curves sharpen what the collapse is and rule out innocent explanations; we illustrate them on the clean reasoning case (Qwen2-VL-2B) and verify they generalize across the prone models. Table 1: The full panel: robustness and learning by model and task (1111 open VLMs, mean over 33 seeds; SmolVLM-500M over 22). stdstd is standard accuracy (robustness), rmprmp is remap accuracy (learning; chance =0.25=0.25), each as K=0→K=maxK=0\!→\!K= , with max=16 =16 for Qwen2-VL-2B/7B, 88 for most others, and 44 for the single-image models LLaVA-1.5-7B and SmolVLM-500M (whose outputs degenerate beyond that point). Falling stdstd indicates the collapse; rising rmprmp indicates genuine learning. Models are grouped into the three regimes of Sec. 4.2.2. Regime Model Task stdstd (0→max0\!→\! ) rmprmp (0→max0\!→\! ) (i) prone, retrieval Qwen2-VL-2B shapes 1.00→0.441.00\!→\!0.44 0.24→0.360.24\!→\!0.36 cifar 0.94→0.440.94\!→\!0.44 0.28→0.350.28\!→\!0.35 fashion 0.88→0.350.88\!→\!0.35 0.34→0.210.34\!→\!0.21 Qwen2.5-VL-3B shapes 1.00→0.801.00\!→\!0.80 0.44→0.350.44\!→\!0.35 cifar 0.92→0.640.92\!→\!0.64 0.28→0.320.28\!→\!0.32 fashion 0.84→0.280.84\!→\!0.28 0.16→0.250.16\!→\!0.25 LLaVA-1.6-Mistral-7B shapes 1.00→0.401.00\!→\!0.40 0.32→0.230.32\!→\!0.23 cifar 0.84→0.250.84\!→\!0.25 0.16→0.160.16\!→\!0.16 fashion 0.96→0.210.96\!→\!0.21 0.36→0.370.36\!→\!0.37 Llama-3.2-11B shapes 1.00→0.451.00\!→\!0.45 0.08→0.240.08\!→\!0.24 cifar 0.84→0.240.84\!→\!0.24 0.28→0.160.28\!→\!0.16 fashion 0.96→0.410.96\!→\!0.41 0.28→0.240.28\!→\!0.24 LLaVA-OV-0.5B shapes 1.00→0.281.00\!→\!0.28 0.20→0.340.20\!→\!0.34 cifar 0.72→0.040.72\!→\!0.04 0.32→0.160.32\!→\!0.16 fashion 0.16→0.160.16\!→\!0.16 0.16→0.300.16\!→\!0.30 LLaVA-1.5-7B cifar 0.92→0.570.92\!→\!0.57 0.32→0.210.32\!→\!0.21 SmolVLM-500M cifar 0.60→0.400.60\!→\!0.40 0.28→0.240.28\!→\!0.24 (i) immune, retrieval SmolVLM-2B shapes 1.00→1.001.00\!→\!1.00 0.28→0.320.28\!→\!0.32 cifar 0.80→0.730.80\!→\!0.73 0.28→0.310.28\!→\!0.31 fashion 0.76→0.680.76\!→\!0.68 0.16→0.130.16\!→\!0.13 LLaVA-OV-7B cifar 1.00→0.711.00\!→\!0.71 0.40→0.210.40\!→\!0.21 (i) immune, learner Qwen2-VL-7B shapes 1.00→0.991.00\!→\!0.99 0.50→0.980.50\!→\!0.98 cifar 0.96→0.790.96\!→\!0.79 0.32→0.800.32\!→\!0.80 fashion 0.88→0.770.88\!→\!0.77 0.32→0.560.32\!→\!0.56 Qwen2.5-VL-7B shapes 1.00→1.001.00\!→\!1.00 0.48→1.000.48\!→\!1.00 cifar 0.96→0.960.96\!→\!0.96 0.24→0.610.24\!→\!0.61 fashion 0.92→0.970.92\!→\!0.97 0.16→0.830.16\!→\!0.83 Figure 6: The in-context collapse and the robustness–learning dissociation, across the panel. All eleven open VLMs in the panel, on CIFAR-4. (a) standard accuracy versus the number of demonstrations K. Collapse-prone models (warm) lose accuracy as demonstrations accumulate, from mild (Qwen2.5-VL-3B) to catastrophic (LLaVA/SmolVLM families fall to or below chance), while collapse-resistant models (cool) remain stable. The effect spans MLP-projector, cross-attention, and high-token architectures and does not track parameter count. (b) On remap (arbitrary labels; chance =0.25=0.25, dotted) only the well-aligned large models (Qwen2-VL-7B, Qwen2.5-VL-7B) rise with K, i.e. genuinely learn the in-prompt rule; the prone models stay at chance, retrieving priors rather than learning. Robustness (panel a) and learning (panel b) are thus separable axes (Table 1). Top row: the full shot range (to K=64K=64); the shaded band is magnified in the bottom row (0≤K≤160\!≤\!K\!≤\!16), where most of the dynamics, the first-shot drop and the slow rise of the learners, occur. The damage is done at the very first demonstration, in the clean-reasoning collapse. In the models whose collapse is a genuine reasoning failure (zero malformed outputs; Fig. 8), the collapse is not a slow erosion that accumulates with context length; it is almost a step change at K=1K=1. For Qwen2-VL-2B on CIFAR-4 the single largest move in the entire curve is the K=0→K=1K=0\!→\!K=1 drop of 0.510.51 (0.94→0.430.94\!→\!0.43), which already accounts for the overwhelming majority of the total 0.560.56 decline out to K=64K=64; no later change between adjacent shot counts exceeds 0.180.18, and the curve only drifts downward from there. This front-loading is the rule across the clean-reasoning models: the first demonstration alone delivers 8686–100%100\% of the entire standard-accuracy drop, averaged over the three tasks, for Qwen2-VL-2B, Qwen2.5-VL-3B, and LLaVA-1.6-Mistral-7B alike. Strikingly, the degeneration-mode models behave oppositely: SmolVLM-2B/500M, whose accuracy crash is coupled to a surge in malformed outputs, lose almost nothing on the first shot (≤2%≤ 2\% of their total drop) and instead erode gradually as their outputs progressively fall apart with context length. Takeaway: the clean-reasoning collapse is triggered by the presence of demonstrations, not by their accumulation past some context budget, a single in-context example is enough to knock a confident zero-shot classifier off its prior, and the timing itself separates the two failure modes: a copy-driven reasoning collapse strikes instantly, whereas output degeneration accrues with length. The first observation already argues against a long-context or token-budget explanation for the clean case, since the effect is saturated long before context length is a concern (we return to this with the K=64K=64 runs below). The performance curves are non-monotonic, and they do not recover. Several collapse-prone models trace a shallow V: Qwen2-VL-2B dips to 0.430.43 at K=1K=1, partially rebounds to 0.610.61 at K=2K=2, then settles in the low-0.40.4s; Qwen2.5-VL-3B and, at frontier scale, Claude Sonnet 4.5 (Fig. 7) show the same dip-and-partial-recovery. This matches the “V-shaped” many-shot curves reported on closed models [22], but with a crucial difference: in our collapse-prone models accuracy never climbs back to the zero-shot value, even at K=64K=64. Takeaway: the collapse is not a transient few-shot artifact that more demonstrations cure; the prior, once displaced, is not recovered by piling on examples. Figure 7: The collapse, and the robustness–learning dissociation, reach frontier scale. Four frontier models (inference APIs) on CIFAR-4. (a) standard accuracy versus K: Nova-Lite/Pro and Claude-Haiku-4.5 are robust (cool), while Claude Sonnet 4.5 (red) collapses from 0.880.88 to 0.120.12 as a handful of demonstrations are added, then only partially recovers. (b) On remap (arbitrary labels; chance =0.25=0.25, dotted) all four, including the collapse-prone Sonnet 4.5, rise with K, i.e. genuinely learn the in-prompt rule (to 0.500.50–0.890.89 at K=8K=8). Thus a frontier model can collapse on natural labels yet still learn an arbitrary mapping: the two axes dissociate even at frontier scale, exactly as in the open panel (Fig. 6). The collapse is recipe/model-specific, not a small-model artifact. The collapse does not wash out at large K. Pushing to K=64K=64 (where prompts exceed 10510^5 visual tokens) does not reverse the collapse: Qwen2-VL-2B sits at 0.380.38 and LLaVA-OV-0.5B at 0.000.00. Takeaway: this is not a small-context phenomenon that many-shot prompting would outgrow; it persists, and in the weakest models deepens, across two orders of magnitude in shot count. Two failure modes, and why this is not a formatting artifact. A natural worry is that the collapse is merely a degradation of output format, the model ceasing to emit a parseable label, rather than a degradation of the underlying decision. The data distinguish two modes (Fig. 8). In the first, the model continues to emit well-formed labels but chooses the wrong one: Qwen2-VL-2B’s malformed-output fraction stays at zero across all K even as its accuracy falls by half, so its collapse is a genuine reasoning failure that no relaxation of output parsing could explain. In the second, the below-chance cases (LLaVA-OneVision-0.5B, SmolVLM-500M) couple their accuracy crash to a surge in malformed outputs (to 9090–98%98\% by the collapse point), so part of their sub-chance accuracy is output degeneration under demonstration overload. We therefore anchor every mechanistic and causal claim in this paper on the clean reasoning collapse (Qwen2-VL-2B), and read the below-chance models as the extreme tail of the same phenomenon rather than as its centre. Because the collapse appears in-distribution, demonstrations and queries are drawn from the same task and label set, it cannot be attributed to the support-set/query mismatch that drives degradation elsewhere [20]. Figure 9 makes the clean reasoning collapse tangible at the level of individual queries: the model names each image correctly with no demonstrations, then mislabels it, usually as another in-task class, once a handful of correctly-labelled demonstrations are prepended. Figure 8: Two failure modes with distinct signatures; the collapse is not a formatting artifact. (left) standard accuracy on CIFAR-4 versus K (log-spaced) for three collapse-prone models. (middle) The fraction of outputs that are not a valid label. Qwen2-VL-2B (red) loses half its accuracy with a malformed-output fraction pinned at zero, a clean reasoning collapse. The below-chance models (LLaVA-OV-0.5B, SmolVLM-500M) instead couple their accuracy crash to a surge in malformed outputs, so their sub-chance scores partly reflect output degeneration. (right) The two modes also differ in timing: the share of the total standard-accuracy drop that lands on the very first demonstration (K=0→1K=0\!→\!1), averaged over the tasks each model collapses on. The clean-reasoning/copy models (red) front-load 8686–100%100\% of the damage onto the first shot, whereas the degeneration models (pink) lose almost nothing there and erode gradually as their outputs fall apart with context length. Mechanistic claims in this paper are anchored on the clean case. Figure 9: The collapse, query by query (Qwen2-VL-2B, standard). For eight held-out images per task, the model’s prediction with no demonstrations (K=0K=0) and after eight correctly labelled demonstrations (K=8K=8). With no demonstrations every image is classified correctly; adding demonstrations flips it to a different in-task class (e.g. airplane→ , cat→ , t-shirt→ , square→ ). The outputs remain well-formed labels throughout, the failure is one of judgement, not formatting. Green/red borders mark the K=8K=8 outcome. The collapse, and the dissociation, reach the frontier. The collapse is not an artifact of small open models (Fig. 7). Of four frontier models probed on CIFAR-4, three are robust, but Claude Sonnet 4.5 exhibits a sharp collapse on standard labels, accuracy 0.880.88 at K=0K=0 falling to 0.120.12 at K=4K=4 before a partial recovery, while remaining perfect on the contamination-free shapes task. Strikingly, the robustness–learning dissociation (Sec. 4.2.2) holds here too: on remap all four models, including the collapse-prone Sonnet 4.5, rise with K and learn the arbitrary mapping (0.500.50–0.890.89 at K=8K=8).333The instruction-tuned frontier models answer verbosely; for the remap probe we use a stricter single-word prompt so the (arbitrary) label is emitted rather than a description. The prompt leaves the already-compliant models’ scores essentially unchanged. That a frontier model collapses on natural labels yet still acquires a novel in-prompt rule shows the collapse is a failure of robustness to prior-conflicting demonstrations, not of in-context learning per se, and that susceptibility is set by the model and its recipe, not by scale alone. 4.2.2 Robustness and in-context learning are dissociable The collapse measures only robustness, whether accuracy survives added demonstrations, and says nothing about whether the model can learn from them. We probe learning with remap: arbitrary class tokens that cannot be retrieved from priors, so above-chance accuracy requires acquiring the in-prompt mapping. The two axes come apart (Fig. 6b), and the full panel populates all three resulting regimes (Table 1). A model can be collapse-resistant yet unable to learn (SmolVLM-2B, LLaVA-OV-7B: flat standard but remap pinned near chance); collapse-prone (Qwen2-VL-2B, LLaVA-1.6-Mistral-7B, Llama-3.2-11B, LLaVA-OV-0.5B: standard collapses, remap at chance); or both robust and a genuine learner (Qwen2-VL-7B, Qwen2.5-VL-7B: standard high and remap rising, Qwen2-VL-7B reaches 0.800.80 on CIFAR-4 and 0.980.98 on shapes at K=16K=16). These combinations define three reproducible regimes that are populated across connector families (MLP-projector, pixel-shuffle, cross-attention) and the full 0.50.5–1111B scale range, so regime membership tracks the alignment recipe more than raw capacity. Learning has the opposite temporal signature to the collapse. The two axes do not merely differ in their endpoints; they evolve in opposite ways as demonstrations accumulate, which is itself evidence that they are distinct computations. The collapse is front-loaded, almost all of it lands on the first demonstration (above). Genuine learning is the mirror image: it is absent at low K and builds slowly. The one clear learner, Qwen2-VL-7B, shows essentially no remap gain through K=2K=2 (0.32→0.310.32\!→\!0.31 on CIFAR-4) and then climbs steadily as evidence accrues, 0.570.57 at K=4K=4, 0.720.72 at K=8K=8, 0.800.80 at K=16K=16; on the contamination-free shapes task it reaches near-ceiling (0.50→0.990.50\!→\!0.99). Takeaway: a collapse-prone model breaks the instant a demonstration appears, whereas a learner needs several demonstrations before the in-prompt rule pays off, so the very demonstrations that destroy one model’s prior are what a learner is only beginning to exploit. The collapse-prone model cannot be coaxed into learning by adding shots. Crucially, the prone model’s failure on remap is not a slow start that more examples would fix: Qwen2-VL-2B’s shapes-remap accuracy is stuck near chance at every shot count (0.24→0.360.24\!→\!0.36 across K=0…16K=0… 16), never approaching the learner’s trajectory. Takeaway: no amount of in-context evidence lets a collapse-prone model acquire a novel image→ rule; the capability is absent, not merely slow, which is exactly what the causal intervention of Sec. 4.2.4 later restores. Where the accuracy goes: copying the most recent label. The shuffled control confirms the reading: for regime-(i)/(i) models shuffled≈standard shuffled\!≈\! standard with high copy rates, i.e. they ignore demonstration content; only regime-(i) models are content-sensitive. The mechanism is visible in a tight coupling between accuracy and the copy rate (Fig. 10). For Qwen2-VL-2B on CIFAR-4 the copy rate jumps from 0 at K=0K=0 to 0.830.83 at K=1K=1, exactly the shot count at which accuracy craters to 0.430.43; thereafter both partially relax in lock-step (copy rate ∼0.35 \!0.35–0.420.42, accuracy ∼0.44 \!0.44–0.610.61). In other words, the lost accuracy does not scatter into random errors, it is captured almost entirely by predictions that echo the label of the most recent demonstration. Takeaway: the collapse is a specific, legible failure, the model substitutes “answer like the last example I was shown” for “read the query”, and the recency-copy is strongest precisely at K=1K=1 where the collapse is steepest. Figure 11 shows this happening on a single prompt: an unmistakable query square is labelled “circle”, the label attached to the most recent demonstration. The dissociation is also a practical diagnostic, it tells a practitioner which models can be adapted by prompting alone, and it sharpens the mechanistic question: what, inside a collapse-prone model, prevents demonstrations from being integrated? Figure 10: Content versus format (Qwen2-VL-2B, CIFAR-4). (left) standard and shuffled accuracy track each other closely, so the model is largely insensitive to whether demonstration labels are correct, it rides priors rather than integrating content. (right) The copy rate (fraction of predictions equal to the most recent demonstration’s label) rises with K, the signature of recency-driven copying rather than rule learning. Figure 11: Anatomy of a failure (Qwen2-VL-2B, K=8K=8, standard). The complete eight (image, true-label) demonstrations and the held-out query (red), for one CIFAR-4 and one synthetic-shapes case. Both queries are classified correctly with no demonstrations; with the demonstrations in context the model mislabels them, and on shapes it copies the most recent demonstration’s label (“circle”) instead of reading the query square. The outputs are valid labels, so this is a reasoning failure, not a parsing one. 4.2.3 The collapse on open-ended VQA benchmarks Sec. 4.2.1 established the collapse across three classification tasks and eleven models; a remaining objection is that it is an artifact of the closed 44-way classification framing itself. This section is a targeted framing-robustness control, and for a control the right design is not a wide panel but a clean comparison that varies only the framing. We therefore use a within-family controlled pair: Qwen2-VL-2B and Qwen2-VL-7B share architecture, connector, tokenizer, and training recipe and differ only in scale, yet sit on opposite sides of the collapse on classification (Table 1). Re-running exactly this pair on recognized open-ended VQA benchmarks, repurposed as many-shot ICL tasks, asks whether the prone/robust split survives the switch to an open-ended format while architecture is held fixed, so any surviving difference cannot be charged to a connector or backbone confound. It survives (Table 2, Fig. 12): the same model ordering carries over. The collapse-prone Qwen2-VL-2B degrades on VQAv2 (0.70→0.620.70\!→\!0.62, ±0.02± 0.02 over 33 seeds) and TextVQA (0.15→0.110.15\!→\!0.11), whereas the robust Qwen2-VL-7B holds on both (0.72→0.740.72\!→\!0.74; 0.20→0.180.20\!→\!0.18). The effect is smaller in absolute terms here than on classification, for two reasons that are worth naming: VQA soft-accuracy is bounded well below 11 even zero-shot (there is less height to fall from), and open-ended answers give the recency-copy heuristic less purchase than a closed 44-way label set. Takeaway: the collapse is a property of many-shot multimodal ICL in general, not of our synthetic protocol, and which side of it a model falls on is the same on VQAv2 as on CIFAR, so the regime is a stable property of the model rather than of the task. The one apparent exception is instructive: on ScienceQA both Qwen models improve with shots (0.18→0.290.18\!→\!0.29; 0.21→0.380.21\!→\!0.38), because ScienceQA is a multiple-choice format whose demonstrations teach the answer format rather than fighting a perceptual prior, exactly the regime in which demonstrations help. The collapse appears when demonstrations must override a strong pretraining prior, and recedes when they instead supply a missing output convention. Table 2: The collapse on standard VQA benchmarks (VQA soft accuracy), K=0→K=16K=0\!→\!K=16 (mean over 33 seeds; 95%95\% CI half-width ≤0.04≤ 0.04 on every cell). A within-family controlled pair (same architecture, differing only in scale): the collapse-prone 22B model degrades on the perceptual benchmarks (VQAv2, TextVQA) while the robust 77B model holds, reproducing their classification ordering with architecture held fixed. Both rise on ScienceQA, a multiple-choice task where demonstrations supply an answer format rather than override a prior. All rows are means over 33 seeds. Model VQAv2 TextVQA ScienceQA Qwen2-VL-2B 0.70→0.620.70\!→\!0.62 0.15→0.110.15\!→\!0.11 0.18→0.290.18\!→\!0.29 Qwen2-VL-7B 0.72→0.740.72\!→\!0.74 0.20→0.180.20\!→\!0.18 0.21→0.390.21\!→\!0.39 Figure 12: The collapse on open-ended VQA benchmarks. VQA soft accuracy versus K (mean ±95%± 95\% CI over 33 seeds) for the collapse-prone Qwen2-VL-2B (red) and the robust Qwen2-VL-7B (green). On the perceptual benchmarks (VQAv2, TextVQA) the 22B model degrades while the 77B holds, the same ordering as on classification. On ScienceQA both rise: its multiple-choice demonstrations teach an answer format rather than fight a perceptual prior, exactly the regime in which demonstrations help. The failures are interpretable at the level of individual questions, and the same pattern holds for both collapse-prone models on all three benchmarks (Fig. 13): a model that answers an open-ended question correctly with no demonstrations gives a wrong, and sometimes degenerate, answer once eight are prepended. On Qwen2-VL-2B, reading a brand correctly as “Coca Cola” at K=0K=0 becomes “koko’s” at K=8K=8, and a fossil-age question answered “crocodile egg” becomes “feather”; on Qwen2.5-VL-3B, a book title read correctly as “Revoltez-vous!” becomes “Harry Potter”, and a temperature read “32∘32 C” becomes “10∘10 C”. As in the synthetic case, the outputs stay fluent; the failure is of judgement, not formatting. Figure 13: Open-ended VQA failures, query by query, for both collapse-prone models. One row per model (Qwen2-VL-2B, Qwen2.5-VL-3B); each row shows two real (image, question) pairs from each of VQAv2, TextVQA, and ScienceQA, with the model’s answer at K=0K=0 (correct, ✓ ) and after eight demonstrations (K=8K=8, wrong, ×); gold answer shown. On both models and across all three benchmarks the collapse turns correct, fluent answers into wrong ones (a correct fact replaced by a wrong fact, or coherent text into a garbled phrase), mirroring the classification gallery (Fig. 9) on open-ended tasks. What the collapse-prone models share. Taken together, the classification panel (Table 1), the VQA benchmarks (Table 2), and the frontier probe (Fig. 7) let us ask what distinguishes the models that collapse from those that do not, across a deliberately heterogeneous set of architectures. Three commonalities emerge. (i) It is the alignment recipe, not the connector design or scale. Collapse-prone models span MLP-projector (Qwen2-VL-2B, LLaVA-1.5/1.6), cross-attention (Llama-3.2-11B), and high-token (LLaVA-OV-0.5B) designs, and range from 0.50.5B to 1111B; collapse-resistant models likewise span families and scales. Within a single family the divide is sharp and consistent (the 22B Qwen collapses, the 77B does not), which points at the post-training that teaches a model how to treat in-context demonstrations rather than at any structural property. (i) The prone models share a recency-copy signature. Wherever the collapse appears, it is accompanied by the same behavioral mechanism, a rising copy rate and shuffled≈ insensitivity to demonstration content (Sec. 4.2.2), so the prone models fail in the same way: they substitute “answer like the most recent example” for “read the query.” (i) The collapse is prior-interference, not task difficulty. It is strongest exactly where demonstrations must override a confident pretraining prior (natural-image classification, perceptual VQA) and recedes or reverses where they instead supply a missing output convention (ScienceQA’s multiple-choice format, on which even prone models improve), and a frontier model that collapses on CIFAR is simultaneously perfect on the prior-free shapes task. Takeaway: the collapse-prone models are not united by what they are built from or how big they are, but by how they were aligned, and they all fail through the same recency-copy shortcut triggered by prior-conflicting demonstrations, which is precisely the behavior the causal analysis of Sec. 4.2.4 localizes and the repair of Sec. 4.3 removes. 4.2.4 The collapse is a causally localized integration failure We now show the collapse is causally localized to the vision–language integration pathway and is separable from the readout. We take a collapse-prone model (Qwen2-VL-2B), train a rank-88 adapter restricted to a single region, and re-measure the remap K-curve: success means a model that could not learn the arbitrary mapping now can (Fig. 14, Table 3). Table 3: The collapse is causally removable at the integration pathway but not the readout. remap accuracy at K=16K=16 (mean ±95%± 95\% CI over 33 seeds) for Qwen2-VL-2B after a rank-88 adapter on each region; chance =0.25=0.25. Adapter region Shapes (remap@16) CIFAR-4 (remap@16) none (baseline) 0.39±0.050.39± 0.05 0.34±0.060.34± 0.06 connector 0.91±0.060.91± 0.06 0.69±0.090.69± 0.09 early 0.96±0.040.96± 0.04 0.64±0.080.64± 0.08 mid 0.63±0.130.63± 0.13 0.59±0.060.59± 0.06 late 0.08±0.060.08± 0.06 0.23±0.070.23± 0.07 all 0.94±0.050.94± 0.05 0.65±0.090.65± 0.09 Figure 14: The collapse is causally removable at the integration interface, not the readout. In-context learning (remap accuracy at K=16K=16; mean ±95%± 95\% CI over 33 seeds; chance =0.25=0.25 dotted, baseline dashed) for Qwen2-VL-2B after a rank-88 adapter on each region. Connector/early/mid (green) and the all-layer adapter restore genuine learning; the equal-capacity late readout adapter (red) does not, and on the contamination-free shapes task drives learning below baseline. The connector, the smallest target, is the most consistent rescue site. Removable at the interface, not the readout. On the contamination-free shapes task, an adapter on the connector lifts learning from 0.390.39 to 0.910.91 and on early layers to 0.960.96, near ceiling, while restoring standard accuracy to ≈0.99≈\!0.99 (the collapse is removed). An equal-capacity adapter on the late layers does the opposite, driving remap to 0.080.08, below the unintervened baseline. CIFAR-4 shows the same ordering (Table 3). Because late targets a parameter count comparable to early/mid (the equal-capacity control of Sec. 4.1: 7070 vs. 6363 adapter modules), the failure of the late intervention is not a capacity artifact; it is a statement about where the collapse lives. Three features of Table 3 deserve emphasis. (i) A monotone front-to-back gradient. Rescue efficacy falls steadily as the adapter moves deeper: connector and early layers nearly saturate the task, mid is intermediate (0.630.63), and late not only fails but actively suppresses learning. The collapse is thus not localized to a single module but to a region, the vision–language interface and the early language layers that first integrate the demonstrations, with the readout playing no constructive role. (i) The smallest intervention is the most effective. The connector is by far the lowest-parameter target (a single projection block), yet it delivers the most consistent rescue across both tasks (shapes 0.910.91, CIFAR 0.690.69), edging out the much larger all-layer adapter. This is strong evidence that the collapse has a specific, compact locus: adding capacity exactly at the modality bottleneck buys more than adding capacity everywhere. Takeaway: you do not need to retrain the network to remove the collapse, you need to touch the right ∼1% \!1\% of it. (i) The late adapter makes learning worse than doing nothing. Driving remap from a 0.390.39 baseline down to 0.080.08, well below chance, is not a null result; it means that spending capacity on the readout while demonstrations are present actively reinforces the recency-copy heuristic rather than the in-prompt rule. Takeaway: the readout is not merely the wrong place to help, it is a place where “adaptation” actively entrenches the failure, which is why the locus of the fix is a substantive claim and not a tuning detail. Together these establish that the collapse is a deficit of the integration computation, surgically correctable at the interface, and it is this locus that Sec. 4.3.1 turns into a reusable repair. 4.3 Resolving the collapse with CircA 4.3.1 From mechanism to method: a transferable integration vaccine Sec. 3 introduced CircA and its three components. We evaluate them across three models, establishing first that the collapse is localized to the same integration pathway on every architecture, then that a one-time vaccine at that pathway repairs it on every architecture. The vaccine carries the load; the gate and inject components, examined afterward on the original model, delimit where the repair must, and must not, live. We select the three models on which the full causal pipeline (lesion-and-rescue plus vaccine) is run before seeing any repair result, so that a positive finding cannot be the product of model search. They are chosen to vary the two factors that could otherwise confound a structural claim: the connector design that bridges vision to language, and the language-model backbone. Qwen2-VL-2B and Qwen2.5-VL-3B are MLP-projector models but from different model generations; InternVL3-2B uses a pixel-shuffle connector and a different backbone entirely. Together they span the two dominant connector families and two backbone lineages at a fixed small scale, so a shared integration-locus mechanism cannot be attributed to a single projector type, tokenizer, or alignment recipe. All three are collapse-prone yet retain enough competence for a rescue to be meaningful (Table 1), which is the regime in which the question is well-posed; deeply degenerate models, already at chance before intervention, leave nothing for a small adapter to restore and are uninformative here. We report all three together throughout this section rather than promoting one and replicating later. The lesion-and-rescue localizes the collapse on every architecture. We run the same lesion-and-rescue on all three models on a common task (CIFAR-4 remap@K=16K=16, Fig. 15): on each, an adapter at the integration locus (connector / early / mid) restores in-context learning while the equal-capacity late readout adapter is the weakest region, the same asymmetry established in Sec. 4.2.4. The ordering is sharp on the two MLP-projector models, where the best integration locus roughly doubles the chance-level baseline (Qwen2-VL-2B: connector 0.690.69 vs. late 0.230.23, at baseline; Qwen2.5-VL-3B: mid 0.770.77 vs. late 0.590.59), and reproduces on the pixel-shuffle InternVL3-2B, where the late intervention sits at baseline (remap 0.190.19) while a mid-layer adapter lifts it to 0.730.73. We note two honest caveats: on Qwen2.5-VL-3B the late adapter, though still the weakest locus and well below the integration peak, does not collapse to baseline as it does on the other two; and InternVL3-2B is noisier across seeds, so we read its positive side as a directional rather than a clean point estimate. The integration-over-readout ordering nonetheless holds on all three, across two connector families and two backbone lineages. This converges with correlational interpretability: Kaduri et al. [23] find cross-modal information flow is dominated by the middle layers, with early and late layers contributing only marginally, the very layers our intervention identifies as the editable integration site; what that work locates by observation, the lesion-and-rescue confirms by manipulation. It also refines the finding that visual information becomes linearly readable only in late layers [35, 12]: the late layers read out the decision but do not integrate the demonstrations, so readability and manipulability dissociate. Figure 15: The lesion-and-rescue localizes the collapse across all three architectures. CIFAR-4 remap accuracy at K=16K=16 (mean ±95%± 95\% CI over 33 seeds) after a rank-restricted adapter on each region, for the two MLP-projector models (Qwen2-VL-2B, Qwen2.5-VL-3B) and the pixel-shuffle InternVL3-2B. On every model the integration loci (connector / early / mid, teal) clear the chance-level baseline (dashed) while the equal-capacity late readout (red) is the weakest region, at or below baseline on Qwen2-VL-2B and InternVL3-2B and below the integration peak on Qwen2.5-VL-3B. The integration-not-readout ordering is thus a property of collapse-prone VLMs rather than of one architecture; InternVL3-2B is noisier across seeds. A one-time vaccine transfers to unseen tasks. We then train a single rank-restricted adapter on the connector and early–mid layers using only the shapes remap task, freeze it, and evaluate remap accuracy with no further training on shapes and on two held-out task families (Figure 16, Table 4). The pattern is the same on all three models: the un-vaccinated model sits at chance across all K on every task (the collapse), while the vaccinated model climbs with K on the trained task and, crucially, on two families the adapter never saw. At K=16K=16 the vaccine lifts the trained shapes task from chance to 0.99/0.76/0.580.99/0.76/0.58 (Qwen2-VL-2B / Qwen2.5-VL-3B / InternVL3-2B) and the held-out families from chance to 0.71/0.68/0.360.71/0.68/0.36 on CIFAR-4 and 0.60/0.40/0.430.60/0.40/0.43 on Fashion-4. Because resistance generalizes to tasks outside the training distribution, the vaccine installs a general “use the in-context mapping” capability at the integration pathway rather than a task-specific lookup. Figure 16: A one-time integration vaccine confers transferable collapse-resistance across architectures. remap accuracy versus K for each model (rows) before (grey) and after (teal) a single integration-locus adapter trained only on shapes remap (mean over 33 seeds, 95%95\% CI bands; chance =0.25=0.25, dotted). On every architecture the vaccine climbs with K on the trained task (left) and lifts two unseen task families (middle, right) from chance, while the baseline stays pinned at chance throughout. The effect is largest on the MLP-projector models and more modest, though still positive on both held-out tasks, on the pixel-shuffle InternVL3-2B. Table 4: The integration vaccine, trained once on shapes remap, transfers to unseen task families across architectures. remap accuracy at K=16K=16 (mean over 33 seeds, n=40n=40 held-out queries per seed; chance =0.25=0.25); each cell is baseline → +vaccine. The localization and the transferable repair hold for an MLP-projector model, a later-generation MLP-projector model, and a pixel-shuffle model. Model Shapes† CIFAR-4 Fashion-4 (trained) (transfer) (transfer) Qwen2-VL-2B (MLP) 0.39→0.990.39\!→\!0.99 0.34→0.710.34\!→\!0.71 0.18→0.600.18\!→\!0.60 Qwen2.5-VL-3B (MLP) 0.29→0.760.29\!→\!0.76 0.38→0.680.38\!→\!0.68 0.29→0.400.29\!→\!0.40 InternVL3-2B (pix.-shuf.) 0.35→0.580.35\!→\!0.58 0.22→0.360.22\!→\!0.36 0.28→0.430.28\!→\!0.43 † each cell: baseline → +vaccine remap accuracy at K=16K=16. Reading the cross-model pattern. Three regularities emerge from comparing the models. First, the direction is invariant: on every architecture the vaccine lifts both the trained task and the two unseen families above chance, and the baseline never does, so the repair is a property of collapse-prone VLMs rather than of one model. Second, the magnitude tracks the connector family: the effect is largest on the two MLP-projector models (held-out gains of +0.37/+0.42+0.37/+0.42 and +0.30/+0.11+0.30/+0.11 on CIFAR/Fashion) and more modest on the pixel-shuffle InternVL3-2B (+0.14/+0.15+0.14/+0.15), mirroring the noisier lesion signal there and suggesting that a simple linear projector exposes a cleaner integration locus for a low-rank edit than a spatially reshuffled one. Third, the gain is monotone in K on all three models, concentrated at the larger demonstration counts where the collapse is deepest (Fig. 16), so the vaccine repairs exactly the many-shot regime it targets rather than shifting the whole curve uniformly. Together these say the integration locus is the same editable site on connectors and backbones that otherwise share no implementation, and that the headroom for a low-rank repair is set by how cleanly that site is exposed. The fix must live at the integration locus. Ablating the vaccine confirms the integration pathway is the causal site (Table 5). Re-targeting the same-capacity adapter to the late layers, the consolidation site, not only fails to transfer but collapses even the trained task to 0.000.00, mirroring the late-adapter failure in the diagnostic lesion (Table 3). Within the integration locus the components are partially redundant for the easier CIFAR transfer (every single-component drop still transfers, CIFAR ≥0.64≥ 0.64 at K=16K=16) but more complementary for the harder Fashion transfer, which erodes as the early or mid language layers are removed (0.60→0.52/0.530.60\!→\!0.52/0.53); the connector is the least load-bearing. The transferable competence is carried by the early–mid language layers, the same locus the lesion study identified, not by the connector alone and not by the readout. Table 5: Locus ablation of the vaccine. remap accuracy at K=16K=16 (Qwen2-VL-2B, mean ±95%± 95\% CI over 33 seeds, n=40n=40 per seed; chance =0.25=0.25) for a single adapter trained on shapes remap and evaluated on the trained task and two held-out families. The repair requires the integration locus: the equal-capacity late variant collapses, while integration-locus variants transfer. Vaccine locus Shapes (trained) CIFAR-4 (transfer) Fashion-4 (transfer) none (baseline) 0.39±0.070.39± 0.07 0.34±0.040.34± 0.04 0.18±0.040.18± 0.04 full (conn+early+mid) 0.99±0.010.99± 0.01 0.71±0.010.71± 0.01 0.60±0.120.60± 0.12 −-connector 1.00±0.001.00± 0.00 0.72±0.070.72± 0.07 0.58±0.090.58± 0.09 −-early 0.83±0.120.83± 0.12 0.64±0.040.64± 0.04 0.52±0.110.52± 0.11 −-mid 1.00±0.001.00± 0.00 0.70±0.140.70± 0.14 0.53±0.070.53± 0.07 late only 0.00±0.000.00± 0.00 0.31±0.050.31± 0.05 0.37±0.090.37± 0.09 Training-free injection does not fix the collapse, and that is itself informative. Task and function vectors transplant a computation a model already performs into a cheaper form [16, 46, 19]; our inject path is their multimodal analogue, extracting the task vector from the model’s own forward pass and adding it at the integration locus. On the collapse-prone model it leaves remap accuracy at chance at both the early and mid loci, identical to the zero-shot baseline. Far from a null result, this pins down the nature of the collapse: a training-free patch can only relocate a computation that exists, so its failure shows the collapse is the absence of the integration computation rather than its misplacement. The fix must therefore add capacity at the integration locus (the vaccine), consistent with the equal-capacity late adapter failing to rescue the collapse (Table 3). Injection remains useful for context amortization once a model can integrate (post-vaccine or collapse-resistant), which the gate exploits at serving time. The repair strategies, head to head. Table 6 places the three CircA components against the two naive baselines in a single transfer setting (train on shapes remap, evaluate remap on the unseen CIFAR-4 at K=16K=16). The ranking is unambiguous: only the vaccine clears the collapse (0.78±0.090.78± 0.09), more than doubling the next-best strategy. Simply adding more demonstrations (many-shot, 0.380.38) is the collapse itself; training-free injection (0.290.29) and the runtime gate (0.210.21) sit at or below the zero-shot prior (0.250.25), confirming that neither relocating an absent computation nor gating the context substitutes for adding integration capacity. The comparison is between repair strategies and naive controls on the same model, not against an external method; we are not aware of a published multimodal many-shot repair baseline to compare against, which is itself indicative of the gap this work addresses. Table 6: Repair strategies versus naive baselines, all on the collapse-prone Qwen2-VL-2B in one transfer setting: trained on shapes remap, evaluated on held-out CIFAR-4 remap at K=16K=16 (mean ±95%± 95\% CI over 33 seeds; chance =0.25=0.25). Only the integration vaccine clears the collapse. Strategy CIFAR-4 remap @K=16K=16 zero-shot (prior) 0.25±0.000.25± 0.00 many-shot (the collapse) 0.38±0.070.38± 0.07 inject (training-free task vector) 0.29±0.010.29± 0.01 gate (runtime context gating) 0.21±0.040.21± 0.04 vaccine (integration adapter) 0.78±0.090.78± 0.09 4.4 Analytic and Ablation Experiments Having established the collapse, its locus, and its repair across the three architectures, we now report finer-grained analyses on the original model that probe the mechanism and its boundaries: how the rescue depends on the demonstration count, how it splits across attention and MLP, and how the integration locus relates to where durable memories are best written. 4.4.1 K-specificity of the rescue The rescue is concentrated where the collapse is. At K≤2K≤2 the integration and late interventions are indistinguishable from baseline; the separation emerges at K=4K=4 and is maximal at K=8,16K=8,16. On shapes, connector remap rises 0.38→0.73→0.90→0.910.38\!→\!0.73\!→\!0.90\!→\!0.91 across K=2,4,8,16K=2,4,8,16, while the late adapter stays at the floor (0.31→0.05→0.07→0.080.31\!→\!0.05\!→\!0.07\!→\!0.08); the full curves are in Table 7. The effect tracks the demonstration-overload regime rather than a uniform change in competence. Table 7: Full lesion-and-rescue K-curves (remap accuracy, Qwen2-VL-2B, mean over 33 seeds) after a rank-88 adapter on each region; chance =0.25=0.25. Connector/early/mid/all rise with K; the equal-capacity late adapter stays at or below chance, and the integration–late gap opens only once demonstrations accumulate (K≥4K≥ 4). Task Region K=0K=0 K=1K=1 K=2K=2 K=4K=4 K=8K=8 K=16K=16 Shapes baseline 0.25 0.25 0.32 0.42 0.37 0.39 connector 0.18 0.28 0.38 0.73 0.90 0.91 early 0.12 0.31 0.46 0.78 0.91 0.96 mid 0.19 0.40 0.38 0.18 0.35 0.63 late 0.00 0.25 0.31 0.05 0.07 0.08 all 0.14 0.29 0.46 0.72 0.90 0.94 CIFAR-4 baseline 0.25 0.28 0.28 0.38 0.35 0.34 connector 0.25 0.23 0.22 0.52 0.66 0.69 early 0.23 0.38 0.25 0.49 0.53 0.64 mid 0.28 0.32 0.23 0.46 0.50 0.59 late 0.18 0.28 0.30 0.39 0.29 0.23 all 0.25 0.40 0.24 0.49 0.54 0.65 4.4.2 Attention and MLP both carry the rescue Splitting each region into attention-only and MLP-only adapters shows the effect is distributed, not isolated to a single sub-module. On shapes (remap@1616), an early-layer MLP-only adapter reaches 0.970.97 and attention-only 0.880.88; a mid-layer adapter favors attention (0.700.70 vs. 0.490.49); in the late layers neither rescues learning (attention 0.280.28, MLP 0.220.22; Fig. 17). CIFAR-4 shows the same ordering. Figure 17: Attention and MLP both carry the rescue. remap accuracy at K=16K=16 (Qwen2-VL-2B, shapes) for adapters restricted to attention-only, MLP-only, or both, within the early, mid and late regions. The integration effect is distributed across attention and MLP at the early/mid interface; neither sub-module rescues learning in the late readout (chance =0.25=0.25, dotted). 4.4.3 Integration is not consolidation Because in-context adaptation is weight-free, the collapse is a capacity bound on the fast path: past it, durable adaptation must be written into weights. Is the same integration circuit that fixes the collapse also the right place to consolidate? We test this on a 55-task class-incremental stream (CIFAR-100 splits) by restricting a sequential LoRA to three loci, the circuit (connector + early/mid), all layers, and late layers, and measuring average accuracy (A) and backward transfer (BWT). The answer is no, and informatively so (Fig. 18). Consolidation is most durable at the late layers: they reach the highest average accuracy (0.7170.717 vs. 0.6680.668 for the circuit), the least forgetting (−0.292-0.292 vs. −0.344-0.344 BWT), and do so with the fewest trainable parameters (3.03.0M vs. 6.46.4M). The locus of in-context integration is thus distinct from the locus of effective weight-based consolidation: the early/mid interface fixes the fast path, while the late readout is the efficient home for the slow path. This is a second dissociation, and a concrete design rule: ride ICL until the collapse, then consolidate at the readout, not at the integration circuit. Figure 18: Integration ≠ consolidation. Continual learning over a 55-task class-incremental stream (mean ± s.d. over 33 seeds), consolidating with a LoRA restricted to each locus. The late readout (red) consolidates best, highest average accuracy (left) and least forgetting (right), at the fewest parameters, not the early/mid integration circuit (green) that removes the collapse in Fig. 14. 5 Conclusion Many-shot multimodal in-context learning is not monotonically helpful: it exhibits a graded collapse that, surprisingly, reaches frontier scale. The collapse exposes a dissociation between robustness and genuine learning, and it is causally localized to the vision–language integration pathway, removable at the connector and early/mid layers but not at the late readout. That same readout, not the integration circuit, is where weight-based consolidation is most durable. The in-context collapse is therefore best understood as an integration failure at the modality boundary, mechanistically distinct from both the readout and the consolidation locus, and it provides a concrete diagnostic for when a vision-language model must stop prompting and start consolidating. The collapse is an in-context integration failure: demonstrations overload the vision–language interface and displace, rather than refine, the model’s prior. It is distinct from catastrophic forgetting, a weight-overwrite failure; the bridge between the two is architectural rather than an equivalence, in that the collapse bounds the weight-free path and thereby motivates the weight-based one. Practically, because the collapse is localized and the fix transfers, immunity can be installed offline: a single small adapter on the integration locus, an integration vaccine, confers collapse-resistance that generalizes to unseen task families without per-task training (§4.3.1), turning the diagnosis into a cheap, deployable intervention. Two boundaries of the present account deserve emphasis. First, the intervention is asymmetric: removing the collapse in a collapse-prone model is clean, but the reverse, inducing a collapse in a robust learner by lesioning the same circuit, is weak and diffuse in our experiments. This asymmetry suggests collapse-immunity is a distributed property not reducible to a single circuit, whereas collapse-proneness is a localized integration deficit that a small adapter can correct, and it means our causal claim is best supported in the prone-to-immune direction. Second, the repair was validated on three small open VLMs spanning two connector families and two backbone lineages; the frontier evidence (§4.2.2) establishes that the phenomenon scales, but whether the same low-rank integration edit repairs proprietary frontier models, which we can probe only through inference APIs (§4.2.2), remains open. Three directions strike us as both unexplored and high-leverage. (i) A collapse predictor from pretraining signatures. The integration locus is identifiable by intervention; if the same locus leaves a measurable fingerprint, for instance the rank or conditioning of the connector-to-early-layer Jacobian, one could predict a model’s collapse-proneness directly from its weights and pre-vaccinate at release time, without ever running the many-shot probe. (i) Closing the prompt–consolidate loop at serving time. We show when a model should stop prompting and start consolidating, but not yet how to do so online; an agent that monitors the copy-rate gate, triggers a lightweight integration update when the gate saturates, and folds the just-seen demonstrations into the vaccine would turn the static diagnostic into a continually adapting controller. (i) Native multimodal and interleaved demonstrations. Our probes place one image per demonstration in a single turn; extending the analysis to interleaved image–text sequences, video frames, and audio would test whether the integration locus is a property of the vision–language boundary specifically or a general signature of any modality fusion under many-shot pressure, the more ambitious structural claim. Data and code availability Panel configurations, the synthetic-concept generator, the intervention harness, and the result files behind every figure and table are released with the paper to enable full reproduction: https://github.com/rostami-m/incontext-collapse. References Agarwal et al [2024] Agarwal R, Singh A, Zhang LM, et al (2024) Many-shot in-context learning. Advances in Neural Information Processing Systems (NeurIPS) Alayrac et al [2022] Alayrac JB, Donahue J, Luc P, et al (2022) Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS 2022 arXiv:2204.14198 Bar et al [2022] Bar A, Gandelsman Y, Darrell T, et al (2022) Visual Prompting via Image Inpainting. NeurIPS 2022 arXiv:2209.00647 Brown et al [2020] Brown TB, Mann B, Ryder N, et al (2020) Language models are few-shot learners. In: Advances in Neural Information Processing Systems (NeurIPS) Buzzega et al [2020] Buzzega P, Boschini M, Porrello A, et al (2020) Dark Experience for General Continual Learning: a Strong, Simple Baseline. NeurIPS 2020 arXiv:2004.07211 Cai and Rostami [2025] Cai Y, Rostami M (2025) CluMo: Cluster-based modality fusion prompt for continual learning in visual question answering. Journal of Artificial Intelligence Research 83. 10.1613/jair.1.17940 Cai and Rostami [2026] Cai Y, Rostami M (2026) Dynamic transformer architecture for continual learning of multimodal tasks. Neurocomputing 10.1016/j.neucom.2025.129533, URL https://arxiv.org/abs/2401.15275 Cai et al [2023] Cai Y, Thomason J, Rostami M (2023) Task-attentive transformer architecture for continual learning of vision-and-language tasks using knowledge distillation. In: Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics, p 6986–7000, 10.18653/v1/2023.findings-emnlp.466 Chaudhry et al [2018] Chaudhry A, Ranzato M, Rohrbach M, et al (2018) Efficient Lifelong Learning with A-GEM. ICLR 2019 arXiv:1812.00420 Chen et al [2024] Chen C, Zhu J, Luo X, et al (2024) CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model. NeurIPS 2024 arXiv:2403.08350 Chen et al [2023] Chen Z, Wu J, Wang W, et al (2023) InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. CVPR 2024 arXiv:2312.14238 Cohen et al [2024] Cohen I, Gottesman D, Geva M, et al (2024) Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models. arXiv 2024 arXiv:2412.14133 Dai et al [2023] Dai W, Li J, Li D, et al (2023) InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. NeurIPS 2023 arXiv:2305.06500 Dubey et al [2024] Dubey A, et al (2024) The llama 3 herd of models. arXiv preprint arXiv:240721783 Golovanevsky et al [2024] Golovanevsky M, Rudman W, Palit V, et al (2024) What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation. NeurIPS 2024 arXiv:2406.16320 Hendel et al [2023] Hendel R, Geva M, Globerson A (2023) In-Context Learning Creates Task Vectors. EMNLP 2023 arXiv:2310.15916 Hu et al [2021] Hu EJ, Shen Y, Wallis P, et al (2021) LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022 arXiv:2106.09685 Hu et al [2025] Hu Z, Rostami M, Thomason J (2025) Multi-modal synthetic data training and model collapse: Insights from VLMs and diffusion models. arXiv preprint arXiv:250508803 URL https://arxiv.org/abs/2505.08803 Huang et al [2024a] Huang B, Mitra C, Arbelle A, et al (2024a) Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning. NeurIPS 2024 arXiv:2406.15334 Huang et al [2025] Huang C, Zhu Y, Zhu S, et al (2025) Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models. arXiv 2025 arXiv:2506.07936 Huang et al [2024b] Huang T, Hu S, Liu L (2024b) Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning. In: Advances in Neural Information Processing Systems (NeurIPS) Jiang et al [2024] Jiang Y, Irvin J, Wang JH, et al (2024) Many-Shot In-Context Learning in Multimodal Foundation Models. arXiv 2024 arXiv:2405.09798 Kaduri et al [2024] Kaduri O, Bagon S, Dekel T (2024) What’s in the Image? A Deep-Dive into the Vision of Vision Language Models. arXiv 2024 arXiv:2411.17491 Kirkpatrick et al [2016] Kirkpatrick J, Pascanu R, Rabinowitz N, et al (2016) Overcoming catastrophic forgetting in neural networks. PNAS 2017 arXiv:1612.00796 Kumaran et al [2016] Kumaran D, Hassabis D, McClelland JL (2016) What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated. Trends in Cognitive Sciences 10.1016/j.tics.2016.05.004 Lange et al [2019] Lange MD, Aljundi R, Masana M, et al (2019) A continual learning survey: Defying forgetting in classification tasks. TPAMI 2022 arXiv:1909.08383 Li et al [2023a] Li B, Zhang Y, Chen L, et al (2023a) Otter: A Multi-Modal Model with In-Context Instruction Tuning. arXiv 2023 arXiv:2305.03726 Li et al [2024] Li B, Zhang Y, Guo D, et al (2024) LLaVA-OneVision: Easy Visual Task Transfer. TMLR 2024 arXiv:2408.03326 Li et al [2023b] Li J, Li D, Savarese S, et al (2023b) BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. ICML 2023 arXiv:2301.12597 Li et al [2025] Li Y, Yang J, Yang Z, et al (2025) Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning. arXiv 2025 arXiv:2505.17097 Li and Hoiem [2016] Li Z, Hoiem D (2016) Learning without Forgetting. TPAMI 2018 arXiv:1606.09282 Liu et al [2023] Liu H, Li C, Wu Q, et al (2023) Visual Instruction Tuning. NeurIPS 2023 arXiv:2304.08485 Lopez-Paz and Ranzato [2017] Lopez-Paz D, Ranzato M (2017) Gradient Episodic Memory for Continual Learning. NeurIPS 2017 arXiv:1706.08840 McClelland et al [1995] McClelland JL, McNaughton BL, O’Reilly RC (1995) Why there are complementary learning systems in the hippocampus and neocortex. Psychological Review 10.1037/0033-295X.102.3.419 Neo et al [2024] Neo C, Ong L, Torr P, et al (2024) Towards Interpreting Visual Information Processing in Vision-Language Models. arXiv 2024 arXiv:2410.07149 Olsson et al [2022] Olsson C, others, Olah C (2022) In-Context Learning and Induction Heads. Transformer Circuits Thread O’Reilly et al [2014] O’Reilly RC, Bhattacharyya R, Howard MD, et al (2014) Complementary Learning Systems. Cognitive Science 10.1111/j.1551-6709.2011.01214.x Palit et al [2023] Palit V, Pandey R, Arora A, et al (2023) Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP. ICCV Workshop 2023 arXiv:2308.14179 Rebuffi et al [2016] Rebuffi SA, Kolesnikov A, Sperl G, et al (2016) iCaRL: Incremental Classifier and Representation Learning. CVPR 2017 arXiv:1611.07725 Rostami and Galstyan [2023] Rostami M, Galstyan A (2023) Cognitively inspired learning of incremental drifting concepts. In: Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI-23), p 3058–3066, 10.24963/ijcai.2023/341 Russin et al [2025] Russin J, Pavlick E, Frank MJ (2025) The dynamic interplay between in-context and in-weight learning in humans and neural networks. PNAS 10.1073/pnas.2502194122 Shin et al [2017] Shin H, Lee JK, Kim J, et al (2017) Continual Learning with Deep Generative Replay. NeurIPS 2017 arXiv:1705.08690 Srinivasan et al [2022] Srinivasan T, Chang TY, Pinto Alva L, et al (2022) CLiMB: A continual learning benchmark for vision-and-language tasks. In: Advances in Neural Information Processing Systems, p 29440–29453, URL https://arxiv.org/abs/2206.09059 Sun et al [2019] Sun FK, Ho CH, Lee HY (2019) LAMOL: LAnguage MOdeling for Lifelong Language Learning. ICLR 2020 arXiv:1909.03329 Tai et al [2023] Tai Y, Fan W, Zhang Z, et al (2023) Link-Context Learning for Multimodal LLMs. CVPR 2024 arXiv:2308.07891 Todd et al [2023] Todd E, Li ML, Sharma AS, et al (2023) Function Vectors in Large Language Models. ICLR 2024 arXiv:2310.15213 Tsimpoukelli et al [2021] Tsimpoukelli M, Menick J, Cabi S, et al (2021) Multimodal Few-Shot Learning with Frozen Language Models. NeurIPS 2021 arXiv:2106.13884 van de Ven et al [2022] van de Ven GM, Tuytelaars T, Tolias AS (2022) Three types of incremental learning. Nature Machine Intelligence 10.1038/s42256-022-00568-3 Wang et al [2024a] Wang P, Bai S, Tan S, et al (2024a) Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution. arXiv 2024 arXiv:2409.12191 Wang et al [2022] Wang X, Wang W, Cao Y, et al (2022) Images Speak in Images: A Generalist Painter for In-Context Visual Learning. CVPR 2023 arXiv:2212.02499 Wang et al [2023a] Wang X, Chen T, Ge Q, et al (2023a) Orthogonal Subspace Learning for Language Model Continual Learning. EMNLP 2023 arXiv:2310.14152 Wang et al [2023b] Wang X, Zhang X, Cao Y, et al (2023b) SegGPT: Segmenting Everything In Context. ICCV 2023 arXiv:2304.03284 Wang et al [2024b] Wang Z, Che C, Wang Q, et al (2024b) SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuning. arXiv 2024 arXiv:2411.13949 Wichers et al [2025] Wichers N, et al (2025) Inoculation prompting: Eliciting undesirable behavior to suppress it. arXiv preprint arXiv:251005024 Zeng et al [2024] Zeng F, Zhu F, Guo H, et al (2024) ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt. arXiv 2024 arXiv:2410.05849 Zenke et al [2017] Zenke F, Poole B, Ganguli S (2017) Continual Learning Through Synaptic Intelligence. ICML 2017 arXiv:1703.04200 Zhai et al [2023] Zhai Y, Tong S, Li X, et al (2023) Investigating the Catastrophic Forgetting in Multimodal Large Language Models. arXiv 2023 arXiv:2309.10313 Zhang et al [2023] Zhang Y, Zhou K, Liu Z (2023) What Makes Good Examples for Visual In-Context Learning? NeurIPS 2023 arXiv:2301.13670 Zhu et al [2024] Zhu D, Sun Z, Li Z, et al (2024) Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models. ICML 2024 arXiv:2402.12048 Zong et al [2025] Zong Y, Bohdal O, Yu T, et al (2025) VL-ICL bench: The devil in the details of multimodal in-context learning. In: International Conference on Learning Representations (ICLR)