Paper deep dive

FLIP: Final Layer Inference-Time Probing for Vision-Language Models

Drandreb Earl O. Juanico, Rowel O. Atienza

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 87

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 9/28/2026, 4:04:48 AM

Summary

The paper introduces FLIP (Final Layer Inference-Time Probing), a method for testing whether the final-layer hidden state of vision-language models (VLMs) supports structured, task-linked computation. FLIP applies elementwise flooring to the post-normalization hidden state before logit computation. The authors propose a four-criterion protocol to validate intervention effects: regime structure, grounding-proxy alignment, feature-coherence dependence, and negative-control contrast. Experiments on Qwen3-VL-4B and other models show that this specific intervention site yields a bounded interior regime where detection recall improves and counting error decreases, a signature not reproduced by raw decoder-layer interventions or negative controls, suggesting it is a valid target for mechanistic interpretability studies rather than a steering method.

Entities (7)

FLIP · method · 98%Qwen3-VL-4B · model · 95%Elementwise Flooring · operation · 93%MS COCO val2017 · dataset · 92%Counting Error (E_count) · metric · 90%Detection Recall (R50) · metric · 90%Mechanistic Interpretability · field · 88%

Relation Signals (6)

FLIP → uses → Elementwise Flooring

confidence 95% · FLIP applies elementwise flooring to the final normalized hidden state before logit computation

FLIP → targets → Qwen3-VL-4B

confidence 92% · main-text experiments use Qwen3-VL-4B

FLIP → evaluateson → MS COCO val2017

confidence 90% · Localization prompts are instantiated on a clustered subset of MS COCO val2017

FLIP → isusedfor → Mechanistic Interpretability

confidence 90% · FLIP is therefore a validation step for intervention-based mechanistic interpretability

FLIP → improves → Detection Recall (R50)

confidence 88% · detection recall at IoU 0.50 (R 50 ) improves

FLIP → reduces → Counting Error (E_count)

confidence 88% · tolerant counting error (E count ) falls

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We present FLIP, a final-layer inference-time probe for testing whether a logit-facing intervention site in an open-weight vision-language model (VLM) supports structured, task-linked computation rather than generic perturbation. Behavioral change under internal intervention is otherwise mechanistically ambiguous: it may reflect improved use of visual evidence, generic output instability, or outright degradation. FLIP applies elementwise flooring to the final normalized hidden state before logit computation, leaving parameters, prompts, and decoding unchanged. On a controlled detection/counting probe, sweeping intervention strength reveals three regions: negligible change, a bounded interior regime in which detection recall at IoU 0.50 ($R_{50}$) improves while tolerant counting error ($\mathcal{E}_{\mathrm{count}}$) falls, and over-suppression. We formalize a four-criterion probe-and-sweep protocol for disciplining the interpretation of intervention effects: regime structure, grounding-proxy alignment, feature-coherence dependence, and failure to reproduce the same positive regime on a performance-based negative control. The post-normalization state passed to the output head is the logit-facing instantiation of this test; under a non-targeted flooring sweep it satisfies the full protocol. Raw decoder-layer interventions, including the last-block output before final normalization, and the singleton-pair left/right control fail to reproduce the Final-site signature, while same-site operators and multiple VLMs replicate it. FLIP is therefore a validation step for intervention-based mechanistic interpretability, not a steering method.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

83,703 characters extracted from source content.

Expand or collapse full text

FLIP: Final Layer Inference-Time Probing for Vision–Language Models Drandreb Earl O. Juanico 1 Rowel O. Atienza 1 2 Abstract We present FLIP, a final-layer inference-time probe for testing whether a logit-facing interven- tion site in an open-weight vision–language model (VLM) supports structured, task-linked compu- tation rather than generic perturbation. Behav- ioral change under internal intervention is other- wise mechanistically ambiguous: it may reflect improved use of visual evidence, generic output instability, or outright degradation. FLIP applies elementwise flooring to the final normalized hid- den state before logit computation, leaving pa- rameters, prompts, and decoding unchanged. On a controlled detection/counting probe, sweeping intervention strength reveals three regions: negli- gible change, a bounded interior regime in which detection recall at IoU0.50(R 50 ) improves while tolerant counting error (E count ) falls, and over- suppression. We formalize a four-criterion probe- and-sweep protocol for disciplining the interpre- tation of intervention effects: regime structure, grounding-proxy alignment, feature-coherence dependence, and failure to reproduce the same positive regime on a performance-based negative control. The post-normalization state passed to the output head is the logit-facing instantiation of this test; under a non-targeted flooring sweep it satisfies the full protocol. Raw decoder-layer interventions—including the last-block output be- fore final normalization—and the singleton-pair left/right control fail to reproduce the Final-site signature, while same-site operators and multi- ple VLMs replicate it. FLIP is therefore a val- idation step for intervention-based mechanistic interpretability, not a steering method. 1 AI Graduate Program, University of the Philippines, Dili- man, Quezon City, Philippines 1101 2 E Institute, College of Engineering, University of the Philippines, Diliman, Quezon City, Philippines 1101. Correspondence to: Drandreb Earl O. Juanico <dojuanico@alum.up.edu.ph>. Mechanistic Interpretability Workshop at the43 rd International Conference on Machine Learning, Seoul, South Korea, 2026. Copyright 2026 by the author(s). 1. Introduction Open-weight vision-language models expose final-layer hid- den states at inference time, but behavioral change under intervention is mechanistically ambiguous: it may reflect im- proved use of visual evidence, generic instability, or outright degradation. Single-point interventions cannot resolve this ambiguity. We therefore evaluate intervention sites under sweeps over intervention strengthθand ask whether they pass a four-criterion probe-and-sweep protocol for struc- tured, task-linked computation rather than generic perturba- tion; a fuller definition of the latter is given in Appendix A. FLIP (Final Layer Inference-Time Probing) applies elemen- twise flooring to the post-normalization Final-site hidden state immediately before logit computation (Section 3). We use the standard linearity of the output head as a baseline for interpreting this site: a hidden-state displacement can be useful only insofar as it projects onto task-relevant output directions. The empirical question is therefore not whether FLIP can perturb outputs, but whether a non-targeted floor- ing sweep reveals a bounded, task-linked response. On a controlled detection/counting probe, we observe three re- sponse regions: negligible change, an interior regime with improved R 50 and reducedE count , and over-suppression. The aim is not to identify a circuit, but to validate whether an intervention effect is structured enough to justify circuit- level follow-up. FLIP therefore treats behavioral change as credible only if it survives independent constraints: regime structure, grounding-proxy alignment, feature-coherence de- pendence, and task-specific negative control. Raw decoder- layer checks (Figure 4; Appendix G, Figure 8) provide site-specific contrasts under the implemented sweep, while cross-architecture and same-site operator replications (Fig- ure 6; Appendices F, M.1) test the protocol-level signature beyond one model or operator. The delta-method prod- uct summary (Table 1) tests compatibility with a shared detection-to-counting grounding factor. Contributions. (1) We propose a probe-and-sweep pro- tocol that treats intervention effects as objects requiring validation, not as self-interpreting performance changes. (2) We instantiate it with FLIP, a minimal Final-site flooring probe grounded in the standard output-head baseline. (3) We show that primary-task gains are insufficient: accepted 1 arXiv:2609.30993v1 [cs.CV] 25 Sep 2026 FLIP for VLMs effects must also satisfy grounding-proxy alignment, feature- coherence dependence, negative-control contrast, and logit- facing site specificity. (4) We replicate this protocol-level signature across same-site magnitude-bounding operators and multiple vision-language models. 2. Background and Related Work Intervention-based interpretability trades off causal speci- ficity and population coverage. Causal abstraction uses interchange interventions to test aligned causal variables (Geiger et al., 2021); causal mediation asks which internal components are implicated in behavior (Vig et al., 2020); activation patching and automated circuit discovery localize task-relevant components after a metric and dataset have been chosen (Zhang and Nanda, 2024; Conmy et al., 2023); and circuit-tracing methods aim to recover prompt-level computational graphs (Ameisen et al., 2025). These meth- ods provide stronger mechanistic follow-up, but their con- clusions can depend on the chosen behavior, metric, exam- ples, and ablation design (Miller et al., 2024). Recent VLM work similarly analyzes visual-token processing, learns in- terpretable feature bases, or intervenes through latent and hidden-state steering to improve grounding or reduce hal- lucination (Neo et al., 2024; Pach et al., 2025; Yang et al., 2026; Liu et al., 2025; Su et al., 2025; Juanico et al., 2025; Li et al., 2025). FLIP targets the preceding selection problem: before assigning an effect to a circuit or using it for steering, it asks whether a candidate site exhibits a population-level, task-linked, control-sensitive signature that warrants mecha- nistic follow-up. 3. FLIP: Final Layer Inference-Time Probing FLIP is an inference-time intervention applied to the post- normalization final hidden-state tensor. Figure 1 shows the operator: components belowθare clamped elementwise be- fore logit computation, while all others are left unchanged. 3.1. Operational Definition Letz (t) ∈ R n t ×d denote the post-normalization Final-site tensor passed to the output head during generation callt, withn t equal to the prompt/image prefill length for the first next-token logits andn t = 1for cached decoding steps. For strength θ∈ R, FLIP outputs e z (t) = max(z (t) ,θ),(1) element-wise, the replacement is applied at every next-token logit computation during generation. Thus, FLIP covers the prefill logits used for the first response token and each subse- quent autoregressive decoding step, while leaving weights, prompts, tokenization, decoding rules, and KV caches un- changed. FLIP (definition used in all experiments). Given the post-normalization Final-site hidden statezand strengthθ, FLIP outputs e z = max(z,θ)element-wise before logit computation. No model weights, prompts, or decoding rules are changed. In the main experiments, FLIP targets the post- normalization final hidden state passed to the output head. We distinguish this Final site from raw decoder-layer out- puts, including the pre-normalization last-block output. FLIP is a probe, not an optimization method. 4.Tasks and Vision-Centric Evaluation Setting Unless otherwise stated, main-text experiments use Qwen3- VL-4B (Bai et al., 2025); additional model results are re- ported in the appendix. 4.1. Intermediate Visual Signals Prompted localization. We query the model with Detect <object>. Provide correct bounding boxes.and parse outputs into a common pixel-space representation;model-specific details are given in Appendix B. Dataset and supervision. Localization prompts are in- stantiated on a clustered subset of MS COCO val2017 (Lin et al., 2014) comprising 7,056 image–prompt pairs, con- structed image-centrically with one query per present object category. Image-level clustering is used because prompts drawn from the same image are not independent; anno- tations are used only to instantiate<object>and score outputs. 4.2. Reasoning-Based Counting Counting prompt.How many <object>? Answer with integer only. isolates errors attributable to vi- sual instance discrimination rather than formatting. 4.3. Vision-Centric Benchmarks We additionally evaluate MMStar, MindCube, and Natural- Bench via LMMS-Eval as intervention probes rather than optimization targets. 1 5. Theory This section gives the simple operator and output-head facts used to motivate the probe. These facts are not themselves 1 Third-party asset credits, licenses, and terms of use for Qwen3- VL, Kimi-VL-A3B, MS COCO val2017, and LMMS-Eval are documented in the supplemental repository. 2 FLIP for VLMs −10 −5 0 5 θ =−2 FLIP z =⟨0.2,1.2,−11,−3.4,0.1,−9,−0.75,−1,3,7,2.5⟩ ̃ z =⟨0.2,1.2, −2, −2,0.1, −2, −0.75,−1,3,7,2.5⟩ Figure 1. Hidden-state flooring view of FLIP. Components belowθare clamped elementwise at the post-normalization Final site before logit computation (here, indices 3, 4, and 6). FLIP uses e z = max(z,θ)as the intervention; attention and rollout maps are used only as downstream diagnostics. the main novelty: they provide the baseline against which the empirical protocol is interpreted. Since FLIP is not trained, task-conditioned, or optimized, any useful effect must come from alignment between the induced displace- ment and task-relevant directions already present in the logit-facing representation. 5.1.Componentwise Flooring and Perturbation Growth For x∈ R and threshold θ∈ R, define φ θ (x) = max(x,θ), δ θ (x) = (θ− x) + .(2) Applied elementwise to z∈ R n×d , e z = φ θ (z),∆ θ = e z− z.(3) Define the active set S θ =(i,k) : z ik < θ, A(θ) = |S θ | nd ,(4) and mean clamp magnitude M (θ) = 1 nd X i,k (θ− z ik ) + .(5) Proposition 5.1 (Monotone clamping). For every (i,k), ez ik = z ik + (θ− z ik ) + ,(6) and∥∆ θ ∥ p is non-decreasing in θ for all p∈ [1,∞]. Asθincreases, FLIP passes from near-inactivity, to a bounded intermediate region, to widespread clamping. This geometry defines the strength axis of the probe but does not by itself determine decoded task metrics. 5.2. Standard Output-Head Alignment Baseline Because FLIP is applied immediately before the output head, its first-order interpretation follows from the standard linear logit map. For a final-token hidden statez t with output matrix W , e ℓ t − ℓ t = W ⊤ ∆ θ,t ,∆ θ,t = (θ− z t ) + .(7) For a task-relevant token setY + and contrast setY − , define the log-sum-exp margin m(z t ) = log X y∈Y + exp(w ⊤ y z t )− log X y∈Y − exp(w ⊤ y z t ). (8) A first-order expansion gives m( e z t )− m(z t )≈⟨ ̄w Y + (z t )− ̄w Y − (z t ), ∆ θ,t ⟩,(9) where ̄w Y + and ̄w Y − are softmax-weighted output-head directions over the two token sets. This calculation is not a new representation theory; it de- fines the baseline for interpreting FLIP. Since the clamp is object-agnostic, any useful change must come from projec- tion onto task-relevant output directions. The protocol then tests whether that necessary condition is accompanied by grounding-proxy alignment, feature-coherence dependence, logit-facing site specificity, and negative-control contrast. 6. Grounding-Proxy Association Analysis This section tests whether FLIP effects are compatible with a grounding-linked pathway, without claiming full causal identification. Let1 FLIP ∈0, 1denote intervention. Detection quality is R 50 = TP 50 gtcount ,∆R 50 = R 50 −E[R 50 | 1 FLIP = 0]. (10) For counting, letybe the parsed model count andy ⋆ = gtcount the ground-truth count; the tolerant error is E count = max(0,|y− y ⋆ |− 1).(11) Localization and counting are elicited by separate prompts and scored from independent outputs, so∆R 50 is not a lit- eral within-pass mediator ofE count . We instead use it as an observable proxy for an unobserved intervention-sensitive grounding state: 1 FLIP → G θ , G θ ⇝∆R 50 ,E count . 3 FLIP for VLMs We estimate the product summary from ∆R 50 ∼ 1 FLIP , E count ∼ 1 FLIP + ∆R 50 . The first model is a Gaussian GLM with image-clustered SEs and supplies thea-term; the second is a Poisson GLM on strict counting responses and supplies theb-term. Confi- dence intervals are large-sample clustered intervals over image-level units, not intervals requiring normally dis- tributed individual errors. 7. Experiments FLIP is applied at inference time by replacing the post- normalization Final-site tensorzwith e z = max(z,θ)prior to logit computation. Implementation details are provided in Appendix E; code, prompts, sweep grids, run manifests, and analysis scripts are available in the anonymized repos- itory athttps://anonymous.4open.science/r/ flip-qwen3vl-EBAC/. 7.1. Experimental Setup Experiments use 7,056 clustered COCO image–query pairs, with localization and counting queries issued per sample un- der matched intervention strengths. Because multiple object queries may be drawn from the same image, statistical un- certainty is computed at the image-cluster level rather than by treating image–query pairs as independent. Reported 95% confidence intervals therefore rely on the large-sample approximation for the image-clustered estimator, not on nor- mality of individual errors or residuals. We use Qwen3-VL- 4B-Instruct as the anchor model; cross-architecture results are in Appendix F. Unless otherwise noted, the core inter- vention sweeps use a fixed, pre-specifiedθgrid and fixed random seeds for replicability; repeated-run dispersion is reported only where runtime-level nondeterminism cannot be controlled in the serving stack. We treat pointwise tests acrossθas descriptive screening rather than as a family of confirmatory discoveries: no claim depends on declaring any single scannedθsignificant. Regime identification is based on the full sweep and the FWHM window around the primary∆R 50 peak, then checked against grounding-proxy alignment, feature-coherence controls, negative controls, raw-layer site specificity, and image-cluster subsampling. Hardware details for the core sweeps are reported in Ap- pendix E. Evaluation Protocol (Behavioral Readout). All inter- ventions are evaluated through final model outputs. For raw decoder-layer checks, we applyh (ℓ) → ̃ h (ℓ) (θ)at the selected layer and run the remaining computation to logits and decoded predictions. For the Final site, we intervene on the post-normalization state passed to the output head. Thus, all layer checks measure downstream behavior, not Figure 2. Attribution redistribution under FLIP for a single image–prompt pair. Top: baseline target-conditioned patch rel- evance. Bottom: relevance after intervention. The figure qualita- tively illustrates the bounded redistribution associated with the inte- rior regime; the saliency construction is described in Appendix H.1, and the aggregate RMR diagnostics are reported in Appendix H. Representative example only. intermediate-state change alone. 7.2. Metrics Primary quantities.We useR 50 /∆R 50 as the detection- side mediator andE count as the downstream outcome. Attribution redistribution (RMR).Relevance mass real- location (RMR) is an appendix-side redistribution diagnos- tic rather than a causal measure. We trackRMR + ,RMR − , andRMR obj ; construction and interpretation are given in Appendices H.1 and H.3. 7.3. Results Primary regime evidence.Figure 3 shows a three-regime response in∆R 50 : negligible change, a bounded interior gain, and collapse. The interior regime is not declared from the single largest point in the sweep, and we do not in- terpret unadjusted pointwisep-values overθas separate discoveries. Instead, the peak defines a descriptive FWHM window that must survive the full protocol: grounding-proxy alignment, feature-coherence disruption, negative-control contrast, logit-facing site specificity, and subsample stabil- ity. Appendix N (Figure 13) shows that the same interior window remains stable under image-cluster subsampling. Grounding-proxy alignment.The interior regime aligns with downstream behavior: intermediate flooring increases ∆R 50 while reducingE count , and over-suppression reverses the effect. Table 1 shows a non-zero product summary in the sameθwindow where recall peaks. Because localization and counting are separate forward passes, this supports a shared grounding-proxy interpretation, not a claim that one 4 FLIP for VLMs Table 1. Delta-method grounding-proxy product summary on Qwen3-VL-4B.a: effect of FLIP on∆R 50 ;b: Poisson-GLM association of∆R 50 withlog E[E count ]on strict counting responses;a×b: product summary; andIRR indirect = exp(a×b). Values below1indicate that improved recall is associated with reduced tolerant counting error. The quantity is interpreted as grounding-proxy compatibility, not direct within-pass causal mediation. θaSE(a) bSE(b) a×bSE(a×b) p-valueIRR indirect [95% CI] −50.00.002080.00054 −2.0700.158 −0.004300.001170.00020.9957 [0.9934, 0.9980] −5.00.026770.00214 −2.0920.152 −0.056000.00606 < 0.00010.9455 [0.9344, 0.9568] −4.00.029620.00219 −2.0870.151 −0.061800.00638 < 0.00010.9401 [0.9284, 0.9519] −2.00.038780.00242 −2.0790.148 −0.080610.00765 < 0.00010.9226 [0.9088, 0.9365] −1.00.048320.00277 −2.0490.146 −0.099030.00905 < 0.00010.9057 [0.8898, 0.9219] −0.50.052830.00294 −2.0150.136 −0.106470.00932 < 0.00010.8990 [0.8827, 0.9156] −0.30.057770.00299 −2.0320.137 −0.117390.01001 < 0.00010.8892 [0.8720, 0.9069] −0.20.057820.00304 −2.0270.137 −0.117230.01002 < 0.00010.8894 [0.8721, 0.9070] 0.00.059000.00321 −2.0740.133 −0.122360.01028 < 0.00010.8848 [0.8672, 0.9028] 0.50.026380.00355 −2.1510.138 −0.056740.00847 < 0.00010.9448 [0.9293, 0.9607] 1.0 −0.5314800.00590 −2.0690.1571.099740.08454 < 0.00013.0034 [2.5447, 3.5447] output directly causes the other. Logit-facing site specificity.Figure 4 shows that hidden- state flooring alone does not induce the Final-site signa- ture. Raw decoder-layer interventions remain near base- line, collapse, or show only partial movement, while the post-normalization Final site exhibits the full interior regime. To test whether this contrast is a raw-threshold artifact, we also overlay percentile-matched sweeps for Layers 8, 16, and 32. These matched raw-layer sweeps do not recover the Final-site dose–response and instead attenuate or worsen the response. Thus, the raw-layer fail- ures are not explained by applying too weak or poorly scaled a threshold. The full protocol—primary-task regime, left/right contrast, grounding-proxy alignment, and feature- coherence dependence—localizes to the post-normalization logit-facing state rather than to raw-layer depth or marginal intervention strength alone. Specificity under controls.At the Final site, permutation controls (Figure 5) show that the positive interior regime is not preserved once feature coherence is disrupted. All∆R 50 and mediator quantities in this control are centered against the same unpermuted no-intervention baseline(λ = 0,θ = −∞) , not against aλ-specific permuted baseline. Thus any baseline damage from permutation is retained rather than subtracted away. Relative toλ = 0, nonzeroλsettings lose the positive window and collapse earlier, though their ordering is not strictly monotonic; the control is therefore interpreted as a feature-coherence stress test rather than as an isolated flooring effect on an already permuted model. Operator comparison. Flooring, symmetric clipping, and additive shift each produce regime-shaped primary- task curves, so non-monotonicity or peak∆R 50 alone is not specific evidence. The full operator replication (Ap- pendix M.1, Figure 11) therefore evaluates the joint sig- nature: grounding-proxy alignment, feature-coherence de- pendence, logit-facing site specificity, and failure on the left/right performance control. Cross-architecture replication and broader spatial con- trol.Figure 6 shows the same qualitative three-phase struc- ture in Qwen3-VL-8B, Qwen3-VL-4B, and Kimi-VL-A3B, with Qwen3-VL-8B the lower-amplitude case. The bottom- panel singleton-pair left/right control does not reproduce the coherent positive interior regime across models; instead, it stays near baseline or drifts downward before late collapse; task details are given in Appendix J.1. Appendix F adds the auxiliary indoor/outdoor response-switch diagnostic and mediation replications (Tables 2 and 3). Appendix-side corroboration.Appendix H.3 shows that, for the lower-amplitude Qwen3-VL-8B case,RMR + tracks the interior regime whileRMR − andRMR obj mark the transition to over-suppression. Appendix I further shows structured sweeps on MMStar, MindCube, and Natural- Bench. These analyses broaden the regime picture but re- main secondary to the mediator-aligned protocol evidence. 8. Discussion FLIP occupies the screening stage between broad behav- ioral probing and circuit-level identification. Primary-task improvement is not enough, and neither is proximity to the last decoder block. The relevant evidence is the joint signa- ture: bounded grounding-proxy-aligned response, Final-site feature-coherence dependence, failure on a performance- based negative control, and specificity to the normalized logit-facing readout. The raw-layer and percentile-matched checks separate depth, dose, and validation: matching marginal intervention strength at Layers 8, 16, and 32 does not recover the Final-site response. Thus, FLIP does not claim that a circuit has been found; it nominates the Final- site interior regime as the most justified target for patching, circuit tracing, or causal-abstraction tests. The controls define the scope of this interpretation. Table 1 supports a shared grounding-proxy account in the interior 5 FLIP for VLMs −50−45−40−35−30−25−20−15−10−50 0 2 4 6 ·10 −2 θ ∆ R 50 −50−45−40−35−30−25−20−15−10−50 −5 0 5 ·10 −2 θ ∆Acc (left/right) Figure 3. Primary dose–response and negative control. Top: FLIP on Qwen3-VL-4B-Instruct yields the expected three-region detection response in∆R 50 : near-baseline behavior, bounded interior gain, and collapse under strong flooring. Bottom: the singleton-pair left/right control, measured by baseline-centered∆Acc, does not reproduce this positive interior regime; it remains near baseline over most non-collapse strengths and degrades under stronger intervention. The shaded band marks the FWHM window from the primary detection peak; bands denote 95% CI where available. −10−5−2−101 −8· 10 −2 0 8· 10 −2 θ ∆ R 50 Final site (post-norm) Layer 35 raw (pre-norm) Layer 34 Layer 33 Layer 32 Layer 32 (percentile-matched) Layer 24 Layer 16 Layer 16 (percentile-matched) Layer 8 Layer 8 (percentile-matched) Figure 4. Raw decoder-layer and percentile-matched interventions versus the post-normalization Final site. FLIP is applied to selected raw decoder-layer outputs and compared with the Final site passed to the output head. The Final site exhibits the bounded interior regime, while raw-layer interventions remain near baseline, collapse, or show only partial movement. Percentile-matched sweeps for Layers 8, 16, and 32 do not recover the Final-site dose–response and instead attenuate or worsen the response, ruling out a simple raw-threshold calibration explanation. Bands denote 95% CI where available. 6 FLIP for VLMs −0.1 0 0.1 −10−50 −0.6 −0.4 −0.2 θ ∆ R 50 λ = 0.00λ = 0.10 λ = 0.20λ = 0.25 λ = 0.33 Figure 5. Graduated permutation control via mixing parameterλ. Both panels show∆R 50 under feature-coherence mixing, with all curves centered against the same unpermuted no-intervention baseline(λ = 0,θ = −∞)rather than againstλ-specific permuted baselines. The upper panel zooms the near-zero/interior-response range, while the lower panel zooms the collapse regime. Relative to the unpermuted case(λ = 0), all permuted conditions lose the positive regime earlier and enter collapse sooner, although the ordering among nonzero λ values is not strictly monotonic. This indicates that feature coherence is necessary for the stability of the interior regime, and that the observed loss is not an artifact of subtracting an already degraded permuted baseline. −10−9−8−7−6−5−4−3−2−101 0 2 4 6 ·10 −2 Qwen3-VL-4B peak ≈ 0.059 Kimi-VL-A3B peak ≈ 0.042 Qwen3-VL-8B≈ 0.008 θ ∆ R 50 (rel. to baseline) −10−8−6−4−2 4 6 8 ·10 −3 −10−9−8−7−6−5−4−3−2−101 −5 0 5 ·10 −2 θ ∆Acc (left/right) Qwen3-VL-8BQwen3-VL-4BKimi-VL-A3B Figure 6. Multi-model dose–response and broader spatial control. Top:∆R 50 (relative to baseline (θ =−∞)) for Qwen3-VL-8B, Qwen3-VL-4B, and Kimi-VL-A3B across intervention strengths. Qwen3-VL-4B and Kimi-VL-A3B exhibit substantially larger interior peaks than Qwen3-VL-8B. Bottom: baseline-centered accuracy change on the singleton-pair left/right spatial control for the same models. Unlike the main detection/counting probe, this broader spatial control does not exhibit a reproducible bounded interior optimum across models; responses remain near baseline over much of the sweep and then diverge into model-dependent mild drift or late collapse. 7 FLIP for VLMs regime and reversal under over-suppression, while Figure 6 shows cross-model persistence despite effect-size variation. The left/right task rules out a broad performance-gain ac- count, and the indoor/outdoor diagnostic is retained only as a baseline-relative instability check. FLIP is therefore a diagnostic triage tool, not a deployment-safety certificate, optimization method, circuit identification result, or proof of model understanding. Limitations. FLIP validates intervention effects; it does not identify the circuit producing them or close the gap from population-level triage to causal mechanism. The grounding-proxy analysis is scoped to detection/counting, where∆R 50 is observable but not a literal within-pass me- diator. Within this scope, we test feature-coherence disrup- tion, performance-based negative controls, same-site opera- tor replication, image-cluster subsampling, multiple VLMs, and raw-layer site specificity. The raw-layer checks are con- ducted under the implemented threshold sweep and do not replace percentile- or norm-matched cross-site interventions. The raw-layer result is therefore best read as a readout-space contrast under this sweep, not as a claim that task-relevant information is absent from earlier layers. Bridging FLIP- positive sites to circuits through matched sweeps, patching, causal abstraction, or circuit tracing remains future work. 9. Conclusion FLIP reframes final layer inference-time intervention as population-level effect validation for mechanistic inter- pretability. Rather than treating a primary-task gain as self- interpreting, the protocol requires grounding-proxy align- ment, feature-coherence dependence, performance-based negative-control contrast, and specificity to the logit-facing readout. Raw-layer and percentile-matched checks further rule out a simple threshold-calibration explanation. This sep- arates structured, task-linked sensitivity from generic pertur- bation and provides a lightweight screen for deciding when downstream patching, circuit tracing, or causal-abstraction analysis is warranted. Acknowledgement DE Juanico acknowledges the funding and technical support from the ERDT and the UP Diliman Artificial Intelligence Graduate Program. References Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URLhttps://transformer-circuits.pub/ 2025/attribution-graphs/methods.html. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025. Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri ` a Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretabil- ity. Advances in Neural Information Processing Systems, 36:16318–16352, 2023. Atticus Geiger, Hanson Lu, Thomas Icard, and Christo- pher Potts. Causal abstractions of neural networks. Ad- vances in Neural Information Processing Systems, 34: 9574–9586, 2021. Drandreb Earl Juanico, Rowel Atienza, and Jeffrey Kenneth Go. Interpretable open-vocabulary referring object detec- tion with reverse contrast attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 723–732, 2025. Zhuowei Li, Haizhou Shi, Yunhe Gao, Di Liu, Zhenting Wang, Yuxiao Chen, Ting Liu, Long Zhao, Hao Wang, and Dimitris N Metaxas. The hidden life of tokens: Re- ducing hallucination of large vision-language models via visual information steering. In Forty-Second Interna- tional Conference on Machine Learning, 2025. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ́ ar, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, pages 740–755. Springer, 2014. Sheng Liu, Haotian Ye, and James Zou.Reduc- ing hallucinations in large vision-language models via latent space steering.In The Thirteenth In- ternational Conference on Learning Representations, 2025. URLhttps://openreview.net/forum? id=LBl7Hez0fF. ICLR 2025 Spotlight. Joseph Miller, Bilal Chughtai, and William Saunders. Trans- former circuit evaluation metrics are not robust. In First Conference on Language Modeling, 2024. Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing in vision-language models. In 8 FLIP for VLMs The Thirteenth International Conference on Learning Representations, 2024. Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, and Zeynep Akata. Sparse autoencoders learn monosemantic features in vision-language models. In The Thirty-Ninth Annual Conference on Neural Infor- mation Processing Systems, 2025. Jingran Su, Jingfan Chen, Hongxin Li, Yuntao Chen, Li Qing, and Zhaoxiang Zhang. Activation steering de- coding: Mitigating hallucination in large vision-language models through bidirectional hidden state intervention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12964–12974, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.634.URLhttps: //aclanthology.org/2025.acl-long.634/. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. In- vestigating gender bias in language models using causal mediation analysis. Advances in Neural Information Pro- cessing Systems, 33:12388–12401, 2020. Jingcheng Yang, Tianhu Xiong, Shengyi Qian, Klara Nahrst- edt, and Mingyuan Wu. Circuit tracing in vision-language models: Understanding the internal mechanisms of multi- modal thinking. arXiv preprint arXiv:2602.20330, 2026. Fred Zhang and Neel Nanda. Towards best practices of acti- vation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Rep- resentations, 2024. 9 FLIP for VLMs A.Generic Perturbation: Definition and Scope We use the term generic perturbation to denote behavioral changes induced by non-specific disruption of the represen- tation, rather than alignment with task-relevant intermediate computations. Formally, leth(z)denote a task-relevant readout and e z = T (z) an intervention. We say that the induced change ∆h = h( e z) − h(z)reflects generic perturbation if it is not systematically mediated by a task-relevant intermediate quantity, such as localization recall in the detection/counting setting studied here. Under this definition, generic perturbation includes: (i) uni- form magnitude suppression, (i) random feature misalign- ment, and (i) global shifts that do not preserve feature semantics. The distinction is operationalized in the main text through grounding-proxy alignment, feature-coherence controls, and negative-control tasks. B. Detection Matching and Mediator Construction Given ground-truth boxesg i n g i=1 and predictionsd j n d j=1 : IoU matrixLwithL ij = IoU(g i ,d j ); class-consistency matrixN ∈ −1, +1; confidence-weighted overlap C ij = s j L ij . In our implementation,s j is the normal- ized predicted-box area, used only to break ties in matching. A one-to-one assignment under IoU threshold0.50produces per-sample true-positive countsTP 50 , from which recall R 50 is computed as in Section 6. Mean IoU is used only for matching eligibility and not as the primary mediator in the main analysis. C. Output-Head Alignment: Scope and Interpretation The output-head alignment baseline used in the main text is a standard pre-decoding calculation, not a new representation theory. Its purpose is to make explicit why useful behavior under FLIP is non-trivial: the flooring operator is not task- conditioned, so any beneficial effect must arise from how the induced displacement projects onto task-relevant output- head directions. For a final-token hidden statez t and output matrixW, FLIP induces e ℓ t − ℓ t = W ⊤ ∆ θ,t . Thus the sign and size of any useful pre-decoding change de- pend on how the flooring displacement projects onto output- head directions. For a log-sum-exp margin between task- relevant tokens Y + and contrast tokens Y − , m(z t ) = log X y∈Y + exp(w ⊤ y z t )− log X y∈Y − exp(w ⊤ y z t ), the first-order change is governed by ⟨ ̄w Y + (z t )− ̄w Y − (z t ), ∆ θ,t ⟩. A systematic improvement therefore requires positive align- ment between the non-targeted flooring displacement and task-relevant output-head directions. This does not imply thatR 50 orE count are continuous or monotone functions ofθ. The empirical pipeline in- cludes autoregressive decoding, string parsing, bounding- box matching, IoU thresholding, and integer count-error scoring. These operations can introduce discontinuities. Accordingly, the paper does not claim a theorem about monotonic behavior of decoded metrics. The role of the theory is instead to identify what a use- ful FLIP effect would require at the logit-facing repre- sentation. Because FLIP does not encode the object cat- egory, box coordinates, or count target, a stable interior gain is unexpected under a generic perturbation account. The empirical protocol then tests whether the behavioral signature is consistent with task-linked alignment: the gain should align with the detection-to-counting mediator, weaken when feature/output-head coherence is disrupted, localize to the normalized logit-facing site, and fail to repro- duce on negative-control tasks. D. Grounding-Proxy Interpretation and Limitations The product summary 1 FLIP → ∆R 50 , E count ∼ 1 FLIP + ∆R 50 is interpreted as grounding-proxy evidence, not proof of direct within-pass mediation. Localization and counting are elicited by separate prompts, so the observed localiza- tion output cannot literally cause the separately generated counting output. Instead,∆R 50 is used as an observable proxy for an unobserved intervention-sensitive grounding state that may be expressed as both improved localization and reduced counting error. This analysis, therefore, supports the narrower claim that the interior regime is compatible with a shared grounding factor. It does not identify the specific circuit, exclude other unobserved mediators, or establish causal sufficiency. A potential concern is that∆R 50 andE count may share a common computational ancestor through predicted bound- ing boxes, making the mediation relationship tautological. In the present design, localization and counting are elicited by separate prompts and scored from independent model outputs, so the observed association is not attributable to a shared output-level pipeline. The remaining alternative is a common representational cause within the model, in 10 FLIP for VLMs which the intervention jointly affects both localization and counting. This possibility is not fully excluded, but is con- strained by the Final-site permutation control and raw-layer checks, which localize the full signature to the normalized logit-facing site rather than raw decoder-layer outputs. E. FLIP Deployment and Runtime Intervention FLIP is implemented as an inference-time patch to logit computation: the post-normalization Final-site tensorz∈ R n×d is replaced by e z = max(z,θ)before multiplication by the output projectionW ⊤ . This realizes the flooring operator directly at the readout site. Strength sweep.θis provided externally (e.g., a config file), enabling deterministic sweeps over a fixed grid without modifying parameters, tokenization, or decoding. Compute resources. All COCO-based core sweeps, in- cluding the primary detection/counting runs, negative con- trols, raw-layer checks, same-site operator comparisons, feature-coherence controls, cross-architecture replications, and image-cluster subsampling runs, were executed as single-GPU inference jobs on one NVIDIA A100-SXM4- 40GB GPU using NVIDIA-SMI/driver 570.195.03 and CUDA 12.8, with GPU memory utilization limited to 0.9. The corresponding wall-clock/GPU-hour requirements are reported in the supplemental run manifest; in our runs, a full model-level core sweep required approximately5-6 GPU-hours,52-53GPU-hours, and189GPU-hours for Qwen3-VL-4B, Qwen3-VL-8B, and Kimi-VL-A3B, respec- tively. Post-hoc parsing, scoring, GLM/mediation analysis, subsample aggregation, and figure generation were CPU- side analyses and did not require GPU acceleration. Prelimi- nary or failed runs outside the reported experiments required approximately 8 GPU-hours. Why the Final site. Final-site interventions are mini- mally invasive: they leave early visual feature extraction unchanged and perturb the normalized readout state where image/text evidence is mapped to logits. F. Cross-Architecture Validation Cross-architecture replication. Figure 6 shows cross- model dose–response for Qwen3-VL-8B, Qwen3-VL-4B, and Kimi-VL-A3B. Kimi-VL-A3B uses a different visual encoder, cross-modal fusion mechanism, and training recipe from the Qwen family, providing genuine cross-architecture replication. All three models exhibit the same qualitative three-phase structure. Importantly, the absolute gains are not uniformly tiny: the inset makes clear that Qwen3-VL- 8B is the lower-amplitude case, while Qwen3-VL-4B and Kimi-VL-A3B show visibly larger interior peaks. Cross-model instability diagnostic. The corresponding three-model indoor/outdoor response-switch diagnostic is shown in Figure 7.This diagnostic measures answer changes relative to baseline rather than task performance. We therefore use it only to check generic output instability; the performance-based negative-control contrast is supplied by the left/right accuracy results in the main text and Ap- pendix J.1. Cross-architecture mediation replications. To test whether the mediated detection-to-counting pathway ex- tends beyond the anchor model, we report the corresponding delta-method product summary tables for Kimi-VL-A3B and Qwen3-VL-8B. These replications preserve the same qualitative interpretation as the main-text Qwen3-VL-4B table: an interior regime withIRR indirect < 1and reversal under over-suppression. G. Raw Decoder-Layer Checks and Logit-Facing Specificity This appendix reports raw decoder-layer checks under the same behavioral readout used in the main text. The goal is to test whether the full protocol is selectively satisfied at the normalized logit-facing Final site or appears trivially in raw decoder-layer spaces. All checks are evaluated through final outputs, so any effect must propagate to decoded behavior. The full result is shown in Figure 8. These checks separate depth, dose, and validation. Early/middle layers remain near baseline before collapse, while late raw layers may show partial∆R 50 movement. We additionally include percentile-matched raw-layer sweeps for Layers 8, 16, and 32 to test whether the failure is ex- plained by marginal threshold mismatch. These matched sweeps do not recover the Final-site response and instead attenuate or worsen the raw-layer dose–response. The ac- cepted protocol, however, is not defined by nonzero∆R 50 alone: it requires negative-control contrast, grounding-proxy alignment, and feature-coherence dependence. The post- normalization Final curve is the logit-facing reference; the raw last-block output is a separate pre-normalization site. Active-set and magnitude calibration.For each site, we track token-level active-set fractionA ℓ,t (θ)and mean clamp magnitudeM ℓ,t (θ)(Section 5.1 and Table 4).Ameasures how many hidden dimensions are touched;Mmeasures average displacement. Layers 8 and 16 are not inactive: at θ = 0, theirAvalues match the Final site, but theirMval- ues are roughly an order of magnitude smaller, followed by rapid saturation. Layer 32 is closer inA/Mand can move ∆R 50 , but still lacks the full feature-coherence-dependent 11 FLIP for VLMs −10−9−8−7−6−5−4−3−2−101 10 −3 10 −2 10 −1 10 0 θ Response-switch rate Qwen3-VL-8BQwen3-VL-4BKimi-VL-A3B Figure 7. Indoor/outdoor response switch-rate comparison. Response switch-rate curves (defined in Appendix S) across Qwen3-VL-8B, Qwen3-VL-4B and Kimi-VL-A3B. Table 2. Delta-method grounding-proxy association for1 FLIP ,∆R 50 , andE count on Kimi-VL-A3B.a: effect of FLIP on the recall proxy∆R 50 ;b: Poisson-GLM association of∆R 50 withlog E[E count ];a×b: product summary;IRR indirect = exp(a×b), interpreted as a grounding-proxy association rather than direct within-pass mediation. Values below 1 indicate that improved recall is associated with reduced tolerant counting error. θaSE(a) bSE(b) a×bSE(a×b) p-valueIRR indirect [95% CI] −3.50.004190.00231 −2.6590.114 −0.011150.006150.06970.9889 [0.9771, 1.0009] −3.00.001890.00241 −2.6450.115 −0.00500.00640.43420.9950 [0.9826, 1.0075] −2.5 −0.000200.00257 −2.6680.1100.000540.006850.93761.0005 [0.9872, 1.0141] −2.00.003080.00269 −2.6590.113 −0.008190.007150.25230.9918 [0.9780, 1.0058] −1.50.008410.00292 −2.6570.108 −0.022360.007800.00420.9779 [0.9630, 0.9930] −1.00.021510.00312 −2.7190.100 −0.058470.00875 < 0.00010.9432 [0.9272, 0.9595] −0.50.039770.00357 −2.7330.095 −0.108660.01047 < 0.00010.8970 [0.8788, 0.9156] −0.40.045430.00368 −2.7720.092 −0.125950.01102 < 0.00010.8817 [0.8628, 0.9009] −0.30.049250.00374 −2.7670.091 −0.136240.01129 < 0.00010.8726 [0.8535, 0.8922] −0.20.050150.00382 −2.6940.103 −0.135100.01153 < 0.00010.8736 [0.8541, 0.8936] −0.10.043750.00383 −2.6350.097 −0.115290.01095 < 0.00010.8911 [0.8722, 0.9104] 0.00.032820.00392 −2.6240.099 −0.086130.01078 < 0.00010.9175 [0.8983, 0.9371] 0.10.014020.00408 −2.6320.103 −0.036900.010830.00070.9638 [0.9435, 0.9844] 0.2 −0.014650.00407 −2.5640.0910.037550.010510.00041.0383 [1.0171, 1.0599] 0.3 −0.055000.00395 −2.5720.0940.141440.01140 < 0.00011.1519 [1.1265, 1.1780] 0.8 −0.232690.00535 −2.6380.1270.613810.03279 < 0.00011.8475 [1.7325, 1.9701] 1.0 −0.232690.00535 −2.6380.1270.613810.03279 < 0.00011.8475 [1.7325, 1.9701] 12 FLIP for VLMs Table 3. Delta-method grounding-proxy association for1 FLIP ,∆R 50 , andE count on Qwen3-VL-8B.a: effect of FLIP on the recall proxy∆R 50 ;b: Poisson-GLM association of∆R 50 withlog E[E count ];a×b: product summary;IRR indirect = exp(a×b), interpreted as a grounding-proxy association rather than direct within-pass mediation. Values below 1 indicate that improved recall is associated with reduced tolerant counting error. θaSE(a) bSE(b) a×bSE(a×b) p-valueIRR indirect [95% CI] −60.00.000360.00024 −2.2400.121 −0.000800.000530.13190.9992 [0.9982, 1.0002] −40.00.001790.00054 −2.2520.119 −0.004020.001240.00120.9960 [0.9936, 0.9984] −20.00.005320.00106 −2.2960.116 −0.012200.00251 < 0.0010.9879 [0.9830, 0.9927] −10.00.005190.00128 −2.2720.119 −0.011800.00297 < 0.0010.9883 [0.9825, 0.9940] −5.00.006930.00155 −2.2720.123 −0.015750.00362 < 0.0010.9844 [0.9774, 0.9914] −4.00.007660.00160 −2.2790.115 −0.017470.00375 < 0.0010.9827 [0.9755, 0.9899] −2.50.008000.00158 −2.2480.118 −0.017990.00368 < 0.0010.9822 [0.9751, 0.9893] −2.00.005920.00153 −2.2540.112 −0.013350.00351 < 0.0010.9867 [0.9800, 0.9935] −1.50.003940.00153 −2.2760.115 −0.008970.003500.0100.9911 [0.9843, 0.9979] −1.00.003270.00157 −2.2490.119 −0.007350.003550.0380.9927 [0.9858, 0.9996] −0.5 −0.002320.00188 −2.2220.1130.005150.004180.2181.0052 [0.9970, 1.0134] 0.0 −0.017990.00255 −2.3120.1100.041600.00622 < 0.0011.0425 [1.0299, 1.0552] signature. Thus raw-layer failures are not explained by too few clamped dimensions; the complete protocol localizes to the post-normalization logit-facing site. The percentile- matched overlays in Figure 4 further show that this localiza- tion is not removed by matching the marginal intervention scale at representative early, middle, and late raw-layer sites. Table 4. Token-level calibration for raw-layer checks. Means of active-set fractionAand mean clamp magnitudeMunder unpermuted flooring. Early/middle layers show matchedAbut much smallerMatθ = 0, then rapid saturation; Layer 32 is closer to the Final site but does not reproduce the full Final-site signature. Layer θ Final A Final MLayer A Layer M 80.00.5000.9680.5000.091 80.50.6021.1220.9790.504 81.00.6621.4530.9991.001 160.00.4980.9630.4970.102 160.50.5911.0390.9800.508 161.00.6311.3170.9981.006 320.00.4540.8210.4980.639 320.50.5351.0000.6430.926 321.00.6081.1920.7581.277 13 FLIP for VLMs −10−50 −8 0 8 ·10 −2 Final ref. (post-norm) θ ∆ R 50 Layer 8 (Early) −10−50 ·10 −2 θ Layer 16 (Mid) −10 −50 ·10 −2 θ Layer 32 (Late) −10−50 −0.6 −0.4 −0.2 0 0.2 Final ref. (post-norm) θ ∆Acc (left/right) −10−50 θ −10−50 θ −10−50 0.8 0.9 1 1.1 1.2 Final ref. (post-norm) θ IRR indirect −10−50 θ −10−50 θ Final ref. (post-norm) λ = 0.00λ = 0.10λ = 0.20λ = 0.25λ = 0.33 Primary task Negative control Mediator Figure 8. Full protocol checks for raw decoder-layer sites. Columns show representative raw decoder-layer outputs; rows show primary-task∆R 50 , left/right∆Acc, andIRR indirect = exp(a× b)under permutation mixing. The post-normalization Final curve is the logit-facing reference. Raw layers may show partial∆R 50 movement, but they do not reproduce the full feature-coherence-dependent Final-site signature. 14 FLIP for VLMs H. Redistribution Breadth and Dispersion Diagnostics H.1. Target-Conditioned Patch Relevance and RMR Computation This appendix describes how target-conditioned patch rele- vance maps are computed and how redistribution diagnostics are derived from them. These quantities are used only as comparative diagnostics of redistribution geometry: they do not replace the primary mediator analysis based on∆R 50 andE count . Target-conditioned saliency maps. For each image– prompt pair, we compute a baseline patch relevance map and a FLIP-intervened patch relevance map conditioned on a specified target token sequence, such as the firstbbox2d field or a generated detection answer. The relevance pipeline first identifies the target token positions in the full sequence (prompt plus generated response). We then definey target as a scalar target score for that sequence, computed as the sum of the logits assigned to the identified target tokens, and compute layerwise gradient-weighted attention relevance maps of the form R (ℓ) = ReLU ∂y target ∂A (ℓ) ⊙ A (ℓ) ,(12) averaged over heads. These per-layer maps are combined into a joint relevance matrix by augmenting with the iden- tity, row-normalizing, and multiplying cumulatively across layers. To sharpen target specificity, the implementation also com- putes token-level Grad×Hidden importance from the final hidden state, g i = h i , ∂y target ∂h i ,(13) with non-target positions masked to zero. The masked token- importance vector is propagated through the joint relevance matrix and restricted to the visual patch-token indices, yield- ing one scalar relevance value per image patch. For Qwen3- VL, these values are arranged on the model’s24× 24patch grid. Qualitative heatmaps are produced by interpolation and overlay, but all reported statistics are computed directly from the underlying patch relevance vectors. Baseline and FLIP conditions. The same target- conditioned relevance procedure is run twice: once under the baseline condition and once under FLIP. Lets −∞ ∈ R P denote the baseline patch saliency map ands θ ∈ R P the FLIP-intervened map, wherePis the number of image patches. We define the patchwise saliency shift δ = s θ − s −∞ .(14) All redistribution diagnostics are computed from δ. Directional RMR diagnostics. LetOdenote the set of object patches under the patch mask induced by the parsed bounding box. We report three directional redistribution statistics: RMR + (θ) = X p∈O max(δ p , 0),(15) RMR − (θ) = X p∈O max(−δ p , 0),(16) RMR obj (θ) = X p∈O δ p − X p/∈O δ p .(17) Thus,RMR + measures saliency mass shifted toward ob- ject patches,RMR − measures saliency mass shifted away from object patches (reported as a positive magnitude), and RMR obj measures the resulting object-versus-background contrast. The object mask is derived from parsed bounding boxes projected onto the patch grid, soRMR obj is a relative contrast measure rather than a raw object saliency total. Interpretive role. These quantities are used to refine the interpretation of the FLIP sweep. In the negligible regime, directional redistribution remains weak. In the in- terior regime, bounded positive redistribution toward ob- ject patches becomes detectable. Under over-suppression, object-aligned gains weaken while off-object loss and re- distribution breadth increase. The diagnostics therefore complement the primary behavioral analysis without being treated as formal mediators or faithful causal explanations. H.2. Comparative role of rollout The target-conditioned saliency maps above are constructed using gradient-weighted attention relevance propagation with rollout. Their most reliable use in this paper is com- parative: differences acrossθfor the same model, prompt, and target are informative, whereas absolute saliency magni- tudes should not be over-interpreted. For coordinate-token targets, tokenization positions can shift across interventions, so target indices are anchored to baseline outputs and reused across FLIP runs. If a target token is absent at a givenθ, that sample is marked missing for RMR computation at that θ. H.3. Regime interpretation of the RMR diagnostics The RMR statistics are used to relate redistribution geometry to the regime structure observed in the main dose–response analysis. They are not formal mediators and do not replace the primary evidence based on∆R 50 andE count . Their role is interpretive: to indicate how target-conditioned relevance mass is reallocated across the FLIP sweep. Negligible regime. At very negativeθ, both∆R 50 and RMR + are small, indicating that FLIP is effectively inactive 15 FLIP for VLMs at the redistribution level. This is the redistribution-side signature of the negligible-change regime. Consistently, RMR − remains comparatively modest andRMR obj stays near zero in this region. Interior regime. In the bounded interior regime,RMR + rises and co-varies with∆R 50 , as shown directly in Fig- ure 9(b). This indicates that the positive detection-side response is accompanied by bounded object-aligned redistri- bution of relevance mass. In this regime,RMR − increases only gradually, andRMR obj remains comparatively modest. The resulting picture is one of constructive redistribution rather than generic suppression. Over-suppression regime. Asθapproaches the col- lapse region,RMR + declines,RMR − rises sharply, and RMR obj spikes late. This transition indicates that the regime has changed qualitatively. The late rise inRMR obj should not be read as evidence of a stronger constructive in- terior regime; instead, it is more consistent with suppression- dominant contrast enhancement, in which object-relative saliency rises because off-object mass is removed broadly. Interpretive role. Taken together, the four-panel anal- ysis refines the regime picture as follows: the negligi- ble regime shows little redistribution; the interior regime is the point of strongest object-aligned reallocation, in- dexed by co-movement ofRMR + and∆R 50 ; and the over- suppression regime is marked by risingRMR − and late spikes inRMR obj . Thus, the RMR diagnostics provide a redistribution-side corroboration of the main regime struc- ture without elevating curve shape alone into a grounding- specificity claim. To connect redistribution geometry more directly to the regime structure, we plotRMR + ,RMR − , andRMR obj againstθfor Qwen3-VL-8B, together with a direct overlay of RMR + and ∆R 50 . I. Vision-Centric Benchmark Consistency This appendix reports additional results on vision-centric benchmarks to assess whether the regime structure observed in the main text extends beyond the detection-based evalua- tion setting. Scope and role. These benchmarks are not designed to isolate the detection-derived mediator used in the main anal- ysis. As such, they do not constitute mediator-aligned evi- dence. Instead, they serve as consistency checks on whether intervention sweeps produce structured, non-monotonic re- sponse patterns in broader evaluation settings. Benchmarks and setup.We evaluate performance under the same intervention sweep across MMStar, MindCube, and NaturalBench. Each benchmark emphasizes different aspects of vision–language performance, including reason- ing, compositionality, and general visual understanding. Un- like the core intervention sweeps, which use fixed random seeds for replicability where supported, these benchmark evaluations are executed through the third-party LMMS- Eval stack. We therefore replicate the benchmark runs mul- tiple times under independent vLLM initializations of the same Qwen3-VL-8B model and report standard deviations across those runs as the error bars in Figure 10. These LMMS-Eval benchmark data were generated on an NVIDIA GB10 system with 128 GB unified memory, using NVIDIA- SMI/driver 580.142 and CUDA 13.0, with GPU memory utilization set to 0.3; the supplemental run manifest reports the corresponding wall-clock/GPU-hour requirements for the MMStar, MindCube, and NaturalBench sweeps. Observed regime patterns.Figure 10 shows that all three benchmarks exhibit structured response curves under in- creasing intervention strength θ. MMStar remains elevated across a relatively wide range ofθ, suggesting a broad plateau rather than a sharply lo- calized interior optimum. MindCube exhibits the clearest three-phase structure, with an identifiable interior regime separating negligible and collapse regions. NaturalBench shows a peak in the interior regime followed by degradation under stronger suppression. While the precise shape and width of the regimes vary across benchmarks, all three display departures from monotonic degradation, consistent with the existence of structured re- sponse regimes. Interpretation. These results provide supporting evi- dence that the emergence of regime structure is not con- fined to the detection-recall metric used in the main anal- ysis. However, because these benchmarks do not isolate the detection-derived mediator, they do not establish causal compatibility in the sense of Section 6. Accordingly, we interpret these findings as regime- consistency evidence rather than primary evidence of grounding-proxy alignment. Because the error bars reflect run-to-run variability in the LMMS-Eval serving stack rather than uncertainty in the core fixed-seed intervention pipeline, they should be read as corroborative robustness information rather than as the primary inferential basis of the paper. Relation to redistribution diagnostics. The regime pat- terns observed here are consistent with redistribution di- agnostics reported in Appendix H, where intermediate in- tervention strengths correspond to structured changes in 16 FLIP for VLMs −60−40−20−10−50 0 2 4 6 θ RMR + (a) −60−40−20−10−50 0 2 4 6 θ RMR + (b) −0.5 0 0.5 1 ·10 −2 ∆ R 50 −60−40−20−10−50 5 10 15 θ RMR − (c) −60−40−20−10−50 0 2 4 6 θ RMR obj (d) Figure 9. Qwen3-VL-8B redistribution diagnostics across the FLIP sweep. (a)θversusRMR + , which peaks in the same broad interior region where the detection-side response is positive. (b) Dual-axis overlay ofθversusRMR + (left axis) andθversus∆R 50 (right axis), showing redistribution-side co-variation with the behavioral dose–response. (c)θversusRMR − , which grows toward collapse and marks increasing off-object saliency loss. (d)θversusRMR obj , which remains modest through the interior regime but spikes late, consistent with suppression-dominant contrast enhancement rather than constructive redistribution. Together, the panels link the appendix RMR diagnostics to the three-regime structure of the main FLIP sweep. attention-derived measures. These diagnostics are presented as complementary geometric indicators rather than causal explanations; Appendix H.3 shows that, for the 8B hard case,RMR + tracks the interior regime whileRMR − and RMR obj characterize the transition to over-suppression. J. Negative-Control Task Design The primary negative control is the singleton-pair left/right spatial task in Appendix J.1. Unlike the indoor/outdoor response-switch diagnostic, this task is scored by accuracy and therefore tests whether the same intervention sweep produces a genuine performance gain on a distinct vision- language control task. The intended contrast is not that every control task must degrade monotonically, but that it should not reproduce the coherent positive interior regime observed for the detection/counting pathway. We retain the indoor/outdoor experiment only as an auxiliary instability diagnostic: its response-switch rate measures whether answers change relative to baseline, not whether performance improves. J.1. Singleton-Pair Left/Right Spatial Control The singleton-pair left/right task is used as a broader spa- tial negative control. Its purpose is not to instantiate the main detection-derived mediator, but to test whether FLIP produces a reproducible bounded interior optimum on a dis- tinct vision-language task that depends on coarse relational judgment rather than full localization/counting structure. Dataset construction. The dataset is built from MS COCO val2017 in clustered-by-image form so that down- stream analysis can still use image-level cluster-robust in- ference. For a candidate image to qualify, it must satisfy the same image-centric filtering logic used in the main clus- tered construction, including minimum category-richness re- quirements. Candidate left/right pairs are then formed only from singleton categories: the image must contain exactly one instance of object 1 and exactly one instance of object 2 . Additional filters require sufficient horizontal separation be- tween the two objects and may also impose minimum area and maximum-overlap constraints. By default, at most one pair is sampled per image to avoid overweighting visually 17 FLIP for VLMs −50−40−30−20−100 −4 −2 0 2 4 ·10 −2 intervention strength θ percent accuracy change MMStar MindCube NaturalBench Figure 10. Dose–response of FLIP on vision-centric benchmarks. Curves show benchmark-specific performance changes relative to baseline. Error bars denote standard deviation across repeated LMMS-Eval runs using independent vLLM initializations of the same Qwen3-VL-8B model. The shaded band indicate the regime partition defined from the primary detection/counting sweep under the current R 50 mediator. MMStar, MindCube, and NaturalBench show structured, non-monotonic responses consistent with the main regime picture. busy scenes. Prompt and label construction. Each retained pair yields a yes/no spatial-relation question of the formIs the <object1> to the left of the <object2>? Answer only yes or no. To maintain near-balanced labels, prompts are generated by alternating the true ordering and the swapped ordering across qualifying pairs, producing an overall dataset that is approximately50/50 yes/no. Querying. The left/right control is queried directly from the stored text field rather than being rebuilt from extracted object names at inference time. The querying infrastructure otherwise matches the local chat-completions pipeline used for the other intervention experiments, including image-path resolution, payload construction, response extraction, and threshold bookkeeping. Interpretive role.Because this control uses unambiguous singleton pairs, errors are less likely to come from mul- tiplicity or ambiguous reference than in general localiza- tion prompts. At the same time, it does not instantiate the detection-to-counting mediator used in the main analysis. We therefore use∆Accon this task as the performance- based negative control: it tests whether the intervention produces a comparable positive interior regime on a dis- tinct spatial judgment, rather than merely causing answer switching. K. Baseline Centering, Paired Designs, and Heterogeneity This subsection clarifies the role of baseline-centered recall in the total-effect model and its relationship to an equiva- lent paired-difference formulation under image-clustered inference. Point-estimate equivalence. LetY i (θ)denote the detection-side mediator for unitiunder intervention strengthθ, with baselineY i (−∞)corresponding to the no- intervention condition. In the present study, each image– query pair is evaluated under both conditions, so the same units appear in both arms. In a binary-treatment regression with an intercept, the co- efficient on the treatment indicator equals the difference in sample means: bα 1 =Y (θ)−Y (−∞).(18) If instead we define per-sample paired differences δ i = Y i (θ)− Y i (−∞),(19) then the sample mean satisfies δ = 1 N N X i=1 δ i =Y (θ)−Y (−∞).(20) At the population level, the same equality follows from linearity of expectation: E[Y i (θ)− Y i (−∞)] = E[Y i (θ)]− E[Y i (−∞)].(21) Thus, when both conditions are evaluated on the same set of units, global centering and paired subtraction target the same estimand and yield the same point estimate. Clustered inference for the paired design. The equiva- lence above concerns the treatment contrast, not a claim that baseline centering is more conservative than paired subtrac- tion. Because the pooled Gaussian GLM uses image-level 18 FLIP for VLMs cluster-robust standard errors, baseline and intervention ob- servations from the same image contribute jointly to the cluster score. The resulting variance estimate therefore accounts for the within-cluster covariance induced by evalu- ating matched image–query units under both conditions. Un- der the balanced paired design used here, a pooled treatment- indicator regression with image-clustered standard errors and an equivalent paired-difference regression with image- clustered standard errors target the same large-sample un- certainty for the treatment contrast, up to finite-sample and implementation differences. We use the baseline-centered pooled specification because it keeps the reported∆R 50 scale aligned with the dose-response figures, mediation sum- maries, and stratified GLM table, not because it is inherently more conservative. What this does and does not resolve about heterogene- ity.The equivalence above establishes identical point esti- mates for the pooled treatment contrast, but it does not by itself rule out heterogeneous effects across subgroups. We therefore examine one interpretable source of heterogene- ity directly: instance count. The updated Qwen3-VL-4B stratification shows that the small-count stratum closely mirrors the pooled pattern across the interior regime. The large-count stratum also remains positive and significant over the main interior window, but its gains are smaller and attenuate earlier, becoming non-significant atθ = 0.5. The over-suppression reversal atθ = 1.0is significant in both strata, indicating that the regime structure—interior im- provement followed by collapse under strong intervention— is preserved across this subdivision. These findings argue against the concern that the main quali- tative conclusions are an artifact of aggregation along this di- mension. If global centering were masking strong opposing subgroup effects, one would expect sign instability across strata; instead, both strata preserve the positive interior di- rection and both exhibit collapse under over-suppression. The heterogeneity is primarily in effect size and attenua- tion: few-object scenes show stronger and more persistent gains, while many-object scenes weaken earlier. At the same time, this analysis addresses heterogeneity only along one interpretable axis. It does not exclude the possibility that other sources of heterogeneity—such as object category, occlusion, or image difficulty—could modulate effect size. A broader distributional heterogeneity analysis remains an important direction for future work. L. Regression Specifications and Robust Inference We estimate: ∆R 50 = α 0 + α 1 1 FLIP + ε, (22) log E[E count | 1 FLIP , ∆R 50 ] = γ 0 + γ 1 1 FLIP + γ 2 ∆R 50 . (23) Standard errors are clustered at the image level to account for dependence among object queries from the same image and for the covariance between matched baseline and inter- vention observations within each image cluster. The first equation is fit as a Gaussian GLM with image-clustered SEs and supplies thea-term; the second is fit as a Poisson GLM on strict counting responses and supplies theb-term. The auxiliary ZeroInflatedPoisson model used in the evaluation code belongs only to the two-part format-failure/tolerant- error decomposition forE combined ; it is not used for the reported product summary. Cluster-robust stratified GLM. Table 5 reports cluster- robust Gaussian GLM estimates stratified by instance count (≤ 3vs.> 3objects) for the Qwen3-VL-4B anchor model. The pooled estimate is practically small atθ =−50, then rises across the interior regime, remaining positive and sig- nificant throughθ = 0.2before attenuating atθ = 0.5 and reversing sharply atθ = 1.0(p < 0.001). The small- count stratum closely tracks the pooled pattern. The large- count stratum is also positive and significant over the main interior window, but attenuates earlier and is not signifi- cant atθ = 0.5, consistent with stronger effects on cleaner few-object scenes and weaker gains as visual multiplicity increases. M. Operator-Comparison Baselines and Feature-Coherence Control This appendix separates two distinct comparisons that serve different roles in the paper. The first is a full four-criterion replication across same-site magnitude-bounding operators; the second is the graduated permutation control for the floor- ing instantiation itself. Together they test whether the proto- col generalizes across operator choice while still isolating feature-coherence dependence. M.1. Full Four-Criterion Replication Across Same-Site Operators All operator-comparison baselines intervene at the same site as FLIP: the post-normalization Final-site tensor passed to the output head. Symmetric clipping.The symmetric-clipping baseline is e z clip = clip(z, −|θ|, +|θ|).(24) 19 FLIP for VLMs Additive shift. The additive-shift baseline is a matched- mean translation: Ψ θ (z) = z + μ θ (z)1, μ θ (z) = 1 nd X i,k (θ− z ik ) + . (25) That is, the hidden state is shifted uniformly by the mean flooring effect that the corresponding flooring operator would induce at the same θ. Interpretation of operator replication. Figure 12 pro- vides the compact operator-baseline comparison at the level of primary-task dose–response alone, while Figure 11 ex- tends the comparison to the full protocol for same-site magnitude-bounding interventions. Across flooring, sym- metric clipping, and additive shift, the primary task exhibits a bounded positive regime under low or no permutation. By contrast, the singleton-pair left/right control, scored by ∆Accrather than response switching, remains near baseline or drifts downward over the non-collapse region and does not reproduce a comparable positive interior regime. The grounding-proxy pathway is likewise altered once feature coherence is disrupted. The operator-side result is therefore not that all operators behave identically, but that the pro- tocol continues to distinguish structured, mediator-aligned regimes from generic perturbation effects across same-site bounded interventions. Figure 11 shows the complete same-site operator-side repli- cation of the four-criterion protocol. M.2. Graduated Permutation Control The graduated permutation control is not an alternative op- erator baseline. It keeps the flooring instantiation fixed and tests whether its positive interior regime depends on feature coherence. Let z∈ R n×d denote the post-normalization Final-site ten- sor, with permutation acting along the feature dimension. For mixing parameterλ ∈ [0, 1], define a partial random permutation operator S λ ∈ R d×d by S λ = Π λ ⊕ I 1−λ ,(26) where: (i)Π λ is a random permutation acting on a randomly selected subset of⌈λd⌉feature dimensions, (i)I 1−λ is the identity on the remaining feature dimensions, and (i)⊕ denotes the direct sum over the partition of1,...,dinto permuted and unpermuted coordinates. The graduated permutation control is then defined by e z (λ) = max(zS λ ,θ),(27) where the maximum is taken element-wise and no inverse permutation is applied. This omission is deliberate: because elementwise flooring is permutation equivariant, an inverse- restored construction would satisfy max(zS λ ,θ)S −1 λ = max(z,θ), and would therefore collapse back to standard flooring rather than stress feature–output-head alignment. Equivalently, a fractionλof feature dimensions is permuted without inverse restoration before flooring, while the remaining(1− λ) fraction is left unchanged. This construction has clean boundary behavior: e z (0) = max(zI,θ) = max(z,θ),(28) e z (1) = max(zP,θ),(29) wherePis a full random permutation on alldfeature di- mensions. Thusλ = 0recovers standard flooring, while largerλvalues progressively destroy feature alignment by increasing the fraction of permuted coordinates. For all reported permutation-control curves, baseline center- ing uses the common unpermuted no-intervention run. Let R (λ) 50 (θ)denote recall after applying e z (λ) = max(zS λ ,θ) . We plot ∆R (λ) 50 (θ) = R (λ) 50 (θ)− R (0) 50 (−∞), notR (λ) 50 (θ)− R (λ) 50 (−∞). Thea-term in the correspond- ingIRR indirect panels uses the same common-baseline ∆R (λ) 50 (θ). Thus any degradation caused by permutation atθ =−∞remains visible in the plotted level rather than being removed by λ-specific re-centering. Figure 5 therefore tests whether the unpermuted interior window survives feature-coordinate misalignment. Relative toλ = 0, all nonzeroλsettings lose the positive regime earlier and enter collapse sooner, although the ordering among nonzeroλvalues is not strictly monotonic. The key contrast is betweenλ = 0andλ > 0: disrupting feature coherence destabilizes the regime-relevant window even when the intervention class is otherwise unchanged. M.3. Related Operator Intuition: Symmetric Clipping and Additive Shift The main text develops the output-head alignment base- line for flooring because FLIP is the primary intervention. Symmetric clipping and additive shift are used as same-site operator-comparison baselines. The relevant comparison is not that these operators are theoretically identical, but that each induces a displacement at the same logit-facing representation. Symmetric clipping.The symmetric-clipping baseline is e z clip = clip(z,−|θ|, +|θ|). 20 FLIP for VLMs It induces a displacement ∆ clip θ = e z clip − z. As|θ|shrinks along the sweep, more coordinates are clipped and the induced displacement becomes broader. Any useful effect must therefore arise from alignment between∆ clip θ and task-relevant output-head directions, not from clipping magnitude alone. Additive shift. The additive-shift baseline applies Ψ θ (z) = z + μ θ (z)1, μ θ (z) = 1 nd X i,k (θ− z ik ) + . Its displacement is global rather than support-local: ∆ shift θ = μ θ (z)1. For fixedz,μ θ (z)is non-decreasing inθ, so the shift magni- tude grows along the sweep. As with flooring and clipping, a useful effect requires the induced displacement to project favorably onto task-relevant output-head directions. Interpretive role.The operator comparison therefore asks whether regime structure and feature-coherence dependence survive changes in the displacement geometry. A regime- shaped curve alone is not sufficient; the key question is whether the effect remains mediator-aligned, coherence- dependent, localized to the logit-facing site, and absent from negative-control tasks. N. Subsample Stability of the Interior Regime This appendix tests whether the interior regime is stable under reductions in effective sample size and under the finite granularity of theθsweep. Because scanning many θvalues can otherwise overstate small peaks, we use the grid only to define a descriptive FWHM regime window and require the resulting window to remain visible under image-cluster subsampling rather than claiming a separate confirmatory discovery at each grid point. Thesubsamplingprocedureaddsanargument --n-subsamples N(default1).Each subsample uses seedbaseseed +s idx , making the full subsampling pipeline reproducible.For each subsample, we draw an independent permutation of the image list without replacement. Fractions within a given subsample are then formed cumulatively from that single permuted ordering, so that10% ⊂ 20% ⊂ 40% ⊂ 60% ⊂ 80% ⊂ 100%within the same subsample. Figure 13 shows that the qualitative regime is preserved across fractions, the within-fraction means track the full- sample curve closely, and variability decreases with scale. This supports the interpretation that the interior regime is a stable signal rather than a small-sample artifact. These image-cluster resampling results also serve as a robustness check on the large-sample clustered-inference interpretation used for the reported 95% confidence intervals. O. Additional Qualitative Illustration O.1. Illustrative Regime Visualization Figure 14. Illustrative regime structure under increasing in- tervention strengthθ. Asθincreases, behavior shifts from neg- ligible change to an interior regime with qualitatively improved detections, and then to over-suppression. Heatmaps show patch- token relevance from attention rollout; solid boxes denote ground truth and dashed boxes denote model detections. Representative example only. P. Proofs and Supporting Analysis This appendix supports the operator and output-head state- ments used in Section 5. We state only hidden-state and pre-decoding facts; no claim is made that decoded metrics are continuous or monotone in θ. P.1.Componentwise Flooring Identity and Monotonicity For a scalar x∈ R and threshold θ∈ R, define φ θ (x) = max(x,θ), δ θ (x) = φ θ (x)−x = (θ−x) + . Applied elementwise to z∈ R n×d , e z = φ θ (z),∆ θ = e z− z. For each coordinate (i,k), ez ik = z ik + (θ− z ik ) + . Proof of Proposition 5.1. The identity follows immedi- ately from the definition of max. If θ 1 ≤ θ 2 , then (θ 1 − z ik ) + ≤ (θ 2 − z ik ) + 21 FLIP for VLMs for every coordinate. Hence each entry of∆ θ is coordinate- wise non-decreasing inθ, the active support expands mono- tonically, and∥∆ θ ∥ p is non-decreasing for anyp∈ [1,∞]. P.2. Output-Head Margin Expansion For a final-token hidden state z t , logits are ℓ t = W ⊤ z t + b. Under FLIP, e z t = z t + ∆ θ,t ,∆ θ,t = (θ− z t ) + , so the logit change is exactly e ℓ t − ℓ t = W ⊤ ∆ θ,t . For token sets Y + and Y − , define m(z t ) = log X y∈Y + exp(w ⊤ y z t )− log X y∈Y − exp(w ⊤ y z t ). The gradient of this margin is ∇m(z t ) = ̄w Y + (z t )− ̄w Y − (z t ), where each ̄wis the softmax-weighted average of output- head vectors within the corresponding token set. A first- order Taylor expansion gives m( e z t )−m(z t ) =⟨ ̄w Y + (z t )− ̄w Y − (z t ), ∆ θ,t ⟩+o(∥∆ θ,t ∥). Thus a useful pre-decoding margin increase requires positive alignment between the FLIP displacement and task-relevant output-head directions. For raw decoder-layer sites, the analogous first-order effect must pass through the remaining computation: e ℓ− ℓ≈ J ℓ→logits ∆ (ℓ) θ . The raw-layer analysis therefore tests whether such propa- gated alignment reproduces the signature observed at the normalized logit-facing representation. Q. FLIP versus Prompt-Based Control Prompt engineering operates exclusively at the level of the input distribution and relies on the model’s learned priors to steer generation. By contrast, FLIP is an inference-time in- tervention applied after tokenization and cross-modal fusion, operating directly on the post-normalization Final-site hid- den state before logit computation. FLIP does not introduce new linguistic content, cannot inject information absent from the model’s internal representations, and fails grace- fully when evidence is weak or over-suppressed—unlike prompt-based control. R. Model Choice, Scope, and Claims The core detection/counting experiments are instantiated on Qwen3-VL-4B-Instruct to enable controlled, auditable inference-time patching. Claims for the mediator-based regime analysis are therefore scoped to: (i) this model fam- ily instance, (i) the post-normalization Final intervention site, and (i) the evaluation protocol. The contribution is the methodology—regime-structured intervention sweeps, a mediator-based probe, and redistribution diagnostics. Cross- architecture validation is reported in Appendix F. S. Auxiliary Instability Diagnostic: Global Scene Classification For the indoors/outdoors diagnostic, answer stability is quantified via a baseline-referenced response-switch rate. Lety (−∞) i andy (θ) i denote the baseline and interven- tion outputs for samplei. Outputs are canonicalized be- fore comparison by lowercasing, extracting the first alpha- betic token, and mapping surface variants to the binary la- bel setindoors/outdoors(e.g.,indoor,indoors, indo 7→ indoors;outdoor,outdoors,out 7→ outdoors). Definingc(·)as this canonicalization map, the response-switch rate at strength θ is RSR(θ) = 1 N N X i=1 1 n c y (θ) i ̸= c y (−∞) i o .(30) This metric measures response instability relative to baseline rather than task accuracy. 22 FLIP for VLMs Table 5. Cluster-robust Gaussian GLM estimates of the intervention effect on∆R 50 , stratified by instance count for Qwen3-VL-4B. Effects are reported pooled and by object-count bin; the coefficient of interest is the slope on 1 FLIP . θSettingˆα 1 SE p-value −50.0Pooled0.00210.0010 < 0.001 Small (≤ 3)0.00230.0010 < 0.001 Large (> 3)0.00100.00100.075 −5.0Pooled0.02680.0020 < 0.001 Small (≤ 3)0.02830.0020 < 0.001 Large (> 3)0.01850.0030 < 0.001 −2.5Pooled0.03500.0020 < 0.001 Small (≤ 3)0.03660.0030 < 0.001 Large (> 3)0.02560.0040 < 0.001 −2.0Pooled0.03880.0020 < 0.001 Small (≤ 3)0.04120.0030 < 0.001 Large (> 3)0.02540.0040 < 0.001 −1.5Pooled0.04220.0030 < 0.001 Small (≤ 3)0.04440.0030 < 0.001 Large (> 3)0.02970.0040 < 0.001 −1.0Pooled0.04830.0030 < 0.001 Small (≤ 3)0.05080.0030 < 0.001 Large (> 3)0.03470.0050 < 0.001 −0.5Pooled0.05280.0030 < 0.001 Small (≤ 3)0.05390.0030 < 0.001 Large (> 3)0.04720.0050 < 0.001 0.0Pooled0.05900.0030 < 0.001 Small (≤ 3)0.06060.0040 < 0.001 Large (> 3)0.05000.0050 < 0.001 0.1Pooled0.05720.0030 < 0.001 Small (≤ 3)0.05940.0040 < 0.001 Large (> 3)0.04540.0050 < 0.001 0.2Pooled0.05760.0030 < 0.001 Small (≤ 3)0.06040.0040 < 0.001 Large (> 3)0.04190.0050 < 0.001 0.5Pooled0.02640.0040 < 0.001 Small (≤ 3)0.03240.0040 < 0.001 Large (> 3) −0.00680.00600.247 1.0Pooled −0.53150.0060 < 0.001 Small (≤ 3) −0.57360.0060 < 0.001 Large (> 3) −0.29920.0090 < 0.001 23 FLIP for VLMs −10−50 −8 0 8 ·10 −2 θ ∆ R 50 Floor −10−50 ·10 −2 θ Symmetric Clip −10−50 ·10 −2 θ Additive Shift −10−50 −0.6 −0.4 −0.2 0 0.2 θ ∆Acc (left/right) −10−50 θ −10−50 θ −10−50 0.8 0.9 1 1.1 1.2 θ IRR indirect −10−50 θ −10−50 θ λ = 0.00λ = 0.10λ = 0.20λ = 0.25λ = 0.33 Primary task Negative control Mediator Figure 11. Full four-criterion replication across same-site magnitude-bounding operators. Columns correspond to flooring (FLIP), symmetric clipping, and additive shift. Rows show the primary-task dose–response∆R 50 , the singleton-pair left/right spatial control scored by∆Acc, and the mediated indirect effectIRR indirect = exp(a× b)under graduated permutation mixing. Solid black denotes λ = 0; gray dash patterns denote increasing mixing. All baseline-centered quantities use the corresponding unpermuted no-intervention baseline, so nonzero-λcurves are not re-centered to already permuted baselines. Across operators, the primary task shows a bounded positive regime under low or no permutation, while coherence disruption destabilizes that regime. The left/right control remains near baseline or drifts downward over the non-collapse region and does not reproduce a comparable positive interior regime. The indirect-effect panels show that the mediated pathway is likewise altered under coherence disruption. 24 FLIP for VLMs −50−45−40−35−30−25−20−15−10−50 −0.6 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 θ ∆ R 50 (relative to baseline) Elementwise flooring Symmetric clip Additive shift Figure 12. Dose–response comparison across same-site operator baselines.∆R 50 under flooring, symmetric clipping, and additive shift, with 95% confidence intervals. All three bounded interventions exhibit a bounded interior regime, but they differ in strength and collapse profile: flooring reaches the largest interior peak, symmetric clipping collapses earliest, and additive shift remains positive over a broader range before delayed collapse. The paper’s specificity claim is therefore narrower than operator uniqueness: curve shape alone is insufficient, and the decisive tests are mediator alignment and feature-coherence dependence. 1020406080 100 4 6 8 ·10 −2 Fraction (%) ∆ R 50 (a) θ =−0.5 −10−50 −8 0 8 ·10 −2 θ (b) 10% −10−50 −8 0 8 ·10 −2 θ (c) 20% −10−50 −8 0 8 ·10 −2 θ ∆ R 50 (d) 40% −10−50 −8 0 8 ·10 −2 θ (e) 60% −10−50 −8 0 8 ·10 −2 θ (f) 80% Figure 13. Subsample stability of the interior regime. (a)∆R 50 at fixedθ =−0.5across subsample fractions. Blue markers show individual resamples; red markers and error bars show the subsample mean and variability. Fractions are formed by subsampling image clusters. The 100% condition is the single full-dataset estimate and therefore has no error bar. (b–f) Dose–response curves for five resamples per fraction, with the red curve denoting the within-fraction mean and the dashed black curve the full-dataset reference. The qualitative regime is preserved across fractions, the means track the full-sample curve closely, and variability decreases with scale, indicating that the effect is a stable signal rather than a small-sample artifact. 25

Instant research discovery