Paper deep dive
Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection
Girish A. Koushik, Diptesh Kanojia, Helen Treharne
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 9/19/2026, 2:33:49 AM
Summary
This paper investigates the 'readout gap' in Large Vision-Language Models (LVLMs) like Gemma-3 and Qwen3.5 for harmful meme detection, demonstrating that models often possess internal evidence of harmfulness but fail to route it to correct outputs. Using sparse autoencoders (SAEs), the authors show that supervised sparse readouts significantly outperform native model predictions across six benchmarks. The study distinguishes between representational failures and routing failures, finding that calibration-only routing and probe-distilled LoRA can recover much of the performance gap, indicating that routing is a primary bottleneck.
Entities (9)
Relation Signals (7)
Sparse Autoencoders → outperforms → Native Prediction
confidence 97% · Sparse readouts outperform native prediction on all six primary binary tasks
Gemma 3 → evaluatedon → Harmful Meme Detection
confidence 96% · We distinguish these cases in Gemma-3 and Qwen3.5... across six harmful content benchmarks
Qwen3.5 → evaluatedon → Harmful Meme Detection
confidence 96% · We distinguish these cases in Gemma-3 and Qwen3.5... across six harmful content benchmarks
Gemma 3 → hasmetric → Macro-F1
confidence 95% · Gemma improves from 0.532 to 0.714 [macro-F1]
Qwen3.5 → hasmetric → Macro-F1
confidence 95% · Qwen averages 0.740 versus 0.432 native macro-F1
Sparse Autoencoders → usedfor → Readout Gap
confidence 95% · We distinguish these cases in Gemma-3 and Qwen3.5 using sparse autoencoders... Sparse readouts outperform native prediction...
LoRA → improves → Native Prediction
confidence 93% · probe-distilled LoRA improves native predictions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:When a large vision-language model misclassifies a harmful meme, the failure may reflect missing internal evidence or an inability to route represented evidence to its output. We distinguish these cases in Gemma-3 and Qwen3.5 using sparse autoencoders, role-conditioned probes, causal interventions, and recovery experiments across six harmful content benchmarks, with additional Spanish and Hindi-English code-mixed evaluations. Sparse readouts outperform native prediction on all six primary binary tasks: Qwen averages $0.740$ versus $0.432$ native macro-F1, while residual reconstruction reaches $0.486$, whereas Gemma improves from $0.532$ to $0.714$. These differences reflect supervised accessibility rather than a pre-existing, native decision rule, and the most influential token role depends on the task. Under the evaluated score scales, Qwen silent-feature ablation is $24-63$ times more probe-sensitive, whereas routed-feature patching on literal yes/no tasks is $16-140$ times more output-sensitive. Calibration-only routing recovers $93.3$% of the mean gap, and probe-distilled LoRA improves native predictions, although shared multi-task adaptation causes negative transfer. A case study of Gemma-3-12B on Facebook Hateful Memes finds a distributed rank-32 image-prompt interaction, reaching $0.756$ versus $0.685$ native macro-F1. Robustness controls show that the signal extends beyond English, is not explained solely by accompanying OCR, and depends on paired visual evidence. Thus, routing, rather than representation alone, is a recurring bottleneck in harmful meme classification.
Tags
Links
- Source: https://arxiv.org/abs/2609.18860v1
- Canonical: https://arxiv.org/abs/2609.18860v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
136,546 characters extracted from source content.
Expand or collapse full text
Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection Girish A. Koushik †thanks: Corresponding Author Diptesh Kanojia Affiliation: Nature-Inspired Computing & Engineering Affiliation: Department of Computer Science Affiliation: University of Surrey Affiliation: Guildford, United Kingdom Email: g.koushik,d.kanojia@surrey.ac.uk Helen Treharne Affiliation: Surrey Centre for Cyber Security Affiliation: Department of Computer Science Affiliation: University of Surrey Affiliation: Guildford, United Kingdom Email: h.treharne@surrey.ac.uk Abstract When a large vision-language model misclassifies a harmful meme, the failure may reflect missing internal evidence or an inability to route represented evidence to its output. We distinguish these cases in Gemma-3 and Qwen3.5 using sparse autoencoders, role-conditioned probes, causal interventions, and recovery experiments across six harmful content benchmarks, with additional Spanish and Hindi-English code-mixed evaluations. Sparse readouts outperform native prediction on all six primary binary tasks: Qwen averages 0.7400.740 versus 0.4320.432 native macro-F1, while residual reconstruction reaches 0.4860.486, whereas Gemma improves from 0.5320.532 to 0.7140.714. These differences reflect supervised accessibility rather than a pre-existing, native decision rule, and the most influential token role depends on the task. Under the evaluated score scales, Qwen silent-feature ablation is 24–63×24--63× more probe-sensitive, whereas routed-feature patching on literal yes/no tasks is 16–140×16--140× more output-sensitive. Calibration-only routing recovers 93.393.3% of the mean gap, and probe-distilled LoRA improves native predictions, although shared multi-task adaptation causes negative transfer. A case study of Gemma-3-12B on Facebook Hateful Memes finds a distributed rank-32 image-prompt interaction, reaching 0.7560.756 versus 0.6850.685 native macro-F1. Robustness controls show that the signal extends beyond English, is not explained solely by accompanying OCR, and depends on paired visual evidence. Thus, routing, rather than representation alone, is a recurring bottleneck in harmful meme classification. Disclaimer: This paper includes references to content that some readers may find triggering or upsetting, consistent with the nature of the task. 1 Introduction Memes compress visual content, written language, social context, and cultural implication into a single communicative unit. Their harmfulness often lies not in the image or text alone, but in the interaction between them. The same image can become hateful under a different caption, while the same caption can become benign or harmful under a different visual context. Benchmarks such as the Hateful Memes Challenge make this ambiguity explicit through benign confounders designed to defeat unimodal shortcuts (Kiela et al., 2020). Yet strong performance from a nominally multimodal model does not, by itself, establish that the model has learned the relevant cross-modal relationship (Hessel and Lee, 2020). Harmful-meme detection therefore provides a demanding setting in which to study not only whether a model produces the correct label, but also what information its internal computation represents and ultimately uses. Large vision-language models (LVLMs) are increasingly used as general-purpose classifiers through prompting, constrained decoding, or label scoring. When such a model returns an incorrect harmfulness label, however, the output alone leaves two distinct explanations unresolved. The model may have failed to represent the relevant evidence, or it may have represented that evidence internally but failed to route it into the final decision, and the distinction between these explanations matters. A representational failure requires better perception, multimodal integration, or pretraining. A readout failure instead suggests that the information is already available and may be recoverable through a better decision pathway. Related discrepancies between internal encoding and external behavior have been observed in language models (Orgad et al., 2025), but remain poorly understood for multimodal safety judgments. Sparse autoencoders (SAEs) offer a useful coordinate system for separating these possibilities. By decomposing dense activations into sparse feature directions, SAEs make it possible to train lightweight readouts, compare feature importance with output alignment, and intervene on selected directions (Huben et al., 2024; Lieberum et al., 2024; Marks et al., 2025). Predictive sparse features are not automatically causal: an interpretable probe does not show that the native model uses the same features. A complete account must connect four levels of evidence: whether harmful-content information is decodable, where it appears across token roles, whether the relevant directions influence the native output, and whether the signal can be converted into improved model behavior. We study these questions across Gemma-3 and Qwen3.5 model families using six benchmarks covering hate speech, harmfulness, misogyny, and offensiveness, followed by Spanish and Hindi-English code-mixed robustness experiments. We compare native constrained prediction with residual-reconstruction hooks and supervised readouts over public, residual, and cross-layer sparse representations. We then localize the signal across prompt, image, generated, and full-sequence states, contrasting probe-important features with independently selected output-routed features and testing both sets through causal interventions. Finally, we evaluate whether the recovered signal can be rerouted directly at the output or distilled into the LVLM itself. Contributions. We make four contributions. 1. Across Gemma-3 and Qwen3.5, we show that sparse readouts outperform native prediction on all six binary tasks; the most informative token role varies by task, while fine-grained labels remain substantially harder. 2. In Qwen, interventions disentangle probe-discriminative SAE directions from those routed to the model’s outputs: ablation of silent features is 24–63×24--63× more sensitive on probes, while patching routed features is 16–140×16--140× more sensitive on native outputs. This separation is actionable: calibration-only routing recovers 93.3%93.3\% of the mean readout gap, and probe-distilled LoRA improves native prediction, although shared multi-task adaptation causes negative transfer. Gemma shows the same geometric split and probe-side separation. 3. Furthermore, we demonstrate that Gemma-3-12B reveals a model-specific, distributed rank-32 image-prompt interaction on FHM, reaching 0.7560.756 versus 0.6850.685 native macro-F1. 4. Finally, through cross-lingual and code-mixed tests, along with OCR ablations and image perturbation studies, we indicate that the effect generalizes beyond English, cannot be accounted for by the provided OCR, and relies substantially on the accompanying visual evidence. Figure 1: Residual-SAE construction and readout pathways. A. A frozen base SAE reconstructs layer-ℓ states, while a trainable SAE models the normalized reconstruction residual, yielding base and joint reconstructions. B. The resulting representation supports native scoring, residual-reconstruction hooks, sparse probing, calibrated output routing, and probe-distilled LoRA. Snowflakes and flames denote frozen and trainable components, respectively. 2 Background and Experimental Framework 2.1 Problem Formulation and Related Work Problem formulation. For each task d∈d , an example is an image-text pair x=(v,t)x=(v,t), where v is the meme image and t contains the task instruction and any supplied OCR; the gold label is y∈dy _d. A frozen LVLM produces a native prediction y^nat(d)(x) y^(d)_nat(x) through constrained decoding or task-specific label scoring. From layer ℓ and token role s, we extract a representation ϕℓ,s(x) _ ,s(x), defined in Section 2.2, and fit a supervised probe using only the task training split. On the locked reporting split, we define the empirical readout gap as Δ^ℓ,s,d=MacroF1(^probe,ℓ,s,d,d)−MacroF1(^nat,d,d), _ ,s,d=MacroF1\! ( y_probe, ,s,d,y_d )-MacroF1\! ( y_nat,d,y_d ), (1) where dy_d denotes the reporting labels and the two prediction vectors are evaluated on identical examples. A positive Δ^ℓ,s,d _ ,s,d means that the selected probe can access label-relevant information beyond what is expressed by the native output. It does not establish model belief, identify a unique causal mechanism, or constitute a like-for-like comparison between equally trained classifiers (Hewitt and Liang, 2019; Belinkov, 2022). We therefore separate three questions: decodability, whether the label is accessible from the representation; routing, whether the corresponding directions influence the native output; and recoverability, whether this signal can improve the model’s own predictions. Harmful meme detection. Harmful memes are compositional objects whose meaning can depend on the juxtaposition of visual entities, overlaid text, sarcasm, cultural references, and the identity of the targeted group. The Hateful Memes Challenge made this property explicit through benign confounders, for which altering either the image or text can reverse the label while preserving the other modality (Kiela et al., 2020). This construction penalizes unimodal priors, although strong multimodal performance alone is not sufficient to demonstrate that a model has learned a genuine cross-modal interaction (Hessel and Lee, 2020). Prior work on harmful-meme detection emphasizes stronger multimodal fusion/alignment, uncertainty- and retrieval-guided training, and LVLM prompting/context/rationale supervision, but typically optimizes the final detector without disentangling representational failures from readout failures (Kumar and Nandakumar, 2022; Koushik et al., 2025; Yang et al., 2024; Huang et al., 2025; Koushik et al., 2026; Kmainasi et al., 2025). Probing, sparse representations, and readout gaps. Diagnostic probes test which properties are recoverable from frozen states, while sparse autoencoders (SAEs) provide an overcomplete, sparsely active coordinate system for analyzing those states (Hewitt and Liang, 2019; Belinkov, 2022; Huben et al., 2024; Marks et al., 2025). Prior work has identified structured multimodal features, failure-predictive internal states, and discrepancies between internal representations and emitted answers in CLIP, LVLM hallucination, and language-model truthfulness settings (Goh et al., 2021; Kogilathota et al., 2026; Orgad et al., 2025). We instead ask whether role-conditioned directions that discriminate harmful-content labels are the same directions that control the LVLM’s native label output. We compare dense, base-SAE, residual-SAE, and cross-layer readouts, and reserve claims about native model use for output-alignment tests and causal interventions. Because our probes are supervised and harmfulness labels depend partly on social conventions, we use readout gap narrowly to mean held-out label accessibility, not latent belief or knowledge. Steering, output control, and distillation. Hidden-state steering and decoding-time control can make internal signals behaviorally effective, while knowledge distillation and low-rank adaptation can absorb an external readout into model parameters (Li et al., 2023; Yang and Klein, 2021; Hinton et al., 2015; Hu et al., 2021). We evaluate three complementary recovery paths: residual reconstruction, calibrated direct logit routing, and probe-distilled LoRA. We also compare dedicated and shared adapters because heterogeneous multi-task adaptation can produce negative transfer (Standley et al., 2020). See Appendix A for expanded related work. 2.2 Sparse Representations, Token Roles and Readout Mechanisms Let hℓ,t(x)∈ℝdℓh_ ,t(x) ^d_ denote the residual-stream state produced by the frozen LVLM at layer ℓ and token position t. Sparse autoencoders provide a sparse coordinate system for these states (Huben et al., 2024; Gao et al., 2025). A frozen public/base SAE is applied first, and a second SAE is trained on its normalized reconstruction residual: zℓ,tb z^b_ ,t =Eb(hℓ,t), =E_b(h_ ,t), h^ℓ,tb h^\,b_ ,t =Db(zℓ,tb), =D_b(z^b_ ,t), rℓ,t r_ ,t =hℓ,t−h^ℓ,tb, =h_ ,t- h^\,b_ ,t, (2) zℓ,tr z^r_ ,t =JumpReLU(Er(N(rℓ,t))), =JumpReLU\! (E_r\! (N(r_ ,t) ) ), r^ℓ,t r_ ,t =N−1(Dr(zℓ,tr)), =N^-1\! (D_r(z^r_ ,t) ), h^ℓ,tjoint h^\,joint_ ,t =h^ℓ,tb+r^ℓ,t. = h^\,b_ ,t+ r_ ,t. Here, N uses statistics computed from the residual-SAE training data. The public/base dictionary remains frozen and only (Er,Dr)(E_r,D_r) is trained. We evaluate the base and residual sparse codes as alternative probe representations, while h^ℓ,tjoint h^\,joint_ ,t is used by the residual reconstruction hook. Figure 1 summarizes the representation construction in panel A and the downstream evaluation and recovery paths in panel B. Objectives, checkpoint source, and complete readout definitions are given in Appendix B. Sparse dictionaries. For Gemma, we use public dictionaries trained on instruction-tuned activations and, where selected on calibration data, sparse crosscoders that map several layers into a shared code with layer-specific decoders (Lieberum et al., 2024; Lindsey et al., 2024). For Qwen, the six headline analyses use the same public layer-20 SAE trained on base-model activations (Deng et al., 2026); we retain a separately trained residual SAE as an ablation. Here, base SAE denotes the first dictionary in Equation 2, whereas base-model SAE describes the training regime of the Qwen checkpoint. The uneven SAE regimes reflect public-checkpoint availability. No instruction-tuned SAE matched the selected Qwen3.5-9B backbone. We therefore perform controlled comparisons on within-family native–hook–probe differences and interpret cross-family agreement only as replication of the readout-gap phenomenon, not as a model or SAE leaderboard. Token roles. We partition positions by their origin into prompt/OCR positions promptT_prompt, image-origin positions imageT_image, and autoregressively generated positions genT_gen. We additionally use pre=prompt∪imageT_pre=T_prompt _image and all=pre∪genT_all=T_pre _gen. These are source labels, not modality-pure states. Contextual processing can place visual evidence in prompt-position states and linguistic evidence in image-position states. Given the token-level feature vector zℓ,t(x)z_ ,t(x) from the selected sparse dictionary, we construct one representation per meme and role by feature-wise max pooling: [ϕℓ,s(x)]j=maxt∈s(x)[zℓ,t(x)]j,s∈prompt,image,gen,pre,all. [ _ ,s(x) ]_j= _t _s(x) [z_ ,t(x) ]_j, s∈\prompt,image,gen,pre,all\. (3) This records whether feature j is strongly active anywhere within a role. The pre-generation representation tests accessibility before decoding, generated-token states describe computation after decoding has begun, and the all-token representation captures the full trajectory. Generated and all-token performance therefore cannot, by itself, establish that the same information was available before answer generation. Dense controls use the corresponding role-conditioned hidden states rather than sparse codes. Readout pathways. Starting from the same frozen LVLM, we compare four paths. The native path uses the model’s constrained label scores. The sparse probe applies a supervised readout to ϕℓ,s(x) _ ,s(x) and measures label decodability. The residual reconstruction hook blends h^ℓ,tjoint h^\,joint_ ,t into the residual stream and then continues through the model’s original upper layers, testing whether reconstruction alone makes the signal usable by the native pathway. Finally, direct logit routing adds a calibrated probe score at the output rather than passing that correction through the upper layers; the native forward pass is still required. Section 5 defines this router and the subsequent probe-distilled LoRA procedure. Together, these paths distinguish information that is accessible, information that survives the model’s native routing, and information that can be converted into improved model behavior. 2.3 Datasets and Evaluation Setup We evaluate six primary binary multimodal harmful-content tasks: CrisisHateMM, FHM, MAMI-A, HarMeme, MMHS150K, and MultiOFF (Bhandari et al., 2023; Kiela et al., 2020; Gasparini et al., 2022; Pramanick et al., 2021; Gomez et al., 2020; Suryawanshi et al., 2020). Table 1 reports the effective train, validation, and test counts used by the primary binary pipeline. Experiment-specific subdivisions of these splits for calibration and reporting are described in Appendix C. For HarMeme, the two harmful classes are merged into Harmful; for MMHS150K, the five hate subtypes are merged into Hate. Their original fine-grained taxonomies and CrisisHateMM target attribution are analyzed separately. Multilingual robustness uses fixed internal holdouts from EXIST-2025 (500 examples, equally divided between English and Spanish) and MultiBully (1,000 Hindi–English examples) (Plaza et al., 2026; Maity et al., 2022); neither contributes to the six-task mean. Table 1: Primary binary tasks and effective split sizes. Counts are usable examples after fixed preprocessing and before any experiment-specific calibration/reporting subdivision. Boldface identifies the locked reporting split. Dataset Binary target Train Validation Test CrisisHateMM11 1 The released CrisisHateMM test directory does not provide usable gold labels. We therefore treat its labeled validation set as the final evaluation split and never use it for fitting the reported probe. hate / no hate 3,600 443† – FHM hateful / not hateful 7,938 469 1000 MAMI misogynous / not misogynous 9,000 1,000 1,000 HarMeme harmful / not harmful 3,013 177 354 MMHS150K Hate / NotHate 134,823 5,000 10,000 MultiOFF offensive / non-offensive 445 149 149 All trainable components use task training data only. Layers, token roles, representations, checkpoints, thresholds, and routing coefficients are selected on training or disjoint calibration data and frozen before reporting. Native predictions use constrained task-specific label scoring; originally multiclass datasets are scored in their full label space before the fixed binary collapse. Macro-F1 is the primary metric, and all systems within a model family are compared on identical example IDs. Full dataset construction, split rules, decoding details, secondary metrics, and comparison scope are provided in Appendix C. 3 Internal Representations versus Generative Head Broad Evaluation Comparison. Table 2 compares locked macro-F1 for the native LVLM, the selected sparse readout, and, for Qwen, the residual reconstruction hook. Sparse readouts outperform native prediction on every task in both model families. Gemma improves from 0.5320.532 to 0.7140.714 mean macro-F1 (+0.182+0.182), while Qwen improves from 0.4320.432 to 0.7400.740 (+0.308+0.308). Qwen provides the more controlled comparison because all six probes use the same public layer-20 SAE and readout family; Gemma replicates the pattern under a distinct model and sparse-representation regime. The Qwen hook improves five tasks and raises the mean to 0.4860.486, but recovers only 17.6%17.6\% of the native-to-probe gap. Sparse reconstruction therefore improves native behavior without making most of the accessible signal usable by the original output pathway. As defined in Section 2.1, these gains measure supervised accessibility rather than a like-for-like comparison between an untuned native classifier and a trained probe. Per-task analysis, base-versus-residual SAE comparisons, and fine-grained results are reported in Appendices D and E. Table 2: Macro-F1 scores on the six primary binary tasks. Gemma uses its calibration-selected sparse readout while Qwen uses the same public layer-20 base-SAE probe across tasks. The hook modifies Qwen’s native pathway through residual reconstruction. Means are unweighted across tasks. Gemma-3-4B-IT Qwen3.5-9B-Base Dataset Native Sparse Native Hook SAE CrisisHateMM 0.606 0.819 0.310 0.349 0.860 FHM 0.648 0.703 0.637 0.692 0.712 MAMI 0.409 0.731 0.408 0.521 0.755 HarMeme 0.719 0.776 0.394 0.394 0.810 MMHS150K 0.292 0.630 0.524 0.556 0.587 MultiOFF 0.519 0.623 0.320 0.404 0.716 Mean 0.532 0.714 0.432 0.486 0.740 Where is the Signal? Token-Role and Representation Ablations. Using the calibration-only protocol of Section 2.3, role-conditioned probes reveal no universal locus of harmful-content information. On Gemma, image-position features outperform generated features on CrisisHateMM and HarMeme, whereas generated features are stronger on MAMI, MMHS150K, and MultiOFF; FHM is nearly tied. The largest shift occurs on MMHS150K, where generated rather than image states raise macro-F1 from 0.5310.531 to 0.6170.617 for the binary task and from 0.2320.232 to 0.4680.468 for the six-class task. MAMI instead peaks at prompt/OCR states (0.8450.845). Because generated states are observed only after decoding begins, they indicate where the computation becomes linearly separable, not necessarily what was available before the decision. Crucially, image or pre-generation readouts still exhibit large locked readout gaps on CrisisHateMM and HarMeme, while the selected FHM readout uses prompt–image cross-layer features. The signal is therefore accessible before generation, but its separability can increase during decoding. Sparse factorization is similarly task-dependent. Relative to probes over corresponding dense hidden states, it helps substantially on CrisisHateMM (+0.283+0.283), modestly on FHM and six-class MMHS150K, and negligibly on MAMI, HarMeme, and MultiOFF; random sparse controls remain near chance. SAEs therefore expose rather than create most of the harmful-content signal, while providing the feature basis needed for the alignment and intervention tests in Section 4.1. FHM additionally resists single-role and simple pooled interaction readouts, motivating the pair-aware low-rank analysis in Section 4.2. Full role sweeps, dense-state controls, and FHM ablations are reported in Appendix F. 4 Mechanistic Structure of Readout Gap Figure 2: Decodability and native-output control dissociate in Qwen. A. Silent-feature knockout primarily changes the probe (n=15n=15 eligible examples per task). Asterisks denote diagnostic yes/no margins, not the native task decision. B. Routed-feature patching primarily changes the native yes/no margin (n=5n=5 donor-target pairs per task). Bars show mean absolute changes; labels give the larger-to-smaller sensitivity ratio. Ratios depend on score scaling and do not measure a mediated fraction or improved correctness. Full results and eligibility rules appear in Appendix G. 4.1 Discriminative versus Routed Directions A strong sparse probe need not rely on the directions that control the LVLM’s own answer. Let djd_j be the decoder direction of SAE feature j, and let wd,jw_d,j be its task-d probe coefficient, oriented so that positive scores favor the harmful class. We compare this discriminative importance with direct access to the yes/no output channel: uyn u_yn =WUf⊤(eyes−eno), =W_U^f (e_yes-e_no ), aj a_j =dj⊤uyn, =d_j u_yn, (4) d _d =j∈TopKk(|wd,⋅|):|aj|<τ, = \j _k\! (|w_d,·| ):|a_j|<τ \, ℛyn _yn =TopKk(|a⋅|) =TopK_k\! (|a_·| ) where WUfW_U^f is the effective unembedding with final normalization folded into it, τ=0.10τ=0.10, and k=20k=20. The task-specific set dS_d contains probe-important but output-silent features, whereas ℛynR_yn contains features selected only for direct access to the native yes/no channel. The yes/no anchor is the actual native decision channel for literal yes/no tasks and only a diagnostic for tasks evaluated by full-label scoring. Moreover, aja_j measures an immediate unembedding effect rather than a complete causal effect through the remaining transformer blocks (Elhage et al., 2021; Belrose et al., 2023). For Qwen’s public base SAE, the region containing both high probe importance and high direct output alignment is empty on five of the six analyzed tasks and contains only 1%1\% of the analyzed MAMI feature records. Figure 2 shows the corresponding causal sensitivity pattern. Knocking out silent features changes the probe score 24.224.2–62.8×62.8× more than the monitored yes/no margin, whereas donor-patching routed features changes the native margin 16.216.2–140.2×140.2× more than the probe across the four tasks with matched literal yes/no decoding. These ratios depend on the two score scales and are not scale-invariant mediation fractions. Under the tested interventions, discriminative features strongly affect the external readout but weakly affect the monitored native margin, while independently selected routed features show the reverse pattern. Routed interventions establish output control, not improved correctness (Vig et al., 2020; Meng et al., 2022; Marks et al., 2025). Gemma replicates the static and probe-side separation: its routed and probe-derived feature sets have zero overlap on all four analyzed tasks, and Jacobian-based (Gurnee et al., 2026) sensitivity places the strongest probe directions outside the dominant output-routing set. Its native-output mediation is less uniform, so we do not claim a universal orthogonal decomposition. The conclusion is local to the analyzed layers, SAE dictionaries, features, and output anchors. Appendix G reports full task-level effects, sample counts, label-score caveats, and patching controls. Figure 3: Confounders and the bilinear readout. A. Holding the background image or supplied text fixed can reverse the gold label when the other component changes; symbols indicate gold classes, not model correctness. B. Two role-conditioned projections form an inner product, scaled by γ and added to the gated baseline. Image and prompt denote token origins, not modality-pure information. Quantitative rank and test results appear in Figure 6 in Appendix H. 4.2 Low-Rank Cross-Modal Structure on FHM We analyze Gemma-3-12B separately because this experiment changes both model scale and readout family and is therefore excluded from the six-task mean in Section 3. FHM’s benign confounders can reverse the label while holding the image or text fixed, so combining two role-specific predictors does not by itself establish a cross-modal interaction (Kiela et al., 2020; Hessel and Lee, 2020). From the layer-31 sparse representation, we construct standardized max-pooled image-position and prompt/OCR vectors ximg,xprompt∈ℝ128x_img,x_prompt ^128 using features selected by a train-only confounder audit. These labels denote token provenance rather than modality-pure states. We augment the pair-aware gated baseline sgates_gate with an explicit rank-r interaction: sr(x) s_r(x) =sgate(x)+γximg⊤UV⊤xprompt =s_gate(x)+γ x_img UV x_prompt (5) =sgate(x)+γ∑q=1r(uq⊤ximg)(vq⊤xprompt),U,V∈ℝ128×r. =s_gate(x)+γ _q=1^r (u_q x_img ) (v_q x_prompt ), U,V ^128× r. This parameterization uses 256r256r interaction parameters rather than a full 1282128^2 matrix (Kim et al., 2016). Training combines sample-level classification with ranking losses on opposite-label same-image and same-text pairs; all choices are made before the locked test split is evaluated. Figure 3 illustrates the confounder problem and the gated-bilinear computation; Figure 6 in Appendix H plots the quantitative comparison. Rank 8 is unstable (0.7110±0.03520.7110± 0.0352 validation macro-F1), whereas rank 32 is more stable (0.7488±0.01200.7488± 0.0120) and approaches the single-seed rank-128 result (0.76210.7621). The rank-32 mean improves over the gated pairwise baseline by 0.01220.0122. The calibration-selected rank-32 checkpoint reaches 0.75600.7560 on the reported test-seen evaluation, compared with 0.68530.6853 for native Gemma-3-12B, a gain of 0.07070.0707 with a paired bootstrap 95% confidence interval of [0.0461,0.0954][0.0461,0.0954]. The sample-count provenance is noted in Figure 6. This larger test gain compares the complete pair-aware readout with the native model; it is not the isolated contribution of the bilinear term. On the validation matched-pair audit, the selected model assigns the higher score to the hateful member in 84.5%84.5\% of same-image pairs and 84.7%84.7\% of same-text pairs. The interaction is distributed rather than attributable to one feature pair: 11 of the 32 factors have measurable leave-one-out effects, but removing any single factor changes validation macro-F1 by at most 0.0070.007, while a within-role quadratic control adds only 0.0040.004. The result is also representation-specific: analogous bilinear readouts underperform their simpler controls for Gemma-3-4B and Qwen. We therefore claim evidence for a distributed rank-32 image-by-prompt interaction in the analyzed Gemma-3-12B layer-31 representation, not a universal FHM mechanism. Complete objectives, rank sweeps, matched-pair metrics, statistical tests, factor interventions, and negative controls appear in Appendix H. 5 Recovering and Stress-Testing the Latent Signal Direct Logit Routing and Probe Distillation. The causal separation in Section 4.1 suggests two recovery paths: bypass the native routing bottleneck at the output, or absorb the external readout into the model. For task d, let md(x)m_d(x) be the signed native decision margin and ud(x)u_d(x) the frozen probe score. We use the calibration-only router mdroute(x)=md(x)+βdud(x)−μdσd+ϵm^route_d(x)=m_d(x)+ _d u_d(x)- _d _d+ε (6) where μd _d, σd _d, and βd _d are estimated on a disjoint calibration partition and frozen before reporting. On matched Qwen reporting subsets, routing improves all six binary tasks, raising mean macro-F1 from 0.44050.4405 to 0.72040.7204, close to the probe ceiling of 0.74040.7404. It therefore recovers 93.3%93.3\% of the native-to-probe gap. FHM is the only task on which the combined score slightly exceeds the probe alone. Four tasks select the largest tested coefficient, βd=8 _d=8, so the experiment establishes that bypassing the native pathway is effective, not that the coefficient sweep found an interior optimum. To remove the SAE and probe at inference, we distill the frozen probe distribution into Gemma-3-4B-IT using LoRA, gold-label supervision, probe KL supervision, and, where available, matched-pair ranking (Hinton et al., 2015; Hu et al., 2021). A dedicated FHM adapter improves the model’s test-seen macro-F1 from 0.64820.6482 to 0.71380.7138. A shared seven-task adapter raises its own unweighted mean from 0.46420.4642 to 0.56860.5686, but underperforms the dedicated FHM adapter on the same development set (0.65480.6548 versus 0.70950.7095) and reduces six-class MMHS150K from 0.37270.3727 to 0.18860.1886 while leaving its binary result nearly unchanged. Probe information can therefore be internalized, but a universal adapter can erase task-specific structure through negative transfer (Standley et al., 2020). Full per-task routing results, objectives, run configurations, and sampling schedules appear in Appendix I. Multilingual and Visio-Textual Robustness. We test whether the readout gap is confined to English or explained by provided OCR through visual credit audit (Liu et al., 2026). On the fixed EXIST holdouts, Qwen cross-language and bilingual probes reach 0.74690.7469–0.81210.8121 macro-F1. Gemma representations remain above chance, but their probe-over-native advantage changes sign across split seeds, so we claim cross-lingual decodability rather than stable superiority. MultiBully gives the clearer cross-model replication: Qwen’s all-token probe reaches 0.72590.7259, compared with 0.34010.3401 natively, while Gemma’s residual-SAE probe reaches 0.6892±0.01430.6892± 0.0143 macro-F1 across five deterministic seeds and improves over native prediction by 0.0915±0.02260.0915± 0.0226. Qwen silent-feature knockout remains probe-dominant on English and Spanish EXIST and on MultiBully (48.548.5–90:190:1), although these checks use only 12–15 eligible examples. Explicit OCR contributes but does not explain the result: removing it changes the strongest text-bearing readouts by at most 0.06710.0671 macro-F1 and changes the MultiBully image-position probe by only 0.00110.0011. In contrast, deterministic same-split image permutation reduces probe macro-F1 by 0.1480.148–0.2060.206 and routed macro-F1 by 0.1600.160–0.2920.292 on every audited dataset. Between 55.8%55.8\% and 88.8%88.8\% of originally correct probe or router predictions become incorrect after image blanking or shuffling, compared with 0.0%0.0\%–38.1%38.1\% for the native classifier. The recovered signal therefore extends beyond English and depends materially on paired visual evidence. These controls do not isolate modality-pure states or a specific cross-modal circuit: the no-OCR condition removes only injected OCR, while image perturbations provide an operational dependence test (Hessel and Lee, 2020). Full split-wise results, metric definitions, matched-pair controls, and perturbation tables appear in Appendix J. 6 Conclusion and Future Work Our results distinguish decodability, native routing, and correctness. Across two LVLM families and six tasks, sparse readouts recover label information weakly expressed by native output. Qwen interventions separate probe-discriminative from output-routed directions (Figure 2); direct routing recovers 93.3%93.3\% of the mean gap. LoRA internalizes part of this signal but can introduce negative transfer. SAEs provide a coordinate system for these tests, while the Gemma-3-12B FHM result supports a representation-specific distributed interaction. Limitations. The readout gap is not an apples-to-apples comparison: probes receive task labels while the native LVLM remains frozen. The mechanistic claims are local to the selected layers, dictionaries, features, and output anchors; some interventions use small eligible subsets and out-of-distribution patching. Token roles identify position provenance rather than modality-pure information, generated-token probes are post-generation readouts, and the FHM interaction does not transfer to every model. Multilingual results rely on internal holdouts, while rare fine-grained classes remain poorly resolved. Improved macro-F1 therefore does not establish readiness for autonomous moderation. Future work. Future work should test larger models, additional layers and dictionaries, and broader multilingual and fine-grained datasets. Position-matched patching and path-level tracing could identify how evidence reaches the output; task-grouped adapters and subtype-preserving objectives could reduce negative transfer. Evaluation should include calibration, subgroup errors, abstention, and human review. References Ashok et al. (2025) D. Ashok, A. Chaubey, H. J. Arai, J. May, and J. Thomason Can VLMs recall factual associations from visual references?. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 15691–15708. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: Appendix A. Belinkov (2022) Y. Belinkov Probing classifiers: promises, shortcomings, and advances. Computational Linguistics 48 (1), p. 207–219. External Links: Link, Document Cited by: §L.2, §2.1, §2.1. Belrose et al. (2023) N. Belrose, I. Ostrovsky, L. McKinney, Z. Furman, L. Smith, D. Halawi, S. Biderman, and J. Steinhardt Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112. Cited by: Appendix A, Appendix G, §4.1. Bhandari et al. (2023) A. Bhandari, S. B. Shah, S. Thapa, U. Naseem, and M. Nasim Crisishatemm: multimodal analysis of directed and undirected hate speech in text-embedded images from russia-ukraine conflict. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 1993–2002. Cited by: Appendix C, §2.3. Bui et al. (2025) M. D. Bui, K. von der Wense, and A. Lauscher Multi3^3Hate: multimodal, multilingual, and multicultural hate speech detection with vision–language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, p. 9714–9731. External Links: Link, Document, ISBN 979-8-89176-189-6 Cited by: Appendix A. Burns et al. (2024) C. Burns, H. Ye, D. Klein, and J. Steinhardt Discovering latent knowledge in language models without supervision. External Links: 2212.03827, Link Cited by: Appendix A. Deng et al. (2026) B. Deng, X. Wang, Y. Wang, Y. Wan, Y. Ma, B. Yang, H. Wei, J. Tang, H. Lin, R. Gao, T. Li, Q. Cao, X. Ren, X. Deng, A. Yang, F. Huang, D. Liu, and J. Zhou Qwen-scope: turning sparse features into development tools for large language models. External Links: 2605.11887, Link Cited by: Appendix B, §2.2. Elhage et al. (2021) N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah A mathematical framework for transformer circuits. Transformer Circuits Thread. Note: https://transformer-circuits.pub/2021/framework/index.html Cited by: Appendix G, §4.1. Gao et al. (2025) L. Gao, T. Dupre la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu Scaling and evaluating sparse autoencoders. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 26721–26754. External Links: Link Cited by: §L.1, §2.2. Gasparini et al. (2022) F. Gasparini, G. Rizzi, A. Saibene, and E. Fersini Benchmark dataset of memes with text transcriptions for automatic detection of multi-modal misogynistic content. Data in brief 44, p. 108526. Cited by: Appendix C, §2.3. Goh et al. (2021) G. Goh, N. C. †, C. V. †, S. Carter, M. Petrov, L. Schubert, A. Radford, and C. Olah Multimodal neurons in artificial neural networks. Distill. Note: https://distill.pub/2021/multimodal-neurons External Links: Document Cited by: Appendix A, Appendix A, §2.1. Gomez et al. (2020) R. Gomez, J. Gibert, L. Gomez, and D. Karatzas Exploring hate speech detection in multimodal publications. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, p. 1470–1478. Cited by: Appendix C, §2.3. Gurnee et al. (2026) W. Gurnee, N. Sofroniew, A. Pearce, M. Piotrowski, I. Kauvar, R. Chen, A. Soligo, P. Bogdan, E. Ong, R. Wang, B. Thompson, D. Abrahams, S. Kantamneni, E. Ameisen, J. Batson, and J. Lindsey Verbalizable representations form a global workspace in language models. Transformer Circuits Thread. External Links: Link Cited by: Appendix G, §4.1. Hessel and Lee (2020) J. Hessel and L. Lee Does my multimodal model learn cross-modal interactions? it’s harder to tell than you might think!. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), B. Webber, T. Cohn, Y. He, and Y. Liu (Eds.), Online, p. 861–877. External Links: Link, Document Cited by: Appendix J, Appendix F, §1, §2.1, §4.2, §5. Hewitt and Liang (2019) J. Hewitt and P. Liang Designing and interpreting probes with control tasks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), p. 2733–2743. Cited by: §L.2, §2.1, §2.1. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: §2.1, §5. Hossain et al. (2022) E. Hossain, O. Sharif, and M. M. Hoque MUTE: a multimodal dataset for detecting hateful memes. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing: Student Research Workshop, Y. Hanqi, Y. Zonghan, S. Ruder, and W. Xiaojun (Eds.), Online, p. 32–39. External Links: Link, Document Cited by: Appendix A. Hu et al. (2021) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen Lora: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Cited by: §L.2, §2.1, §5. Hua et al. (2025) Z. Hua, J. He, Z. Yao, T. Han, H. Guo, Y. Jia, and J. Fang Steering LVLMs via sparse autoencoder for hallucination mitigation. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 10808–10828. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: Appendix A. Huang et al. (2025) J. Huang, J. Pan, Z. Wan, H. Lyu, and J. Luo Evolver: chain-of-evolution prompting to boost large multimodal models for hateful meme detection. In Proceedings of the 31st International Conference on Computational Linguistics, O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, and S. Schockaert (Eds.), Abu Dhabi, UAE, p. 7321–7330. External Links: Link Cited by: §2.1. Huben et al. (2024) R. Huben, H. Cunningham, L. Smith, A. Ewart, and L. Sharkey Sparse autoencoders find highly interpretable features in language models. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024, p. 7827–7845. External Links: Link Cited by: §L.1, §1, §2.1, §2.2. Kiela et al. (2020) D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, and D. Testuggine The hateful memes challenge: detecting hate speech in multimodal memes. Advances in neural information processing systems 33, p. 2611–2624. Cited by: §L.4, Appendix C, Appendix F, §1, §2.1, §2.3, §4.2. Kim et al. (2016) J. Kim, K. On, W. Lim, J. Kim, J. Ha, and B. Zhang Hadamard product for low-rank bilinear pooling. arXiv preprint arXiv:1610.04325. Cited by: §L.4, §4.2. Kirk et al. (2021) H. Kirk, Y. Jun, P. Rauba, G. Wachtel, R. Li, X. Bai, N. Broestl, M. Doff-Sotta, A. Shtedritski, and Y. M. Asano Memes in the wild: assessing the generalizability of the hateful memes challenge dataset. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), A. Mostafazadeh Davani, D. Kiela, M. Lambert, B. Vidgen, V. Prabhakaran, and Z. Waseem (Eds.), Online, p. 26–35. External Links: Link, Document Cited by: Appendix A. Kmainasi et al. (2025) M. B. Kmainasi, A. Hasnat, M. A. Hasan, A. E. Shahroor, and F. Alam MemeIntel: explainable detection of propagandistic and hateful memes. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 30263–30279. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: Appendix A, §2.1. Kogilathota et al. (2026) S. A. Kogilathota, S. V. E. G, L. Sun, and J. Zhou HALP: detecting hallucinations in vision-language models without generating a single token. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), V. Demberg, K. Inui, and L. Marquez (Eds.), Rabat, Morocco, p. 6067–6085. External Links: Link, Document, ISBN 979-8-89176-380-7 Cited by: Appendix A, §2.1. Koushik et al. (2025) G. A. Koushik, D. Kanojia, and H. Treharne Towards a robust framework for multimodal hate detection: a study on video vs. image-based content. W ’25, New York, NY, USA, p. 2014–2023. External Links: ISBN 9798400713316, Link, Document Cited by: §2.1. Koushik et al. (2026) G. A. Koushik, H. Treharne, A. Joshi, and D. Kanojia TRACE: textual relevance augmentation and contextual encoding for multimodal hate detection. In Proceedings of the Fortieth AAAI Conference on Artificial Intelligence and Thirty-Eighth Conference on Innovative Applications of Artificial Intelligence and Sixteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’26/IAAI’26/EAAI’26. External Links: ISBN 978-1-57735-906-7, Link, Document Cited by: §2.1. Kumar and Nandakumar (2022) G. K. Kumar and K. Nandakumar Hate-CLIPper: multimodal hateful meme classification based on cross-modal interaction of CLIP features. In Proceedings of the Second Workshop on NLP for Positive Impact (NLP4PI), L. Biester, D. Demszky, Z. Jin, M. Sachan, J. Tetreault, S. Wilson, L. Xiao, and J. Zhao (Eds.), Abu Dhabi, United Arab Emirates (Hybrid), p. 171–183. External Links: Link, Document Cited by: §2.1. Li et al. (2023) K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg Inference-time intervention: eliciting truthful answers from a language model. Advances in neural information processing systems 36, p. 41451–41530. Cited by: Appendix A, §2.1. Lieberum et al. (2024) T. Lieberum, S. Rajamanoharan, A. Conmy, L. Smith, N. Sonnerat, V. Varma, J. Kramar, A. Dragan, R. Shah, and N. Nanda Gemma scope: open sparse autoencoders everywhere all at once on gemma 2. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, Y. Belinkov, N. Kim, J. Jumelet, H. Mohebbi, A. Mueller, and H. Chen (Eds.), Miami, Florida, US, p. 278–300. External Links: Link, Document Cited by: Appendix B, §1, §2.2. Lindsey et al. (2024) J. Lindsey, A. Templeton, J. Marcus, T. Conerly, J. Batson, and C. Olah Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread. Cited by: §L.1, Appendix B, §2.2. Liu et al. (2021) A. Liu, M. Sap, X. Lu, S. Swayamdipta, C. Bhagavatula, N. A. Smith, and Y. Choi DExperts: decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli (Eds.), Online, p. 6691–6706. External Links: Link, Document Cited by: Appendix A. Liu et al. (2026) F. Liu, Q. Qiu, L. Sun, N. Wei, H. Shen, and X. Cheng Visual credit audit for multimodal spatial reasoning. External Links: 2607.27069, Link Cited by: §L.4, §5. Lou et al. (2025) H. Lou, C. Li, J. Ji, and Y. Yang SAE-v: interpreting multimodal models for enhanced alignment. External Links: 2502.17514, Link Cited by: Appendix A. Maity et al. (2022) K. Maity, P. Jha, S. Saha, and P. Bhattacharyya A multitask framework for sentiment, emotion and sarcasm aware cyberbullying detection from multi-modal code-mixed memes. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, New York, NY, USA, p. 1739–1749. External Links: ISBN 9781450387323, Link, Document Cited by: Appendix J, Appendix C, §2.3. Marks et al. (2025) S. Marks, C. Rager, E. Michaud, Y. Belinkov, D. Bau, and A. Mueller Sparse feature circuits: discovering and editing interpretable causal graphs in language models. In International Conference on Learning Representations, Vol. 2025, p. 23888–23923. Cited by: Appendix A, Appendix G, §1, §2.1, §4.1. Mei et al. (2024) J. Mei, J. Chen, W. Lin, B. Byrne, and M. Tomalin Improving hateful meme detection through retrieval-guided contrastive learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 5333–5347. External Links: Link, Document Cited by: Appendix A. Mei et al. (2025) J. Mei, J. Chen, G. Yang, W. Lin, and B. Byrne Robust adaptation of large multimodal models for retrieval augmented hateful meme detection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 23806–23828. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: Appendix A. Meng et al. (2022) K. Meng, D. Bau, A. Andonian, and Y. Belinkov Locating and editing factual associations in gpt. Advances in neural information processing systems 35, p. 17359–17372. Cited by: Appendix G, §4.1. Orgad et al. (2025) H. Orgad, M. Toker, Z. Gekhman, R. Reichart, I. Szpektor, H. Kotek, and Y. Belinkov LLMs know more than they show: on the intrinsic representation of llm hallucinations. External Links: 2410.02707, Link Cited by: Appendix A, §1, §2.1. Pach et al. (2026) M. Pach, S. Karthik, Q. Bouniot, S. Belongie, and Z. Akata Sparse autoencoders learn monosemantic features in vision-language models. Advances in Neural Information Processing Systems 38, p. 95706–95742. Cited by: Appendix A. Parfenova et al. (2021) I. Parfenova, D. Elliott, R. Fernández, and S. Pezzelle Probing cross-modal representations in multi-step relational reasoning. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), A. Rogers, I. Calixto, I. Vulić, N. Saphra, N. Kassner, O. Camburu, T. Bansal, and V. Shwartz (Eds.), Online, p. 152–162. External Links: Link, Document Cited by: Appendix A. Plaza et al. (2026) L. Plaza, J. Carrillo-de-Albornoz, I. Arcos, P. Rosso, D. Spina, E. Amigó, J. Gonzalo, and R. Morante Overview of exist 2025: learning with disagreement for sexism identification and characterization in tweets, memes, and tiktok videos. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, J. Carrillo-de-Albornoz, A. García Seco de Herrera, J. Gonzalo, L. Plaza, J. Mothe, F. Piroi, P. Rosso, D. Spina, G. Faggioli, and N. Ferro (Eds.), Cham, p. 266–289. External Links: ISBN 978-3-032-04354-2 Cited by: Appendix J, Appendix C, §2.3. Pramanick et al. (2021) S. Pramanick, D. Dimitrov, R. Mukherjee, S. Sharma, M. S. Akhtar, P. Nakov, and T. Chakraborty Detecting harmful memes and their targets. arXiv preprint arXiv:2110.00413. Cited by: Appendix A, Appendix C, §2.3. Shen et al. (2025) S. Shen, J. Sun, Q. Huang, and S. Wang VL-sae: interpreting and enhancing vision-language alignment with a unified concept set. External Links: 2510.21323, Link Cited by: Appendix A. Standley et al. (2020) T. Standley, A. Zamir, D. Chen, L. Guibas, J. Malik, and S. Savarese Which tasks should be learned together in multi-task learning?. In International conference on machine learning, p. 9120–9132. Cited by: §2.1, §5. Su et al. (2025) X. Su, Y. Li, D. Inkpen, and N. Japkowicz A context-aware contrastive learning framework for hateful meme detection and segmentation. In Findings of the Association for Computational Linguistics: NAACL 2025, L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, p. 5216–5230. External Links: Link, Document, ISBN 979-8-89176-195-7 Cited by: Appendix A. Suryawanshi et al. (2020) S. Suryawanshi, B. R. Chakravarthi, M. Arcan, and P. Buitelaar Multimodal meme dataset (multioff) for identifying offensive content in image and text. In Proceedings of the second workshop on trolling, aggression and cyberbullying, p. 32–41. Cited by: Appendix C, §2.3. Vig et al. (2020) J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and S. Shieber Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems 33, p. 12388–12401. Cited by: Appendix G, §4.1. Wu et al. (2025) J. Wu, Y. Ding, G. Liu, T. Xia, Z. Huang, D. Sui, Q. Liu, S. Wu, L. Wang, and T. Tan SHARP: steering hallucination in LVLMs via representation engineering. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 14346–14361. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: Appendix A, Appendix A. Yang et al. (2024) C. Yang, Y. Liu, F. Zhu, J. Han, and S. Hu Uncertainty-guided modal rebalance for hateful memes detection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 4361–4371. External Links: Link, Document Cited by: §2.1. Yang and Klein (2021) K. Yang and D. Klein FUDGE: controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou (Eds.), Online, p. 3511–3535. External Links: Link, Document Cited by: Appendix A, §2.1. Appendix Contents 1 Introduction 2 Background and Experimental Framework 2.1 Problem Formulation and Related Work 2.2 Sparse Representations, Token Roles and Readout Mechanisms 2.3 Datasets and Evaluation Setup 3 Internal Representations versus Generative Head 4 Mechanistic Structure of Readout Gap 4.1 Discriminative versus Routed Directions 4.2 Low-Rank Cross-Modal Structure on FHM 5 Recovering and Stress-Testing the Latent Signal 6 Conclusion and Future Work References A Expanded Related Work B Sparse-Representation and Readout Details C Dataset and Evaluation Details D Additional Broad-Comparison Analysis E Fine-Grained Harmful-Content Classification F Token-Role and Representation Ablations G Feature Alignment and Causal Intervention Details H FHM Low-Rank Cross-Modal Readout Details I Direct-Routing and Probe-Distillation Details I.1 Calibration-Only Direct Logit Routing I.2 Probe-Distilled LoRA J Multilingual, OCR, and Visual-Credit Robustness Details K Qualitative Feature Examples L Glossary of Terms L.1 Representations and token roles. L.2 Readouts and output pathways. L.3 Discriminative and routed directions. L.4 Cross-modal and robustness analyses. Appendix A Expanded Related Work Hateful meme detection, localization, explanation, and intervention. The harmful-meme literature has expanded beyond binary classification. MOMENTA (Pramanick et al., 2021) jointly predicts harmfulness and the targeted social entity, while HateSieve localizes hateful regions and text-image evidence through context-aware contrastive learning (Su et al., 2025). MemeIntel introduces explanation-enhanced resources and jointly optimizes label detection and rationale generation (Kmainasi et al., 2025). These objectives are complementary to mechanistic analysis but answer a different question: generated rationales or localized regions describe a model’s claimed evidence, whereas our probes and interventions examine the internal coordinates that support a decision and whether those coordinates affect the native output. Domain, language, and cultural variation. Harmful-meme systems are sensitive to changes in meme format, OCR quality, language, and annotation culture. Models trained on the Hateful Memes Challenge degrade on naturally occurring Pinterest memes whose captions must be recovered through OCR (Kirk et al., 2021). MUTE supplies Bengali and Bengali-English code-mixed memes (Hossain et al., 2022), while Multi3Hate presents parallel memes in five languages with annotations from culturally distinct populations (Bui et al., 2025). Retrieval-guided and retrieval-augmented methods attempt to improve adaptation to new and evolving examples (Mei et al., 2024; Mei et al., 2025). Our multilingual experiments should consequently be read as evidence about representational transfer under fixed internal holdouts, not as a claim that harmfulness has a single language- or culture-independent decision boundary. Interpretability of vision-language representations. Early analyses of CLIP identified neurons responding to concepts across visual objects, written words, symbols, and associated entities (Goh et al., 2021). Later SAE-based approaches learned sparse features in vision encoders and used feature interventions to alter multimodal generation (Pach et al., 2026). SAE-V studies multimodal alignment and data quality through sparse features (Lou et al., 2025), while VL-SAE learns a shared concept set with modality-specific decoders (Shen et al., 2025). These methods focus primarily on feature coherence, alignment, or controllability. Our analysis instead asks whether task-discriminative sparse directions coincide with the directions through which a decoder-style LVLM communicates a harmful-content decision. Diagnostic probing and latent readout gaps. Multimodal probing has been used to examine relational information across visual contexts (Parfenova et al., 2021), while neuron-level analyses of CLIP identified units responsive to related concepts across objects, rendered text, and visual symbols (Goh et al., 2021). More recent LVLM work uses internal states to predict hallucination risk before generation, diagnose failures to connect visual references with factual associations, or intervene on hallucination-related activation patterns (Kogilathota et al., 2026; Ashok et al., 2025; Wu et al., 2025). Related language-model studies show that truth-related or answer-relevant information can be recovered from internal states even when the emitted response is incorrect (Burns et al., 2024; Belrose et al., 2023; Orgad et al., 2025). These results motivate separating representational availability from external behavior. Our setting is narrower because the probes are supervised and harmfulness labels are socially defined; we therefore test benchmark-label accessibility and then independently measure whether the identified directions affect the native output. Validity of sparse explanations. Neither sparse reconstruction nor a human-readable feature description guarantees causal relevance. Probe coefficients can reflect correlated features, SAE dictionaries can split one concept across multiple latents, and feature identities can vary with architecture, width, training data, or initialization. We therefore use sparse features as a testable coordinate system rather than as a complete ontology of the model. Raw-state controls test whether sparsification improves accessibility, base-versus-residual comparisons test dependence on the dictionary construction, and ablation or patching tests whether selected coordinates are functionally load-bearing. This follows the broader move from descriptive feature visualization toward sparse causal circuits (Marks et al., 2025). Alternative routes from representation to behavior. Representation engineering methods modify hidden states directly, while decoding-time methods modify output probabilities without changing the underlying computation (Li et al., 2023; Yang and Klein, 2021; Liu et al., 2021). In LVLMs, SHARP steers dense hallucination-related representations (Wu et al., 2025), whereas SAE-based hallucination mitigation intervenes on sparse latent directions (Hua et al., 2025). Our experiments include both a hidden-state reconstruction path and a logit-space path. The subsequent LoRA experiment asks a stricter question: whether the external readout can be internalized so that neither the SAE nor the probe is required during inference. Appendix B Sparse-Representation and Readout Details SAE objectives. For a token-level target uℓ,tu_ ,t, an SAE encoder-decoder pair produces zℓ,t=E(uℓ,t),u^ℓ,t=D(zℓ,t)z_ ,t=E(u_ ,t), u_ ,t=D(z_ ,t) and uses a reconstruction–sparsity objective of the form ℒSAE=x,t[‖uℓ,t(x)−D(E(uℓ,t(x)))‖22+λ‖E(uℓ,t(x))‖0].L_SAE=E_x,t [ \|u_ ,t(x)-D(E(u_ ,t(x))) \|_2^2+λ \|E(u_ ,t(x)) \|_0 ]. (7) Equation equation 7 is a schematic reconstruction–sparsity objective. The residual-SAE implementation uses variance-normalized reconstruction error (FVU) and a JumpReLU sparsity penalty; the logged coefficients therefore apply to that normalized objective, not to an unnormalized squared-error loss. For a standard dictionary, uℓ,t=hℓ,tu_ ,t=h_ ,t. For the residual SAE trained in this work, uℓ,t=N(rℓ,t)u_ ,t=N(r_ ,t), where rℓ,t=hℓ,t−h^ℓ,tbr_ ,t=h_ ,t- h^\,b_ ,t. The decoded residual is mapped back to the original activation scale with N−1N^-1. The public/base SAE remains frozen throughout residual SAE training. Figure 4: Selected residual-SAE training diagnostics. Training FVU (log scale, left) and mean active features L0L_0 (right) for Gemma-3-4B layer 22 (λ=0.0003λ=0.0003, selected step 1,500) and Qwen3.5-9B layer 20 (λ=0.0001λ=0.0001, selected step 2,000). Gemma uses the saved per-step metrics; Qwen markers are the training checkpoints recorded in the experiment log, joined only to guide the eye. Curves stop at the selected checkpoints; no other runs are included. Reconstruction metrics were not used as a substitute for downstream checkpoint evaluation: better SAE-objective values did not monotonically improve downstream behavior. Checkpoint sources. The Gemma dictionaries are derived from instruction-tuned Gemma activations and include both single-layer SAEs and sparse crosscoders (Lieberum et al., 2024; Lindsey et al., 2024)22 2 https://huggingface.co/google/gemma-scope-2. The Qwen headline analyses use a public SAE trained on base-model activations (Deng et al., 2026)33 3 https://huggingface.co/collections/Qwen/qwen-scope. The exact checkpoint, layer, dictionary width, sparsity setting, normalization statistics, and selected token role for every reported experiment are retained in the corresponding saved run configurations; the present appendix describes the common construction rather than an exhaustive configuration table. Cross-layer sparse representations. For selected Gemma experiments, a sparse crosscoder constructs one shared token-level code from activations at layers ℒL: ztcc=ρ(∑ℓ∈ℒWenc(ℓ)hℓ,t+benc),h^ℓ,t=Wdec(ℓ)ztcc+bdec(ℓ),z^c_t=ρ\! ( _ W^( )_ench_ ,t+b_enc ), h_ ,t=W^( )_decz^c_t+b^( )_dec, (8) where ρ is the checkpoint-specific sparse nonlinearity. The shared code tests whether harmful-content information is represented across model depth rather than being confined to one analyzed layer. Native and external readouts. For class c, let τ(c)τ(c) denote its textual verbalization. The native label score is snat,c(x)=∑q=1|τ(c)|logpθ(τ(c)q∣x,τ(c)<q).s_nat,c(x)= _q=1^|τ(c)| p_θ (τ(c)_q x,τ(c)_<q ). (9) A supervised linear probe maps a role-conditioned representation to task scores: probe(s)(x)=Wsϕℓ,s(x)+bss^(s)_probe(x)=W_s _ ,s(x)+b_s (10) For the residual reconstruction intervention, the activation passed to the remaining transformer blocks is hℓ,t′=hℓ,t+αhook(h^ℓ,tjoint−hℓ,t),αhook∈[0,1]h _ ,t=h_ ,t+ _hook ( h^\,joint_ ,t-h_ ,t ), _hook∈[0,1] (11) Section 5 introduces the direct logit router; Appendix I specifies its calibration statistics, signed task margin, mixing coefficient, and locked selection procedure. Hardware and runtime. Experiments were run on a shared compute cluster with access to 2×2× NVIDIA B200 and 2×2× NVIDIA A100 GPUs; individual jobs used between one and four accelerators. Activation caching was the dominant computational cost: the full Qwen dense cache required approximately three to four 24-hour Slurm runs, followed by residual-cache construction. Most individual probe-fitting, SAE-training, inference, routing, causal-intervention, and LoRA runs completed within 12 hours, although the largest MMHS150K sweeps required longer. Appendix C Dataset and Evaluation Details Primary benchmarks. CrisisHateMM contains text-embedded images from the Russia–Ukraine conflict annotated for hate speech and attack targets (Bhandari et al., 2023). FHM is a binary hate-speech benchmark constructed with benign image and text confounders intended to reduce unimodal shortcuts (Kiela et al., 2020). MAMI evaluates misogyny identification in memes (Gasparini et al., 2022); HarMeme annotates COVID-19 memes for harmfulness severity and targeted social entities (Pramanick et al., 2021); MMHS150K contains paired image-text social-media posts labeled as NotHate or one of five hate categories (Gomez et al., 2020); and MultiOFF consists of US-election memes annotated for offensiveness (Suryawanshi et al., 2020). Table 1 reports the frozen split used for each headline binary task. Binary and fine-grained targets. The headline benchmark contains one binary task per dataset. For HarMeme, the two harmfulness categories are merged into Harmful, while the harmless class becomes Not Harmful. For MMHS150K, Racist, Sexist, Homophobe, Religion, OtherHate are merged into Hate, with NotHate retained as the negative class. The original three-class HarMeme task, six-class MMHS150K taxonomy, and CrisisHateMM target-attribution task are treated as secondary fine-grained evaluations and are excluded from the primary six-task mean. Multilingual robustness datasets. EXIST-2025 contains English and Spanish memes annotated for sexism under a learning-with-disagreement framework (Plaza et al., 2026). Because official test labels were unavailable, we derive binary targets by strict majority vote over Task 2.1 annotations, remove tied or invalid instances, and construct a fixed language-stratified holdout of 500 examples, comprising 250 English and 250 Spanish memes, from 3,420 valid examples. The remaining 2,920 examples are used for probe training. MultiBully contains Hindi-English code-mixed memes annotated for cyberbullying, sentiment, emotion, and sarcasm (Maity et al., 2022). We use its joint image-text Bully/Nonbully label rather than either unimodal annotation. After removing 53 unreadable and 8 missing images, a deterministic label-stratified split assigns 4,793 of the 5,793 valid examples to training and 1,000 to evaluation. EXIST and MultiBully use internal holdouts and are reported only as robustness studies. Model selection and locked evaluation. Probes and trainable adapters are fitted only on task training data. Any data-dependent choice, including layer, token role, representation, checkpoint, decision threshold, or routing coefficient, is made using training or calibration data and frozen before the reporting split is evaluated. When official validation and test sets both exist, validation is used for calibration and test for reporting. Methods requiring additional scalar calibration divide the available held-out data into disjoint calibration and reporting partitions. A result described as locked uses no reporting example for threshold selection, checkpoint selection, or other post-hoc tuning. Native decoding and label normalization. Genuinely binary tasks use constrained positive-versus-negative label scoring. For datasets whose original annotation space is multiclass, including HarMeme and MMHS150K, the LVLM first scores every original class label and then applies the fixed binary mapping described above. This avoids evaluating a multiclass dataset through a semantically mismatched yes/no question about one arbitrarily selected class. Primary evaluations use the original image and natively supplied OCR where available. OCR removal, image blanking, and image permutation are used only as paired robustness controls. Metrics and comparison scope. The primary metric is macro-averaged F1, which weights each class equally despite the substantial class and prediction imbalance in several datasets. Accuracy, class-wise precision and recall, predicted class frequencies, and paired uncertainty analyses are treated as secondary diagnostics. Native, reconstruction-hook, probe, routed, and adapted predictions are compared on identical example IDs within each model family. Because Gemma and Qwen differ in scale, training regime, SAE source, and analyzed layer, cross-family results are interpreted as replication of the phenomenon rather than as a model leaderboard. Appendix D Additional Broad-Comparison Analysis Task-level readout gaps. The positive sparse-readout advantage is not concentrated in one benchmark. For Gemma-3-4B-IT, the largest improvements over native macro-F1 occur on MMHS150K (+0.338+0.338), MAMI (+0.322+0.322), and CrisisHateMM (+0.213+0.213); the gains on FHM (+0.055+0.055), HarMeme (+0.057+0.057), and MultiOFF (+0.104+0.104) are smaller but remain positive. For Qwen3.5-9B-Base, the largest gains occur on CrisisHateMM (+0.550+0.550), HarMeme (+0.416+0.416), MultiOFF (+0.396+0.396), and MAMI (+0.348+0.348), with smaller positive gains on FHM (+0.075+0.075) and MMHS150K (+0.064+0.064). The unweighted mean gives every dataset equal weight and therefore prevents MMHS150K from dominating the aggregate through its substantially larger reporting split. Residual reconstruction. The Qwen reconstruction hook raises mean macro-F1 from 0.4320.432 to 0.4860.486, compared with 0.7400.740 for the external SAE probe. Its aggregate recovery fraction is 0.486−0.4320.740−0.432=0.176. 0.486-0.4320.740-0.432=0.176. The hook improves CrisisHateMM, FHM, MAMI, MMHS150K, and MultiOFF, while remaining unchanged on HarMeme. Reconstruction can therefore make some internally represented information behaviorally useful, but most of the probe-accessible signal is not recovered after the modified activation passes through the model’s original upper layers and output head. Public base SAE versus residual SAE. For Qwen, the public base-SAE and residual-SAE probes obtain nearly identical six-task means of 0.7400.740 and 0.7370.737, respectively. The public base SAE is stronger on CrisisHateMM, MultiOFF, HarMeme, and MMHS150K, while the residual SAE is stronger on FHM and MAMI. We retain the public base SAE as the primary interpretability object because it provides one standardized feature space across all six tasks, not because residualization is uniformly weaker. Their near-tie also shows that the broad readout gap is not specific to one SAE construction. Fine-grained targets. The original three-class HarMeme and six-class MMHS150K evaluations are excluded from the primary six-task mean because they require distinguishing harmfulness severity or hate subtypes rather than the coarser harmful-versus-benign boundary. Complete aggregate and class-wise results are reported in Appendix E. Appendix E Fine-Grained Harmful-Content Classification Aggregate fine-grained results. Table 3 reports the locked held-out comparison for the original HarMeme and MMHS150K taxonomies. These results are kept separate from the binary benchmark because the fine-grained tasks require distinguishing severity levels or hate subtypes rather than detecting harmful content alone. Figure 5: Gemma validation-to-test generalization. Paired dots preserve the original eight task-level macro-F1 comparisons; connectors join only the two splits of the same categorical task. MAMI drops from 0.83900.8390 to 0.73110.7311, whereas MMHS150K and MultiOFF transfer more closely. These are descriptive split differences, not paired-example uncertainty intervals. Table 3: Fine-grained macro-F1. For Gemma, Selected sparse denotes the frozen validation-selected residual-SAE readout. For Qwen, the public base SAE is the primary sparse representation and the residual SAE is an ablation. The binary counterpart uses the corresponding sparse representation on the collapsed binary task. Qwen binary values are shown as base-SAE/residual-SAE. Family Task n Native Hook Base SAE Selected/ residual SAE Binary counterpart Gemma HarMeme 3-class 354 0.3538 0.3103 0.5432 0.5627 0.7760 MMHS150K 6-class, flat 10,000 0.3168 0.3118 0.3588 0.4699 0.6299 MMHS150K 6-class, hierarchical 10,000 NAa 0.3609 0.4105 0.4882 0.6299 Qwen HarMeme 3-class 354 0.2626 0.2626 0.5292 0.5476 0.8096 / 0.7998 MMHS150K 6-class 10,000 0.1661 0.2071 0.3693 0.3602 0.5871 / 0.5731 a The hierarchical hook/base-SAE runs use a genuine two-stage readout: ‘NotHate’ versus ‘Hate’, then a five-way hate-type classifier when stage 1 selects ‘Hate’. Native hierarchical remains ‘NA’ because no corresponding two-stage native LVLM decoder was run. The sparse readouts remain substantially stronger than native generation on the fine-grained tasks, but the gap between binary and fine-grained performance is large. For Gemma, the selected binary HarMeme readout reaches 0.77600.7760, compared with 0.56270.5627 for the selected three-class readout, a difference of 0.21330.2133. These use the image and generated tokens, respectively, so the difference does not isolate label collapse in a fixed classifier. The analogous MMHS150K differences are 0.16000.1600 relative to the flat six-class probe and 0.14170.1417 relative to the hierarchical probe. For Qwen’s primary base-SAE representation, the binary readouts exceed the fine-grained readouts by 0.28040.2804 on HarMeme and 0.21780.2178 on MMHS150K. The residual-SAE differences are 0.25220.2522 and 0.21290.2129, respectively. Class-wise error structure. Table 4 reports class-level metrics from the saved locked Gemma evaluation summary. The MMHS150K values correspond to the hierarchical six-class readout, whose class F1 values average to its reported macro-F1 of 0.48820.4882. The HarMeme values correspond to the selected generated-token three-class readout. Table 4: Class-level metrics explicitly recorded for the locked Gemma fine-grained evaluation. Task Class Precision Recall F1 Support / diagnostic HarMeme 3-class not harmful 0.9337 0.7957 0.8592 230 gold; 183 TP somewhat harmful 0.5986 0.8544 0.7040 103 gold; 88 TP very harmful 0.1818 0.0952 0.1250 21 gold; 2 TP MMHS hierarchical Homophobe 0.6202 0.6763 0.6470 1,007 gold; 681 TP NotHate 0.7275 0.7192 0.7233 6,090 gold; 4,380 TP OtherHate 0.4446 0.5356 0.4858 801 gold; 429 TP Racist 0.4458 0.3594 0.3979 1,614 gold; 580 TP Religion 0.4444 0.3333 0.3810 24 gold; 8 TP; low-frequency class Sexist 0.2613 0.3362 0.2941 464 gold; 156 TP HarMeme’s aggregate accuracy of 0.77120.7712 masks a severe minority-class failure. Only 22 of the 2121 very harmful examples are classified correctly, and the class F1 is 0.1250.125. The model therefore captures the broad harmfulness boundary much more reliably than the distinction between somewhat harmful and very harmful. MMHS150K shows a less concentrated but still uneven error profile. NotHate and Homophobe are the strongest classes, with F1 values of 0.7230.723 and 0.6470.647. Sexist is the weakest at 0.2940.294, while Racist, Religion, and OtherHate remain between 0.3810.381 and 0.4860.486. The hierarchical formulation improves aggregate macro-F1 from 0.46990.4699 to 0.48820.4882, but it does not eliminate the substantial variation between hate subtypes. Completeness of the Qwen class report. The locked Qwen summary records macro-F1 for the native model, reconstruction hook, public base-SAE probe, and residual-SAE probe, but does not include their class-wise precision, recall, support, or confusion matrices. These class-level results are not reported here. This limits our ability to attribute Qwen’s aggregate fine-grained gains to particular classes or characterize its minority-class errors. Earlier validation reports are not substituted because they use different splits, metrics, or model-selection protocols. Appendix F Token-Role and Representation Ablations Calibration-only role selection. For each task, candidate probes over prompt/OCR, image, generated, pre-generation, and all-token representations use the same training examples and probe family. The token role and representation used for locked evaluation are selected exclusively on calibration data and are then frozen. The values below are therefore representation-selection diagnostics rather than final reporting results. Table 5: Gemma-3-4B token-role ablation on the validation/calibration split. Values are macro-F1. Positive Δgen−img _gen-img indicates that generated-token states outperform image-position states. Dataset Image Generated Δgen−img _gen-img CrisisHateMM 0.817 0.811 −0.006-0.006 FHM 0.645 0.651 +0.006+0.006 MAMI 0.789 0.839 +0.050+0.050 HarMeme 0.719 0.668 −0.051-0.051 MMHS150K 0.531 0.617 +0.086+0.086 MultiOFF 0.600 0.615 +0.016+0.016 Task-dependent token roles. Table 5 rules out a simple account in which harmful-content information is consistently concentrated in either image-origin or generated states. Image features are strongest for HarMeme and marginally stronger for CrisisHateMM. Generated features outperform image features on MAMI, MMHS150K, and MultiOFF, while the two FHM readouts differ by less than one macro-F1 point. The clearest shift occurs on MMHS150K. Moving from image to generated states raises binary macro-F1 from 0.5310.531 to 0.6170.617. On the original six-class taxonomy, the corresponding increase is from 0.2320.232 to 0.4680.468. The decoding trajectory therefore makes hate subtypes such as Racist, Homophobe, and NotHate considerably easier to separate than they are in image-position states. This result does not imply that generated tokens supply the gold label; rather, the model’s own classification or explanation trajectory reorganizes the input into a more label-separable representation. MAMI exhibits a different pattern. Its prompt/OCR probe reaches 0.8450.845 macro-F1, slightly exceeding the generated-token result and clearly exceeding the image-position result of 0.7890.789. The strongest calibration signal is therefore associated with text-bearing states. Prompt positions may incorporate visual information through contextual processing, so this is not a text-only intervention; nor is it direct evidence of visual misogyny recognition. The subsequent validation-to-test decrease indicates limited generalization but does not, on its own, identify reliance on lexical cues. Pre-generation and post-generation interpretations. Generated-token representations are observed only after answer generation begins and may encode an emerging label, explanation, or confidence. We therefore use them to localize when the representation becomes separable, not as sole evidence that the same information was available before decoding. Pre-generation controls show that the broad readout gap is not merely a probe over the model’s own answer. Image-position probes are selected for CrisisHateMM and binary HarMeme and retain substantial advantages over native output on their locked reporting splits. The selected Gemma FHM readout uses prompt and image positions from a cross-layer representation rather than generated states. Both pre-generation and post-generation states therefore contain useful harmful-content information, although the strongest role varies by task. FHM multi-role controls. FHM was designed with benign image and text confounders, so a useful classifier must distinguish examples whose label changes when one component is held fixed (Kiela et al., 2020). Gemma’s image-generated and all-token layer-22 probes remain in a similar performance range. Its final 4B candidate instead combines prompt–image crosscoder features with complementary layer-22 and layer-29 readouts. Qwen provides a separate control using its public layer-20 base SAE. Image- and prompt-position probes obtain 0.6280.628 and 0.6310.631 macro-F1, respectively, while their concatenation reaches 0.6410.641. A rank-16 bilinear probe falls to 0.5280.528. The two roles therefore contain useful additive information, but this pooled bilinear parameterization does not recover a useful interaction. More generally, using multiple modality-associated representations does not itself establish a genuine cross-modal interaction (Hessel and Lee, 2020). Section 4.2 tests this question using matched confounders and a stronger Gemma-3-12B representation. Sparse versus dense representations. We compare each sparse readout with a probe over the corresponding role-pooled dense hidden state. For Gemma, sparse factorization improves macro-F1 by 0.2830.283 on CrisisHateMM, 0.0410.041 on six-class MMHS150K, and 0.0220.022 on FHM. Its effect on MAMI, HarMeme, and MultiOFF lies between −0.006-0.006 and +0.008+0.008. Random sparse controls remain near chance, ruling out dimensionality or sparsity alone as the source of the strong probe results. These controls indicate that most harmful-content information is already present in the dense residual stream. Sparse factorization can make that information easier to separate on selected tasks, but its primary value for the subsequent analysis is that it expresses the signal in a coordinate system supporting feature-level output alignment, ablation, and activation patching. The complementary Qwen public-base versus residual-SAE comparison is reported in Appendix D. Appendix G Feature Alignment and Causal Intervention Details Static feature-output alignment. For an SAE feature with decoder direction dj∈ℝdℓd_j ^d_ , we measure its direct alignment with the yes/no output channel. Let myn(x)=ℓyes(x)−ℓno(x)m_yn(x)= _yes(x)- _no(x) (12) denote the monitored output margin, where ℓv(x) _v(x) is the logit for token v. With the final normalization scale folded into the unembedding, the corresponding hidden-space direction and feature effect are uyn=WUf⊤(eyes−eno),aj=dj⊤uyn.u_yn=W_U^f (e_yes-e_no ), a_j=d_j u_yn. (13) The signed agreement between this effect and task-d probe weight wd,jw_d,j is qd,j=sign(wd,j)aj.q_d,j=sign(w_d,j)a_j. (14) A probe-important feature is called aligned when qd,j≥τq_d,j≥τ, misaligned when qd,j≤−τq_d,j≤-τ, and output-silent when |aj|<τ|a_j|<τ, using τ=0.10τ=0.10. Silence is defined only relative to this output anchor; it does not imply that the feature is globally inactive or incapable of affecting another output channel. For Qwen’s public base SAE, the aligned subset is empty for CrisisHateMM, FHM, MultiOFF, HarMeme, and MMHS150K, and contains only 1%1\% of analyzed feature records for MAMI. No task has a substantial misaligned subset. The secondary residual SAE gives the same qualitative result. Because Equations equation 12 and equation 13 bypass the intervening transformer blocks, they are used as feature-selection diagnostics rather than as complete causal attributions (Elhage et al., 2021; Belrose et al., 2023). Independent feature-set construction. The routed set is selected independently of labels and probe weights: ℛyn=TopKk(|dj⊤uyn|j=1m).R_yn=TopK_k ( \|d_j u_yn| \_j=1^m ). (15) For task d, the silent discriminative set is d=j∈TopKk(|wd,j|j=1m):|aj|<τ.S_d= \j _k (\|w_d,j|\_j=1^m ):|a_j|<τ \. (16) All reported Qwen interventions use k=20k=20. Thus, dS_d is selected for task discrimination subject to weak output alignment, while ℛynR_yn is selected for output alignment without using the task labels. Knockout and donor patching. Following causal-mediation and activation-patching analyses (Vig et al., 2020; Meng et al., 2022; Marks et al., 2025), the knockout intervention removes a selected set S: zt,j−S(x)=0,j∈S,zt,j(x),j∉S.z^-S_t,j(x)= cases0,&j∈ S,\\ z_t,j(x),&j∉ S. cases (17) For donor x′x and target x, routed-feature patching assigns the donor’s max-pooled activation to the selected target features: zt,jx←x′=maxt′zt′,j(x′),j∈ℛyn,zt,j(x),j∉ℛyn.z^\,x← x _t,j= cases _t z_t ,j(x ),&j _yn,\\ z_t,j(x),&j _yn. cases (18) The patched value is broadcast over target positions. This is useful for testing whether the selected feature family can control a downstream quantity, but it does not reproduce the donor’s natural position-dependent activation pattern. For intervened computation x~ x, we record Δprobe=sprobe(x~)−sprobe(x),Δnative=myn(x~)−myn(x) _probe=s_probe( x)-s_probe(x), _native=m_yn( x)-m_yn(x) (19) and summarize the mirror-image interventions using ρ=|Δprobe||Δnative|,ρℛ=|Δnative||Δprobe|. _S= E| _probe|E| _native|, _R= E| _native|E| _probe|. (20) Qwen intervention results. Silent-feature knockout uses at most 15 eligible examples per task. Where eligible examples exist, removing only 20 features changes the probe score by approximately 2.02.0–2.52.5 units while changing the monitored yes/no margin by only 0.040.04–0.080.08. This gives probe-to-margin ratios of 24.2:124.2:1 to 62.8:162.8:1. For FHM, the monitored margin is the actual native decision channel. For HarMeme and MMHS150K, it is only a diagnostic because their native predictions use full-label scoring. Routed-feature patching uses at most five eligible donor-target pairs. On the four literal yes/no tasks, it changes the native margin by 0.340.34–1.091.09 while changing the probe score by only 0.0060.006–0.0620.062, yielding native-to-probe ratios from 16.2:116.2:1 to 140.2:1140.2:1. The MMHS150K routed result is excluded from the headline comparison because both its output channel and routed feature set are mismatched to the task’s label-score decision rule. These effects establish control of the monitored output, not an improvement in prediction correctness. Why silent-feature donor patching is not headline evidence. Broadcasting a donor’s max-pooled feature value to every target token position can create activation patterns that do not occur during ordinary inference. This intervention produces inconsistent signs and, for CrisisHateMM, an unusually large movement of the monitored margin. We therefore use silent-feature knockout to establish probe dependence and routed-feature patching to establish output control. Silent-feature patching is retained in Table 6 as a transparency and sanity-check result rather than as primary evidence. Gemma replication and limits. For Gemma’s layer-22 residual-SAE dictionary, the top probe-derived and directly routed sets have zero overlap on all four analyzed tasks. Perturbing the silent set changes the probe score 1313–180×180× more than perturbing the routed set. We additionally compute an upper-layer-aware output-sensitivity diagnostic: ad,jJ=dj⊤x[∇hℓmd(x)].a^J_d,j=d_j E_x [ _h_ m_d(x) ]. (21) This Jacobian-based (Gurnee et al., 2026) analysis also places the strongest probe directions outside the dominant output-routing set. Native-output mediation is less uniform: routed-feature ablation has a larger effect than silent-feature ablation on MMHS150K, but not consistently on MAMI, HarMeme, or CrisisHateMM. Gemma therefore replicates the geometric and probe-side separation, but does not support a universal two-subspace causal decomposition. Table 6: Complete causal interventions for Qwen’s public layer-20 base SAE. |Δprobe|E| _probe| and |Δnative|E| _native| are mean absolute changes in the external probe score and monitored yes/no margin. The final column is always written as probe sensitivity: native-margin sensitivity. Thus, 50.2:150.2:1 is probe-dominant, whereas 1:16.21:16.2 is output-dominant. YN denotes literal yes/no decoding and LS denotes full-label scoring. Task Decode n |Δsprobe|E| s_probe| |Δmyn|E| m_yn| Probe:YN A. Silent-feature knockout CrisisHateMM YN –a – – – FHM YN 15 2.5121 0.0500 50.2:1 MAMI YN –a – – – HarMeme LS† 15 2.4995 0.0398 62.8:1 MMHS150K LS† 15 2.0190 0.0833 24.2:1 MultiOFF YN –a – – – B. Silent-feature donor patch CrisisHateMM YN 5 0.3416 1.5377 0.2:1 FHM YN 5 0.4010 0.1373 2.9:1 MAMI YN 5 0.6764 0.1124 6.0:1 HarMeme LS –b – – – MMHS150K LS† 5 0.4248 0.2737 1.6:1 MultiOFF YN 5 0.5354 0.1253 4.3:1 C. Routed-feature donor patch CrisisHateMM YN 5 0.0125 0.3998 1:32.0 FHM YN 5 0.0208 0.3376 1:16.2 MAMI YN 5 0.0624 1.0880 1:17.4 HarMeme LS –b – – – MMHS150K LS‡ 5 0.3191 0.0900 3.5:1 MultiOFF YN 5 0.0065 0.9116 1:140.2 a No eligible gold-positive, probe-correct, native-wrong examples occurred under the Intervention A selection rule. The native classifiers for these tasks already predicted the positive class at a high rate, leaving few or no relevant false negatives. b No eligible patch targets occurred for HarMeme because the native Qwen classifier predicted not harmful for every reporting example. † HarMeme and MMHS150K use label-score decoding rather than literal yes/no decoding. Their reported Δmyn m_yn therefore measures a diagnostic yes/no direction, not the complete native task decision. ‡ In addition to the diagnostic-margin caveat, the routed feature set itself was selected using the yes/no unembedding direction. It is therefore mismatched to MMHS150K’s label-score channel and is not included in the routed-feature headline conclusion. Appendix H FHM Low-Rank Cross-Modal Readout Details Figure 6: FHM rank sweep and test comparison. A. Validation macro-F1 on 375 examples after threshold selection on a disjoint 94-example calibration partition. Rank-8 and rank-32 dots show five seeds; error bars show sample SD. The gated baseline and rank-128 values are single runs. B. The displayed test-seen count is n=1000n=1000, following the manuscript’s current reporting convention; the saved paired-analysis artifact underlying the unchanged scores contains 931 paired predictions. This denominator discrepancy remains to be reconciled. The locked readout reaches 0.75600.7560 versus 0.68530.6853 natively; the difference compares the complete pair-aware readout with the native model, not the isolated bilinear contribution. Representation construction. We use the Gemma-3-12B layer-31 sparse representation. A train-only confounder audit selects K=128K=128 features, after which image-position and prompt-position activations are feature-wise max-pooled and standardized: ximg,xprompt∈ℝK.x_img,x_prompt ^K. The prompt representation includes both the task instruction and supplied OCR. Image and prompt refer to token origin; by layer 31, both state families may contain information exchanged through the transformer. Gated pairwise baseline. Separate image and prompt-position scores are simg(x)=wimg⊤ximg+bimg,sprompt(x)=wprompt⊤xprompt+bprompt.s_img(x)=w_img x_img+b_img, s_prompt(x)=w_prompt x_prompt+b_prompt. (22) An input-dependent gate combines them: g(x) g(x) =σ(wg⊤[ximg;xprompt]+bg), =σ\! (w_g [x_img;x_prompt]+b_g ), (23) sgate(x) s_gate(x) =g(x)simg(x)+(1−g(x))sprompt(x). =g(x)s_img(x)+ (1-g(x) )s_prompt(x). Because g(x)g(x) depends on both role representations, we refer to this as a gated pairwise baseline rather than a strictly additive linear model. Low-rank interaction. The full readout is defined in Equation equation 5. Equivalently, its interaction matrix is Wbil=UV⊤,U,V∈ℝK×r.W_bil=UV , U,V ^K× r. The rank r bounds the number of multiplicative image-by-prompt directions and reduces the interaction parameter count from K2K^2 to 2Kr2Kr. Matched-pair objective. Let imgP_img contain opposite-label pairs sharing the same image, and let textP_text contain opposite-label pairs sharing the same meme text. For (x+,x−)(x^+,x^-), x+x^+ is the hateful member and x−x^- is the benign member. Training minimizes ℒ=ℒcls+∑c∈img,textλc|c|∑(x+,x−)∈c[δ−sr(x+)+sr(x−)]+L=L_cls+ _c∈\img,text\ _c|P_c| _(x^+,x^-) _c [δ-s_r(x^+)+s_r(x^-) ]_+ (24) where [a]+=max(0,a)[a]_+= (0,a), δ is the required pair margin, and λc _c controls the contribution of each pair family. Development and locked-test protocol. Model parameters are fitted on the FHM training set. Decision thresholds are selected on a fixed 94-example calibration partition and evaluated on a disjoint 375-example validation partition. Training seeds change only parameter initialization. Rank 32 is chosen from the development rank sweep based on its accuracy-stability trade-off. Within that rank, the seed-0 checkpoint and threshold 0.520.52 are selected from calibration results and then frozen before evaluating the 10001000 test-seen examples with paired predictions. Table 7: Gemma-3-12B FHM rank sweep. Validation macro-F1 is measured on the same 375-example partition after threshold selection on the disjoint 94-example calibration partition. Rank-8 and rank-32 values are the mean ± standard deviation over five seeds. Pair BC reports the mean both-correct rate for same-image and same-text opposite-label pairs. The † test result is from the calibration-selected rank-32, seed-0 checkpoint, not an average over test runs. Readout Validation F1 Pair BC, Img/Txt Test-seen F1 Native 12B label score – – 0.6853 Gated pairwise baseline 0.7366 0.571 / 0.435 – Bilinear r=8r=8 0.7110±0.03520.7110± 0.0352 0.471 / 0.407 – Bilinear r=32r=32 0.7488±0.01200.7488± 0.0120 0.555 / 0.464 0.7560†0.7560 Bilinear r=128r=128 0.7621 0.583 / 0.506 – Locked test comparison. The selected rank-32 checkpoint transfers from 0.75910.7591 validation macro-F1 to 0.75600.7560 on the 10001000-example paired test-seen subset. Native Gemma-3-12B obtains 0.68530.6853, giving Δtest=0.7560−0.6853=0.0707. _test=0.7560-0.6853=0.0707. A label-stratified paired bootstrap gives a 95% confidence interval of [0.0461,0.0954][0.0461,0.0954]. An exact McNemar test favors the pair-aware bilinear readout on 103 discordant examples, compared with 37 examples favoring the native model (p=2.18×10−8p=2.18× 10^-8). These statistics compare the complete pair-aware readout with native generation. They should not be interpreted as the isolated effect of adding the bilinear component to the gated baseline. Figure 7: FHM calibration curve (Gemma-3-12B). The locked bilinear readout attains Brier score 0.17210.1721 and ECE 0.03750.0375, compared with 0.30840.3084 and 0.30650.3065 for the native model. Matched-confounder behavior. For the selected rank-32 seed, the hateful member receives the larger score in 84.5%84.5\% of same-image pairs and 84.7%84.7\% of same-text pairs. The stricter both-correct rates are 58.3%58.3\% and 49.4%49.4\%, respectively. The model therefore captures a strong relative-ordering signal when either the image or text is held fixed, while exact classification of both members remains more difficult. Cross-covariance interpretation. For centered role vectors, define the class-conditional cross-covariance difference ΔΣip=[ximgxprompt⊤∣y=1]−[ximgxprompt⊤∣y=0]. _ip=E [x_imgx_prompt y=1 ]-E [x_imgx_prompt y=0 ]. (25) The class separation contributed by the explicit interaction component is Δint=γtr(Wbil⊤ΔΣip). _int= (W_bil _ip ). (26) The bilinear term therefore reads class-dependent co-variation between the two role representations, rather than only their separate marginal activations. Nonlinear and factor-level controls. A within-role quadratic control adds squared image, prompt, and generated-feature activations without multiplying features across roles. It changes validation macro-F1 by only 0.0040.004 and leaves the matched-pair both-correct rates unchanged. The improvement is therefore not explained by generic nonlinear rescaling within an individual role. Leave-one-rank-out analysis finds that 11 of the 32 interaction factors make a measurable contribution to validation macro-F1. Removing any single factor changes macro-F1 by at most 0.0070.007. Follow-up patching and polarity tests do not support a collection of narrowly bound rules of the form “specific image feature i plus specific text feature j.” The more conservative interpretation is a redundant, distributed interaction over broader image- and prompt-side subspaces. Cross-model negative controls. The result does not transfer uniformly to other representations. For Gemma-3-4B, the corresponding bilinear readout obtains 0.6540.654 macro-F1, below its simpler baseline at 0.6880.688. For Qwen’s public layer-20 base SAE, a rank-16 bilinear readout obtains 0.5280.528, below image-prompt concatenation at 0.6410.641. These results limit the interaction claim to the analyzed Gemma-3-12B layer-31 representation and pair-aware training procedure; they do not show that other models lack cross-modal structure at different layers or under different readouts. Appendix I Direct-Routing and Probe-Distillation Details I.1 Calibration-Only Direct Logit Routing For task d, let d,calD_d,cal be the calibration partition, md(x)m_d(x) the signed native decision margin, and ud(x)u_d(x) the decision-function score of the frozen probe. The calibration statistics are μd _d =1|d,cal|∑x∈d,calud(x), = 1|D_d,cal| _x _d,calu_d(x), (27) σd2 _d^2 =1|d,cal|∑x∈d,cal(ud(x)−μd)2, = 1|D_d,cal| _x _d,cal (u_d(x)- _d )^2, u~d(x) u_d(x) =ud(x)−μdσd+ϵ. = u_d(x)- _d _d+ε. The routed margin is given by Equation equation 6. Its mixing coefficient is selected only on the calibration partition: βd⋆=argmaxβ∈MacroF1([md(x)+βu~d(x)≥0]x∈d,cal,d,cal), _d = _β MacroF1 ( \I [m_d(x)+β u_d(x)≥ 0 ] \_x _d,cal,y_d,cal ), (28) where A is the finite coefficient grid. The normalization statistics, coefficients, and zero decision threshold are then frozen before evaluation on the disjoint reporting partition. Table 8: Qwen direct logit routing on disjoint reporting partitions. Macro-F1 is evaluated after selecting βd⋆ _d on a separate 30% calibration partition. Native, routed, and probe values use identical reporting examples and therefore differ from the full-split results in Table 2. Task Native Routed Probe βd⋆ _d CrisisHateMM 0.3527 0.8413 0.8582 4 FHM 0.6660 0.7387 0.7269 8 MAMI 0.4020 0.7703 0.7952 8 HarMeme 0.3922 0.7650 0.8251 4 MMHS150K 0.5097 0.5708 0.5745 8 MultiOFF 0.3201 0.6365 0.6623 8 Mean 0.4405 0.7204 0.7404 – The routed mean recovers 0.7204−0.44050.7404−0.4405=0.933 0.7204-0.44050.7404-0.4405=0.933 of the native-to-probe gap. FHM is the only task on which the native and probe scores are complementary enough for the routed system to exceed the probe alone. Because FHM, MAMI, MMHS150K, and MultiOFF select the maximum tested value βd=8 _d=8, these runs do not establish that the selected coefficients are interior optima. Table 9: Recorded configuration of the dedicated and shared probe-distillation runs. Configuration FHM-dedicated adapter Shared seven-task adapter Base model Gemma-3-4B-IT Gemma-3-4B-IT Training tasks FHM binary hatefulness FHM, MAMI, MultiOFF, CrisisHateMM-A, CrisisHateMM-B, HarMeme, and MMHS150K Available target rows 7,9387,938 160,761160,761 across seven tasks Optimization steps 2,0002,000 4,0004,000 Task-sampling schedule Single-task sampling Equal round-robin task exposure; each task receives 1/71/7 of task-selection steps LoRA rank rLoRAr_LoRA 1616 1616 Adapted modules Seven projection types in all 3434 transformer layers Same seven projection types across all 34 layers Trainable parameters 29,802,49629,802,496 of 4,329,881,9684,329,881,968 parameters (0.6883%0.6883\%) 29,802,49629,802,496 of 4,329,881,9684,329,881,968 parameters (0.68830.6883%) Gold-label loss Cross-entropy; λgold=0.5 _gold=0.5 Cross-entropy; λgold=0.5 _gold=0.5 Probe-target loss KL distillation from the frozen FHM probe; λprobe=0.5 _probe=0.5 KL distillation from the corresponding frozen task probe; λprobe=0.5 _probe=0.5 Pair-ranking loss Active on 4,3304,330 pseudo-confounder pairs; λpair=0.5 _pair=0.5 Active only for tasks with valid pairs; λpair=0.5 _pair=0.5 Pair margin δ 1.01.0 1.01.0 Pairs sampled per eligible task per step 4 FHM pairs 11 pair, reduced from 22 after the initial run Pair-loss accumulation Four sampled pair losses are averaged into the final microbatch loss, then scaled by 1/nacc1/n_acc Each sampled pair is backpropagated separately with effective weight λpair/(nvalidnacc) _pair/(n_validn_acc), where nvalidn_valid is the number of valid sampled pairs and naccn_acc is the gradient-accumulation factor Output-rate regularization None Task-dependent; see Table 10 Output-rate coefficient λrate _rate 00 0.50.5; EMA decay 0.980.98 Batch size 11 11 Gradient-accumulation factor naccn_acc 88 88 Effective main-loss batch per optimizer step 8 examples 8 examples Optimizer and learning rate AdamW, lr=10−4lr=10^-4, weight decay 00; 5050-step linear warmup AdamW, lr=10−4lr=10^-4, weight decay 00; 100100-step linear warmup I.2 Probe-Distilled LoRA Let pθ,ϕ,d(⋅∣x)p_θ,φ,d(· x) be the label distribution of the adapted LVLM, where θ is frozen and ϕφ contains the LoRA parameters, and let qd(⋅∣x)q_d(· x) be the frozen probe distribution. The task-d objective is ℒ(d)= ^(d)= λgoldCE(y,pθ,ϕ,d(⋅∣x))+λprobeKL(qd(⋅∣x)∥pθ,ϕ,d(⋅∣x)) _goldCE (y,p_θ,φ,d(· x) )+ _probeKL (q_d(· x)\;\|\;p_θ,φ,d(· x) ) (29) +[|d|>0]λpairℒ(d)pair+[d∈rate]λrateℒ(d)rate. +I[|P_d|>0]\, _pairL^(d)_pair+I[d _rate]\, _rateL^(d)_rate. The gold-label term limits blind imitation of probe errors, while the KL term transfers the probe’s graded class preferences. For tasks with matched opposite-label pairs, ℒpair(d)=1|d|∑(x+,x−)∈d[δ−mθ,ϕ(d)(x+)+mθ,ϕ(d)(x−)]+L^(d)_pair= 1|P_d| _(x^+,x^-) _d [δ-m^(d)_θ,φ(x^+)+m^(d)_θ,φ(x^-) ]_+ (30) where x+x^+ and x−x^- are the harmful and benign pair members, respectively, and δ is the ordering margin. Dedicated FHM adapter. The dedicated Gemma-3-4B-IT run uses rank-16 LoRA updates and contains 29,802,496 trainable parameters, corresponding to 0.6883%0.6883\% of the 4.33-billion-parameter model. It is trained for 2,000 steps using 7,938 available target rows and 4,330 pseudo-confounder pairs. The adapted model improves macro-F1 on the scored test-seen subset from 0.64820.6482 to 0.71380.7138 without requiring an SAE or probe at inference. On the separate 469-example development evaluation, it obtains 0.70950.7095, compared with 0.71100.7110 for the teacher probe. Similar macro-F1 does not imply identical predictions: the adapter is more precise but less sensitive on the hateful class and reaches a similar aggregate score at a different operating point. Shared seven-task adapter. The shared adapter is trained for 4,000 steps with equal round-robin task sampling. Each task therefore receives one seventh of task-selection steps, irrespective of its share of the 160,761 available target rows. Under its own evaluation setup, it raises the unweighted seven-task mean from 0.46420.4642 to 0.56860.5686, including large improvements on MAMI and binary HarMeme. The gains are not uniform. On the same FHM development evaluation, the shared adapter obtains 0.65480.6548, compared with 0.70950.7095 for the dedicated adapter. On MMHS150K, six-class macro-F1 decreases from 0.37270.3727 to 0.18860.1886, although the binary result changes only from 0.39400.3940 to 0.38950.3895. The shared adapter therefore preserves much of the coarse hate-versus-non-hate boundary while destroying subtype discrimination. Equal task exposure may contribute: MMHS150K accounts for 83.865%83.865\% of available target rows but receives the same nominal number of task-selection steps as MultiOFF, which accounts for only 0.277%0.277\%. These results motivate task-specific or task-grouped adapters rather than a single universal adapter. Table 10: Per-task target pools and auxiliary supervision for the shared adapter. Pool share is computed over the 160,761160,761 available target rows. “Configured, inactive” means that the task was eligible for a pair loss but no valid training pairs were found. Output-rate regularizers are reported only where their status is explicitly documented. Task Targets NdN_d Pool share Step share Valid pairs Pair loss Output-rate regularizer FHM 7,9387,938 4.938%4.938\% 1/71/7 4,3304,330 Active NR; no task-specific form is stated in the experiment observations MAMI 9,0009,000 5.598%5.598\% 1/71/7 2323 Active Active; EMA prediction rate compared with the approximately 0.50.5 empirical positive-class prior MultiOFF 445445 0.277%0.277\% 1/71/7 11 Active, but extremely sparse Not used CrisisHateMM -A 3,6003,600 2.239%2.239\% 1/71/7 00 Configured, inactive Not used HarMeme 3,0133,013 1.874%1.874\% 1/71/7 5151 Active Active on the binary harmful/not harmful collapse; it does not regularize the somewhat harmful/very harmful boundary MMHS150K 134,823134,823 83.865%83.865\% 1/71/7 5757 Active Active using a one-sided EMA overshoot statistic relative to the approximately 0.5830.583 empirical Hate prior; the term was largely inactive because the EMA rate remained below the prior CrisisHateMM -B 1,9421,942 1.208%1.208\% 1/71/7 00 Inactive Not used because the three-way target task has no natural harmful-versus-benign output-rate constraint Total 160,761160,761 100.000%100.000\% – 4,4624,462 – – Appendix J Multilingual, OCR, and Visual-Credit Robustness Details Perturbation protocol. For prediction system S∈native,probe,routeS∈\native,probe,route\, each example is evaluated under four conditions: origorig uses the original image and supplied OCR; blankblank replaces the image with a uniform grey image; shuffleshuffle assigns a deterministic unrelated image from the same reporting split; and noOCRnoOCR removes the supplied OCR string while retaining the original image. Let FS,d(c)F^(c)_S,d be system S’s macro-F1 on dataset d under condition c. We report ΔS,dOCR=FS,d(orig)−FS,d(noOCR),ΔS,dshuf=FS,d(orig)−FS,d(shuffle). ^OCR_S,d=F^(orig)_S,d-F^(noOCR)_S,d, ^shuf_S,d=F^(orig)_S,d-F^(shuffle)_S,d. (31) Aggregate score changes need not identify whether the affected examples were originally correct. We therefore define qS,iimg= q^img_S,i= [y^(orig)S,i=yi][y^(blank)S,i≠yi∨y^(shuffle)S,i≠yi], [ y^(orig)_S,i=y_i ]I [ y^(blank)_S,i≠ y_i\; \; y^(shuffle)_S,i≠ y_i ], (32) CS,dimg= C^img_S,d= ∑i=1ndqS,iimg∑i=1nd[y^(orig)S,i=yi]. _i=1^n_dq^img_S,i _i=1^n_dI [ y^(orig)_S,i=y_i ]. Thus, CS,dimgC^img_S,d is the fraction of originally correct predictions that become incorrect under at least one image perturbation. It measures operational image dependence rather than a complete allocation of causal credit between modalities. Cross-lingual decodability on EXIST. We use the deterministic internal holdouts defined in Appendix C (Plaza et al., 2026). With Qwen’s public base SAE, an English-trained all-token probe reaches 0.74690.7469 macro-F1 on Spanish, while a Spanish-trained generated-token probe reaches 0.77390.7739 on English. Bilingual training reaches 0.81210.8121 on English using prompt/OCR states and 0.76230.7623 on Spanish using all-token states. The strongest role therefore changes across transfer directions, but substantial decodability is not restricted to the probe’s training language. Gemma gives a less stable result. Across five independently constructed holdouts, every analyzed representation remains above chance, but each transfer direction contains at least one split on which the probe-minus-native difference is negative, and the standard deviation across splits exceeds the mean gain. We therefore treat the Gemma EXIST experiment as evidence of cross-lingual internal decodability rather than a stable improvement over native prediction. Table 11: EXIST transfer stability over split seeds (Gemma). Mean ± sample SD of the paired probe-minus-native macro-F1 gain over deterministic holdout seeds 13, 29, 42, 73, and 101, not initialization seeds or confidence intervals. English-to-Spanish uses all-token states; the other two settings use generated-token states. Transfer direction Gain (macro-F1) English→ 0.0145±0.03020.0145± 0.0302 Spanish→ 0.0095±0.03730.0095± 0.0373 Bilingual (generated-token) 0.0139±0.02130.0139± 0.0213 Code-mixed decodability on MultiBully. On Hindi-English code-mixed MultiBully (Maity et al., 2022), Qwen’s all-token base-SAE probe obtains 0.72590.7259 macro-F1, compared with 0.34010.3401 for native constrained output. Gemma’s all-token residual-SAE probe obtains 0.71090.7109, compared with 0.59900.5990 natively. The Gemma probe outperforms native prediction on all five deterministic split seeds, with a mean paired difference of 0.0915±0.02260.0915± 0.0226, where the uncertainty is the sample standard deviation over seeds. Multilingual causal replication. Using the silent-feature intervention protocol from Appendix G, Qwen gives probe-to-native sensitivity ratios of approximately 62:162:1 on English EXIST, 90:190:1 on Spanish EXIST, and 48.5:148.5:1 on MultiBully. The two EXIST estimates use 12 probe-correct positive examples per language, while MultiBully uses 15 examples. These small intervention sets support a mechanistic replication of the probe-dominant pattern, not a population-level effect estimate. Table 12: Qwen visual-credit audit on disjoint reporting partitions. P denotes the frozen public base-SAE probe and R the calibration-only logit router. Δshuf=F(orig)−F(shuffle) ^shuf=F^(orig)-F^(shuffle). The final column reports the conditional image-credit rates CPimg/CRimgC^img_P/C^img_R. All probe and router values within a row use identical reporting examples; they differ from full-split headline results because router calibration uses a separate partition. Dataset ndn_d FPorigF^orig_P ΔPshuf ^shuf_P FRorigF^orig_R ΔRshuf ^shuf_R CPimg/CRimgC^img_P/C^img_R FHM 651 0.679 0.162 0.687 0.160 70.4/67.8% MAMI 700 0.664 0.206 0.801 0.292 88.8/64.7% MultiBully 700 0.618 0.148 0.720 0.200 65.8/69.1% EXIST-ES 175 0.729 0.182 0.713 0.178 55.8/64.0% HarMeme 248 0.698 0.162 0.782 0.211 56.7/62.6% Supplied-OCR ablations. On Qwen EXIST, removing supplied OCR lowers prompt-probe macro-F1 by 0.05070.0507 for English-to-Spanish transfer, 0.06710.0671 for bilingual training evaluated on English, and 0.02180.0218 for bilingual training evaluated on Spanish. Spanish-to-English transfer is the exception: macro-F1 changes from 0.75830.7583 to 0.76840.7684 after OCR removal. On MultiBully, the prompt-position probe loses 0.05230.0523, the all-token probe loses 0.02360.0236, and the image-position probe changes by only 0.00110.0011. Supplied OCR therefore contributes most clearly to text-bearing representations, but neither uniformly nor sufficiently to explain the complete signal. The no-OCR condition removes only the OCR string inserted into the prompt. Meme text remains visible in the image, and its removal can indirectly alter subsequent generated-token states; this condition is consequently not a text-free input. Table 13: Gemma-3-4B visual-credit classification controls. Macro-F1 on fixed audit splits, not uniformly the headline test splits. FHM uses its dedicated LoRA; MAMI and HarMeme use the shared task-trained adapter; MultiBully and EXIST use that shared adapter zero-shot. All systems and conditions within a task use the same n examples. Task / split System / adapter n Original Blank Shuffled No-OCR FHM / validation Native 469 0.657 0.585 0.553 0.650 LoRA, dedicated 469 0.710 0.502 0.517 0.696 Probe 469 0.701 0.591 0.476 0.656 MAMI / test Native 1,000 0.409 0.354 0.360 0.416 LoRA, shared in-domain 1,000 0.818 0.700 0.537 0.812 Probe 1,000 0.694 0.333 0.482 0.703 HarMeme / validation Native 177 0.462 0.475 0.440 0.492 LoRA, shared in-domain 177 0.792 0.715 0.553 0.775 Probe 177 0.710 0.396 0.487 0.727 MultiBully / internal Native 1,000 0.599 0.594 0.549 0.538 LoRA, shared zero-shot 1,000 0.547 0.448 0.445 0.523 Probe 1,000 0.711 0.323 0.539 0.681 EXIST-ES / internal Native 500 0.658 0.608 0.547 0.660 LoRA, shared zero-shot 500 0.648 0.581 0.511 0.591 Probe 500 0.600 0.272 0.490 0.596 Table 14: Gemma prompt-prior diagnostics, not classification baselines. Each cell lists native / LoRA / probe macro-F1 under a changed prompt. Splits, sample counts, and adapters match Table 13. These prompts do not preserve the classification instruction, so their scores cannot isolate OCR or visual contributions to task classification. Task Blank + OCR-only prompt Neutral-description prompt FHM 0.406 / 0.395 / 0.338 0.338 / 0.585 / 0.558 MAMI 0.379 / 0.433 / 0.333 0.333 / 0.433 / 0.699 HarMeme 0.285 / 0.623 / 0.396 0.349 / 0.396 / 0.715 MultiBully 0.422 / 0.438 / 0.480 0.355 / 0.447 / 0.600 EXIST-ES 0.473 / 0.441 / 0.272 0.385 / 0.387 / 0.594 Visual-credit results. As shown in Table 12, image permutation reduces probe macro-F1 by 0.1480.148–0.2060.206 and routed macro-F1 by 0.1600.160–0.2920.292 on every audited dataset. Conditional image credit ranges from 55.8%55.8\% to 88.8%88.8\% for the probe and from 62.6%62.6\% to 69.1%69.1\% for the router. The corresponding native-model rate is only 0.0%0.0\%–38.1%38.1\%, partly because several native classifiers have collapsed or strongly biased output distributions. The probe advantage is therefore not adequately explained as post-hoc classification of supplied OCR. Gemma visual-credit and prompt-prior controls. Table 13 complements the Qwen probe-router audit with native, probe, and LoRA results for Gemma. Image shuffling produces substantially larger losses than OCR removal for the strongest systems, while the blank-image results show that visual credit is material but not exclusive. The shared zero-shot adapter does not outperform native Gemma on MultiBully (0.5470.547 versus 0.5990.599) or EXIST-ES (0.6480.648 versus 0.6580.658); neither is a task-specific LoRA result. The altered-prompt conditions in Table 14 are reported only as prompt-prior diagnostics. FHM matched-pair controls. On opposite-label FHM pairs sharing the same image, Qwen’s probe and router assign the hateful member the larger score in 86.4%86.4\% and 88.4%88.4\% of cases, respectively. On pairs sharing the same text but using different images, the corresponding rates are 83.8%83.8\% and 86.3%86.3\%. These ordering results show sensitivity to the component that changes within each pair, although correctly classifying both members remains more difficult. Gemma independently reproduces the qualitative visual-sensitivity pattern. Image shuffling reduces macro-F1 by approximately 0.170.17–0.280.28 for its strongest FHM, MAMI, HarMeme, and MultiBully probe or LoRA systems. Blanking versus shuffling. Blank and shuffled images are distinct out-of-distribution interventions. Blanking removes most natural visual content and can change a model’s class prior, whereas shuffling supplies a plausible but semantically mismatched image. This distinction explains cases such as the Qwen HarMeme probe, which improves slightly under blanking but degrades substantially under shuffling. We therefore use the two conditions jointly to establish visual dependence and do not interpret either as identifying a specific cross-modal circuit (Hessel and Lee, 2020). Appendix K Qualitative Feature Examples Selection and interpretation. Before rendering the revised atlas, we fix a retrospective four-feature case-study set: 02261 as an FHM group-antagonism feature included in the causal audit, 01867 as a CrisisHateMM domain contrast, 02205 as a MAMI relationship contrast, and 02629 as a HarMeme feature also listed in the FHM confounder audit. This is not a preregistered or globally highest-|w||w| selection. In particular, discovery scores in the earlier cards are not interchangeable with fitted probe coefficients. We report actual fitted weights and their absolute ranks instead of claiming that these four are the most probe-important features. For each feature, we select the two highest max-pooled activations within its source task’s fixed validation pool, with sample ID breaking ties and no filtering by label or image appearance. Exemplars and probe coefficients use the same token role: all tokens for the saved FHM probe and image tokens for the other three probes. Signed weights refer to the stated class on the probe’s scaled inputs; ranks are among all 20,480 coefficients of that class. Confounder ranks and causal inclusion come from separate, explicitly role-labeled audits, not from these validation exemplars. The matched examples narrow the qualitative claims: 01867 illustrates conflict-related context, but 02205’s strongest displayed MAMI examples are benign, and its coefficient for misogyny is negative. Likewise, 02629’s displayed HarMeme examples are non-harmful, and its coefficient for very harmful is small. These cards illustrate contextual features rather than pure harmfulness detectors. Only 02261 has documented inclusion in the saved causal-card audit at image positions; that does not establish causal efficacy for its all-token probe shown here. Absence from the saved audit is not evidence of causal irrelevance. Figure 8: Four-feature qualitative atlas. Same-dataset, role-matched validation maxima for the selected Gemma-3-4B layer-22 residual SAE. Each card reports its source task and class, token role, fitted probe rank and signed weight, causal-audit inclusion, and train-only confounder-overlap status. The class naming the coefficient need not be the gold class of a top-activating example. Feature descriptions are qualitative context summaries, not claims of monosemanticity or complete explanations of model behavior. Original FHM confounder examples. Figure 9 shows two matched pairs from the original 02261 card. The anchors were retrieved among that feature’s generated-token training exemplars, whereas the revised atlas reports all-token FHM validation activations. We keep these sources separate: the pairs illustrate the dataset’s compositional contrasts, not new role-matched activation measurements or causal effects. The labels are the dataset annotations. Figure 9: Original FHM image and text confounders associated with feature 02261. A. Training examples 19726 and 84176 share a background image but differ in text and gold label (same-image group 867). B. Examples 69035 and 91683 share the supplied text but differ in image and gold label (same-text group 952). These illustrative pairs are not selected from the validation maxima in the atlas and do not establish that feature 02261 alone distinguishes either pair. Warning: offensive content is reproduced for analysis. Appendix L Glossary of Terms This glossary defines terms according to their operational meaning in this paper. In particular, internal information means information that a specified readout can recover on held-out data; it does not imply human-like knowledge, belief, or native model use. L.1 Representations and token roles. Native LVLM / native pathway. The original Gemma or Qwen model and its own route from the input, through the transformer and output head, to the final label. No external probe, activation hook, output router, or trained adapter determines the prediction. Residual stream. The sequence of hidden-state vectors passed through the transformer layers. We denote the activation at layer ℓ and token position t by hℓ,th_ ,t. It should not be confused with an SAE reconstruction residual. Sparse autoencoder (SAE). An encoder–decoder model that maps a dense activation h to a sparse latent vector z=E(h)z=E(h) and reconstructs it as h^=D(z) h=D(z) (Huben et al., 2024; Gao et al., 2025). Sparse feature or SAE latent. One coordinate zjz_j in the SAE representation. Each feature has an activation value and a decoder direction djd_j in the LVLM’s hidden space. A sparse feature is not necessarily a perfectly monosemantic human-interpretable concept. Feature dictionary. The collection of SAE decoder directions d1,…,dm\d_1,…,d_m\. Feature indices are specific to a particular model, layer, and SAE; the same index in two dictionaries need not represent the same pattern. Base SAE. The public SAE applied directly to an LVLM hidden state. The term base refers to its position in our SAE pipeline, not to the base LVLM itself. Reconstruction residual. The information left after the base SAE reconstructs a hidden state: rℓ,t=hℓ,t−h^ℓ,tb.r_ ,t=h_ ,t- h^\,b_ ,t. This is an SAE reconstruction error, not the transformer’s residual stream. Residual SAE. A second SAE trained on the reconstruction residual rℓ,tr_ ,t. It captures structure not reconstructed by the base dictionary and is treated as an alternative representation rather than an automatically superior one. Joint reconstruction. The sum of the base-SAE reconstruction and decoded residual-SAE reconstruction: h^joint=h^b+r^. h^\,joint= h^\,b+ r. This is the activation used by the residual reconstruction hook. Crosscoder. A sparse model that constructs one latent representation from several transformer layers and uses separate decoders for those layers. It is intended to organize information shared across model depth rather than simply concatenate independent layer representations (Lindsey et al., 2024). Token role. The provenance of a sequence position. We distinguish prompt/OCR, image-origin, and generated positions. Token role identifies where a state originated, not which modality it contains after contextual processing. Prompt/OCR tokens. Positions containing the task instruction, class wording, and supplied OCR. At intermediate layers, these states may already contain visually conditioned information. Image-position states. Positions originating from the visual encoder. Because they interact with prompt states inside the transformer, they should not be interpreted as modality-pure or strictly image-only representations. Generated-token states. Hidden states produced after answer generation begins. They may encode the model’s emerging label, explanation, or confidence. Gold labels are not supplied during their extraction, but the states may encode the model’s predicted answer and are not pre-decision representations. Pre-generation representation. A representation pooled over prompt and image positions while excluding generated positions. It tests what can be decoded before the model starts producing its answer. All-token representation. A representation pooled over prompt, image, and generated positions. It provides a full-computation decodability estimate, but is not a purely pre-decision representation. Pooling. The operation that converts token-level activations into one vector per meme. Our default is featurewise max pooling, [ϕs(x)]j=maxt∈s(x)zt,j,[ _s(x)]_j= _t _s(x)z_t,j, where s denotes the selected token role. Pooling discards token order and exact activation position. L.2 Readouts and output pathways. Readout. Any mapping from a representation to a task score or prediction. The LVLM output pathway is a native readout; a probe is an external readout; and the output router combines the two. Native prediction. The label obtained from the frozen LVLM using the task’s constrained decoding or label-scoring rule. Native does not necessarily mean free-form generation. Output head / unembedding. The final transformation from the model’s hidden state to vocabulary logits. Its directions indicate how changes in hidden space immediately affect candidate output tokens. Native decision margin. A signed difference between competing native scores. For a literal yes/no task, myn(x)=ℓyes(x)−ℓno(x),m_yn(x)= _yes(x)- _no(x), where ℓyes _yes and ℓno _no are the two output logits. A positive value favors the designated positive class. External probe. A supervised classifier trained on frozen internal representations. A probe measures how accessible a label is to the selected classifier family; it does not establish that the native model uses the same decision rule (Hewitt and Liang, 2019; Belinkov, 2022). Sparse readout / SAE probe. A probe whose input is a pooled vector of SAE feature activations. The input is sparse-derived, although the fitted classifier weights need not themselves be sparse. Decodability. The extent to which a specified readout can recover a target from a frozen representation on held-out data. Decodability is relative to the chosen layer, token role, representation, probe family, and training set. Internal signal. Shorthand for label-associated structure that is decodable from a model representation. It does not imply that the model’s native output pathway uses that structure. Probe ceiling. The best held-out performance obtained by the selected probe and representation. It is an empirical reference point, not a theoretical upper bound on all possible classifiers. Internal-to-output gap / readout gap. The difference between probe and native performance on the same reporting examples: Δreadout=Fprobe−Fnative. _readout=F_probe-F_native. Because the probe receives supervised labels while the native LVLM remains frozen, this measures supervised accessibility rather than a like-for-like model comparison. Hook. A mechanism attached to an intermediate layer to record or modify an activation during the model’s forward pass. The term identifies where an intervention occurs, not the type of classifier used afterward. Residual reconstruction hook. An intervention that moves the original hidden state toward its joint SAE reconstruction: hℓ,t′=hℓ,t+αhook(h^ℓ,tjoint−hℓ,t).h _ ,t=h_ ,t+ _hook ( h^\,joint_ ,t-h_ ,t ). The modified state then passes through the remaining LVLM layers and its native output head. Direct logit routing / output router. An output-level combination of the native margin and standardized probe score: mdroute(x)=md(x)+βdu~d(x),m_d^route(x)=m_d(x)+ _d u_d(x), where md(x)m_d(x) is the native task margin, u~d(x) u_d(x) is the calibrated probe score, and βd _d is selected on a calibration set. No hidden state is modified. Probe distillation. Training the LVLM to reproduce useful probe behavior using gold labels and soft probe targets. Unlike direct routing, the resulting model no longer requires the SAE or external probe at inference. LoRA adapter. A parameter-efficient update that freezes an original weight matrix W0W_0 and learns a low-rank correction, W=W0+γrLoRABA,W=W_0+ γr_LoRABA, where A and B are trainable matrices, rLoRAr_LoRA is the adapter rank, and γ is a scaling factor (Hu et al., 2021). L.3 Discriminative and routed directions. Discriminative direction. A feature or direction that helps the supervised probe separate task labels. Discriminative importance does not imply influence on the native output. Routed direction. A feature direction selected because it has a large projection onto the native output direction. It may strongly affect the output margin without being predictive of the correct label. Feature-to-output alignment. The direct projection aj=dj⊤uout,a_j=d_j u_out, where djd_j is the decoder direction of feature j and uoutu_out is the relevant native output direction. It is a static approximation to immediate logit influence, not a complete causal effect. Silent feature. A probe-important feature whose direct output alignment |aj||a_j| lies below a fixed threshold. Silent means weakly connected to the particular output channel being tested, not globally inactive or causally irrelevant. Readout or routing misalignment. A situation in which the directions most useful to an external probe differ from those that control the native output. This provides one possible explanation for high decodability but weak native performance. Output anchor. The output contrast used to define a routed direction, such as yes minus no. An anchor is mismatched when the task actually uses a different rule, such as full-label string scoring. Donor and target. The donor is the example from which an activation is taken. The target is the example whose computation is modified by that activation. Ablation / knockout. An intervention that removes selected components, usually by setting their feature activations to zero. Probe-side ablation tests dependence of the external classifier; online ablation tests effects on the LVLM’s subsequent computation. Activation patching. Replacing part of a target example’s internal representation with a representation obtained from a donor example. Patching tests whether transplanting a candidate feature pattern changes a downstream quantity. Sensitivity ratio. The ratio between an intervention’s effect on the probe and its effect on the native margin, or vice versa. It measures relative responsiveness under the chosen score scales, not a scale-invariant mediation fraction or whether the intervention improves correctness. L.4 Cross-modal and robustness analyses. Benign confounder. An image–text combination that resembles a harmful example in one modality but is benign when both modalities are interpreted together. FHM was designed around such confounders (Kiela et al., 2020). Matched confounder pair. Two memes that share one component, differ in the other, and have opposite labels. A pseudo-image pair shares an image but changes the text; a pseudo-text pair shares text but changes the image. Additive readout. A classifier of the form f(ximg)+g(xprompt)f(x_img)+g(x_prompt), with no input-dependent cross-role coupling. Using both roles additively does not by itself demonstrate an interaction. Our gated pairwise baseline is not strictly additive because its gate depends on both role representations. Low-rank bilinear readout. A readout containing an explicit image-by-prompt term, xi⊤UV⊤xpx_i UV x_p where xix_i and xpx_p are image- and prompt-position representations and U,V∈ℝK×rbilU,V ^K× r_bil. The rank rbilr_bil limits the number of multiplicative interaction factors (Kim et al., 2016). Distributed interaction. An interaction whose predictive contribution is spread across several bilinear factors. Small single-factor ablation effects show that no one tested factor is necessary for the full performance gain; they do not establish that every factor is individually insufficient or define a unique semantic factorization. Pair margin. The score difference s(x+)−s(x−)s(x^+)-s(x^-) between harmful and benign members of a matched pair. A positive margin means that the pair is ordered correctly even when one member crosses the wrong hard-decision threshold. Positive-margin, separated, and both-correct rates. The positive-margin rate measures correct score ordering. The separated rate measures whether the pair receives different hard labels. The both-correct rate requires both members to receive their correct labels and is therefore the strictest measure. No-OCR control. A condition in which supplied OCR is removed from the prompt while the original image remains. Since meme text is still visible in the image, this tests reliance on explicit OCR injection rather than providing a text-free input. Visual credit. A paired measure of image dependence. An originally correct prediction is image-credited when it becomes incorrect after image blanking or same-split image shuffling. It is an operational robustness measure, not a complete causal allocation of credit between modalities (Liu et al., 2026). Locked evaluation. A protocol in which the representation, token role, classifier, threshold, router coefficient, and checkpoint are frozen before the reporting split is scored. No reported examples are used for post-hoc model selection.