Paper deep dive

Using OCR Heads to Verbalize Image Semantics

Sheridan Feucht, Benno Krojer, Sarah Wang, Henry Abrahamsen, Byron C. Wallace, David Bau

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 88

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 9/19/2026, 2:30:49 AM

Summary

This paper investigates how Vision-Language Models (VLMs) map pixels to semantics by studying Optical Character Recognition (OCR). The authors identify specific attention heads in models like Qwen3-VL-8B that are causally necessary for OCR but function as general-purpose 'verbalization heads' capable of outputting semantic features for non-text image tokens (e.g., 'feathers' for a bird wing). They propose a 'verbalization lens' transformation that combines these heads' parameters to reveal interpretable semantic labels from early layers (Layer 0), demonstrating that image representations are aligned with language space immediately upon entering the language backbone. This method also enables causal editing of image concepts.

Entities (10)

Qwen3-VL-8B · model · 98%Verbalization Lens · method · 97%Attention Heads · model-component · 96%OCR · task · 95%Qwen3-VL-2B · model · 95%Logit Lens · method · 92%Llava-Next-34B · model · 90%Molmo2-7B · model · 90%COCO · dataset · 88%ImageNet · dataset · 85%

Relation Signals (7)

Qwen3-VL-8B → isevaluatedin → This Study

confidence 95% · Across four models, we identify attention heads... For example... Qwen3-VL-8B

Verbalization Lens → uses → Attention Heads

confidence 95% · We collapse these heads' attention weights into a single verbalization lens transformation

Attention Heads → iscausallynecessaryfor → OCR

confidence 94% · identify attention heads causally necessary for OCR

Attention Heads → enables → Verbalization

confidence 93% · discover that these are in fact general-purpose heads that output interpretable semantic features

Verbalization Lens → reveals → Semantic Features

confidence 92% · verbalization lens transformation that reveals interpretable semantic features in hidden states

Verbalization Lens → outperforms → Logit Lens

confidence 90% · Unlike vanilla logit lens, our approach reveals information about image tokens starting from layer 0

Verbalization Lens → enables → Object Editing

confidence 89% · use the inverse of this transformation to edit non-word concepts, e.g., replacing a tractor with a revolver

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:How do VLMs map from pixels to semantics? To understand this general question, we focus on a narrow one: studying how VLMs perform optical character recognition (OCR). Across four models, we identify attention heads causally necessary for OCR, and discover that these are in fact general-purpose heads that output interpretable semantic features across all image tokens. For example, pointing these heads at an image token containing the word "bike" causes Qwen3-VL-8B to output "bike," but pointing them at a bird wing causes the model to output the token "feathers." We collapse these heads' attention weights into a single verbalization lens transformation that reveals interpretable semantic features in hidden states across all layers. When combined with projection to vocabulary space, we can obtain interpretable labels starting from layer 0, showing that image representations are in fact aligned with language in early layers. We find that we can also use the inverse of this transformation to edit non-word concepts, e.g., replacing a tractor with a revolver in a naturalistic image, providing causal evidence that this subspace is useful for more than just OCR. Our results are an example of how the study of specific mechanisms can shed light on broader interpretability problems.

PDF

Open source PDF →

PDF not stored locally. Use the link above to view on the source site.

Full Text

87,348 characters extracted from source content.

Expand or collapse full text

Using OCR Heads to Verbalize Image Semantics Sheridan Feucht Northeastern University Benno Krojer Northeastern University Sarah Wang Independent Henry Abrahamsen Northeastern University Byron C. Wallace Northeastern University David Bau Northeastern University Abstract How do VLMs map from pixels to semantics? To understand this general question, we focus on a narrow one: studying how VLMs perform optical character recognition (OCR). Across four models, we identify attention heads causally necessary for OCR, and discover that these are in fact general-purpose heads that output interpretable semantic features across all image tokens. For example, pointing these heads at an image token containing the word “bike” causes Qwen3-VL-8B to output “bike,” but pointing them at a bird wing causes the model to output the token “feath- ers.” We collapse these heads’ attention weights into a single verbalization lens transformation that reveals inter- pretable semantic features in hidden states across all lay- ers. When combined with projection to vocabulary space, we can obtain interpretable labels starting from layer 0, showing that image representations are in fact aligned with language in early layers. We find that we can also use the inverse of this transformation to edit non-word concepts, e.g., replacing a tractor with a revolver in a naturalistic im- age, providing causal evidence that this subspace is useful for more than just OCR. Our results are an example of how the study of specific mechanisms can shed light on broader interpretability problems. 1 1. Introduction Reading is a specialized form of perception that involves mapping from very specific visual forms to semantic con- cepts. Because it is closely tied to language, it occupies a special place in neuroscience research, with a large body of work studying specific areas of the brain used to rep- resent word forms [8, 9, 41]. The central point of inter- est is the interface between visual and linguistic representa- tions: a question that is also a concern for AI interpretability 1 For code and interactive demos, see https://ocr.baulab.info. Corre- spondence: feucht.s@northeastern.edu Table 1 clean ' bike' 1.000 ' ' 0.000 ' bicycle' 0.000 'bike' 0.000 ' Bike' 0.000 ' ride' 0.000 " '" 0.000 ' bake' 0.000 ' bi' 0.000 ' b' 0.000 (a) Prompting Qwen3-VL-8B to “Transcribe the word in this image.” (b) Verbalization lens: decoding image activations at layer 0 Original generation with vanilla attention Word: bike OCR Heads on Word: branch 喙 beak 喙 beak 眼睛 eye 紫色 purple 褐色 brown 灰色 gray 红色 red 头发 hair 朦胧 murky 红色 red ⻦类 birds 羽毛 plumage birds snowy 翅膀 wing 翅膀 wing 红色 red 红色 red 红色 red 红色 red 红色 red 红色 red 红色 red Raw logit lens probs. (baseline) Verbalization lens probs. (ours) refuge 0.04, 并将 and will 0.03, 刺 thorn 0.02, 鹑 quail birds 0.46, ⻦类 birds 0.28, ⻦ bird 0.13, bird 0.08 birds (All heads: ‘_’) OCR heads on Word: feathers (All heads: \\) Figure 1. We find that VLM attention heads responsible for OCR are actually generic verbalization heads that can read out seman- tics of image tokens that do not contain text. (a) Verbalization heads attend to text when prompted to transcribe words, but man- ually redirecting their attention to image tokens that do not con- tain text causes the model to output words corresponding to the semantics of those tokens (top 10% of Qwen3-VL-8B verbaliza- tion heads). (b) We repurpose these heads’ attention weights to create a verbalization lens that can be applied to image represen- tations from any layer. Here, our approach reveals alignment of image representations with language space starting from layer 0, which logit lens is unable to see. [44, 50, 58]. In this work we study how vision-language models “read” (i.e., perform optical character recognition, or OCR) [20, 29, 36], which we show can help us understand how these models map from pixels to semantics in general. We discover that attention heads responsible for OCR can also arXiv:2609.18823v1 [cs.CV] 16 Sep 2026 be used to verbalize semantic information across a wide range of image tokens. We first isolate a set of attention heads that are causally responsible for OCR; if we prompt Qwen3-VL-8B [1] to transcribe a word in, e.g., Figure 1a, it will correctly output “bike,” and we can observe these heads attending to the tokens that correspond to that word. How- ever, if we intervene on these attention heads to attend to a bird wing, we find that Qwen3-VL-8B will output the to- ken feathers. Similarly, it will outputbranch if we force these heads to attend to a branch. Due to these heads’ ability to describe image tokens that do not contain text, we refer to them as general verbalization heads. We study the subspace of activation space read by ver- balization heads across four models, and find that it contains a wealth of high-level semantic information about inputted images. Specifically, we combine the parameters of identi- fied verbalization heads into a single matrix that can be ap- plied to any hidden state, following techniques described in prior work on language-only models [11–13]. When com- bined with logit lens [27, 44, 45] (which projects hidden ac- tivations to vocabulary space), our verbalization matrix re- veals semantic information across all image tokens, as seen in Figure 1b. Unlike vanilla logit lens, our approach reveals informa- tion about image tokens starting from layer 0 of the lan- guage backbone. This means that when images are passed from the image encoder to the language decoder, they al- ready contain information aligned with language space. In other words, our results provide evidence that image repre- sentations are immediately aligned with language represen- tations starting from early layers [32, 62], despite the appar- ent modality gap claimed in prior work [25, 50, 54, 58, 59]. These verbalizations are not just correlational, but can also be used to edit latent image representations for Qwen3- VL models. By adding and subtracting latent vectors within the subspace read by verbalization heads, we can edit a model’s activations to replace objects in an image, e.g., causing Qwen3-VL-2B to describe an ant as “an iPod rest- ing on a green plant stem” (Figure 8a). This indicates that the verbalizations obtained using our approach are causally important for more than just OCR. Although reading letters appears to be a narrow task, it provides a clean way to identify more general VLM compo- nents that map from vision to language space. Obtaining su- pervision for OCR tasks is uniquely straightforward: while an image of a bird might be mapped to the concepts “ani- mal,” “feathers,” or “small,” an image of the letters b-i-r-d is uniquely associated with the word “bird.” Thus, identifying attention heads in this tractable setting allows us to under- stand the semantic representations of VLMs more broadly. We provide code and an interactive demo of verbalization lens across models at https://ocr.baulab.info. Qwen3-2B OCR ScoresQwen3-8B OCR Scores LayerLayer Head Index Head Index Qwen3-2B Head Mean AblationsQwen3-8B Head Mean Ablations OCR Accuracy Percent of model heads ablatedPercent of model heads ablated Figure 2. OCR Scores (Equation 1) for all heads in Qwen3-VL models. We prompt models to transcribe random words w i in im- ages x i and measure P(w i ) according to logit lens at the final token position for each individual head (n=1024). Many heads in late layers promote the word contained in an OCR dataset image. See Figure 11 for scores for Molmo2-7B and Llava-Next-34B. Qwen3-2B OCR ScoresQwen3-8B OCR Scores LayerLayer Head Index Head Index Qwen3-2B Head Mean AblationsQwen3-8B Head Mean Ablations OCR Accuracy Percent of model heads ablatedPercent of model heads ablated Molmo2-7B OCR ScoresLlava-Next-34B Scores Layer Layer Head IndexHead Index Qwen3-2B Head Mean AblationsQwen3-8B Head Mean Ablations OCR Accuracy Percent of model heads ablatedPercent of model heads ablated Figure 3. Top-scoring attention heads from Figure 2 are causally necessary for doing OCR. Across models, we find that heads re- sponsible for OCR comprise 10% of total attention heads. These ablations are run for 100 images across 10 seeds for random base- lines; see Figure 12 for Molmo2-7B and Llava-Next-34B. 2. Finding Verbalization Heads 2.1. Approach We first search for attention heads causally responsible for extracting text from image representations. We set up a sim- ple OCR task by placing random English words on top of ImageNet [49] images. 2 We filter words, keeping only those corresponding to a single token in the model vocabulary, then pre-fill the input with “Word:” and read out the re- sponse. Table 3 shows that all models we study can reliably complete our OCR task; see Figure 10 for dataset examples. To find attention heads that are responsible for OCR, we apply logit lens [27, 44, 45] to each head’s output and measure the probability of the desired word. Each image in our OCR dataset x i ∈ X OCR contains a word w i . Let h (l,h) x i ∈R d head refer to the output of attention head h at layer l at the final token position for image x i , when the model is about to output its transcription. We project this vector to the residual stream dimension using the output projection for that head W (l,h) O ∈R d model ×d head , and directly decode to vocabulary space to obtain P (w i ). We take the mean of this 2 We use image backgrounds because OCR performance is poor for words on blank backgrounds; see Table 3. value across images: s(l,h) = 1 |X OCR | X x i ∈X OCR LOGITLENS(W (l,h) O h (l,h) x i )[w i ] (1) which yields a score s(l,h) for attention head h at layer l, measuring how strongly this head outputs words w i across the OCR dataset. Logit lens [45] is defined as: LOGITLENS(a) = softmax(W U norm(a))(2) for a hidden state a ∈R d model , where “norm” refers to the final norm, and W U ∈R |V|×d model is the “unembedding” matrix mapping to the token vocabulary spaceV . The soft- max induces a distribution over tokens. The intuition behind our approach is that when the model is about to transcribe a word, some set of attention heads are responsible for “fetching” w i from the image and bringing that information into the column space of the final decoder head. Equation 1 is designed to find these heads, assuming they exist. 2.2. Results Figure 2 shows OCR scores for all attention heads in two Qwen3-VL models. Notably, heads with high scores are concentrated in late layers. To assess whether these heads are causally important for OCR, we progressively mean- ablate 3 heads with the highest scores and measure the trend in task performance in Figure 3. Across all models, mean- ablating the top 10% of heads ranked by OCR scores fully degrades task performance. OCR heads are concentrated in late layers, which might be a confound, but ablating an equivalent number of random late-layer heads is less effec- tive. This suggests that OCR heads are most responsible for the model’s performance on this task. Results are similar for Molmo2-7B and Llava-Next-34B (see Appendix B). 2.3. Head Behavior on Non-Text Although we identify these heads based on a small syn- thetic OCR task, they appear to be broader verbalization heads used for more general image understanding (see Fig- ure 1). While delineating the exact purpose of these heads is beyond the scope of this work, we now briefly discuss some intuition for why these heads are able to verbalize non-textual tokens. In Figure 4a, we visualize attention patterns for top- scoring Qwen3-VL-2B OCR heads on a random image. As expected, these heads attend to image tokens containing text when prompted to transcribe the text in the image. How- ever, we also find that when the model is prompted “What 3 To “mean-ablate” [12, 60] an attention head, we replace the output of that head across all token positions with its mean over 1000 examples from our OCR dataset (also taken across token positions). Transcribe the text in this image. What is the animal in this image? What color is the lifesaver in this image? Top OCR Head (L24, H6) 2nd OCR Head (L25, H5) Top OCR Head (L24, H6) 2nd OCR Head (L25, H5) Top OCR Head (L24, H6) 2nd OCR Head (L25, H5) (a)(b) Figure 4. Top-scoring heads for OCR attend to text—but un- der different prompt settings, they can also attend to animals and objects. We show the two top-OCR-scoring attention heads in Qwen3-VL-2B as an example. (a) These heads attend to the sub- title “[steamboat sound]” when prompted to transcribe text. (b) The same heads also attend to a mouse when prompted “What is the animal in this image?” and a lifesaver when prompted “What color is the lifesaver in this image?” (Qwen3-VL-2B’s answers to these questions are “Mouse” and “White,” respectively). This be- havior suggests that these heads are not purely dedicated to OCR. is the animal in this image?,” the same attention heads be- gin to attend to tokens corresponding to the mouse in the image, rather than the subtitles. This attention behavior is analogous to work on filter heads in LLMs [34, 52], which “retrieve” items from the prior context based on a prompt query (see Appendix C for discussion). What do verbalization heads output when they attend to non-text tokens? Figure 1a shows a qualitative example for Qwen3-VL-8B: redirecting the top 10% of verbaliza- tion heads to attend to, e.g., a token containing a bird wing causes the model to output the token feathers. This motivates a question: can we use these attention heads to create a “verbalizer” for image tokens in general? We note that it is likely one could also identify this set of attention heads using tasks other than OCR, e.g., asking “What is in this image?” with supervision from image anno- tations. However, OCR provides a straightforward one-to- one mapping from visual inputs to language outputs. If we were to run the same procedure with natural images, a given image token of a bird might plausibly map to many differ- ent tokens, including “bird,” “feathers,” “cardinal,” “wing,” or “小鸟” (“bird” in Chinese). The OCR task offers un- ambiguous supervision, enabling broader understanding of how VLMs map from pixels to semantics. 3. Building a Verbalization Lens 3.1. Verbalization Transformation Instead of redirecting attention maps as in Figure 1a (which requires a separate forward pass for every image token), we compact the parameters of these verbalization heads into a (a) 蓝天 blue sky signage 旗帜 flag diamondplatespathway endingsdiamond mountain ⻛筝 kite mountain farm- house 山坡 hillside Montana mountainmountainmountain 天空 sky (b) (e) (c) (d) (f) (a)Raw Logit Lens+Verbalization Heads L0 schließen close 0.06, angep 0.06, gef 0.04, clinic 0.04 outdoors 0.26, geography 0.20, 地理位置 location 0.16 L16 ⻛景 landscape 0.02, 自驾 drive oneself 0.02 road 0.39, travel 0.21, roadside 0.08, ⻛景 landscape 0.05 L32 scenic 0.44, outdoors 0.34, scene 0.13, 场景 scenario 0.03 outdoors 0.67, ⻛景 landscape 0.15, 景色 view 0.09, outdoor 0.06 (c)Raw Logit Lens+Verbalization Heads L0 fur 0.60, frei 0.02, 宾客 guests 0.04, pageTitle 0.01 cloud 0.63, clouds 0.23, 云 cloud 0.11, cloud 0.02 L16 建 [garbage] 0.08, [garbage] 0.01 clouds 0.90, cloud 0.09, 云 cloud 0.00, .cloud 0.00 L32 四五 four-five 0.04, ubi 0.01, dent 0.01, .ibm 0.01 4 0.37, cloud 0.28, 云 cloud 0.13, clouds 0.09 (b)Raw Logit Lens+Verbalization Heads L0 stu 0.78, minster 0.03, kür 0.03, 谣言 rumor 0.01 天空 sky 0.26, 蓝天 blue sky 0.12, sky 0.01, skies 0.00 L16 bell 0.03, unos 0.02, space 0.01, 抜け omission 0.01 天空 sky 0.98, sky 0.01, 蓝天 blue sky 0.01, 晴 clear 0.00 L32 蓝天 blue sky 0.38, 藍 blue 0.26, 天空 sky 0.20 天空 sky 0.84, 蓝天 blue sky 0.15, sky 0.01, blue 0.00 (e)Raw Logit Lens+Verbalization Heads L0 两端 both ends 0.15, 桥梁 bridge 0.07, 而去 and go 0.05, 逶 0.04 道路 path 0.61, roadway 0.29, road 0.05, 公路 highway 0.03 L16 两条 two 0.10, 尽头 end 0.10, 停⻋位 parking space 0.02, 三条 three 0.02 道路 path 0.83, 的道路 the path of 0.04, road 0.04, roadway 0.04 L32 6 0.56, 路况 road conditions 0.14, Sixth 0.12, 6 0.09 道路 path 0.47, 公路 highway 0.47, road 0.04, roads 0.01 (d)Raw Logit Lens+Verbalization Heads L0 realpath 0.05, 部⻓ head of department 0.02, tee 0.02, weed 0.02 pathway 0.98, pathways 0.02, 路径 path/route 0.00, path 0.00 L16 下乡 go to countryside 0.04, finder 0.03, gate 0.03, oret 0.02 pathway 1.00, pathways 0.02, 路径 path/route 0.00, roadway 0.00 L32 way 0.80, ways 0.18, WAY 0.01, WAYS 0.01 pathway 0.98, 路径 path/ route 0.01, pathways 0.00, path 0.00 (f)Raw Logit Lens+Verbalization Heads L0 出入境 entry and exit 0.06, await 0.04, 晚期 late stage 0.03 Wyoming 0.72, 乡村旅游 rural tourism 0.16, Yellowstone 0.10, mountain 0.01 L16 小镇 small town 0.05, 公共服务 public services 0.03, 自驾 drive oneself 0.02 tourist 0.43, 旅游 travel 0.11, travel 0.11, pilgrimage 0.11 L32 trails 0.92, trail 0.05, Trail 0.04 徒步 on foot 0.86, hiking 0.12, trail 0.01, walking 0.00 蓝天 blue sky signage 旗帜 flag diamondplatespathway endingsdiamond mountain ⻛筝 kite mountain farm- house 山坡 hillside Montana mountainmountainmountain 天空 sky Figure 5. Verbalization lens reveals semantic information contained within VLM image tokens starting from early layers. We show several token readouts for one image for Qwen3-VL-8B, comparing raw logit lens to the top 10% of verbalization heads. Applied to image tokens that contain text, verbalization lens shows tokens corresponding to the appropriate word; see subfigure (d). Applied to other image tokens, verbalization lens reveals interpretable semantic information, outputting cloud for a token containing clouds (c), or more general information likeoutdoors above a possible register token [7] in (a). Outputs are often in Chinese for Qwen3-VL models, likely because they are Chinese models. Gray text means probability is below 0.10; top verbalization lens predictions are bolded for emphasis. See Appendix D for qualitative results across models. single linear transformation that can reveal semantic infor- mation contained within any hidden state. Specifically, we sum the output-value (OV) matrices [11] of verbalization heads across layers. This is roughly equivalent to interven- ing on attention maps, except that heads can “attend” to ac- tivations from any layer [12, 13]. Let O p be a set containing the top-p% of OCR heads; across models, we use p=10% based on Sec. 2.2. We con- struct a matrix L p ∈R d model ×d model comprising the parame- ters of these verbalization heads. We calculate L p = X (l,h)∈O p W (l,h) O W (l,h) V ,(3) based on the output and value projections of heads in O p , which have shape W (l,h) O ∈R d model ×d head , W (l,h) V ∈ R d head ×d model . Multiplied together, each “OV matrix” deter- mines what this head will output if it attends to a particular token; adding them together is equivalent to summing the outputs of multiple heads. Thus, applying this matrix to a hidden state is equivalent to “forcing” all heads inO p to at- tend to that hidden state and summing the resulting outputs. 3.2. Decoding to Vocabulary Space After applying the verbalization transformation from Equa- tion 3 to an activation a (l) ∈R d model , we can project the resulting vector to vocabulary space (Equation 2) to obtain probabilities for each token: VERBALLENS(a (l) ) = LOGITLENS(L p a (l) ).(4) This approach can be thought of as a quick approxima- tion of the proof-of-concept in Figure 1a. There are two key differences: first, heads across layers can “attend” to a (l) re- gardless of whether they are also at layer l. This still yields interpretable results, perhaps because layer order has been shown to be mostly interchangeable in LLMs [12, 33, 56]. Second, we directly decode to vocabulary space, rather than allowing the model to further process these head outputs. This gives us the advantage of being able to run a single forward pass and decode any activation with two simple ma- trix multiplications (the naive approach requires a separate forward pass for each image token). We can conceptualize L p as an operation that “focuses” logit lens on the correct subspace of model activations. 3.3. Qualitative Results Figure 5 shows an example of logit lens applied to image hidden states across layers with and without our verbaliza- tion transformation. Applying our verbalization heads re- veals semantic information both for image tokens contain- ing written text (e.g., pathway for a patch containing the text “PATHWAY”, Figure 5d) and image tokens containing more generic visual information (e.g., cloud for a patch containing clouds, Figure 5c). Our approach yields inter- pretable results as soon as layer 0, whereas raw logit lens is quite noisy in early-middle layers. This corroborates ev- idence from prior work showing that image representations are aligned with language in early VLM layers [32, 62]. 3.4. Object Confidence To quantify how well this approach recovers relevant se- mantic concepts from image tokens, we set up an object de- tection task using the COCO validation split [37], similar to the setup from [27]. For each COCO category (e.g., “back- pack”), we sample n images with that object and n random images that do not contain that object. Let o be the token (a)(b) Max Across Image at Layer 0 P(o) Raw Logit Lens+Verbalization Lens (top 10% heads) Count Layerwise Prob. Diff: P(o) with −P(o) without Mean Prob. Diff Max Across Image Tokens P(o) Max Across Image Tokens P(o) Qwen3-VL-8B Qwen3-VL-2B FINAL VERSIN with n=512 per category (there are about 47 single tok categories) stride across layers of 4. top 10% of heads Max Across Image Tokens P(o) Max Across Image Tokens P(o) Qwen3-VL-8B Molmo2-O-7B Count Mean Prob. Diff Molmo2-O-7BLlava-Next-34B Random 10% of heads All heads Raw Logit Lens Verbalization Lens Figure 6. Verbalization lens is sensitive to whether an image contains an object: it yields high probabilities for images that contain objects, and low probabilities for images that do not. We apply logit lens (Equation 2) to VLM representations of images from the COCO dataset [37] with and without this transformation, and extract the probability of the COCO class label token for all single-token categories, similar to [27]. (a) At layer 0, verbalization lens is better able to detect whether an image contains an object—P(o) is consistently higher for the red histogram for Qwen3-VL-8B and Molmo2-7B. (b) Average difference in maximum P(o) across layers for images containing an object vs. random images not containing that object. Verbalization lens increases the gap between the two settings more than raw logit lens, especially in early-middle layers. Using the weights of all attention heads gives a weaker effect than just verbalization heads. for a particular category. For each image, we take the maxi- mum P (o) under verbalization lens across all image tokens as the model’s confidence that an image contains o, focus- ing on one layer at a time. We filter to include only single- token categories (56/80 COCO classes for Qwen3 models). As found in prior work [27], raw logit lens can distinguish COCO objects. But verbalization lens leads to much higher probabilities for images that contain o (Figure 6a). Verbal- ization lens also achieves consistent separation across lay- ers, whereas raw logit lens only begins to work in late lay- ers (Figure 6b). Figure 19 shows similar results with ROC- AUC: raw logit lens offers some discrimination, but verbal- ization lens realizes near perfect AUC across layers. 3.5. Object Localization We find that verbalization lens generally reveals informa- tion localized to relevant regions of an image. Similar to Section 3.4, we calculate P (o) across all image tokens at a given layer, which yields a heatmap of confidence scores across image tokens. We can use this approach to obtain probabilities at a particular location for any token in the model’s vocabulary. For example, in Figure 7a, the to- ken with the maximum P (orange) is the token containing a small orange. Localization is also sensitive to nuances in word meaning. While the tokenphone localizes image tokens that appear to contain a mobile or landline phone, “手机” (a Chinese word specifically meaning cell phone) localizes only the cell phone and “电话” (a word mostly used for landlines) localizes only the landline. Table 1. Token-level localization results for Qwen3-VL models. Here, we follow [27] by taking the maximum probability for each token across layers. We use the top 10% of verbalization heads based on results from Figure 3, and thus also sample 10% of heads for the random baseline. For mIoU, we choose the best thresh- old based on 128 random images and report results for a different sample of 512 images, all from the COCO 2014 validation set. See Appendix F for details. Qwen3-VL-8BQwen3-VL-2B MethodmIoUmAPmIoUmAP Raw Logit Lens0.1610.3740.2230.462 Random Heads0.1670.3020.1570.224 Verbalization Lens0.1900.4070.2490.463 Following [27], we calculate segmentation metrics for our approach using the COCO 2014 validation set [37]. For a given image, we retrieve all of the single-token categories 4 o in that image and use our lens to generate segmentation maps for each category in the image. To generate a seg- mentation map, we obtain a score P (o) for each token by taking the maximum probability of o across layers. We then binarize these probabilities to score lens predictions against ground truth masks by choosing a threshold τ on a disjoint set of images (or sweep across τ for AP), and average re- sulting scores across all objects/images. Unlike prior object localization work [4, 14, 27], we evaluate at the token-scale 4 This number is roughly similar across models; see Table 2. Original Image 手机 (cell phone) +Verbalization Lens _phone +Verbalization Lens 电话 (telephone) +Verbalization Lens _orange +Verbalization Lens _orange Raw Logit Lens (a) (b) Figure 7. We use verbalization lens to measure P(o) across image tokens for an object o, and find that high probabilities are concentrated in image tokens containing o. (a) P(orange)=0.05 for a token containing an orange, whereas raw logit lens probabilities are lower and scattered across the image. (b) Localization is sensitive to the specific word used: P(phone) is high for both phones in the image, but “手机” (a Chinese word specifically meaning cell phone) localizes only the cell phone, and “电话” (a word mostly used for landlines) localizes only the landline. Results are for Qwen3-VL-2B (layer 15), with the top 10% of verbalization heads. instead of upsampling our predictions to the pixel scale. 5 We compare to logit lens and an equivalent number of random heads. Table 1 shows that verbalization lens is slightly more spatially localized than baselines. However, numbers are overall quite low—this may in part be due to register token behavior [7, 26], where general information about an image is stored in arbitrary tokens (e.g., Figure 5f). 3.6. Comparison to LatentLens LatentLens [32] is an alternative approach to decoding im- age activations: instead of using unembeddings, it measures cosine similarity of image tokens against a large pool of contextual text embeddings from intermediate LLM layers. However, Krojer et al. [32] report failure modes in the early and mid-layers of some larger models such as Llava-Next- 34B. Adopting the same LLM judge interpretability metric as Krojer et al. [32], we find that verbalization lens produces interpretable tokens even in these difficult settings, with a 40.5 point increase in judge scores for Llava-Next-34B on average across layers. For all other models, our approach is on par with LatentLens; see Appendix G for details. 5 We define a patch as containing an object if more than 10% of its pixels correspond to the ground-truth COCO mask. 4. Editing Conceptual Information Are the representations read out by these verbalization heads a mirage, or are they causally relevant for the model’s “understanding” of an image? Instead of merely verbaliz- ing hidden state representations, we use the subspace read by L p to edit Qwen3-VL models’ perception of an image, finding that we can, e.g., cause the model to describe an ant in an image as an iPod. 4.1. Approach Let c rem correspond to a concept we want to remove, and c add be a concept we want to add in its place. Here, we focus on single-token concepts. If we take the unembed- ding vector u c ∈R d model corresponding to the token for the concept c, we can obtain a latent vector v c for that concept using the inverse of the transformation from Equation 3: v c = L −1 p u c .(5) where v c ∈R d model is a direction in model activation space that encodes the concept c. Although L p is typically full- rank (and thus invertible), we find that in practice it has a long tail of small singular values (e.g., Figure 14). This means that directly taking the inverse of L p would cause an rank 892, sample 5 — coherence: 10 remove_prevalence: 2 add_prevalence: 2 specificity: 9 This image shows a whimsical and creative cake designed to look like a vintage computer setup, presented inside an open cardboard rank 234, sample 2 — coherence: 10 remove_prevalence: 0 add_prevalence: 3 specificity: 8 This is a dynamic, low-light photograph capturing a live percussion performance on a dark stage. Main Subjects: • Center Stage: A male percussionist is the focal point, energetically playing a large, traditional-looking set of wooden hammers or mallets. He is wearing a distinctive, patterned vest over a dark shirt and white pants. He is actively striking the instruments with both hands, suggesting a powerful and rhythmic performance. • Left Side: A female percussionist stands to the left, also engaged in playing. She appears to be using a smaller set of instruments, possibly a set of hand drums or a similar percussion setup, and is holding a mallet or stick. She is wearing a light-colored top. Stage Setup: • The stage is dominated by large, wooden percussion instruments, including what appears to be a massive, multi-tiered xylophone or marimba on the right, and a set of horizontal wooden bars on the left. • The instruments are illuminated by warm, reddish-orange stage lighting, which creates a dramatic contrast against the black background. • Various stands, cables, and microphones are visible, indicating a professional live performance setup. Audience: • The foreground is dark, showing the silhouettes of audience members' heads, indicating the photo was taken from the audience perspective. Overall Impression: The image conveys the energy and intensity of a live musical performance, highlighting the skill and focus of the percussionists. The dramatic lighting and dark background emphasize the performers and their instruments, creating a powerful and immersive scene. The style suggests a traditional or cultural music performance, possibly from a non-Western tradition, given the type of instruments and the performers' attire.<|im_end|> rank 234, sample 5 — coherence: 10 remove_prevalence: 0 add_prevalence: 10 specificity: 9 This is a close-up, studio-style photograph of a sleek, black mouse or computer mouse, presented against a clean white background. The mouse has a glossy, dark finish that reflects light, giving it a modern and premium appearance. It features prominent silver-colored metal accents, including: • A large, curved metal pull handle on the top surface, secured with four small, round, gold-colored rivets. • Silver-colored metal corner caps on each of the four corners, which also appear to be fastened with small gold-colored rivets. • A smaller, matching silver metal latch mechanism on the left side, which likely serves as a button or release mechanism. On the left side of the mouse, there is a decorative panel with an intricate, golden, gear-like or mechanical design, adding a touch of vintage or steampunk-inspired detail to the otherwise minimalist and modern aesthetic. The lighting is soft and even, highlighting the mouse’s contours and the contrast between the (b) (a) Qwen3-2B: “ant” “iPod”← rank 219, sample 4 — coherence: 9 remove_prevalence: 0 add_prevalence: 10 specificity: 9 This is a close-up, macro photograph of a single, red-brown, human- made object, specifically an iPod, resting on a green plant stem. The image is taken in a dark environment, with the iPod and the stem being the primary subjects against a black background. The iPod is positioned diagonally across the frame, with its left side facing the viewer. It is a classic model with a glossy, reddish-brown finish. The device has a distinct design with a rounded, curved body, a small, dark, circular earpiece on the left side, and a visible earphone jack on the right. The iPod is attached to a green plant stem, which is visible in the foreground and background. The stem has a smooth, slightly glossy surface and a light green color. The background is completely black, which makes the iPod and the stem stand out prominently. The image is a detailed, high-resolution photograph, likely taken with a macro lens, capturing the intricate details of the iPod and the plant stem. The lighting is focused on the subjects, creating a strong contrast with the dark background. The image is marked with the text "UGA1366066" in the bottom right corner, which appears to be a digital watermark or identifier. In summary, the image is a detailed, high-resolution macro photograph of a red-brown iPod resting on a green plant stem, set against a black background.<|im_end|> 2B Qwen3-8B: “hare” “cucumber” ← This is a close-up, macro photograph of a single, red-brown, human-made object, specifically an iPod, resting on a green plant stem. ... The iPod is positioned diagonally across the frame, with its left side facing the viewer. It is a classic model with a glossy, reddish-brown finish. ... The iPod is attached to a green plant stem, which is visible in the foreground and background. This is a charming and whimsical photograph of a Snowshoe cucumber (Cucumis sativus) in a natural outdoor setting. Main Subject: The central focus is a large, healthy-looking Snowshoe cucumber. It is sitting upright on a patch of grass ... The cucumber has a robust, rounded form with a textured, mottled brownish-green skin. Its most striking features are its large, dark, expressive eyes and its prominent, upright, greenish-brown ears (leaves). rank 463, sample 4 — coherence: 10 remove_prevalence: 0 add_prevalence: 10 specificity: 9 This is a charming and whimsical photograph of a Snowshoe cucumber (Cucumis sativus) in a natural outdoor setting. Main Subject: The central focus is a large, healthy-looking Snowshoe cucumber. It is sitting upright on a patch of grass and soil, with its body angled slightly to the left. The cucumber has a robust, rounded form with a textured, mottled brownish-green skin. Its most striking features are its large, dark, expressive eyes and its prominent, upright, greenish-brown ears (leaves). A black leash is attached to its ear, suggesting it is being walked or is a pet, which adds a humorous and anthropomorphic element to the image. Background: The background is softly blurred, drawing attention to the cucumber. It consists of a mix of green grass, brown soil, and scattered leaves. In the upper right background, a small, bright red bird (possibly a male Rose-breasted Grosbeak) is visible, adding a splash of color and life to the scene. Text and Context: At the bottom of the image, there is a caption that reads: "Snowshoe Cucumber (Churchill, Canada) Copyright 2009 - Monte M. Taylor." This text provides context, indicating the location (Churchill, Canada) and the photographer (Monte M. Taylor) and the year the photo was taken (2009). Overall Impression: The image is playful and endearing, capturing a moment of quiet observation in nature. The use of a leash on the cucumber is clearly a humorous artistic choice, transforming a vegetable into a cute, walkable pet. The natural lighting and shallow depth of field create a soft, pleasant atmosphere.<|im_end| > Qwen3-8B Qwen3-2B Qwen3-8B: “pineapple” “washer” ← This is a close-up, top-down photograph of a white ceramic plate with a decorative green leaf or vine pattern around its rim. ... The most striking feature is the arrangement of objects on the plate: - A large, white, plastic or ceramic washer/dishwasher part (possibly a washing machine agitator or a decorative piece) is prominently displayed in the center. It has a complex, layered, star-like or flower-like shape with multiple ridges and grooves. prevalence: 0Specificity: 9 prevalence: 10Coherence: 9 c rem c add prevalence: 0Specificity: 9 prevalence: 10Coherence: 9 c rem c add rank 463, sample 5 — coherence: 10 remove_prevalence: 0 add_prevalence: 10 specificity: 4 prevalence: 0Specificity: 4 prevalence: 10Coherence: 10 c rem c add Mean Diff. Prevalence Mean Diff. Prevalence Scaling Factor α Scaling Factor α Figure 8. We can use our verbalization transformation to edit image activations, e.g., replacing the concept of “ant” with “iPod.” (a) Qualitative examples for editing objects within an image. Rank is constant based on our initial sweep; we show α=4 for the first two images, and α=5 for the third. (b) Quantitative LLM judge scores across random ImageNet [49] images (n=256). Verbalization lens yields latent vectors that can effectively edit image representations without heavily damaging irrelevant concepts, or degrading caption coherence. Edits are most effective with a larger scaling factor α. We use a rank for each model that explains the top ρ=60% of verbalization energy, based on an initial sweep over 100 images (Appendix I). explosion of singular values in L −1 p . Instead of taking the full inverse, we can take a pseudo-inverse: L p = UΣV ∗ (6) L + p,k = V[:, : k]Σ −1 U ∗ [: k](7) where 1≤ k ≤ d model is an integer hyperparameter, chosen via sweep on a held-out set. Here, we interpret bottom sin- gular values as noise, manually setting them to zero rather than letting them affect our transformation. This approach implicitly views L p as reading from a low-rank subspace of the residual stream, as described in previous work [21]. It also means that our edit is more surgical, as we are only affecting activations within a low-rank subspace. Once we have obtained our latent vectors v c rem , v c add for the concepts we would like to add and remove, we edit ac- tivations a (l) t ∈R d model across all layers l and image token positions t. For a given activation vector, we remove the component of that activation in the direction of v c rem , and add in v c add at the same relative magnitude: λ = (a (l) t · v c rem )/||v c rem || 2 (8) a (l)∗ t = a (l) t − λv c rem + αλv c add (9) where we allow for a scaling factor α ∈R on the added concept c add as an additional hyperparameter. Note that if c rem is not present in this activation vector, λ will be close to zero, and Equation 9 becomes a no-op. This is by design: we do not want to edit token positions where there is noth- ing to replace. Thus, while we apply this edit to all image tokens, it only has an effect at relevant token positions. 4.2. Evaluation To evaluate the efficacy of our edits, we prompt models to free generate descriptions with intervened image represen- tations. We prompt o4-mini to score the generated captions according to four metrics on a scale from 0 to 10: • Prevalence of c add . e.g., how prominently does the con- cept of ‘dog’ feature in this caption? • Prevalence of c rem . e.g., how prominently does the con- cept of ‘motorcycle’ feature in this caption? • Specificity. Apart from any differences regarding the concepts c rem and c add , how similar is the counterfactual caption to the ground truth vanilla caption? • Coherence. In terms of writing quality, how coherent is this caption? See Appendix H for full prompts. We obtain scores for all generated and vanilla captions. To measure edit success, we measure the ∆ in prevalence scores for a counterfactual caption compared to scores for the vanilla generation. We evaluate on ImageNet [49] images as they are generally fo- cused on one subject at a time: images are assigned c rem based on their class label, sampling only from images with single-token class labels. Then, c add is randomly sampled from all other single-token ImageNet class names. 4.3. Results Editing with latent vectors obtained via the verbalization head subspace is more effective than directly editing with unembeddings. After performing an initial sweep over rank and α (see Appendix I for details), we select the rank that explains 60% of the energy of L p and run our edit with this setting for n=256 random images, with results shown in Figure 8b. We find using α > 1 is most effective. In- tuitively, this means that c add must be added more strongly than c rem was present in order to have a causal effect; per- This is a highly stylized and surreal image of a revolver that has been creatively modified to resemble a full-sized, classic car — specifically, a red and silver revolver that looks like a miniature, toy-like car with wheels and a license plate. Here’s a detailed description: - The Central Object: The main subject is a revolver, but it has been artistically altered to look like a car. It has a red body, a silver front bumper, and a license plate that reads “JN1 209”. The revolver’s barrel has been extended to look like a front bumper, and the cylinder is visible as part of the engine block. The revolver’s trigger guard and grip have been transformed into a car’s steering wheel and dashboard. - The Wheels: The revolver’s cylinder and the revolver’s hammer have been replaced with large, detailed wheels. The front wheel is silver and has a realistic tire tread, while the rear wheel is black and has a more rugged, off-road look. The revolver’s trigger has been replaced with a car’s gear shift. - The Driver: Inside the “car,” there is a man sitting in the driver’s seat, looking out the window. He is wearing a black shirt and has a serious expression. His hand is on the steering wheel, and he appears to be driving the car. - The Setting: The “car” is parked on a paved road, with trees and bushes in the background. The trees are green and brown, suggesting it is autumn. The sky is overcast, and the lighting is soft and diffused. - The Style: The image has a surreal, dreamlike quality. It blends the familiar with the fantastical, creating a sense of wonder and imagination. The revolver, which is typically associated with danger and violence, has been transformed into a playful, toy-like car, which is a striking contrast. Overall, this image is a creative and imaginative piece of art that plays with the viewer’s expectations and perceptions. It is a fun and whimsical image that invites the viewer to look closer and appreciate the details. Editing Object Perception for Qwen3-VL-8B-Instruct Edited with Verbalization Transformation: “tractor” “revolver” ← This is a photograph of a red vintage tractor driving on a paved road, set against a backdrop of dense, leafy trees. Key Details: - The Tractor: The central subject is a red Massey Ferguson tractor, likely a model from the 1970s or 1980s. It features a prominent, enclosed cab with a black frame and glass windows. The front grille has the "MF" logo, and the model number "135" is visible on the side. The tractor has large, rugged rear tires and smaller front wheels, typical of agricultural machinery. A small orange warning light is mounted on the roof of the cab. - The Driver: A man is visible inside the cab, sitting in the driver's seat and looking towards the camera. He is wearing a dark shirt. - The License Plate: The tractor has a UK-style license plate that reads "JN1 203". - The Setting: The tractor is on a grey asphalt road. The background is filled with lush green and autumnal trees and bushes, suggesting the photo was taken in late summer or early autumn. The lighting appears to be natural daylight, possibly overcast. - Overall Impression: The image captures a classic, well-maintained piece of farm equipment in a rural setting, evoking a sense of nostalgia and the enduring presence of agricultural machinery. The red color of the tractor stands out vividly against the green and brown foliage. Qwen3-VL-8B Vanilla Generation (top 10% OCR heads, rank 463, alpha=4) proper version where we have the right rank! rank 463, sample 4 — coherence: 9 remove_prevalence: 0 add_prevalence: 10 specificity: 2 “Describe this image.” (ImageNet Class: Tractor) Figure 9. We can use our verbalization transformation to edit Qwen3-VL-8B image activations, replacing the concept of “tractor” with “revolver.” We obtain latent vectors for these concepts using the inverse of our verbalization matrix. After subtracting the latent vector for “tractor” and adding the vector for “revolver,” Qwen3-VL-8B describes the main subject of the image as a revolver. We leave all other image features untouched. The model reconciles remaining visual features (like wheels, a license plate, and a driver) by describing the image as a “surreal image of a revolver that has been creatively modified to resemble a full-sized, classic car.” Both captions are generated with greedy decoding. This generation uses α=4; across all examples, we use the top ρ=60% ofL p energy, where p=10%. haps this is to override remaining information in the residual stream related to c rem but not captured in v c rem . We show three abridged qualitative examples in Fig- ure 8a. We can see that, e.g., the concept of “ant” has been cleanly replaced by the concept of “iPod,” while other vi- sual features (like background descriptions) remain intact. These edits preserve low-level visual features of the original object: i.e., the color and texture of the ant is now applied to the iPod, with the model describing the iPod as being “a classic model with a glossy, reddish-brown finish.” Figure 9 shows a full example edit for Qwen3-VL- 8B alongside the model’s vanilla generation.Here, c rem =“tractor,” and we randomly sample c add =“revolver”. Our edit is successful: the model does not describe the im- age as containing a tractor, but rather as an image of a re- volver. Interestingly, because other aspects of the image remain untouched (wheels, license plate, bumper, and man inside the tractor), Qwen synthesizes these attributes into a “surreal” description of a “revolver that has been creatively modified to resemble a full-sized, classic car.” 5. Related Work Lenses. Logit lens [45] was the first to find that LLM in- ternal hidden states can be projected to vocabulary direc- tions with interpretable results, and was followed by a large body of work on other “lenses” to decode internal states [2, 10, 12, 13, 16, 19, 21, 30, 45, 46]. Prior work has ap- plied logit lens approaches to VLMs [27, 35, 44]; one ap- proach [32] provides an alternative to vocabulary decoding using cached latent representations of text. Earlier, [28] also showed that CLIP’s classifier head can act as a “logit lens” for its intermediate states. Our work augments the inter- pretability of these vanilla logit lens approaches. Language and Vision. What is the alignment of language and vision in models? For CLIP vision models, previ- ous work studied “multimodal” neurons that respond to both text referring to a concept and images of that con- cept [17, 40]. Several works have shown that with just a small adapter module, one can map from image to text [39, 43, 57], even with both models frozen [42, 50]. Other works study vision representations through the lens of lan- guage [5, 14], or differentiation between vision and lan- guage representations [22, 24, 47]. Others have studied how language priors can conflict with visual features [23, 53], particularly for OCR [20, 29, 36]. Our results are evoca- tive of recent work on how VLMs use language as a me- diator for visual understanding [48, 51, 55]. One question is whether image representations are aligned with language in early VLM layers: while earlier work claims that align- ment arises only in middle layers [25, 44, 50, 59], others as- sert early-layer alignment [32, 62], which our method also shows. Other heads. We note conceptual overlap between ver- balization heads and two types of attention heads studied in prior work: filter heads [52] and gaze heads [15]. We com- pare against these and find modest overlap in Appendix C, suggesting that there may be a relationship between these components. This further supports our claim that the narrow setting of OCR provides a straightforward way of identify- ing heads responsible for more general VLM mechanisms. 6. Conclusion In this work, we find that attention heads responsible for OCR can be used to verbalize general semantics of non-text image tokens across four VLMs. We collapse the weights of these attention heads into a single matrix that yields more interpretable labels than raw logit lens, and show that the subspace this matrix reads from is causally relevant for model descriptions of images. Taken together, our results show how study of narrow mechanisms can shed light on broader interpretability problems. Acknowledgements SF thanks Si Wu for discussions on language and letters throughout this project, as well as Rohit Gandikota and Arnab Sen Sharma for feedback on early experiments. We thank Andy Arditi, Eric Todd, and Grace Proebsting for feedback on initial drafts. SF is funded by a grant from Co- efficient Giving. BK and DB are funded by NSF #2403304. References [1] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhao- hai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Jun- yang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jian- qiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yux- uan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report, 2025. 2, 13 [2] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Ja- cob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. CoRR, abs/2303.08112, 2023. 8 [3] Adrian Chang, Sheridan Feucht, Byron C Wallace, and David Bau. Does FLUX know what it’s writing? In NeurIPS Mechanistic Interpretability Workshop, San Diego, 2025. 14 [4] Hila Chefer, Shir Gur, and Lior Wolf. Transformer inter- pretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 782–791, 2021. 5 [5] Haozhe Chen, Junfeng Yang, Carl Vondrick, and Chengzhi Mao. Interpreting and controlling vision foundation models via text explanations, 2023. 8 [6] Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammadreza Salehi, Rohun Tripathi, Sangho Lee, Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, Vincent Shao, Yue Yang, Weikai Huang, Ziqi Gao, Taira Anderson, Jianrui Zhang, Jitesh Jain, George Stoica, Winson Han, Ali Farhadi, and Ranjay Krishna. Molmo2: Open weights and data for vision-language models with video understanding and grounding. CoRR, abs/2601.10611, 2026. 13 [7] Timoth ́ e Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net, 2024. 4, 6 [8] Stanislas Dehaene and Laurent Cohen. The unique role of the visual word form area in reading. Trends in Cognitive Sciences, 15(6):254–262, 2011. 1 [9] Stanislas Dehaene, Laurent Cohen, Jos ́ e Morais, and R ́ egine Kolinsky.Illiterate to literate: behavioural and cerebral changes induced by reading acquisition. Nature Reviews Neuroscience, 16(4):234–244, 2015. 1 [10] Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to conclusions: Short-cutting transform- ers with linear transformations. In Proceedings of the 2024 Joint International Conference on Computational Linguis- tics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pages 9615–9625. ELRA and ICCL, 2024. 8 [11] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Das- Sarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.https://transformer- circuits.pub/2021/framework/index.html. 2, 4 [12] Sheridan Feucht, Eric Todd, Byron Wallace, and David Bau. The dual-route model of induction. In Second Conference on Language Modeling, 2025. 3, 4, 8, 14 [13] Sheridan Feucht, Byron Wallace, and David Bau. Vector arithmetic in concept and token subspaces. In Second Mech- anistic Interpretability Workshop at NeurIPS, 2025. 2, 4, 8 [14] Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting clip’s image representation via text-based de- composition. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. 5, 8 [15] Rohit Gandikota and David Bau. Gaze heads: How vlms look at what they describe. arXiv preprint arXiv:2606.14703, 2026. 8, 12 [16] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Gold- berg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space, 2022. 8 [17] Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah.Multimodal neurons in artificial neural net- works. Distill, 2021. https://distill.pub/2021/multimodal- neurons. 8 [18] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the V in VQA matter: Ele- vating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 13 [19] Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, Ben Thomp- son, David Abrahams, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, and Jack Lindsey. Verbalizable representations form a global workspace in language mod- els. Transformer Circuits Thread, 2026. 8 [20] Zhentao He, Can Zhang, Ziheng Wu, Zhenghao Chen, Yufei Zhan, Yifan Li, Zhao Zhang, Xian Wang, and Minghui Qiu. Seeing is believing? mitigating ocr hallucinations in multi- modal large language models, 2025. 1, 8 [21] Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Be- linkov, and David Bau. Linearity of relation decoding in transformer language models. In Proceedings of the 2024 In- ternational Conference on Learning Representations, 2024. 7, 8 [22] Etha Tianze Hua, Tian Yun, and Ellie Pavlick.Source- modality monitoring in vision-language models, 2026. 8 [23] Tianze Hua, Tian Yun, and Ellie Pavlick. How do vision- language models process conflicting information across modalities? CoRR, abs/2507.01790, 2025. 8 [24] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola.The platonic representation hypothesis.CoRR, abs/2405.07987, 2024. 8 [25] Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model, 2024. 2, 8 [26] Nick Jiang, Amil Dravid, Alexei A. Efros, and Yossi Gan- delsman. Vision transformers don’t need trained registers. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Sys- tems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, 2025. 6 [27] Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision-language rep- resentations to mitigate hallucinations, 2025. 2, 4, 5, 8 [28] Sonia Joseph and Neel Nanda. Laying the foundations for vision and multimodal mechanistic interpretability & open problems, 2024. Accessed: 2024-06-28. 8 [29] Antonia Karamolegkou, Nicolas Angleraud, Beno ˆ ıt Sagot, and Thibault Cl ́ erice. Reading or guessing? visual grounding failures of vision-language models for ocr in ancient greek editions, 2026. 1, 8 [30] Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf. Backward lens: Projecting language model gradients into the vocabulary space, 2024. 8 [31] Benno Krojer, Dheeraj Vattikonda, Luis Lara, Varun Jam- pani, Eva Portelance, Chris Pal, and Siva Reddy. Learn- ing action and reasoning-centric image editing from videos and simulation. In Advances in Neural Information Process- ing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. 14 [32] Benno Krojer, Shravan Nayak, Oscar Ma ̃ nas, Vaibhav Ad- lakha, Desmond Elliott, Siva Reddy, and Marius Mosbach. Latentlens: Revealing highly interpretable visual tokens in LLMs. In Forty-third International Conference on Machine Learning, 2026. 2, 4, 6, 8, 13, 14, 19 [33] Vedang Lad, Jin Hwa Lee, Wes Gurnee, and Max Tegmark. Remarkable robustness of llms: Stages of inference? In Ad- vances in Neural Information Processing Systems 38: An- nual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, 2025. 4 [34] Andrew Lee, Yonatan Belinkov, Fernanda B. Vi ́ egas, and Martin Wattenberg. Decomposing query-key feature interac- tions using contrastive covariances. CoRR, abs/2602.04752, 2026. 3 [35] Yueyan Li, Chenggong Zhao, Zeyuan Zang, Caixia Yuan, and Xiaojie Wang. Reading images like texts: Sequential im- age understanding in vision-language models. arXiv preprint arXiv:2509.19191, 2025. 8 [36] Yunhao Liang, Ruixuan Ying, Bo Li, Hong Li, Kai Yan, Qingwen Li, Min Yang, Okamoto Satoshi, Zhe Cui, and Shi- wen Ni. Visual merit or linguistic crutch? a close look at deepseek-ocr, 2026. 1, 8 [37] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ́ ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Computer Vision - ECCV 2014 - 13th Eu- ropean Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, pages 740–755. Springer, 2014. 4, 5, 13 [38] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 13 [39] Oscar Ma ̃ nas, Pau Rodriguez Lopez, Saba Ahmadi, Aida Ne- matzadeh, Yash Goyal, and Aishwarya Agrawal. MAPL: Parameter-efficient adaptation of unimodal pre-trained mod- els for vision-language few-shot prompting. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 2523–2548, Dubrovnik, Croatia, 2023. Association for Computational Linguistics. 8 [40] Joanna Materzynska, Antonio Torralba, and David Bau. Disentangling visual and written concepts in CLIP.In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18- 24, 2022, pages 16389–16398. IEEE, 2022. 8 [41] Bruce D. McCandliss, Laurent Cohen, and Stanislas De- haene. The visual word form area: expertise for reading in the fusiform gyrus. Trends in Cognitive Sciences, 7(7): 293–299, 2003. 1 [42] Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. In The Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- view.net, 2023. 8 [43] Ron Mokady, Amir Hertz, and Amit H. Bermano. Clipcap: CLIP prefix for image captioning. CoRR, abs/2111.09734, 2021. 8 [44] Clement Neo, Luke Ong, Philip H. S. Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual in- formation processing in vision-language models.ArXiv, abs/2410.07149, 2024. 1, 2, 8 [45] Nostalgebraist.Interpreting gpt:The logit lens. https://w.alignmentforum.org/posts/ AcKRB8wDpdaN6v6ru/interpreting-gpt-the- logit-lens, 2020. Accessed: 23 Sep 2024. 2, 3, 8 [46] Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C Wallace, and David Bau. Future lens: Anticipating subsequent to- kens from a single hidden state. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 548–560, 2023. 8 [47] Isabel Papadimitriou, Huangyuan Su, Thomas Fel, Naomi Saphra, Sham M. Kakade, and Stephanie Gil. Interpret- ing the linear structure of vision-language model embedding spaces. CoRR, abs/2504.11695, 2025. 8 [48] Athulith Paraselli, Etha Tianze Hua, and Ellie Pavlick. Slow to see, slow to suppress: Understanding the effects of modal- ity in context-memory conflicts, 2026. 8 [49] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition chal- lenge. CoRR, abs/1409.0575, 2014. 2, 7, 14 [50] Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. Multimodal neurons in pre- trained text-only transformers. In IEEE/CVF International Conference on Computer Vision, ICCV 2023 - Workshops, Paris, France, October 2-6, 2023, pages 2854–2859. IEEE, 2023. 1, 2, 8 [51] Haz Sameen Shahgir, Xiaofu Chen, Yu Fu, Erfan Shayegani, Nael B. Abu-Ghazaleh, Yova Kementchedjhieva, and Yue Dong.Vlms need words: Vision language models ig- nore visual detail in favor of semantic anchors.CoRR, abs/2604.02486, 2026. 8 [52] Arnab Sen Sharma, Giordano Rogers, Natalie Shapira, and David Bau. Llms process lists with general filter heads. The Fourteenth International Conference on Learning Represen- tations (ICLR 2026), 2025. 3, 8, 12 [53] Yan Shu, Hangui Lin, Yexin Liu, Yan Zhang, Gangyan Zeng, Yan Li, Yu Zhou, Ser-Nam Lim, Harry Yang, and Nicu Sebe. When semantics mislead vision: Mitigating large mul- timodal models hallucinations in scene text spotting and un- derstanding, 2025. 8 [54] Mustafa Shukor and Matthieu Cord. Implicit multimodal alignment: On the generalization of frozen llms to multi- modal inputs. In Advances in Neural Information Process- ing Systems, pages 130848–130886. Curran Associates, Inc., 2024. 2 [55] Alexa R. Tartaglini, Satchel Grant, Daniel Wurgaft, Christo- pher Potts, and Judith E. Fan. Diagnosing bottlenecks in data visualization understanding by vision-language models, 2025. 8 [56] Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau.Function vectors in large language models.In The Twelfth Inter- national Conference on Learning Representations, 2024. arXiv:2310.15213. 4 [57] Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In Advances in Neu- ral Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 200–212, 2021. 8 [58] Constantin Venhoff, Ashkan Khakzar, Sonia Joseph, Philip Torr, and Neel Nanda. How visual representations map to language feature space in multimodal llms, 2025. 1, 2 [59] Constantin Venhoff, Ashkan Khakzar, Sonia Joseph, Philip H. S. Torr, and Neel Nanda. Too late to recall: Explaining the two-hop problem in multimodal knowledge retrieval. In Advances in Neural Information Processing Systems 38: An- nual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, 2025. 2, 8 [60] Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. CoRR, abs/2211.00593, 2022. 3 [61] Eliot Weinberger. 19 ways of looking at Wang Wei: with more ways. New Directions Books, New York, NY, 2016. Afterword by Octavio Paz. 14 [62] Ev ˇ zen Wybitul, Javier Rando, Florian Tram ` er, and Stanislav Fort. Representations of text and images align from layer one, 2026. 2, 4, 8 [63] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jia- long Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tian- hao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. 12 A. Model Details In this work, we study four models of varying sizes with three distinct model families. See Table 2 for architectural details of models studied in this paper. B. Finding Verbalization Heads OCR dataset examples are shown in Figure 10. We use large font sizes and randomly sample colors that contrast with the background. Table 3 shows model accuracy for these stimuli. Due to models having poor OCR performance on stimuli with white backgrounds, we opt for realistic back- grounds. Figure 10. Random examples from our OCR dataset used to iden- tify heads in Section 2. Results from Section 2 are shown for Molmo2-7B and Llava-Next-34B in Figure 11 and Figure 12. We also pro- gressively ablate heads responsible for OCR for Qwen3- VL-2B on VQA in Figure 13. Qwen3-2B OCR ScoresQwen3-8B OCR Scores LayerLayer Head Index Head Index Qwen3-2B Head Mean AblationsQwen3-8B Head Mean Ablations OCR Accuracy Percent of model heads ablatedPercent of model heads ablated Molmo2-7B OCR ScoresLlava-Next-34B Scores Layer Layer Head IndexHead Index Figure 11. OCR scores from Equation 1 for Molmo2-O-7B and Llava-Next-34B. The takeaway is similar to Figure 2 for Qwen3- VL models: OCR heads appear in late layers. Qwen3-2B OCR ScoresQwen3-8B OCR Scores LayerLayer Head Index Head Index Qwen3-2B Head Mean AblationsQwen3-8B Head Mean Ablations OCR Accuracy Percent of model heads ablatedPercent of model heads ablated Molmo2-7B OCR ScoresLlava-Next-34B Scores Layer Layer Head IndexHead Index Qwen3-2B Head Mean AblationsQwen3-8B Head Mean Ablations OCR Accuracy Molmo2-7B Head Mean AblationsLlava-Next-34B Head Mean Ablations OCR Accuracy Percent of model heads ablatedPercent of model heads ablated Figure 12. Progressively mean-ablating top-scoring OCR heads for Molmo2-O-7B and Llava-Next-34B. Results are similar to Qwen3-VL results from Figure 3. C. Overlap with Other Heads Attention patterns from Figure 4 are reminiscent of filter heads from prior work [52]. Are they the same heads? We find that of the 11 filter heads found in Qwen3-1.7B [63] (the base LLM for Qwen3-VL-2B-Instruct), six of them are also in the top-10% of verbalization heads. This suggests that vision fine-tuning may cause filter heads to “evolve” into verbalization heads. We also check for overlap with gaze heads [15], a type of attention head in VLMs that keeps track of which portion of an image the VLM is captioning. For Qwen3-VL-2B, 30% of gaze heads are also verbalization heads (3/10), and for Qwen3-VL-8B, 29% of gaze heads are also verbalization heads (29/100). This is higher than the expected overlap when sampling the same number of random heads. 6 Al- though this means that verbalization heads are somewhat distinct from gaze heads, the modest overlap also supports our claim that analysis of OCR can identify heads more gen- erally responsible for mapping from pixels to semantics. D. Full Verbalization Lens Examples In Figures 16, 17, and 18, we show full verbalization lens outputs for models not shown in Figure 5. All outputs are for layer 0, and compared to raw logit lens outputs at layer 0. E. Object Confidence For each category, we sample n=512 images containing o and n images without o, except for Llava-Next-34B, for which n=128 due to computational constraints. Figure 19 shows these results using ROC-AUC curves, instead of cal- culating average difference in maximum P (o) as in the main paper. Interestingly, ROC-AUC shows that raw logit lens has more signal for Qwen models, despite verbalization lens still showing marked improvements. F. Object Localization Details Here, we provide implementation details for Section 3.5. For each image, we score one segmentation map per object category present in that image, where the ground truth mask is the union of all instances of that category (e.g., if there are multiple people in an image, our ground truth mask includes all of those people). For the 512 images that we display numbers for in Table 1, there are 1216 (image, category) pairs, an average of just over two object categories per im- age. Averaging over these pairs ensures that each object counts once regardless of image/object size. 6 For Qwen3-VL-2B, we have 45 verbalization heads and 10 gaze heads, so the expected number of overlapping heads is (10×45)/448, or 1/10 gaze heads. For Qwen3-VL-8B, we have 115 verbalization heads and 100 gaze heads, so the expected overlap with random samples is (100×115)/1152, or 10/100 gaze heads. Table 2. Architectural details of models used in this paper. “Tied” refers to tied embeddings, “Img Encoder” refers to whether the image encoder is trained end-to-end or frozen, and “Single-tok COCO” is the number of COCO [37] categories that are single-token for this tokenizer (which determines results in Sections 3.4-3.5). Full Model NameCiteAbbrv.Total HeadsTied?Img Encoder?Single-tok COCO Qwen/Qwen3-VL-2B-Instruct[1]Qwen3-VL-2B28 lyrs× 16 hds = 448YesEnd-to-end56/80 Qwen/Qwen3-VL-8B-Instruct[1]Qwen3-VL-8B36 lyrs× 32 hds = 1152NoEnd-to-end56/80 allenai/Molmo2-O-7B[6]Molmo2-7B32 lyrs× 32 hds = 1024NoEnd-to-end56/80 llava-hf/llava-v1.6-34b-hf[38]Llava-Next-34B60 lyrs× 56 hds = 3360NoEnd-to-end51/80 020406080100 Heads ablated (%) 0.0 0.2 0.4 0.6 0.8 1.0 VQAv2 accuracy yes/no (n=408) 020406080100 Heads ablated (%) number (n=111) 020406080100 Heads ablated (%) other (n=505) Original (58.2%) Top OCR-ranked heads Random - all layers Random - last 50% Qwen3-VL-2B-Instruct Head Mean Ablations by answer type Figure 13. We also measure accuracy for VQAv2 [18] when progressively mean-ablating the top-scoring OCR heads throughout generation. While ablation of these heads does not affect performance for questions that have Yes/No or numeric answers, it does affect performance for other types of questions (e.g., “What is the type of food in this image?” → “Mexican”) slightly more than an equal number of random heads. This suggests that verbalization heads may be useful for more general-purpose question answering tasks. Table 3. OCR Accuracy for randomly-selected English words su- perimposed on either ImageNet or white backgrounds (n = 100). Accuracies are obtained with the prompt “Transcribe the word in this image.” and assistant prefill “Word:”. ModelImageNet BgWhite Bg Qwen3-VL-2B0.890.07 Qwen3-VL-8B0.930.05 Molmo2-O-7B0.900.44 Llava-Next-34B0.800.75 We pick a threshold τ for mIoU based on a disjoint set of 128 images. We show chosen mIoU thresholds in Table 4. We omit token-level accuracy, as in our dataset, only 15.7% of tokens are positive. Therefore, a a baseline that never segments any objects can trivially achieve 84.3% token ac- curacy. G. LatentLens Comparison Details We describe our setup for the LatentLens [32] comparison from Section 3.6 in more detail here. Table 4. Thresholds τ for each approach in Table 1, chosen based on a disjoint set of 128 images. We sweep over a log-spaced range of 60 thresholds based on the range of probability scores for that method. Qwen3-VL-8BQwen3-VL-2B Random Heads7.2×10 −6 5.1×10 −8 Raw Logit Lens3.0×10 −4 4.4×10 −3 Verbalization Lens3.4×10 −3 1.1×10 −3 LatentLens replication. For each of the four models, we build a LatentLens contextual index following Krojer et al. [32]’s released implementation and corpus. We extract con- textual embeddings at every model’s embedding layer plus a set of decoder layers space across the model, and store up to 30 contexts per unique token. Following [32], we retrieve the top-k nearest neighbors by cosine similarity across all indexed layers jointly at query time. As in their evaluation, LatentLens neighbor tokens are often subwords; we expand each to its full containing word using the neighbor’s own source sentence as context (e.g. “ing”→ “rendering”). We find that a small fraction of nearest-neighbor matches (concentrated in Qwen3-VL-8B’s middle layers) are whitespace tokens carrying no semantic content despite high cosine similarity. Thus, we search k = 32 neighbors and take the top-5 non-blank words. Layers. For each model we evaluate six layers: Qwen3-VL-2B 0, 8, 12, 16, 20, 27,Qwen3-VL-8B 0, 8, 12, 20, 24, 35, Molmo2-O-7B 0, 8, 12, 19, 22, 31, Llava-Next-34B 0, 10, 24, 36, 42, 59.See Table 2 for model details. Images. We sample 100 random images (fixed seed) from the same COCO val2014 pool used elsewhere in the pa- per. For each image we decode one patch, chosen pseu- dorandomly (seeded per image) from the central 60% of that model’s patch grid to avoid picking pure back- ground/padding; the same patch is used across all 6 layers for a given image. LLM judge. We use the same LLM-judge as Krojer et al. [32]. We show GPT-5 the full image with a red bound- ing box over the target patch, a cropped close-up of that region, and the top-5 candidate words for one lens at a particular layer. The judge labels each candidate as con- crete (literally visible in the box), abstract (an implied con- cept/activity/quality), or globally related (visible elsewhere in the image but not the box). A patch is scored interpretable if at least one candidate word receives any of these three la- bels. Results. Figure 20 shows the resulting interpretable- token percentage across layers for all four models. For all models, verbalization lens is on par with or better than La- tentLens in terms of LLM judge scores. We note that ver- balization lens is especially helpful for Llava-Next-34B. H. LLM Judge for Editing Table 5 lists LLM judge prompts for evaluating image ed- its from Section 4.2. For each question, the judge returns a structured response containing an integer answer (its rat- ing). I. Editing Sweep for Rank/Scaling Factors We run an initial sweep over ranks and scaling factors for the edit in Section 4. For each setting, we greedy-decode a caption per image. We choose to sweep across scaling factors α∈1, 2, 3, 4, 5, 10. Ranks are chosen for each model based on the number of dimensions it takes to explain ρ% of the energy of L p . That is, if Σ=[σ 1 ,...σ d model ], we take the minimum top-k di- mensions such that sum(Σ[: k] 2 )/sum(Σ 2 ) > ρ/100 for ρ ∈ 1, 5, 10, 20, 40, 60, 80, 90, 100. For reference, we 0500100015002000 Dimension 0 5 10 15 σ Singular Values of Verbalization Lens Qwen3-VL-2B-Instruct ρ=60% of energy ρ=90% of energy Figure 14. Singular values ofL p for Qwen3-VL-2B. We plot dot- ted lines to indicate the ranks that explain 60% and 90% of energy. See Section 4 for discussion of low-rank approximations ofL p . show the singular values of L p for Qwen3-VL-2B in Fig- ure 14. Figures 21 and 22 show results for LLM judge met- rics from Section 4.2 across n=100 randomly-sampled Im- ageNet images/object pairs [49]. We find that our edit gen- erally requires a scaling factor α > 1, and does not have an effect for full-rank L −1 p . We also find that a large scaling factor α=10 is damaging, as it causes specificity to drop. Thus, we choose ρ=60% of verbalization lens energy and perform edits for n=256 additional images across scaling factors (excluding α=10) in Figure 8b. ρ is constant for all qualitative generations shown in the paper. J. Behind the Scenes Following [31] and [32], we describe the trajectory of this project in hopes of humanizing this work and making the research process transparent. This project began because SF was interested in writing and letters. They had worked on letter representations in text-to-image diffusion models before [3], but had just re- turned from an internship and were trying to decide how exactly to continue this line of work. With encouragement from Si Wu and John Cayley, they began working on a project (which is still in-progress) on VLM representations of Chinese poetry; specifically, the poem 鹿柴 by Wang Wei, which has an interesting history of analysis in terms of its translations [61]. In trying to understand how VLMs might process an im- age of this poem, SF set out to first figure out how VLMs do OCR in general. The plan was just to figure out how OCR worked, so that this knowledge could be leveraged to cre- ate interesting visualizations for this art project. But then, SF realized that the heads they had picked out for OCR were much more general when they were playing around with the OV lens approach used in prior work [12]. Clicking on the word “Hey” in a test image yielded that word, as expected. QuestionPrompt template Prevalence of cOn a scale from 0–10, how prominently does the concept of ‘c’ feature in this image caption? Let’s say that a score of 10 means that the concept is the main focus, 0 means that the concept is not mentioned at all, and in-between scores mean that the concept is mentioned to some extent. If the caption is complete nonsense, rate it as 0. Caption: caption SpecificityYou will be shown two captions. The first is a Ground Truth caption describing a real image. The second is a Counterfactual version of the first caption, where the concept of ‘c rem ’ has been replaced by the concept of ‘c add ’. Please rate how similar the Counterfactual caption is to the Ground Truth caption on a scale from 0–10. If the two captions are the exact same, give a rating of 10; close paraphrases can also get high ratings. You can still give a high score if the Counterfactual caption appears to describe a similar scene to the Ground Truth caption when you ignore the edited concepts: for example, if we removed the concept “motorcycle” and replaced it with “dog,” then the captions “a motorcycle in an empty garage, dimly lit” and “a dog in a low-lit empty garage” would get a score of 9, because they are close paraphrases of each other when you ignore the edited concepts. In the worst case, if the Counterfactual Caption has nothing to do with the Ground Truth Caption at all, give a rating of 0. Ground Truth Caption: clean caption Counterfactual Caption: edited caption CoherenceOn a scale from 0–10, ignoring the semantics and focusing on grammar and writing quality, how coherent is this image caption (with 10 being perfectly coherent and 0 being complete gibberish)? Caption: caption Table 5. Question templates for LLM judging from Section 4.2. We use the “Prevalence of c” template both for c rem and c add . For Specificity scores, we also give the judge the model’s original cleancaption, so that it can evaluate whether any important details have changed. But out of curiosity they clicked on an emoji in the image, expecting the result to be garbage. Surprisingly, they saw the tokenemoji in early layers! In that moment, the au- thors realized that they had a pretty interesting little finding that could provide a nice approach to “logit lens for image tokens.” At the same time, HA and SW had been helping out SF with other lines of attack on understanding text in images (studying emoji and lexical representations). With this sur- prise, the authors decided to band together and write a paper on this finding. They decided to submit to a conference in a location SF wanted to visit for personal reasons. They re- cruited BK, who was just beginning his postdoc at the lab, because of his expertise from prior work on this topic, and started writing this paper. Even though BK was in the mid- dle of moving in, and SF came down with a fever in the last few weeks of work on this paper, it ended up coming together in the end. Figure 15. The original image that SF was using, which was just a random image with the text “Hey” and an emoji overlaid using Photoshop. Clicking on the emoji yieldedemoji, even though SF expected to just see noise. Qwen3-VL-2B: Raw Logit Lens (Layer 0) Qwen3-VL-2B: +Verbalization Lens (Layer 0) Figure 16. Verbalization lens results for image in Figure 5 for Qwen3-VL-2B. Molmo2-O-7B: Raw Logit Lens (Layer 0) Molmo2-O-7B: +Verbalization Lens (Layer 0) Figure 17. Verbalization lens results for image in Figure 5 for Molmo2-O-7B. Llava-Next-34B: Raw Logit Lens (Layer 0) Llava-Next-34B: +Verbalization Lens (Layer 0) Figure 18. Verbalization lens results for image in Figure 5 for Llava-Next-34B. Image is slightly cropped for readability. Layerwise ROC-AUC: Object Confidence Qwen3-VL-8B Qwen3-VL-2B FINAL VERSIN with n=512 per category (there are about 47 single tok categories) stride across layers of 4. top 10% of heads Llava-Next-34B Random 10% of heads All heads Raw Logit Lens Verbalization Lens Molmo2-O-7B Figure 19. ROC-AUC scores for the experiment in Figure 6b. For raw logit lens, layers where ROC-AUC is high but probability differences in Figure 6b are low indicate that logit lens is badly calibrated—even if the ranking between tokens is correct, probability differences are very small, indicating low confidence. We see a similar effect when using all attention heads (a superset that includes our verbalization heads): using all attention heads can achieve high ROC-AUC, but the lower overall probability differences in Figure 6b indicate lower confidence. For verbalization lens, ROC-AUC is close to ceiling across layers in addition to large probability deltas in Figure 6b. This indicates that verbalization lens has higher confidence, which also makes our approach more qualitatively useful (i.e., semantic predictions are assigned high probabilities, making them visible in qualitative visualizations; see interactive demo). 0510152025 Layer 0 20 40 60 80 100 Interpretable Visual Tokens (%) Qwen3-VL-2B VerbalLens LatentLens 05101520253035 Layer 0 20 40 60 80 100 Interpretable Visual Tokens (%) Qwen3-VL-8B VerbalLens LatentLens 051015202530 Layer 0 20 40 60 80 100 Interpretable Visual Tokens (%) Molmo2-O-7B VerbalLens LatentLens 0102030405060 Layer 0 20 40 60 80 100 Interpretable Visual Tokens (%) LLaVA-NeXT-34B VerbalLens LatentLens Figure 20. Verbalization lens is on par with or better than LatentLens [32] across all models. We show LLM judge interpretability (% of patches judged interpretable) across layers (100 images per point). LatentLens shows a pronounced early/mid-layer dip for Llava-Next- 34B, but verbalization lens scores consistently high across layers. 1234510 α Scaling Factor 1% (2) 5% (10) 10% (22) 20% (48) 40% (115) 60% (219) 80% (418) 90% (627) 100% (2048) Rank ( ρ % of Verbal. Energy) -1.0-2.3-3.0-4.0-4.4-5.2 0.7-0.1-1.1-1.8-2.9-4.8 0.4-0.1-1.0-1.8-2.2-4.2 -0.3-0.6-0.8-1.4-2.1-4.4 -1.8-3.2-3.9-5.0-5.2-5.5 -2.5-4.3-5.2-5.5-5.7-5.9 -1.9-3.4-4.9-5.2-5.6-5.8 -0.7-2.3-3.2-4.7-5.3-5.9 -2.0-3.3-3.9-4.6-4.9-5.4 Δ Prevalence of Removed Object ↓ 1234510 α Scaling Factor 1% (2) 5% (10) 10% (22) 20% (48) 40% (115) 60% (219) 80% (418) 90% (627) 100% (2048) Rank ( ρ % of Verbal. Energy) 0.00.00.00.00.00.0 0.00.00.00.00.00.0 0.00.00.10.10.10.0 0.00.00.10.30.40.1 0.21.12.02.02.11.7 0.52.94.24.64.23.3 0.42.83.94.35.13.6 0.11.23.13.94.53.3 0.00.00.00.00.00.0 Δ Prevalence of Added Object ↑ 1234510 α Scaling Factor 1% (2) 5% (10) 10% (22) 20% (48) 40% (115) 60% (219) 80% (418) 90% (627) 100% (2048) Rank ( ρ % of Verbal. Energy) 6.65.13.82.82.11.1 8.67.76.45.84.21.8 8.68.17.36.45.32.7 8.78.38.06.96.33.5 8.37.86.65.95.23.1 8.07.36.76.05.52.8 8.37.66.66.25.43.2 8.58.07.26.45.93.6 5.53.82.61.91.60.8 Specificity ↑ 1234510 α Scaling Factor 1% (2) 5% (10) 10% (22) 20% (48) 40% (115) 60% (219) 80% (418) 90% (627) 100% (2048) Rank ( ρ % of Verbal. Energy) 9.18.88.48.38.47.6 9.49.38.88.98.38.2 9.69.39.38.78.68.3 9.39.49.18.48.37.7 9.38.58.07.87.67.7 9.28.58.07.27.26.5 9.28.47.77.47.06.7 9.58.98.17.77.46.8 8.98.47.97.97.46.0 Coherence ↑ −10 −5 0 5 10 −10 −5 0 5 10 0 2 4 6 8 10 0 2 4 6 8 10 Qwen3-VL-2B-Instruct: Mean Judged Scores Across Ranks/Scales Figure 21. Sweep across pseudo-inverse ranks and scales for editing objects in Qwen3-VL-2B. We generate edits across every combination of rank and scaling factor for 100 random ImageNet images, score the edited captions, judge outputs, and present the mean of scores across images. We focus on results for ρ=60% for Qwen3-VL-2B in the main paper. 1234510 α Scaling Factor 1% (4) 5% (19) 10% (41) 20% (92) 40% (234) 60% (463) 80% (892) 90% (1319) 100% (4096) Rank ( ρ % of Verbal. Energy) 0.0-1.2-2.7-3.4-3.8-5.2 0.30.2-0.8-1.6-2.4-4.3 0.2-0.1-1.2-2.3-3.5-5.5 -0.3-1.1-2.4-3.7-4.4-5.8 -1.3-2.8-3.9-5.0-5.3-6.2 -1.3-2.3-3.8-4.8-5.6-6.2 -0.8-1.8-2.9-3.2-4.4-5.9 -0.6-0.4-1.0-2.3-3.1-4.7 -2.5-3.6-4.5-4.9-5.5-6.2 Δ Prevalence of Removed Object ↓ 1234510 α Scaling Factor 1% (4) 5% (19) 10% (41) 20% (92) 40% (234) 60% (463) 80% (892) 90% (1319) 100% (4096) Rank ( ρ % of Verbal. Energy) 0.00.00.00.00.00.0 0.00.00.10.10.00.0 0.00.00.20.20.20.1 0.00.30.81.21.21.3 0.20.82.33.23.64.5 0.21.22.63.64.36.1 0.10.41.72.63.44.4 0.00.10.31.62.54.0 0.00.00.00.00.00.0 Δ Prevalence of Added Object ↑ 1234510 α Scaling Factor 1% (4) 5% (19) 10% (41) 20% (92) 40% (234) 60% (463) 80% (892) 90% (1319) 100% (4096) Rank ( ρ % of Verbal. Energy) 9.17.25.94.63.92.2 9.49.38.77.46.73.4 9.49.38.87.66.53.7 9.39.28.78.27.64.4 9.38.98.58.27.25.5 9.38.78.88.17.45.8 9.49.28.78.48.05.0 9.49.49.38.38.26.8 5.63.72.62.11.50.7 Specificity ↑ 1234510 α Scaling Factor 1% (4) 5% (19) 10% (41) 20% (92) 40% (234) 60% (463) 80% (892) 90% (1319) 100% (4096) Rank ( ρ % of Verbal. Energy) 9.89.79.69.69.68.5 9.99.89.79.79.89.7 9.99.79.89.79.69.6 9.89.89.69.49.59.3 9.89.69.49.59.19.3 9.89.49.39.38.99.0 9.89.59.39.28.88.5 9.89.89.89.28.98.9 9.58.78.47.87.35.0 Coherence ↑ −10 −5 0 5 10 −10 −5 0 5 10 0 2 4 6 8 10 0 2 4 6 8 10 Qwen3-VL-8B-Instruct: Mean Judged Scores Across Ranks/Scales Figure 22. Sweep across pseudo-inverse ranks and scales for editing objects in Qwen3-VL-8B. We generate edits across every combination of rank and scaling factor for 100 random ImageNet images, score the edited captions, judge outputs, and present the mean of scores across images. See Section 4.3 for details. Qwen3-VL-8B appears to require a larger scaling factor α. We focus on results for rank ρ =60% for Qwen3-VL-8B in the main paper.

Instant research discovery