Paper deep dive

Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style

Marvin Limpijankit, Milad Alshomary, Yassin Oulad Daoud, Amith Ananthram, Tim Trombley, Elias Stengel-Eskin, Mohit Bansal, Noam M. Elcott, Kathleen McKeown

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 44

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/22/2026, 6:19:59 AM

Summary

This paper investigates how Vision-Language Models (VLMs) recognize artistic style by employing a latent-space decomposition approach to extract interpretable, patch-level visual concepts. Through an interdisciplinary collaboration with art historians, the authors evaluate whether these model-learned concepts align with human expert criteria, finding that 73% of concepts are semantically meaningful and 90% are relevant to style prediction, while also identifying cases where models rely on formal visual regularities (e.g., light/dark contrast) that diverge from traditional art-historical categorization.

Entities (5)

Vision-Language Models · technology · 100%WikiArt · dataset · 99%LLaVA-1.5 · model · 98%Qwen3 · model · 98%Semi-Nonnegative Matrix Factorization · method · 95%

Relation Signals (3)

Art Historians → evaluated → Visual Concepts

confidence 98% · 73% of the extracted concepts are judged by art historians to exhibit a coherent and semantically meaningful visual feature

Qwen3 → performstask → Art Style Classification

confidence 95% · Finding that Qwen3 performs strongly, we extend the concept decomposition framework

Semi-Nonnegative Matrix Factorization → usedtoextract → Visual Concepts

confidence 92% · Semi-Nonnegative Matrix Factorization (Semi-NMF) can yield concepts that are grounded in both visual and textual modalities

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:VLMs have become increasingly proficient at a range of computer vision tasks, such as visual question answering and object detection. This includes increasingly strong capabilities in the domain of art, from analyzing artwork to generation of art. In an interdisciplinary collaboration between computer scientists and art historians, we characterize the mechanisms underlying VLMs' ability to predict artistic style and assess the extent to which they align with the criteria art historians use to reason about artistic style. We employ a latent-space decomposition approach to identify concepts that drive art style prediction and conduct quantitative evaluations, causal analysis and assessment by art historians. Our findings indicate that 73% of the extracted concepts are judged by art historians to exhibit a coherent and semantically meaningful visual feature and 90% of concepts used to predict style of a given artwork were judged relevant. In cases where an irrelevant concept was used to successfully predict style, art historians identified possible reasons for its success; for example, the model might "understand" a concept in more formal terms, such as dark/light contrasts.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

43,332 characters extracted from source content.

Expand or collapse full text

Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style Marvin Limpijankit 1 , Milad Alshomary 1 , Yassin Oulad Daoud 2 Amith Ananthram 1 , Tim Trombley 2 , Elias Stengel-Eskin 3 Mohit Bansal 4 , Noam M. Elcott 2 , Kathleen McKeown 1 1 Columbia University, Department of Computer Science 2 Columbia University, Department of Art History & Archaeology 3 University of Texas at Austin 4 UNC Chapel Hill Abstract VLMs have become increasingly proficient at a range of computer vision tasks, such as visual question answering and object detec- tion. This includes increasingly strong ca- pabilities in the domain of art, from analyz- ing artwork to generation of art. In an in- terdisciplinary collaboration between com- puter scientists and art historians, we char- acterize the mechanisms underlying VLMs’ ability to predict artistic style and assess the extent to which they align with the crite- ria art historians use to reason about artis- tic style. We employ a latent-space decom- position approach to identify concepts that drive art style prediction and conduct quan- titative evaluations, causal analysis and as- sessment by art historians. Our findings in- dicate that 73% of the extracted concepts are judged by art historians to exhibit a coher- ent and semantically meaningful visual fea- ture and 90% of concepts used to predict style of a given artwork were judged rele- vant. In cases where an irrelevant concept was used to successfully predict style, art historians identified possible reasons for its success; for example, the model might “un- derstand” a concept in more formal terms, such as dark/light contrasts. 1 Introduction Humans understand artistic style by considering both local features, such as texture and color, as well as global properties like overall compo- sition (Barnet, 2015). As vision-language mod- els (VLMs) steadily approach human-level perfor- mance on tasks such as object detection and visual question answering (Kim et al., 2025; Zou et al., 2023), the fundamental question of how models process visual inputs and generate their responses becomes increasingly relevant. This understand- ing is critical to model interpretability, and the in- ternal mechanisms and decision-making processes of VLMs remain an open and challenging research problem (Lin et al., 2025). Identifying artwork style is particularly chal- lenging for VLMs as these images often con- tain many rich, fine-grained details (Strafforello et al., 2025). Visual style is also complex, lacking explicit grounding compared to tasks like object recognition, and models often fail to generalize, relying on patterns from pre-training data rather than faithfully reasoning over the visual source image (Bin et al., 2024). Figure 1, for example, shows a complex scene of many people, where de- tails of its content as well as its form (e.g., smooth brushwork, soft rendering of drapery, sepia tones) contribute to classification as Renaissance. While prior work has focused on whether VLMs can accurately classify artwork style, less is known about what visual features drive their prediction and whether this aligns with domain knowledge. This raises the question: have the models learned to see like human experts, or does their vision op- erate according to different patterns and logics— perhaps even a fundamentally nonhuman world- view? In this interdisciplinary collaboration in- volving art historians and computer scientists, we take a model interpretability approach to investi- gate the following research questions: RQ1 What visual concepts do VLMs rely on when predicting artistic style? RQ2 Do these concepts reflect criteria that art his- torians use when analyzing artistic style? RQ3 What kind of misalignment is there between VLMs and art historians, and how does this affect style prediction? We first evaluated open- and closed-source VLMs on art style classification across early mod- ern art, modern art, and architecture datasets. Finding that Qwen3 performs strongly, we extend the concept decomposition framework of Parekh arXiv:2603.11024v1 [cs.CV] 11 Mar 2026 Figure 1: A motivating example. Top: The VLM classifies the image as Renaissance, however, image concepts offer little explanatory insight—they display visually similar images but do not reveal why the model made this prediction. Relevant tokens obtained via logit lens are similarly non-descriptive. Bottom: Our method extracts interpretable, patch-level concepts from the image, assigning each a label that captures both content and form. et al. (2024), which was previously applied to ob- ject classification, to the art domain, extracting the task-relevant visual concepts that VLMs use when predicting style. A key feature of our approach is the integration of localized image patches in our interpretability framework in order to disentangle complex visual interplay of content and form that characterize different artistic styles. Our concept decomposition pipeline is outlined in Figure 2. To validate the extracted concepts, we perform in- tervention experiments confirming that these con- cepts causally affect the style prediction, and sup- plement this with a correlational analysis based on linear probing to identify the most style-relevant concepts. In the second part of our analysis, we collab- orate with a team of six art historians, compris- ing graduate students and faculty, to condcut two complementary user studies. In the first study, art historians examine and label each concept, assess- ing its coherence and meaningfulness with respect to art-historical knowledge. Our findings indicate that 73% of the extracted concepts are perceived as meaningful. These concepts reflect a range of thematic dimensions, from content-based features such as specific objects and scenes, to form-based features such as color palette, texture, and lighting. For the second study, we investigate the alignment between the model’s concept-based prediction and domain expert judgments. We find that in almost all cases, the activated concepts were judged as relevant both to the images themselves as well as the model’s style prediction. Notably, the cases in which expert judgment diverged from the model’s concept relevance assessments proved particularly revealing, suggesting that the model captures cer- tain visual regularities that, while predictively use- ful, fall outside conventional art-historical catego- rization These observations are further examined in the qualitative analysis in §6.2. Figure 2: Overview of the concept decomposition pipeline. In the training stage, we (1) split training images into 4x4 patches, (2) extract their VLM latent representations and decompose them to obtain patch-level concept activations, and (3) generate text labels describing the top activating images of each concept. At test time, given an image, we identify its corresponding image-level concept and map it to the patch-level concepts that are most strongly represented within it and display the top results. Our contributions are as follows: • An extension of VLM concept decomposi- tion to art style classification that operates at the patch level and a method for identifying relevant concepts to full image predictions • A causal and correlational analysis demon- strating that concepts causally influence style classification performance and identifying which concepts drive the model toward pre- dicting certain styles • An interdisciplinary comparison between the model’s stylistic analysis and the canonical analysis of art historians revealing that most identified concepts are relevant to prediction of style as well as explanations for why con- cepts judged irrelevant might nonetheless re- sult in succesful style prediction. 2 Related Works Automatic recognition of artistic style has been studied for over a decade (Castellano and Vessio, 2021). While early approaches relied on hand- crafted visual features such as color histograms (Li and Chen, 2009), since then, learned local and global features of convolutional neural networks (CNNs) have yielded substantial improvements in accuracy (Karayev et al., 2013; Lecoutre et al., 2017; Menis-Mastromichalakis et al., 2020). The strength of these representations have expanded the frontier of tractable domains beyond paint- ings to architecture (Xu et al., 2014) and recent advances in vision-language models have made open generation tasks such as question answering and formal analysis possible (Garcia et al., 2020; Bleidt et al., 2024; Bin et al., 2024). A recent eval- uation of state-of-the-art VLMs on recognition of artistic style revealed impressive capabilities while also highlighting systematic disagreements with ground-truth labels that may reflect contested hu- man categorizations (Strafforello et al., 2025). Our work sheds light on this question by shifting the focus from whether VLMs can classify style to how they do so, comparing the mechanisms under- lying VLM style predictions directly against the informed judgments of expert art historians. We do so by leveraging methods for model in- terpretability. These techniques, which include linear probing (Alain and Bengio, 2016), activa- tion patching (Wang et al., 2022), dictionary learn- ing (Lee and Seung, 1999; Olah et al., 2020; Fel et al., 2023) and sparse autoencoders (Bricken et al., 2023; Cunningham et al., 2023), aim to ex- plain model decisions by decomposing dense net- work activations into human-understandable con- cepts.Parekh et al. (2024) extend these ap- proaches to vision-language models by show- ing that Semi-Nonnegative Matrix Factorization (Semi-NMF) can yield concepts that are grounded in both visual and textual modalities. We adapt their framework to visual art via a novel patch- DatasetStyles WikiArt (Early Modern) Baroque, Renaissance, Realism, Ro- coco, Romanticism WikiArt (Modern) Abstract Expressionism, Color Field, Cubism, Fauvism, Minimalism Architecture Art Nouveau, Baroque, Byzantine, Gothic, Romanesque Table 1: Datasets and associated style categories used in our experiments. level decomposition that localizes visual features, informed by the spatially distributed nature of stylistic signals. Crucially, our work pairs this computational approach with interdisciplinary in- quiry, contributing to an emerging literature that measures model alignment with categories that human experts actually use (Orgad et al., 2026). 3 Data We curate three datasets each consisting of im- ages from five style categories. Two datasets are sourced from WikiArt, 1 which focuses on art- works, whereas the third is from the Architec- ture dataset, which contains images of architec- tural styles (Xu et al., 2014).We used 2,500 images for each artwork dataset and 1,500 im- ages from the architecture dataset sampled evenly across styles. As detailed in §4.1, these images are then split into 16 patches each, resulting in 40,000 and 24,000 samples respectively. Table 1 summa- rizes the styles in each dataset. We group styles that share visual characteristics and belong to the same broad category, creating a challenging fine- grained classification task. 4 Methods 4.1 Discovering Concepts The goal of concept decomposition is to learn a set of interpretable concepts that capture recur- ring patterns in model representations. This is of- ten achieved via dictionary learning based meth- ods (Bussmann et al., 2024; Rajamanoharan et al., 2024). In particular, Semi-Nonnegative Matrix Factorization (Semi-NMF) has been shown to ef- fectively find concepts that VLMs associate with identifying specific objects (Parekh et al., 2024). We extend this approach to the domain of artistic style. 1 https://w.wikiart.org Formally, for a dataset of images X = x 1 ,...,x n with corresponding ground truth styles y = y 1 ,...,y n , we prompt a VLM to classify each image x i as one of five style cate- gories. Following Parekh et al. (2024), we extract the residual-stream representation from a speci- fied layer L upon generating the first token of a target word, specifically the model’s style pre- diction (e.g., ‘Baroque’). The result is a matrix Z ∈R d×n , where the i-th column contains the d-dimensional latent representation used by the VLM to predict the style of image x i . This ma- trix is then decomposed (Z ≈ UV) under the op- timization, U ∗ , V ∗ = arg min U,V ∥Z− UV∥ 2 F + λ∥V∥ 1 s.t. V≥ 0, ∥u k ∥ 2 ≤ 1, ∀k ∈1,...,K (1) where K is a pre-specified number of concepts, U ∈R d×K is the learned dictionary of concepts (normalized d-dimensional vectors), and V ∈ R K×n contains the concept activations for each image. Activations are strictly nonnegative and the λ parameter encourages a few concepts to be active per image (sparsity). These concepts are then represented using prototyping, where images with maximal concept activation are selected (Al- varez Melis and Jaakkola, 2018; Kim et al., 2018). Image Patching. Identifying concepts across multiple images of artwork is more difficult than in other genres-such as the photographs used for ob- ject recognition-given the many detailed elements and relations that are depicted. Thus, as in Kon- dapaneni et al. (2025), who examine shared object concepts across models, we first split each image into a 4x4 grid of patches and then apply con- cept decomposition at the patch-level (i.e., each input is an image patch). This localizes image features, making it easier to generate interpretable, high quality labels for concepts. Hyperparameters. We experiment with mul- tiple layers L at which the latent representations are extracted and the number of concepts K. The goal is that concepts should help explain model outputs. We evaluate configurations using linear probe accuracy (see §4.2), which measures how well concept activations predict the model’s out- put style. Activation Threshold. Semi-NMF encourages sparsity but results in many near-zero activations that can create noise. Therefore, to determine an ModelLayerThreshold Percentile Avg. Active Concepts Llava-1.5200.6015.64 Llava-1.5200.807.82 Llava-1.5200.903.91 Llava-1.5300.6014.67 Llava-1.5300.807.33 Llava-1.5300.903.67 Qwen3200.6015.60 Qwen3200.807.80 Qwen3200.903.90 Qwen3300.6011.46 Qwen3300.805.73 Qwen3300.902.86 Table 2: Sparsity statistics across models and layers. activation threshold, we run experiments using dif- ferent percentile thresholds and calculate the aver- age number of concepts activated per patch (Table 2). Since the patches are small, we expect each patch to correspond to only a few concepts and ap- ply a threshold of 0.90 throughout. 4.2 Concept-Style Associations We assess VLMs’ concept-style associations via probing, where a linear classifier is trained to pre- dict the model’s response (i.e., the predicted style) based only on concept activations–a single vec- tor indicating which concepts the image activated. Finding that this classifier identifies the model’s style prediction reliably (see §5.2), we inspect the learned weights of the classifier and extract the most positively associated concepts for each style. 4.3 Causal Analysis via Intervention To validate whether concepts causally affect the VLM’s style prediction, we conduct an interven- tion analysis on a held-out set of images. First, we pass the image and prompt to the VLM and mea- sure the logits and log-probs of each style name by considering the first generated token. Namely, for each style s (e.g., ‘Romanticism’) with tokens s = t 1 ,t 2 ,... (‘Rom’, ‘antic’, ... ), we compute logit(s) = z 1 [t 1 ](2) logp(s) = log softmax(z 1 )[t 1 ](3) where z 1 are the raw logit outputs of the first gen- erated token. These serve as the baseline for the model’s confidence that the image belongs to style s. Then, for each of the top three activated con- cepts, we modify the final prompt token’s hidden state at the corresponding layer (h L ) by subtract- ing out scaled amounts of the concept to create the modified hidden state ̃ h L = h L −α·(a i v i ), where v i is concept i’s vector, a i is the activation, and α is a scaling parameter (α = 1 means the concept is fully suppressed, whereas a negative value means it is promoted). This intervened hidden state is propagated through the model until the final layer where we re-measure the logits and log-probs of each style. The causal effect of a concept on a style is the mean difference in logits or log-probs between the intervened and baseline conditions. Since any intervention will affect token proba- bilities, we calibrate each intervention against 10 random ones, where a uniformly random vector with equal magnitude is used in place of the con- cept direction. Thus, the reported value measures the difference in probability of removing the con- cept on a style over random directions. We exper- iment with α = [−0.5,−0.25, 0.25, 0.5, 0.75, 1] including negative values to additionally test boosting rather than suppressing a concept. 4.4 From Patches to Images Patch concepts enable interpretability by localiz- ing features, but leveraging them to understand model predictions on full images introduces a new challenge which we address in this section. One approach is to similarly extract the VLM’s latent representation of the full image and directly ap- ply the concept decomposition learned at the patch level. However, we find that this leads to non- sparse activations and certain concepts activating on nearly all images, producing incoherent results. We attribute this to the domain shift from patches, which are lower quality and visually simple, to full-resolution images of artworks. Therefore, we propose the following method to identify the relevant patch concepts from a full im- age. Given the dataset of images, we run con- cept decomposition at both the full image and patch level, yielding K f ull and K patch concepts respectively (K f ull ≪ K patch ). Then, the patch- level concept activations are binarized using a percentile-based threshold τ patch computed across all non-zero activations. This results in a K patch - dimensional 0, 1 vector for each patch, where the i-th entry indicates the activation of concept i. We then aggregate patch vectors into a single representation for the full-image by taking their element-wise OR, yielding a K patch -dimensional binary vector indicate which concepts were ac- tive in any of the image’s patches.We apply a similar threshold τ f ull for full image concepts, and, using the result of the previous step, compute P(c patch i | c f ull j )–the probability that a patch con- cept i is present in the image given it activated full image concept j–empirically from co-occurence counts. We experiment with thresholds and set τ patch to the 95th percentile and τ f ull to the 80th percentile of non-zero activations. 5 Experiments 5.1 Model Benchmarking Performance on art style classification across sev- eral popular VLMs is reported in Figure 3. Addi- tionally, we introduce a control set of WikiArt im- ages consisting of 5 significantly different styles. We find that GPT5 and Qwen3 demonstrate the Figure 3: Overview of VLM performance on zero-shot art style classification (full image). best accuracies with a notable drop-off in Molmo2 and Llava-1.5. Models tend to perform bet- ter on the architecture dataset relative to WikiArt datasets. We also verify that models perform very well on the control group, suggesting that diffi- culty on WikiArt stems from the use of closely re- lated styles rather than just the difference between the architecture and artwork. We note that prior work has observed disagreement among domain experts on the WikiArt style labels, which may partially explain these results (Strafforello et al., 2025). The following analyses focuses on Qwen3 and Llava-1.5, two open-source models with a notable difference in accuracy. 5.2 Concept Analysis Can concepts predict the model’s style classi- fication? We train concept decomposition mod- els using different configurations and construct a meta-model by fitting a linear classifier to predict the model’s style prediction based only on an im- age patch’s concept activations. At test time, we pass unseen image patches to the VLM, extract their latent representations, decompose them into concept activation vectors, and measure the ac- curacy of the trained probe. Figure 4 shows lin- ear probe accuracies on our Early Modern sub- set of WikiArt. Results on other settings are de- ferred to the appendix due to space constraints. Figure 4: Accuracy of a linear probe trained to predict model output style from its concept activations (• raw activations;▲ binarized activations). We find that concept activations from later layers are able to predict the model’s output with high accuracy (0.95). This predictive ability improves throughout layers, which is consistent with previ- ous notions that representations get enriched with task-relevant information as they approach the fi- nal layer, where they are used to generate the next token (Wang et al., 2022). For Llava-1.5, even concept activations in early layers in some cases are sufficient. This is likely due to that model’s tendency to predict certain styles nearly all the time, contributing to the poor classification per- formance observed in Figure 3. Furthermore, we experiment with only using the binary represen- tation of concept activations (i.e., after applying the threshold) and find that relying on only the top few activated concepts remains sufficient for pre- dicting model outputs (0.85). How do models associate concepts with art styles? Figure 5 visualizes the space of concepts Figure 5: The space of concepts visualized in 2D via t-SNE. Concepts more similar to each other in high-dimension are closer. The size of each concept indicates how frequently the concept is activated across the dataset. Colors indicate style-specific concepts–instances where > 70% of the activated images correspond to 1 (or if not, 2) styles. These styles either correspond to the model’s prediction (rows 1, 3) or the ground truth style (rows 2, 4). (high-dimensional vectors) at different layers in 2D using t-SNE. The concepts specific to a cer- tain style, either the model’s predicted style or the ground truth, are colored. We observe that as rep- resentations progress throughout layers, Qwen3 identifies concepts that it associates strongly with a outputting a specific style (i.e., single-colored concepts). These, however, may not always re- flect their ground truth annotations. For exam- ple, concepts in the bottom-right of the space for Qwen3 layer 35 (Renaissance, Rococo, Baroque) seem to align with ground truth labels whereas in the top-left, the model mistakenly associates some concepts with Romanticism when these examples are annotated as Realism. In their analysis, the art historians also note that the dataset has over- laps between Romanticism and Realism and that these styles are not clearly delineated (see Sec- tion 6.2). The concepts are broadly similar in activation frequency, with only a small number that activate few images. Furthermore, Qwen3 more successfully identifies concepts that corre- spond to images of a specific ground truth style than Llava-1.5. Additionally, we once again observe that Llava-1.5 exhibits a strong bias towards predicting the same two styles (Baroque and Romanticism) across all images. 5.3 Concept-Style Associations What concepts drive the model’s style pre- dictions? The linear probe weights from §5.2 identify concepts that strongly correlate with the model’s prediction of a specific style. We com- plement this with a causal analysis, where the ef- fects of subtracting varying amounts of the con- cepts from the latent representation are studied. Figure 6: Left: The causal effect over style logits when intervening on concept 9 (human figures, drapery, and chiaroscuro). The x-axis shows α, the scaling parameter of the intervention (negative: concept boosting; positive: concept suppression), and the y-axis shows the change in logits relative to baseline, calibrated against 10 random interventions per instance. Right: Causal slope (from left) versus linear probe weights for styles, where each point represents a concept. The concepts with the strongest causal and correlational relationships are labeled. Namely, for each concept we observe that remov- ing an increasing amount of the concept (α → 1) causes the logits of, on average, 1.14 styles to de- crease while the rest to increase. This suggests that concepts tend to be causally tied to a single style, since its removal selectively suppresses that style’s logits while leaving others unaffected (or slightly increased). An example is shown in Figure 6 (left), where concept nine, which is related to human fig- ures, drapery, and chiaroscuro, is strongly associ- ated with the Baroque style in Qwen3. We further investigate whether these causally relevant concepts align with the correlated con- cepts derived from our linear probe. For every concept and style, we similarly plot the causal ef- fect graph and fit a linear function (with no inter- cept), finding that on average R 2 = 0.96. The weights of this line, referred to as the ‘causal slope’—where a negative value indicates an as- sociation between that concept and style—is plot- ted against the weights from the correlation anal- ysis (Figure 6 right). The concepts with large positive classifier weights—those most correlated with a particular style—consistently show nega- tive causal slopes, validating that their removal de- creases the probability that the style will be pre- dicted. The Spearman’s ρ is displayed on each graph, ranging from -0.50 to -0.83 and all with p < 0.05. A few examples of concepts relevant to art styles in Qwen3 are visualized in Figure 7. 6 User Studies In the following, we present two user studies we conducted with art historians on the team: an in- trinsic study to analyze the quality of the concepts used by VLMs and an extrinsic study to assess their alignment with art historians’ expertise for the task of style prediction. 6.1 Concept Quality To evaluate the quality of extracted concepts, we conducted a study with six art historians from our research team. The experts were asked to assess the semantic and stylistic coherence of the ex- tracted concepts. Specifically, for each extracted concept, we presented the art historians with the 24 most strongly activating image patches and in- structed them to (1) provide up to three textual la- bels characterizing the shared visual or thematic properties of these images, and (2) rate the degree to which the images collectively represent a single, coherent art-historical concept on a 5-point Lik- ert scale. Each concept was evaluated by three art historians. For convenience, we split the 128 con- cepts into two batches and each batch was evalu- ated by a different group of three art historians. 6.1.1 Results We collect a majority label for each concept and report our results accordingly.Regarding inter-annotator agreement, we computed Krippen- dorff’s alpha, which yielded 0.52, indicating mod- erate agreement. Figure 9 shows the distribution Figure 7: Top-four concepts associated with each style according to our correlational analysis. Concepts are represented in 4x4 grids where each image is a strongly activating image. The ‘No Style Associations’ category refer to concepts that are not linked to a specific style. Concept 111 Concept 95 Concept 33Concept 34 Figure 8: Example concepts discussed by art historians of scores given by art historians. Of the 128 con- cepts assessed, 93 (73%) received a majority score of 3 or higher, indicating the majority of concepts do represent a single, coherent art-historical con- cept. In Figure 10, we show the distribution of concepts across content, form, or style. Visual in- spection of the image patches associated with the 128 concepts reveals that they are not all united by aspects of style (Figure 8). In fact, relatively few concepts seem primarily bound by a specific style (e.g. concept 111). A vast majority instead seem primarily united by form (e.g. concept 33) or by content (e.g. concept 95). Most often the concepts are united by some combination of form, content, and style, with each present to varying de- grees (e.g. in concept 111 all of the top-activating patches are “Rococo” but in addition most con- vey a certain kind of content—here mainly female subjects). For only a few, it is unclear, from a hu- 12345 Coherence Scores 0 10 20 30 40 Number of concepts 14 21 33 43 17 Figure 9: Distribution of scores given by Art Historians for the coherence of the extracted concepts man perspective, how they are united (e.g., con- cept 34). We would like to highlight that style in art history is a way of categorizing artworks histor- ically based on their intrinsic qualities, and which includes not only aspects of the artworks’ form (countour, scale, color, tone, texture, etc.) but also their content (what the artwork represents, means, or otherwise communicates). The definitions of style, form, and content are all contested within art history, but for the sake of clarity, here we use the terms as just described. 6.2 Alignment of Concepts for Style Prediction In this user study, we evaluate the plausibility of concepts for style prediction. To this end, we de- signed an experiment in which we presented five art historians with a piece of artwork together with the model’s style prediction and three associated concepts, shown in the bottom half of Figure 12 39.8% 37.5% 17.2% 5.5% 128 concepts Concept Category Distribution Content (51) Form (48) Style (22) Unclear (7) Figure 10: Percentage of concepts for each of the style, concent, form, and unclear categories For each image, up to two of these concepts were selected from the most highly activated concepts identified in the image, while the remaining con- cepts were randomly chosen from non-activated concepts for calibration. The art historians were first asked to infer the artwork’s artistic style, then to evaluate the extent to which each concept is re- flected in the artwork, and finally to rate the rele- vance of each concept to their own predicted style, as well as to the model’s predicted style when it differed from their own. If annotators assign high ratings to the associated concepts but low ratings to the control concepts, we take this as evidence that the concepts are plausibly associated with the predicted style and aligned with art historians’ do- main knowledge. We assembled 50 cases for the study, selecting ten from each of the examined art styles; for each style, the model correctly pre- dicted seven cases and failed to predict three. 6.2.1 Results From the five annotations, we aggregate the ma- jority label from the art historian’s prediction of style as well as for their ranking of the concept’s relevancy. Concept Relevance Figure 11 shows the distri- bution of majority scores achieved by the top rele- vant concepts compared to the random ones. Only 6% (5 out of 80) of the top concepts were marked 1 2 (a) Relevance to model prediction 0 10 20 30 40 50 60 70 73 11 1 8 34 38 3 0 10 20 30 40 Number of concepts 36 25 8 15 1 2 3 4 33 38 3 (b) Relevance to user's prediction 0 12 10 20 30 40 50 60 70 72 12 1 12 38 30 Random Concepts Top Concepts (c) Relevance to the painting Strongly Disagree Neutral Strongly Agree Strongly Agree Disagree Figure 11: User Study 2: Concepts’ ranking on a scale from 1 to 3 based on their relevance to the model pre- diction (a) and the user’s prediction (b) as well as their scores for being relevant to the painting (c). as not reflected in the painting compared to 72% of the random ones (Figure 11.c). This result demon- strates that our approach extracts concepts that art historians consider reflected in the artwork. Out of 80 concepts that the model determined were relevant to the style prediction, only 10% (8 con- cepts) received a score of one (not relevant) by the art-historians (Figure 11.b). This percentage in- creases to 15% when the user’s prediction is con- sidered (Figure 11.a). Compared to randomly se- lected concepts, which were mostly considered ir- relevant to the prediction, we conclude that there is substantial alignment between the model’s con- cepts and art historians’ domain knowledge. 6.3 Qualitative analysis of Concepts In a detailed error analysis, two art historians on our team examined concepts drawn from our alignment study (study 2). Many of the cases where the model is found unable to accurately classify the style of an art- work in this study ultimately stem from the fact that the “ground truth” styles used to assess classi- fication accuracy come from WikiArt, whose clas- sifications often do not align with art historical consensus. Other cases of inaccuracy point to the style categories themselves. For example, gener- ally Realism and Romanticism are confused much more often than the other styles, partly because of how those styles are classified in WikiArt and represented in our “ground truth” set, but also partly because those two styles overlap historically and in general are not easily distinguishable on the patch level. The difference between Romanti- cism and Realism is partially formal in nature but largely representational, such that without seeing the whole artwork and thereby its overall subject matter, it can be very difficult to say whether a painting belongs to Realism or not. Bias toward one or two styles. The model’s style classifications of a set of artworks often are biased toward one or two styles when the set really comes from a greater variety of styles. The oppo- site scenario, where a model’s guesses are more varied than the “ground truth” styles represented among the artworks, rarely occurs. Detail bias. One reason the above issue occurs is likely related to the fact that certain styles are better captured in local details than others. For ex- ample, several concepts are linked to cases where the model guesses Romanticism more often than it should (e.g. concept 64 (Figure 12)—captur- ing forest details—was associated with guesses by the model that are far disproportionately bi- ased toward Romanticism, even though the top- activating patches for this concept came from a variety of styles, mostly from works classed as “Realism.” There is some art historical legitimacy to this: forests and nature generally are conceptu- ally linked to Romanticism in a way that isn’t true for the other style categories, since nature plays an important role in much Romanticist art. And yet the art historians exclusively described concept 64 in terms of its content—trees and forests—rather than any artistic style.This and similar cases seem to indicate that the model associates certain content-based concepts or details with a particular style, even when they are reflected in any number of styles. Misalignment between how the model and art historians perceived the type of unity in a concept. In a few cases revealed by user study 2, art histo- rians were not readily able to see how a concept was associated with the model’s classification of a given work of art; and it may be because the model’s concept is united by aspects of an image that human users tend to look at differently. For example, in the case below (Figure 12), the con- cept labeled “concept 3” (concept 67) in user study 2 was the top-activated concept in the model’s classification of a John Singer Sargent painting as Realism (which is correct). Art historians tended to rank this concept as not relevant in this clas- sification, likely since the image patches in that concept do not appear to come particularly from examples of Realism and do not capture details Concept 64 Concept 67 Example Artwork with top-activated concept 3 and two other random concepts 1 &2 Figure 12: Example artwork and concept associations found in the painting. On longer inspection, one can see how the model might “understand” a con- cept in more formal terms: the kinds of dark/light contrasts in the image patches from concept 67 do appear in the Sargent painting; and it is possible the activation is due to a similarity of form rather than style or content. 7 Conclusion Recent work has shown that VLMs can classify artistic style; however, the visual features driving these predictions remain difficult to interpret, and little work has examined whether they align with art-historical expertise. To fill this gap, we present a method to extract interpretable visual concepts from latent representations in models and identify their concept-style associations. We demonstrate the usefulness of our approach on Qwen3 and Llava-1.5 using the WikiArt dataset, finding that style-specific visual concepts emerge through- out the model’s layers and can partially explain ob- served differences in model performance. A fur- ther analysis reveals that activated concepts are strongly correlated with the model’s predictions (80–90% accuracy) and causally influential. In two user studies, we examined the coherence of the extracted concepts from an art-historical perspective and the alignment between what these concepts represent and the art historian’s domain of knowledge. Our findings reveal that a large per- centage of these concepts is comprehended by art historians. Further analysis of misalignment sce- narios reveals interesting insights. References Guillaume Alain and Yoshua Bengio. 2016. Understandingintermediatelayersusing linear classifier probes.arXiv preprint arXiv:1610.01644. David Alvarez Melis and Tommi Jaakkola. 2018. Towards robust interpretability with self- explaining neural networks. Advances in neural information processing systems, 31. Sylvan Barnet. 2015. A short guide to writing about art. Pearson Upper Saddle River, NJ, USA. Yi Bin, Wenhao Shi, Yujuan Ding, Zhiqiang Hu, Zheng Wang, Yang Yang, See-Kiong Ng, and Heng Tao Shen. 2024.Gallerygpt: Analyz- ing paintings with large multimodal models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 7734–7743. Tibor Bleidt, Sedigheh Eslami, and Gerard De Melo. 2024. Artquest: Countering hidden language biases in artvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 7326–7335. Trenton Bricken, Adly Templeton, Joshua Bat- son, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Deni- son, Amanda Askell, Robert Lasenby, Yi- fan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield- Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. 2023. Towards monosemanticity: Decompos- ing language models with dictionary learning. Transformer Circuits Thread. Bart Bussmann, Patrick Leask, and Neel Nanda. 2024. Batchtopk sparse autoencoders. arXiv preprint arXiv:2412.06410. Giovanna Castellano and Gennaro Vessio. 2021. Deep learning approaches to pattern extraction and recognition in paintings and drawings: an overview. Neural Computing and Applications, 33(19):12263–12282. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable fea- tures in language models.arXiv preprint arXiv:2309.08600. Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. 2023. Craft: Concept recursive activation factoriza- tion for explainability. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2711–2721. Noa Garcia, Chentao Ye, Zihua Liu, Qingtao Hu, Mayu Otani, Chenhui Chu, Yuta Nakashima, and Teruko Mitamura. 2020.A dataset and baselines for visual question answering on art. In European conference on computer vision, pages 92–108. Springer. Sergey Karayev, Matthew Trentacoste, Helen Han, Aseem Agarwala, Trevor Darrell, Aaron Hertzmann, and Holger Winnemoeller. 2013. Recognizing image style.arXiv preprint arXiv:1311.3715. Been Kim, Martin Wattenberg, Justin Gilmer, Car- rie Cai, James Wexler, Fernanda Viegas, et al. 2018. Interpretability beyond feature attribu- tion: Quantitative testing with concept activa- tion vectors (tcav). In International conference on machine learning, pages 2668–2677. PMLR. Byeong Su Kim, Jieun Kim, Deokwoo Lee, and Beakcheol Jang. 2025. Visual question answer- ing: A survey of methods, datasets, evalua- tion, and challenges. ACM Computing Surveys, 57(10):1–35. Neehar Kondapaneni, Oisin Mac Aodha, and Pietro Perona. 2025.Representational simi- larity via interpretable visual concepts. arXiv preprint arXiv:2503.15699. Adrian Lecoutre, Benjamin Negrevergne, and Flo- rian Yger. 2017. Recognizing art style auto- matically in painting with deep learning. In Asian conference on machine learning, pages 327–342. PMLR. Daniel D Lee and H Sebastian Seung. 1999. Learning the parts of objects by non-negative matrix factorization. nature, 401(6755):788– 791. Congcong Li and Tsuhan Chen. 2009. Aesthetic visual quality assessment of paintings. IEEE Journal of selected topics in Signal Processing, 3(2):236–252. Zihao Lin, Samyadeep Basu, Mohammad Beigi, Varun Manjunatha, Ryan A Rossi, Zichao Wang, Yufan Zhou, Sriram Balasubramanian, Arman Zarei, Keivan Rezaei, et al. 2025. A survey on mechanistic interpretability for multi- modal foundation models.arXiv preprint arXiv:2502.17516. Orfeas Menis-Mastromichalakis, Natasa Sofou, and Giorgos Stamou. 2020.Deep ensemble art style recognition.In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill, 5(3):e00024–001. Hadas Orgad, Fazl Barez, Tal Haklay, Isabelle Lee, Marius Mosbach, Anja Reusch, Naomi Saphra, Byron C Wallace, Sarah Wiegreffe, Eric Wong, et al. 2026. Interpretability can be ac- tionable. Jayneel Parekh, Pegah Khayatan, Mustafa Shukor, Alasdair Newson, and Matthieu Cord. 2024. A concept-based explainability framework for large multimodal models. Advances in Neural Information Processing Systems, 37:135783– 135818. Senthooran Rajamanoharan,Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. 2024. Jumping ahead:Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435. Ombretta Strafforello, Derya Soydaner, Michiel Willems, Anne-Sofie Maerten, and Stefanie De Winter. 2025. Have large vision-language models mastered art history? In International Conference on Image Analysis and Processing, pages 524–544. Springer. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022.Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593. Zhe Xu, Dacheng Tao, Ya Zhang, Junjie Wu, and Ah Chung Tsoi. 2014. Architectural style clas- sification using multinomial latent logistic re- gression. In European conference on computer vision, pages 600–615. Springer. Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. 2023. Object detection in 20 years: A survey. Proceedings of the IEEE, 111(3):257–276.

Instant research discovery