Paper deep dive
When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs
Tanay Sodha, Aditya Sharma, Ramya Hebbalaguppe, Vinti Agarwal, Pranav Murthy Yeluripaty
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/18/2026, 8:26:43 AM
Summary
The paper proposes ArgTca, a graph-based framework for improving calibration in Vision-Language Models (VLMs) during test-time adaptation. It constructs a Symbolic Attribute Graph (SAG) representing (class, attribute) pairs and trains a Graph Attention Network (GAT) to capture inter-attribute dependencies. Two selection strategies, ArgTca-Div (intra-class diversity) and ArgTca-Disc (inter-class discrimination), are used to select attributes that reduce Expected Calibration Error (ECE) compared to baselines like TCA, C-TPT, and O-TPT.
Entities (13)
Relation Signals (10)
ArgTca → includes → ArgTca-Div
confidence 95% · We introduce two attribute selection strategies: ArgTca-Div for intra-class diversity
ArgTca → includes → ArgTca-Disc
confidence 95% · ArgTca-Disc for inter-class discrimination
ArgTca → uses → Graph Attention Network
confidence 95% · train a Graph Attention Network (GAT) via supervised contrastive learning
ArgTca → uses → Symbolic Attribute Graph
confidence 95% · We propose ArgTca, which represents (class, attribute) pairs as nodes in a Symbolic Attribute Graph
ArgTca → improves → Expected Calibration Error
confidence 90% · ArgTca-Div reduces average Expected Calibration Error (ECE) by approximately ~37% over baselines
CLIP → isbackboneof → ArgTca
confidence 90% · using ViT-B/16 as CLIP backbone
TCA → ispredecessorof → ArgTca
confidence 90% · Our method builds on and improves upon TCA’s attribute selection procedure.
ArgTca → outperforms → TCA
confidence 90% · outperforming existing prompt-tuning baselines
ArgTca → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reliable confidence estimation remains a key limitation of test-time adaptation in vision-language models (VLMs), where prompt tuning improves zero-shot accuracy but often degrades calibration due to entropy-driven overconfidence. Prior approaches mitigate this using LLM-derived class attributes and contrastive regularization, yet treat attributes independently, ignoring their relational structure. We propose ARGTCA, which represents (class, attribute) pairs as nodes in a Symbolic Attribute Graph and trains a Graph Attention Network (GAT) using contrastive objectives to produce structurally informed embeddings that capture inter-attribute dependencies. We introduce two attribute selection strategies: ARGTCA-DIV for intra-class diversity and ARGTCA-DISC for inter-class discrimination. Experiments across nine benchmarks show that ARGTCA-DIV reduces average Expected Calibration Error (ECE) by approximately ~37% over baselines, while ARGTCA-DISC consistently performs as the second-best variant, reducing average ECE by approximately ~17% over baselines. These results suggest that modeling symbolic attribute interactions provides a principled approach for reliable test-time adaptation in VLMs.
Tags
Links
- Source: https://arxiv.org/abs/2607.07395v1
- Canonical: https://arxiv.org/abs/2607.07395v1
Trouble viewing inline? Open PDF directly →
Full Text
50,193 characters extracted from source content.
Expand or collapse full text
When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs Tanay Sodha1 Equal contribution. Aditya Sharma11footnotemark: 11 Ramya Hebbalaguppe2 Vinti Agarwal1 Pranav Murthy Yeluripaty1 1Department of Computer Science and Information Systems, Birla Institute of Technology and Science, Pilani, India 2TCS Research, New Delhi, India Abstract Reliable confidence estimation remains a key limitation of test-time adaptation in vision–language models (VLMs), where prompt tuning improves zero-shot accuracy but often degrades calibration due to entropy-driven overconfidence. Prior approaches mitigate this using LLM-derived class attributes and contrastive regularization, yet treat attributes independently, ignoring their relational structure. We propose ArgTca, which represents (class, attribute) pairs as nodes in a Symbolic Attribute Graph and trains a Graph Attention Network (GAT) via contrastive objectives to produce structurally informed embeddings capturing inter-attribute dependencies. We introduce two attribute selection strategies: ArgTca-Div for intra-class diversity and ArgTca-Disc for inter-class discrimination. Experiments across 99 benchmarks show that ArgTca-Div reduces average ECE(↓ ) by ∼37% 37\% over baselines, whereas ArgTca-Disc consistently performs as the second-best variant, reducing average ECE by ∼17% 17\% over baselines. These results suggest that modeling symbolic attribute interactions provides a principled approach for reliable test-time adaptation in VLMs. When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs Tanay Sodha†thanks: Equal contribution.1 and Aditya Sharma11footnotemark: 11 and Ramya Hebbalaguppe2 and Vinti Agarwal1 and Pranav Murthy Yeluripaty1 1Department of Computer Science and Information Systems, Birla Institute of Technology and Science, Pilani, India 2TCS Research, New Delhi, India 1 Introduction Vision-language models (VLMs) such as CLIP (Radford et al., 2021) have shown strong zero-shot image recognition by aligning images and text in a shared embedding space through large-scale contrastive pretraining. CLIP performs zero-shot classification of an image by computing cosine similarities between its visual embedding and class-conditioned text features generated from prompt templates such as “a photo of a class”. While effective, such templates are suboptimal and domain-agnostic. Test-time prompt tuning (TPT) (Shu et al., 2022) addresses this by optimizing prompt tokens for each test image using only its own augmented views, with no labeled data. However, TPTs entropy minimization objective inherently drives the model toward overconfident predictions, producing miscalibrated outputs, quantified by the Expected Calibration Error (ECE) (Guo et al., 2017). It poses a fundamental barrier to deploying VLMs in safety-sensitive applications such as healthcare diagnostics and autonomous systems, where unreliable uncertainty estimates can have serious consequences. TPT (4.59) C-TPT (4.24) TCA (2.56) O-TPT (3.62) A-TPT (2.67) Ours (1.62) Figure 1: Reliability diagrams on Caltech101 dataset (ViT-B/16, ECE(↓ ) in parentheses). TCA is overconfident (bars below diagonal); while TPT, C-TPT, O-TPT, and A-TPT overcorrect into underconfidence (bars above diagonal). ArgTca-Disc (ours) achieves the closest alignment with the calibration diagonal across all confidence bins. Full results across all datasets are provided in Figure 8 in Appendix. A key insight motivating recent work is that calibration and accuracy are largely decoupled: different prompts can achieve nearly identical top-1 accuracy while exhibiting significantly divergent ECE (Yoon et al., 2024). This implies that the geometry of the textual feature space, rather than predictive accuracy alone, governs how well-calibrated a VLMs predictions are. C-TPT (Yoon et al., 2024) and O-TPTSharifdeen and others (2025) exploit this by adding a feature-dispersion loss and angular dispersion respectively, that spreads class-conditioned text features across the hypersphere. TCA (Hebbalaguppe et al., 2025) complements these geometric objectives by adding LLM-extracted visual attributes for semantically grounded prompt initialization, combined with a contrastive intra-class and inter-class regularization over the resulting text embeddings, achieving strong calibration. Structural limitations of TCA. Despite strong calibration performance, TCA exhibits two structural failures that directly degrade the geometry of class-conditioned text features. First, attributes are selected as a flat set based on cosine-similarity without any relational context: on texture-rich datasets such as DTD, attributes such as rough, bumpy, and grainy are nearly collinear in CLIP’s embedding space, so top-M′M selection clusters attribute embeddings tightly and undermines the separation that calibration requires. Second, attributes shared ubiquitously across different classes—such as mammal across leopard, dolphin, and beaver in Caltech101—carry diminished discriminative capacity, leaving their embeddings proximate across class boundaries. Both the failure modes are invisible to TCA’s selection procedure, which operates purely on similarity based ordering of LLM generated attributes without any notion of relational geometry between attributes. We propose ArgTca (Attribute Relation Graph for Test-time Calibration Adaptation), a graph-based framework that addresses both structural failures through a Symbolic Attribute Graph (SAG) constructed by representing (class, attribute) pairs as nodes, trained offline via a Graph Attention Network with no task supervision. Two selection strategies derived from the graph — ArgTca-Div for intra-class diversity and ArgTca-Disc for inter-class discrimination. Both strategies operate at zero additional test-time cost — Note: the graph is trained once offline and selection is performed before tuning begins. We validate ArgTca across nine benchmark datasets spanning fine-grained recognition, texture classification, and domain-shifted imagery and demonstrate empirically that graph-structured relational reasoning over attributes is a key driver of calibration improvement — as shown in Figure 1, ArgTca produces reliability diagrams most closely aligned with the perfect calibration diagonal, outperforming existing prompt-tuning baselines. Our key contributions are: • We identify two structural failures in TCA: near-synonymous intra-class attributes that cluster on the hypersphere, and ubiquitous cross-class attributes that carry no discriminative capacity yet receive no suppression. We construct a Symbolic Attribute Graph (SAG) from class specific attributes and train a Graph Attention Network via supervised contrastive learning to produce relational embeddings capturing intra-class complementarity and cross-class redundancy. • We introduce two selection strategies from graph-refined embeddings: ArgTca-Div, maximising intra-class semantic complementarity; and ArgTca-Disc, maximising distance from all other-class embeddings. Both strategies operate at zero additional test-time cost and are evaluated across nine benchmarks using CLIP ViT-B/16. 2 Related Work 2.1 Test-Time Adaptation of VLMs Foundation models such as CLIP (Radford et al., 2021; Jia et al., 2021) enable zero-shot classification by aligning image and text in a shared embedding space. While learned prompt methods like CoOp (Zhou et al., 2022b) and CoCoOp (Zhou et al., 2022a) improve over hard prompts, they require labeled data, limiting zero-shot applicability. TPT (Shu et al., 2022) removes this constraint by adapting prompts via entropy minimization over augmented views of a single test image, extended further by DiffTPT (Feng et al., 2023) and PromptAlign (Mirza et al., 2023) through improved augmentation and distribution alignment. However, entropy minimization drives overconfident predictions across all TPT-family methods, significantly degrading calibration relative to the zero-shot baselineYoon et al. (2024). 2.2 Calibration of Vision-Language Models Post-hoc methods such as temperature scaling (Guo et al., 2017) and Platt scaling (Platt, 1999) require labeled validation data, making them impractical at test time. C-TPT (Yoon et al., 2024) establishes that TPT degrades calibration, and that well-calibrated prompts exhibit higher Average Text Feature Dispersion (ATFD). O-TPT (Sharifdeen and others, 2025) refines this, showing that ATFD displaces the feature centroid without ensuring pairwise angular separation — the operative quantity for calibration on the unit hypersphere. A-TPT (Ahamed et al., 2026) addresses a further degeneracy in O-TPT when N>|D|N>|D| (N: #datapoints, D: feature dimensions), enforcing pairwise angular separation without dimensional constraints. TCA (Hebbalaguppe et al., 2025) initializes prompts with LLM-extracted visual attributes and applies intra- and inter-class contrastive regularization, but treats attributes as a flat set ignoring relationships — a gap our work addresses. 2.3 Graph-Based VLM Adaptation GNNs (Kipf and Welling, 2017; Veličković et al., 2018) have proven effective for learning over relational data. In VLM adaptation, GraphAdapter (Li et al., 2023) uses dual knowledge graphs for supervised few-shot transfer, HGCLIP (Zheng et al., 2024) encodes label taxonomies for hierarchical classification, and VCGPrompt (Wang et al., 2026) builds visual concept graphs for prompt learning. These works demonstrate that relational structure over the label space improves representation quality in VLMs, however are designed to operate in supervised settings, not for zero-shot calibration. In contrast, ArgTca utilizes symbolic graph edges in a fully label-free setting, specifically targeting calibration improvement. ArgTca occupies the intersection of all three research threads retaining TCA’s semantically grounded prompt initialization, corrects its flat-set attribute selection through a Symbolic Attribute Graph, and directly optimizes the geometric properties of text feature space that govern calibration — none of the prior works simultaneously achieves. 3 Methodology §4§7§6§8§9 Figure 2: Overview of ArgTca attribute selection pipeline. ❶ An LLM generates M candidate attributes per class. ❷ These are used to construct the Symbolic Attribute Graph (SAG) with intra-class (EintraE_intra) and inter-class (EinterE_inter) edges. ❸ A Graph Attention Network (GAT) is trained via supervised contrastive loss (ℒSupConL_SupCon) to propagate structural relationships across the SAG, producing attribute embeddings. ❹ Two selection criteria — ArgTca-Div (intra-class diversity) and ArgTca-Disc (inter-class discrimination) — select the most informative M′M attributes per class. ❺ The final selected attributes form the matrix ∈ℝM′×KA ^M × K, used for downstream prompt tuning. ArgTca replaces TCA’s Hebbalaguppe et al. (2025) similarity-based attribute selection with a graph-informed selection via two phases. In Phase 1, a Symbolic Attribute Graph (SAG) is constructed over all (class, attribute)(class, attribute) pairs, with intra-class edges (EintraE_intra) encoding within-class attribute relations and inter-class edges (EinterE_inter) encoding cross-class sharing. A Graph Attention Network (GAT) is trained with a supervised contrastive objective (Eq. 7), producing relational embeddings that capture intra-class complementarity and inter-class redundancy. These drive two principled selection strategies: ArgTca-Div, which selects the most semantically complementary attribute pair per class, spanning the widest semantic range for diverse prompt adaptation signals; and ArgTca-Disc, which scores each attribute by its mean angular distance from all other-class nodes and selects the top-M′M per class — these maximally class-specific descriptors strengthen TCA’s inter-class regularizer. Figure 2 gives an overview of the proposed graph-informed attribute selection. In Phase 2, selected attributes initialize the standard TCA test-time tuning procedure without modification. The graph’s contribution is purely structural: deciding which attributes appear, not how they are represented. Moreover, graph training and attribute selection are performed once offline before testing, incurring no additional test-time cost. 3.1 Preliminaries Notation Let C=c1,…,cKC=\c_1,…,c_K\ be a set of K class labels and Ak=a1,…,aMA_k=\a_1,…,a_M\ be the set of M LLM-generated visual attributes for class k, giving N=K×MN=K× M total (ck,ajc_k,a_j) pairs. A full list of notations is provided in Table 5 in Appendix. TCA background TCA (Hebbalaguppe et al., 2025) initializes soft prompts using visual attributes and defines per-class text features via attribute-conditioned prompt embeddings. For each class k and each of its M′M selected attributes, a prompt is assembled as tk,m=⊕m⊕k,t_k,m=p _m _k, (1) where p is the learnable soft context (initialised to "a photo of a"), and ma_m, kc_k are the token embeddings of the attribute and class name strings, respectively. The text feature is fk,m=g(tk,m)f_k,m=g(t_k,m), where g(⋅)g(·) denotes the frozen CLIP text encoder. Class probabilities combine all attribute prompts via a bag-of-attributes softmax aggregation: p(y=k∣x)=∑m=1M′exp(fimg(x)⋅fk,m/τ)∑k′=1K∑m=1M′exp(fimg(x)⋅fk′,m/τ)p(y=k x)= _m=1^M \! (f_img(x)· f_k,m/τ ) _k =1^K _m=1^M \! (f_img(x)· f_k ,m/τ ) (2) where τ is CLIP’s learned logit scale. The test-time objective minimizes: ℒTCA=ℒTPT−αℒinter+βℒintraL_TCA=L_TPT-α\,L_inter+β\,L_intra (3) with α weights how aggressively class centroids μk _k are pushed away from the global centroid μ, and β weights how tightly each class’s attribute embeddings fk,mf_k,m are pulled toward their class centroid μk _k. ℒTPTL_TPT is the entropy of the average prediction over the ⌊B⋅psel⌋ B· p_sel most confident of B augmented views. ℒinter=1K∑k‖μk−μ‖2L_inter= 1K _k _k-μ _2 is the mean distance of class centroids μk=1M′∑mfk,m _k= 1M _mf_k,m from the global centroid μ; maximizing it spreads classes apart. ℒintra=1K∑k1M′∑m‖fk,m−μk‖2L_intra= 1K _k 1M _m f_k,m- _k _2 is minimized to make each class’s attribute prompts consistent. Only p is updated; all other parameters are frozen. 3.2 Phase 1 — Symbolic Attribute Graph and GAT Training 3.2.1 Node Initialization We adopt the M visual attributes per class provided by Hebbalaguppe et al. (2025), generated offline via GPT-4 prompting across all nine benchmark datasets. Specifically, for each class ckc_k, an LLM is prompted with the class name to produce a list of descriptive visual attributes ama_m, which are subsequently ranked in descending order of relevance by cosine similarity between the attribute and class name embeddings (Hebbalaguppe et al., 2025). Each class-attribute pair (ckc_k, aja_j) is represented as a graph node, initialized with ℓ2 _2-normalized EOS hidden state obtained by encoding the phrase “a aja_j of a ckc_k” through frozen CLIP text encoder,g(⋅)g(·): i(0)=g("a aj of a ck")∈ℝdh^(0)_i\;=\;g\! ( "a a_j of a c_k " )\;∈\;R^d (4) 3.2.2 Symbolic Attribute Graph (Sag) We define the edge set E via a symbolic rule over the node vocabulary: ℰ=(i,j)|i≠j and ci=cj⏟Eintra or ai=aj⏟EinterE= \(i,j) |i≠ j and c_i=c_j_E_intra or a_i=a_j_E_inter \ (5) where both directions are included and self-loops are omitted. In equation 5 ℰintraE_intra connects all attribute nodes belonging to the same class (ci=cjc_i=c_j), enabling the GAT to jointly aggregate every attribute perspective of a class; ℰinterE_inter connects nodes that sharing a common attribute token across different classes — for instance, mammal links leopard, dolphin, and beaver in Caltech101 — allowing the GAT to propagate how the same descriptor is contextualized differently across class neighborhoods. For K classes each with M attributes, |ℰintra|=K⋅M⋅(M−1)|E_intra|=K· M·(M-1) is guaranteed regardless of the data distribution. 3.2.3 GAT & hybrid Objective We use GAT to propagate feature information across ℰintraE_intra and ℰinterE_inter edges so that each attribute node aggregates both the complementary perspectives of its own class and the cross-class context of any shared attribute, producing node embeddings (L)h^(L) where same-class attributes are pulled into a compact, coherent cluster while cross-class attribute representations are pushed apart. Concretely, at layer l, the updated embedding of node i is computed as: il=ELU(∥h=1H∑j∈(i)αjh⋅hjl−1)h^l_i=ELU ( \|_h=1^H _j (i)α^h_j·W_hh_j^l-1 ) (6) where ∥\| denotes head-wise concatenation over H attention heads, (i)N(i) is the set of neighbors of node i under ℰE, and h(l)∈ℝd/H×dW^(l)_h ^d/H× d is the head-specific linear projection at layer l. The attention coefficient αji(l,h)α^(l,h)_ji measures the relative importance of neighbor j to node i under head h, and ELUELU induces non-linearity. Full architectural details including the attention coefficient computation are given in section B of appendix. GAT Training Objective The GAT is trained using supervised contrastive loss (Khosla et al., 2020) where same-class (ck,aj)(c_k,a_j) nodes form positives, and all inter-class nodes form negatives. ℒSupCon=1||∑i∈−1|P(i)|∑p∈P(i)logexp(~i⊤⋅~p/τ)∑j≠iexp(~i⊤⋅~j/τ)L_SupCon= 1|A| _i -1|P(i)| _p∈ P(i)\\ ( h_i · h_p/τ ) _j≠ i ( h_i · h_j/τ ) (7) where ~(L)=(L)/‖(L)‖2 h^(L)=h^(L)/\|h^(L)\|_2 and A = nodes that have at least one positive. This trains the GAT to produce class-cohesive representations in (L)h^(L) space while discriminating against all other classes. 3.3 Attribute Selection (Post training) After GAT training, the learned (L)h^(L) is used to select M′M of the M attributes per class using one of two criteria. Both strategies operate entirely offline at zero test-time cost. ArgTca-Div — Intra-class diversity For each class ck∈Cc_k∈ C, select the pair (aik,ajk)(a_i^k,a_j^k) with the minimum cosine similarity in the graph-refined embedding space — i.e., the most semantically complementary pair: (aik,ajk)=argmini≠j∈Ikh~i(L)⋅h~j(L).(a_i^k,a_j^k)= _i≠ j∈ I_k\; h^(L)_i· h^(L)_j. (8) ArgTca-Disc: For each node aika_i^k in class ckc_k, we compute discriminability score measuring its mean angular distance from all other-class nodes: score(aik)=1|Ω¯k|∑j∉Ωk(1−cos(h~i(L),h~j(L))),score(a_i^k)\;=\; 1| _k| _j∉ _k (1- \! ( h^(L)_i,\; h^(L)_j ) ), (9) where Ωk _k denotes the global node indices for class k and Ω¯k _k all other-class indices. The top M′M attributes by score are selected per class. 3.4 Phase 2 — Test-Time Tuning Prompt construction For each ckc_k and its M′M attributes selected in Phase 1, prompts are assembled via Eq. 1, with ma_m and kc_k as frozen CLIP token embeddings. The per-class text feature is fk,m=g(tk,m)f_k,m=g(t_k,m) and class probabilities follow Eq. 2. The full positional breakdown is given in Table 4. prompttk,m=[SOS]⏟pos 0‖⏟pos 1-nctx‖amk,cls tokens,…⏟ pos nctx+1⋯ 76prompt\,t_k,m= [SOS]_pos 0 \| p_pos 1-n_ctx \| a_m^k,cls tokens,…_ pos n_ctx+1\;·s\;76 (10) Test-time objective & inference For each test image, the top ⌊B⋅psel⌋ B· p_sel most confident of B augmented views are selected and a single AdamW step minimizes ℒTCAL_TCA (Eq. 3) on p. Final prediction uses the non-augmented image with the updated p, which is reset before each new test image. Figure 7 in appendix illustrates the prompt-tuning process. 4 Experimental Setup 4.1 Datasets and Baselines We evaluate on nine benchmarks datasets, using ViT-B/16 as CLIP backbone and following the evaluation protocol of baselines (He et al., 2016; Radford et al., 2021). Dataset details are provided in section A in appendix. Furthermore, we compare our proposed approach against recent, state-of-the-art approaches: • TCA (Hebbalaguppe et al., 2025): The direct predecessor, which initializes prompts with LLM-generated visual attributes and applies contrastive intra- and inter-class regularization. Our method builds on and improves upon TCA’s attribute selection procedure. • C-TPT (Yoon et al., 2024): Adds a text feature dispersion loss (ATFD) to TPT Shu et al. (2022) to improve calibration. • O-TPT (Sharifdeen and others, 2025): Replaces ATFD with pairwise angular separation between class centroids. • A-TPT (Ahamed et al., 2026): Fixes O-TPT’s degeneracy when classes exceed embedding dimension. All baselines are re-evaluated under identical hyperparameters tuned on the Caltech101 validation set, following the protocol of TCA (Hebbalaguppe et al. (2025)). 4.2 Implementation Details All experiments are done on a single A100 80GB GPU. The CLIP backbone is fully frozen throughout both phases. Phase 1 — Offline Graph Training. Each dataset’s attribute graph is constructed from Nattr=10N_attr=10 LLM-generated attributes per class (from Hebbalaguppe et al. (2025)), giving K×10K× 10 graph nodes. The Graph Attention Network (AttributeGAT; 2 layers, 4 heads, 512-dimensional node features) is trained for 100 epochs with supervised contrastive loss (Eq. 7), temperature τ=0.07τ=0.07, and the Adam optimizer with learning rate 10−310^-3. Phase-1 training is done offline and completes in under two minutes per dataset. Phase 2 — Test-Time Prompt Tuning. We follow the standard TCA evaluation protocol: soft context initialised to “a photo of a” (nctx=4n_ctx=4), 64 augmented views per test image with the top 10% selected by entropy, and a single gradient step per image with batch size 64. Regularization weights: ViT-B/16 uses α=10α=10, β=35β=35; M′=2M =2 attributes per class are selected from the graph-refined embeddings via either the diversity (ArgTca-Div) or discriminability (ArgTca-Disc) criterion. Metrics We report top-1 accuracy (Acc, %, ↑ ) and Expected Calibration Error (ECE, %, ↓ ) computed with 20 equal-width bins following Hebbalaguppe et al. (2025). Lower ECE indicates better-calibrated predictions. 5 Results 5.1 Main Results Figure 3: Radar plot comparing ECE (lower is better, smaller area is better) of TCA, O-TPT, A-TPT, and ArgTca-Div (Ours) across nine datasets (ViT-B/16). ArgTca-Div consistently occupies the innermost region, with the largest gains on Aircraft, Flower, and EuroSAT. Table 1 shows that ArgTca-Div achieves the best average ECE of 4.45%4.45\%, reducing miscalibration by 26.7%26.7\%, 31.5%31.5\%, 42.4%42.4\%, and 48.0%48.0\% over C-TPT, TCA, O-TPT, and A-TPT respectively, while improving average accuracy simultaneously. ArgTca-Disc achieves the highest average accuracy of 64.68%64.68\%, with an ECE of 5.90%5.90\% — a 2.8%2.8\%, 9.2%9.2\%, 23.7%23.7\%, and 31.0%31.0\% reduction over C-TPT, TCA, O-TPT, and A-TPT respectively. Notably, A-TPT achieves competitive accuracy (64.84%64.84\%) but at a significantly higher calibration cost (ECE: 8.55%8.55\%). ArgTca-Disc breaks this trade-off — matching A-TPT’s accuracy within 0.2%0.2\% while reducing its ECE by 31%31\%, confirming that graph-structured attribute reasoning provides consistent calibration gains. Method Metric Caltech Pets Cars Flower Food101 Aircraft DTD EuroSAT UCF101 Average Pre-trained Backbone: CLIP Vit-B/16 | Embedding dimension: 1024-d TPT (NeurIPS’22) Acc. 93.83 87.08 66.32 69.31 84.70 23.61 46.70 42.79 67.27 64.62 ECE 4.59 5.77 5.25 13.27 4.05 16.48 21.35 21.49 13.01 11.70 C-TPT (ICLR’24) Acc. 93.35 84.14 65.45 69.79 83.28 23.85 46.04 27.8 59.7 61.49 ECE 4.24 2.77 1.94 5.21 3.77 4.38 12.27 15.16 3.89 6.07 TCA (ECML’25) Acc. 93.82 90.51 65.92 69.18 69.18 24.96 44.73 45.52 66.9 63.31 ECE 2.56 6.3 7.85 3.67 5.28 4.52 11.26 11.35 5.25 6.50 O-TPT (CVPR’25) Acc. 93.67 88.55 65.65 69.96 83.52 23.46 46.87 42.85 66.90 64.60 ECE 3.62 1.92 1.80 7.15 3.13 7.75 19.75 20.92 3.56 7.73 A-TPT (ICLR’26) Acc. 93.75 88.49 66.44 69.83 83.57 23.34 47.22 43.29 67.64 64.84 ECE 2.67 2.23 2.54 10.38 2.20 13.64 19.81 17.53 6.11 8.55 ArgTca-Div (Ours) Acc. 92.82 89.48 65.07 70.36 83.64 24.69 47.34 32.84 69.63 63.99 ECE 2.52 3.44 2.75 2.74 1.75 4.83 3.17 15.83 3.05 4.45 ArgTca-Disc (ours) Acc. 93.19 88.91 66.47 69.59 83.85 25.05 46.63 39.53 68.70 64.68 ECE 1.62 3.37 4.29 3.57 2.06 4.25 6.36 19.81 3.81 5.90 Table 1: Top-1 accuracy (Acc.%, ↑ ) and Expected Calibration Error (ECE, %, ↓ ) on nine benchmarks on ViT-B/16. red: best per dataset and metric; orange: second best. 5.2 Calibration Quality: Confidence Distributions and Reliability The confidence bin plots (Figure 4) reveal that TCA, O-TPT, and A-TPT accumulate incorrect (red) predictions at high confidence levels, indicating systematic overconfidence. ArgTca-Disc achieves the lowest ECE of 1.62%1.62\% at 93.2%93.2\% accuracy, with incorrect predictions concentrated at low confidence and correct predictions (green) dominating the high-confidence region. By contrast, O-TPT (ECE: 3.61%3.61\%) and TCA (ECE: 2.56%2.56\%) exhibit red-green mixing in mid-confidence bins (2–4), while A-TPT (ECE: 2.67%2.67\%) partially improves but still misclassifies samples at low confidence. The radar plot (Figure 3) confirms this trend across all nine benchmarks: ArgTca consistently occupies the innermost region, with gains most pronounced on Aircraft (ECE: 1.21%1.21\% vs. 8.19%8.19\% for A-TPT), Food101 (1.75%1.75\% vs. 2.02%2.02\% for A-TPT), and Flower (2.88%2.88\% vs. 10.38%10.38\% for A-TPT). Reliability diagrams (Figures 1, 8) further corroborate this, with ArgTca showing the closest alignment to the calibration diagonal — particularly in high-confidence bins (0.8–1.0) — confirming that graph-based attribute selection suppresses overconfidence without inducing underconfidence. 5.3 Accuracy–Calibration Trade-off The two selection strategies encode complementary objectives. ArgTca-Div selects the most semantically complementary attribute pair per class, maximizing angular spread in (L)h^(L) space. The resulting diverse prompt initialization provides strong gradient signals for both ℒintraL_intra and ℒinterL_inter, driving higher accuracy alongside well-calibrated text distributions. ArgTca-Disc instead targets attributes maximally distant from other-class embeddings, initializing prompts with high inter-class separation and more conservative updates — effective on datasets with severe inter-class confusion (e.g., Caltech101, Aircraft) but less consistent across all nine benchmarks. Recommendation: ArgTca-Div for broad cross-dataset calibration; ArgTca-Disc when inter-class confusion dominates. (a) TCA (b) O-TPT (c) A-TPT (d) ArgTca (Ours) Figure 4: Confidence bin plots on Caltech101 (ViT-B/16). Green/red indicate correct/incorrect predictions. ArgTca-Disc achieves the lowest ECE with the cleanest separation between correct and incorrect predictions across confidence bins. Figure 5: ECE vs. α (inter-class regularization weight) on Caltech101, OxfordPets, and DTD (β=35β=35 fixed). Figure 6: ECE vs. β (intra-class regularization weight) on Caltech101, OxfordPets, and DTD (α=10α=10 fixed). 5.4 Ablation Studies Variant Sel. Caltech101 OxfordPets DTD Acc ECE Acc ECE Acc ECE ArgTca (Full) Div 92.82 2.52 89.48 3.44 47.34 3.17 Disc 93.19 1.62 88.91 3.37 46.63 6.36 (i) Zero edges Div 93.18 2.36 89.32 3.84 45.04 4.91 Disc 93.51 1.74 90.00 3.50 45.74 5.31 (i) Raw EOS Div 92.21 2.73 88.74 3.87 46.81 5.65 Disc 93.39 2.60 88.99 3.71 45.98 8.95 (i) Random sel. Div 93.10 2.53 88.63 3.44 45.92 5.16 Disc 92.25 2.43 88.50 3.55 44.27 6.33 Table 2: Ablation study on Caltech101, OxfordPets, and DTD (ViT-B/16). Acc (%, ↑ ) / ECE (%, ↓ ). red: best per column; orange: second best. Div = ArgTca-Div; Disc = ArgTca-Disc. All ablations use ViT-B/16 on Caltech101, OxfordPets, and DTD. We ablate three components of ArgTca: (i) edge structure — Sag trained with ℰ=∅E= , removing message passing while retaining contrastive training and Div/Disc selection; (i) GAT training — Div/Disc applied directly to raw CLIP EOS embeddings (0)h^(0), skipping Phase 1 entirely; and (i) selection criterion — Phase 1 run in full but with random attribute selection (M′=2M =2, seed 42). We additionally sweep α∈5,10,20,40α∈\5,10,20,40\ (β=35β=35 fixed) and β∈15,25,35,50β∈\15,25,35,50\ (α=10α=10 fixed). A detailed per-ablation description and hyperparameter sweep figures are in shown Appendix C. The ablation results (Table 2) confirm the contribution of each component. Removing edge structure (i) degrades ECE on DTD for both Div (3.17→4.913.17→4.91) and Disc (6.36→4.916.36→4.91). Skipping GAT training entirely (i) causes the largest degradation: DTD ECE rises to 5.655.65 (Div) and 8.958.95 (Disc), confirming that relational contrastive training is critical for texture-heavy datasets. Random selection (i) degrades DTD Div ECE from 3.173.17 to 5.165.16 with minimal accuracy impact, directly isolating the geometric Div/Disc criteria as the operative calibration mechanism. Dataset Class TCA (top-2) ArgTca-Div ArgTca-Disc Caltech101 airplane aircraft, travel transportation, altitude jet, airline camera image, digital digital, vintage dslr, digital chair design, chair design, director’s elegant, chair face images, human natural, adults emotion, casual motorbike motorbike, pillion seat, suspension suspension, motorbike OxfordPets Abyssinian Abyssinian, cats restful, observant cats, feline Boxer Dog, Boxer Athletic, Friendly Boxer, Pet Maine Coon Feline, Cat Tabby, Fluffy Kitten, Coon Persian Adorable, Persian Adorable, Lying Persian, Adorable Shiba Inu Shiba, Dog Cute, Foxy Inu, Shiba DTD banded design, casual material, colorful casual, colorful bumpy uneven, unequal unequal, geometric knobby, indented crystalline natural, crystals natural, glittering shimmering, crystals dotted variation, design variation, fashion dots, stylized woven interwoven, craftsmanship material, organic handwoven, craftsmanship Table 3: Representative attribute selection per class across three datasets. (ArgTca-Div, ArgTca-Disc) avoid near-synonymous descriptors that cluster tightly in CLIP embedding space, unlike TCA. Attributes selected by TCA vs. ArgTca. TCA selects top-M′M attributes by cosine similarity to the class name, surfacing near-synonymous descriptors that cluster tightly on the hypersphere. ArgTca-Div selects the maximally complementary pair in (L)h^(L) space; ArgTca-Disc selects the pair most distant from all other-class embeddings. Per-class examples are in Table 3. Hyperparameter sensitivity (α, β). Both weights are stable across α∈5,10,20,40α∈\5,10,20,40\ and β∈15,25,35,50β∈\15,25,35,50\: ECE varies by at most ∼2% 2\% across the full sweep, with the ViT-B/16 defaults (α=10α=10, β=35β=35) lying in the stable plateau (see Figures 5–6). Full ECE sweep for both selection strategies are provided in Appendix C. 6 Conclusion We present ArgTca, a test-time calibration framework for CLIP that replaces flat attribute selection in TCA with graph-relational reasoning. A GAT trained via supervised contrastive loss over the Symbolic Attribute Graph (Sag) produces attribute embeddings encoding intra-class diversity and inter-class discriminability, giving rise to two selection criteria: ArgTca-Div and ArgTca-Disc. Across nine benchmarks, ArgTca-Div achieves the lowest average ECE (4.454.45 ), reducing miscalibration by up to 48%48\%. Ablations confirm that edge topology, contrastive training, and geometric selection each contribute independently. Future works include a unified selection objective jointly optimiing diversity and discriminability, and cross-dataset graph transfer to remove the per-dataset offline build requirement. Limitations LLM attribute quality. The method inherits TCA’s dependence on LLM-generated attribute quality. Domains where LLMs produce generic or noisy attributes (e.g., satellite imagery) remain challenging. Richer, domain-specific attribute generation strategies may be needed for such cases. Accuracy–calibration tradeoff. ArgTca-Div and ArgTca-Disc are complementary but not jointly optimal: across benchmarks, ArgTca-Div tends to improve accuracy while ArgTca-Disc tends to reduce ECE, and no single variant dominates on both metrics simultaneously. Practitioners must choose a selection strategy based on deployment priority, which limits plug-and-play usability without prior knowledge of the target domain. Dataset-specific graph. The Sag and GAT are trained independently per dataset, so applying ArgTca to a new benchmark requires re-running Phase 1. The learned graph structure does not transfer across datasets or domains, and Phase 1 cannot be run when the class vocabulary is unavailable at build time, constraining applicability in fully zero-shot settings. References S. A. Ahamed, U. S. K. P. M. Thanthrige, R. Rodrigo, and M. H. Khan (2026) A-tpt: angular diversity calibration properties for test-time prompt tuning of vision-language models. ICLR. Cited by: §2.2, 4th item. L. Bossard, M. Guillaumin, and L. Van Gool (2014) Food-101 – mining discriminative components with random forests. In Proceedings of the 13th European Conference on Computer Vision, ECCV, p. 446–461. Cited by: Appendix A. M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi (2014) Describing textures in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3606–3613. Cited by: Appendix A. L. Fei-Fei, R. Fergus, and P. Perona (2004) Learning generative visual models from few training examples. In Proceedings of the CVPR 2004 Workshop on Generative-Model Based Vision, Cited by: Appendix A. C. Feng, K. Yu, Y. Liu, S. Khan, and W. Zhong (2023) Diverse data augmentation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, p. 2704–2714. Cited by: §2.1. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger (2017) On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML, p. 1321–1330. Cited by: §1, §2.2. K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 770–778. Cited by: §4.1. R. Hebbalaguppe, T. Kandar, A. Nagpal, and C. Arora (2025) Prompting without panic: attribute-aware, zero-shot, test-time calibration. arXiv preprint arXiv:2506.22819. Cited by: Appendix A, §1, §2.2, §3.1, §3.2.1, §3, 1st item, §4.1, §4.2, §4.2. P. Helber, B. Bischke, A. Dengel, and D. Borth (2019) EuroSAT: a novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12 (7), p. 2217–2226. Cited by: Appendix A. C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig (2021) Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML. Cited by: §2.1. P. Khosla, Y. Tian, H. Wang, C. Liu, J. Valmadre, C. Tian, M. Norouzi, et al. (2020) Supervised contrastive learning. In Advances in Neural Information Processing Systems, NeurIPS, Vol. 33, p. 18661–18673. Cited by: §3.2.3. T. N. Kipf and M. Welling (2017) Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, ICLR. Cited by: §2.3. J. Krause, M. Stark, J. Deng, and L. Fei-Fei (2013) 3D object representations for fine-grained categorization. In ICCV Workshop on 3D Representation and Recognition, Cited by: Appendix A. X. Li, W. Lian, Z. Lu, J. Bao, J. Liu, J. Li, J. Lan, and L. Zhu (2023) GraphAdapter: tuning vision-language models with dual knowledge graph. In Advances in Neural Information Processing Systems, NeurIPS. Cited by: §2.3. S. Maji, E. Rahtu, J. Kannala, M. B. Blaschko, and A. Vedaldi (2013) Fine-grained visual classification of aircraft. CoRR abs/1306.5151. Cited by: Appendix A. M. U. Mirza, F. S. Khalid, F. Siddiqui, and S. Khan (2023) LAFTER: label-free test-time adaptation with language feedback for robust visual prompt tuning. In Advances in Neural Information Processing Systems, NeurIPS. Cited by: §2.1. M. Nilsback and A. Zisserman (2008) Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Cited by: Appendix A. O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar (2012) Cats and dogs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3498–3505. Cited by: Appendix A. J. C. Platt (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, p. 61–74. Cited by: §2.2. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021) Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML, p. 8748–8763. Cited by: §1, §2.1, §4.1. A. Sharifdeen et al. (2025) O-TPT: orthogonality-regularized test-time prompt tuning for zero-shot generalization in vision-language models. In International Conference on Learning Representations, ICLR. Cited by: §1, §2.2, 3rd item. M. Shu, W. Nie, D. Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao (2022) Test-time prompt tuning for zero-shot generalization in vision-language models. In Advances in Neural Information Processing Systems, NeurIPS, Vol. 35, p. 14274–14289. Cited by: §1, §2.1, 2nd item. K. Soomro, A. R. Zamir, and M. Shah (2012) UCF101: a dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402. Cited by: Appendix A. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio (2018) Graph attention networks. In International Conference on Learning Representations, ICLR. Cited by: Appendix B, §2.3. M. Wang, F. Liu, L. Jiao, S. Li, L. Li, P. Chen, X. Liu, and W. Ma (2026) VCGPrompt: Visual Concept Graph-Aware Prompt Learning for Vision-Language Models. Pattern Recognition 170, p. 112012. Cited by: §2.3. J. Yoon, J. Oh, B. Kim, I. S. Kweon, and J. Choi (2024) C-TPT: calibrated test-time prompt tuning for vision-language models via text feature dispersion. In International Conference on Learning Representations, ICLR. Cited by: §1, §2.1, §2.2, 2nd item. X. Zheng, L. Ji, C. Hong, N. Ying, M. Liu, and Z. Zhang (2024) HGCLIP: exploring vision-language models with graph representations for hierarchical understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR. Cited by: §2.3. K. Zhou, J. Yang, C. C. Loy, and Z. Liu (2022a) Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, p. 16816–16825. Cited by: §2.1. K. Zhou, J. Yang, C. C. Loy, and Z. Liu (2022b) Learning to prompt for vision-language models. International Journal of Computer Vision 130, p. 2337–2348. Cited by: Appendix A, §2.1. Appendix Contents Section Page A Datasets A B GAT Architecture Details B Attention and Aggregation B Residual, Normalisation, and Projection B Node Attention Score B C Ablation Studies C C.1 Attribute Comparison: TCA vs. ArgTca-Div vs. ArgTca-Disc C C.2 Zero Graph Edges C C.3 No GAT Training (Raw CLIP EOS Embeddings) C C.4 Random Attribute Selection C C.5 Hyperparameter Sensitivity (α, β) C D Prompt Token Layout 4 E Notation Reference 5 Figures TCA Pipeline Overview 7 Reliability Diagrams Across All Datasets (ViT-B/16) 8 Tables Attribute Comparison: TCA vs. ArgTca-Div vs. ArgTca-Disc 3 Ablation Results 2 Prompt Token Layout — ArgTca 4 Notation Reference 5 Position Content Notes 0 SOS token embedding Frozen 11–nctxn_ctx Soft prompt ∈ℝnctx×Dtp ^n_ctx× D_t Learnable (tuned at test time) nctx+1n_ctx+1 amka_m^k graph selected attribute nctx+2n_ctx+2 – L−1L\!-\!1 Class tokens + EOS + padding Frozen Table 4: Full positional breakdown of a prompt in ArgTca. Appendix A Datasets We evaluate on nine benchmarks: Caltech101 Fei-Fei et al. (2004) (general objects), StanfordCars Krause et al. (2013), FGVC-Aircraft Maji et al. (2013), OxfordPets Parkhi et al. (2012), and Flowers102 Nilsback and Zisserman (2008) (fine-grained recognition), Food101 Bossard et al. (2014) (food categories), DTD Cimpoi et al. (2014) (perceptual textures), UCF101 Soomro et al. (2012) (action recognition), and EuroSAT Helber et al. (2019) (satellite land-use). We follow the standard Zhou et al. (2022b) test splits used in TCA (Hebbalaguppe et al., 2025), evaluating on the test set only with no labeled data. Figure 7: Test time prompt calibration TPT TCA C-TPT A-TPT O-TPT ArgTca (Ours) DTD Stanford Cars Food101 Oxford Pets EuroSAT Aircraft Flower102 Figure 8: Reliability diagrams across all datasets using ViT-B/16. Each row is a dataset; each column is a method. Bars above/below the diagonal (pink) indicate under/overconfidence. ArgTca consistently shows better alignment with the diagonal across all datasets. Notation Reference A full list of all the notations used in the document are provided in Table 5. Symbol Shape / Type Meaning K scalar Number of classes M scalar Attributes loaded per class (num_attributes=10=10) M′M scalar Attributes selected for injection (prompt_attributes=2=2) N=K×MN=K× M scalar Total graph nodes d=512d=512 scalar CLIP token embedding dimension ckc_k string Class name for class k aja_j string j-th attribute string (ck,aj)(c_k,a_j) tuple Node i in the graph i(0)h^(0)_i ℝdR^d Initial node embedding (CLIP EOS hidden state) i(L)h^(L)_i ℝdR^d GAT output (refined node embedding) p ℝnctx×dR^n_ctx× d Soft prompt context (trainable, Phase 2) g(⋅)g(·) ℝdR^d CLIP text encoder EOS hidden state f(⋅)f(·) ℝdoutR^d_out CLIP text encoder full output (after projection T) τ scalar InfoNCE temperature α,βα,\,β scalars TCA inter / intra loss weights Table 5: Notation used throughout this document. Appendix B GAT Architecture Details Our graph-based attribute selection follows the Graph Attention Network (GAT) framework of Veličković et al. (2018), which assigns learned importance weights to neighborhood edges during message passing, enabling the model to selectively aggregate information from the most relevant neighbors. Our Sag is an undirected graph; each undirected edge u,v∈ℰ\u,v\ is represented as two directed edges (u→v)(u→ v) and (v→u)(v→ u) during message passing, following standard practice in GAT implementations. Attention and aggregation. For each edge u,v\u,v\ and head h, the normalized attention coefficient is: αuv,h=exp(LeakyReLU(h⊤[~u,h∥~v,h]))∑u′∈(v)exp(LeakyReLU(h⊤[~u′,h∥~v,h])), _uv,h= \! (LeakyReLU(a_h [ h_u,h\| h_v,h]) ) _u (v) \! (LeakyReLU(a_h [ h_u ,h\| h_v,h]) ), (11) where ~=(l)Wlayer h=h^(l)W_layer is the linearly projected embedding. Node representations are updated as: ~(l)=ELU(concath=1H[∑u∈(v)drop(αuv,h)⋅~u,h]). h^(l)=ELU\! (concat_h=1^H\! [ _u (v)drop( _uv,h)· h_u,h ] ). (12) Residual, normalisation, and projection. After L layers, a residual connection (with WresW_res initialised to I) mitigates over-smoothing, followed by ℓ2 _2 normalisation: (L)=normalize(~(L)+Wres(0)).h^(L)=normalize\! ( h^(L)+W_res\,h^(0) ). (13) A two-layer MLP projects to the final embedding: =W2(GELU(W1(L)+1))z=W_2(GELU(W_1h^(L)+b_1)). Node attention score. A scalar importance score per node, used for attribute selection in ArgTca-Div and ArgTca-Disc, is computed as the mean attention coefficient across heads and layers: attn[v]=1LH∑l=1L∑h=1H1|(v)|∑u∈(v)αuv,h.attn[v]= 1LH _l=1^L _h=1^H 1|N(v)| _u (v) _uv,h. (14) This scalar serves as the basis for attribute selection in both ArgTca-Div and ArgTca-Disc (Section 3.2). Appendix C Ablation Studies C.1 Attribute comparison: TCA vs. ArgTca-Div vs. ArgTca-Disc. The table 3 lists the two attributes selected per method for a representative subset of classes from Caltech101, OxfordPets, and DTD. TCA ranks attributes by cosine similarity between each attribute’s frozen CLIP embedding and the class name embedding, selecting the top-M′M ; ArgTca-Div selects the most complementary pair by minimum pairwise cosine similarity in (L)h^(L) space; ArgTca-Disc selects the pair with highest mean cosine distance to all other-class embeddings in (L)h^(L) space. C.2 Zero graph edges. To isolate the contribution of the Sag edge structure, we train the GAT with ℰ=∅E= : all nodes are present and trained via supervised contrastive loss, but no message passing occurs. Without edges, each GAT layer reduces to an independent per-node linear transform; no cross-class or within-class structural context propagates. Attribute selection uses the same Div/Disc criteria on these edge-free embeddings. The calibration drop on DTD relative to the full model (Table 2) confirms that the Sag’s edge topology — not the contrastive objective alone — drives selection quality. C.3 No GAT training (raw CLIP EOS embeddings). We skip Phase 1 entirely and apply the Div/Disc selection criteria directly to the initial embeddings (0)h^(0) (frozen CLIP EOS hidden states, Eq. 4), without any contrastive training. This tests whether GAT training adds value beyond the semantic geometry already present in CLIP’s encoder. The DTD ECE degrades substantially (Disc: 8.958.95 vs. 6.366.36 for the full model), showing that contrastive relational training is necessary for reliable attribute discrimination on texture-heavy datasets. C.4 Random attribute selection. We run Phase 1 in full (SAG construction, GAT training) but replace Div/Disc selection with a random draw of M′=2M =2 attributes per class (seed 42). This isolates the selection criterion: random selection degrades ECE on DTD (Div: 5.165.16 vs. 3.173.17 for the full model) while leaving Caltech accuracy largely unchanged, confirming that the geometric Div/Disc criteria are the operative mechanism for calibration improvement, not merely the graph training. C.5 Hyperparameter sensitivity (α, β). Figures 5 and 6 show ECE as α and β are swept independently (α∈5,10,20,40α∈\5,10,20,40\ with β=35β=35 fixed; β∈15,25,35,50β∈\15,25,35,50\ with α=10α=10 fixed), evaluated on Caltech101, OxfordPets, and DTD for both ArgTca-Div and ArgTca-Disc. Both metrics are stable across the swept range: ECE varies by at most ∼1.5% 1.5\% on Caltech101 and ∼2.0% 2.0\% on DTD for α, and by at most ∼1.8% 1.8\% on DTD for β. The optimal region is α∈[10,20]α∈[10,20] and β∈[25,35]β∈[25,35]; the ViT-B/16 defaults (α=10α=10, β=35β=35) lie within this plateau, confirming robustness to precise hyperparameter choice. Figure 9: Radar plot comparing ECE (lower is better, smaller area is better) of TCA, O-TPT, A-TPT, and ArgTca-Disc (Ours) across nine datasets (ViT-B/16).