Paper deep dive
SHOE: Semantic HOI Open-Vocabulary Evaluation Metric
Maja Noack, Qinqian Lei, Taipeng Tian, Bihan Dong, Robby T. Tan, Yixin Chen, John Young, Saijun Zhang, Bo Wang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/2/2026, 11:53:10 PM
Summary
SHOE (Semantic HOI Open-Vocabulary Evaluation) is a novel evaluation framework for human-object interaction (HOI) detection that replaces rigid, exact-match categorical evaluation with a semantic similarity-based approach. By decomposing HOI predictions into verb and object components and leveraging LLM-derived similarity scores mapped to WordNet synsets, SHOE enables flexible, scalable evaluation of both traditional HOI models and open-vocabulary generative models, achieving 85.73% agreement with human judgments.
Entities (5)
Relation Signals (3)
SHOE → evaluates → HICO-DET
confidence 95% · This enables a flexible and scalable evaluation of both existing HOI detection methods and open-ended generative models using standard benchmarks such as HICO-DET.
SHOE → uses → WordNet
confidence 95% · We do so by assigning each verb and object to its WordNet synset, a sense-specific semantic unit
SHOE → surpasses → mAP
confidence 90% · SHOE scores align more closely with human judgments than existing metrics, including LLM-based and embedding-based baselines
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Open-vocabulary human-object interaction (HOI) detection is a step towards building scalable systems that generalize to unseen interactions in real-world scenarios and support grounded multimodal systems that reason about human-object relationships. However, standard evaluation metrics, such as mean Average Precision (mAP), treat HOI classes as discrete categorical labels and fail to credit semantically valid but lexically different predictions (e.g., "lean on couch" vs. "sit on couch"), limiting their applicability for evaluating open-vocabulary predictions that go beyond any predefined set of HOI labels. We introduce SHOE (Semantic HOI Open-Vocabulary Evaluation), a new evaluation framework that incorporates semantic similarity between predicted and ground-truth HOI labels. SHOE decomposes each HOI prediction into its verb and object components, estimates their semantic similarity using the average of multiple large language models (LLMs), and combines them into a similarity score to evaluate alignment beyond exact string match. This enables a flexible and scalable evaluation of both existing HOI detection methods and open-ended generative models using standard benchmarks such as HICO-DET. Experimental results show that SHOE scores align more closely with human judgments than existing metrics, including LLM-based and embedding-based baselines, achieving an agreement of 85.73% with the average human ratings. Our work underscores the need for semantically grounded HOI evaluation that better mirrors human understanding of interactions. We will release our evaluation metric to the public to facilitate future research.
Tags
Links
- Source: https://arxiv.org/abs/2604.01586v1
- Canonical: https://arxiv.org/abs/2604.01586v1
Trouble viewing inline? Open PDF directly →
Full Text
86,771 characters extracted from source content.
Expand or collapse full text
SHOE: Semantic HOI Open-Vocabulary Evaluation Metric Maja Noack 1 Qinqian Lei 2 Taipeng Tian 3 Bihan Dong 1 Robby T. Tan 2,4 Yixin Chen 1 John Young 1 Saijun Zhang 1 Bo Wang 1 1 University of Mississippi 2 National University of Singapore 3 Independent Researcher 4 ASUS Intelligent Cloud Services (AICS) https://github.com/majnoa/SHOE Abstract Open-vocabulary human–object interaction (HOI) detec- tion is a step towards building scalable systems that gen- eralize to unseen interactions in real-world scenarios and support grounded multimodal systems that reason about human–object relationships. However, standard evalua- tion metrics, such as mean Average Precision (mAP), treat HOI classes as discrete categorical labels and fail to credit semantically valid but lexically different predictions (e.g., “lean on couch” vs. “sit on couch”), limiting their ap- plicability for evaluating open-vocabulary predictions that go beyond any predefined set of HOI labels. We introduce SHOE (Semantic HOI Open-Vocabulary Evaluation), a new evaluation framework that incorporates semantic similarity between predicted and ground-truth HOI labels. SHOE de- composes each HOI prediction into its verb and object com- ponents, estimates their semantic similarity using the aver- age of multiple large language models (LLMs), and com- bines them into a similarity score to evaluate alignment be- yond exact string match. This enables a flexible and scal- able evaluation of both existing HOI detection methods and open-ended generative models using standard benchmarks such as HICO-DET. Experimental results show that SHOE scores align more closely with human judgments than ex- isting metrics, including LLM-based and embedding-based baselines, achieving an agreement of 85.73% with the av- erage human ratings. Our work underscores the need for semantically grounded HOI evaluation that better mirrors human understanding of interactions. We will release our evaluation metric to the public to facilitate future research. 1. Introduction Human–object interaction (HOI) detection is a fundamen- tal problem in computer vision and artificial intelligence, aiming to understand how humans interact with objects in images. Such interaction understanding is also important Figure 1. (a) Standard mAP metric considers all mismatches as false positives, even when predictions are semantically similar to ground truths. (b) Our SHOE metric assigns soft credit to such cases based on similarity scores, resulting in partial true positives. for grounded multimodal systems, where models must rea- son about human–object relationships rather than only iden- tifying objects [33, 48, 55]. Understanding human–object interactions is critical for a wide range of applications, in- cluding assistive robotics, autonomous agents, behavioral analysis, and healthcare [13, 15, 31, 37, 45, 53, 54]. The combinatorial nature of HOI classes, spanning diverse verbs and objects across varied contexts, makes it infeasible to ex- haustively annotate every possible interaction. This intrin- sic open-endedness highlights the need for models that gen- eralize beyond fixed label sets, a challenge that is increas- ingly recognized in HOI research and critical for deploying HOI systems in real-world applications [21, 52]. Recent advances in vision-language models (VLMs) and multimodal large language models (MLLMs) have shown strong visual understanding capabilities [2, 24, 25, 30], pre- senting a promising avenue for addressing the challenges of open-vocabulary HOI detection. The standard metric in HOI detection, mean Average Precision (mAP) [6], relies on exact categorical matching, where a prediction is con- sidered correct only if it exactly matches a predefined HOI class in the label set. This rigid formulation overlooks the fact that similar or even synonymous interactions, such as 1 arXiv:2604.01586v1 [cs.CV] 2 Apr 2026 “race motorcycle” vs “speed motorcycle” are assigned a zero score (see Figure 1 a(1)). In open-vocabulary HOI de- tection, existing exact-match evaluation protocols become fundamentally inadequate, as they undervalue the general- izable outputs of VLMs and MLLMs which are trained on massive datasets instead of limited predefined HOI classes. As a result, existing evaluation frameworks fail to capture the linguistic variability inherent in HOI detection. In addition to the linguistic variability challenge, existing benchmarks also present challenges for evaluating open- vocabulary predictions. Datasets such as HICO-DET [7] and SWIG-HOI [49] often suffer from incomplete annota- tions: not all valid human–object interactions in an image are labeled, which means that reasonable predictions may be penalized as false positives (see Figure 1 a(2)). For ex- ample, the ground truth for an image of a person riding a motorcycle may include only “race a motorcycle”, while missing equally valid interactions such as “ride a motor- cycle” or “straddle a motorcycle”. In this case, a model that predicts “ride a motorcycle” or “straddle a motorcycle” would be incorrectly penalized. As a result, semantically valid predictions that differ from the labeled ground truth can be wrongly treated as errors, introducing false negatives and discouraging models from generating diverse or expres- sive interaction descriptions. These limitations not only un- dermine the accurate evaluation of open-vocabulary meth- ods but also raise concerns about the reliability of current benchmarks for assessing supervised HOI models, high- lighting the need for more flexible and semantically-aware evaluation frameworks. To address the limitations of existing evaluation metrics in open-vocabulary HOI detection, we propose SHOE (Se- mantic HOI Open-Vocabulary Evaluation), a new evalua- tion framework that supports graded, semantically aware matching. Instead of requiring exact categorical matches, our approach compares predicted and ground-truth HOI classes by measuring the similarity between their verb and object components. Specifically, we compute fine- grained similarity scores using averaged ratings from mul- tiple LLMs, capturing nuanced relationships such as syn- onymy and relatedness (see Figure 1 b). We do so by assign- ing each verb and object to its WordNet synset [35], a sense- specific semantic unit, which removes ambiguity from pol- ysemous words. These similarity scores for individual HOI predictions are then aggregated across the dataset to pro- duce a final evaluation score. Importantly, our metric is compatible with both structured HOI class predictions and open-vocabulary outputs, making it broadly applicable to existing HOI detection methods as well as emerging VLMs and MLLMs. This enables flexible evaluation across di- verse models in open-vocabulary HOI detection. Our proposed evaluation metric introduces several tech- nical innovations that distinguish it from existing ap- proaches. First, unlike traditional metrics that rely solely on exact label matches, our method leverages graded sim- ilarity scores derived from LLM-based ratings, enabling a more nuanced and semantically aware assessment of model predictions. Second, we extend our framework to support per-class evaluation, offering insights aligned with exist- ing metrics like mAP. Third, by decomposing HOI predic- tions into verb and object components, our metric enables scalable similarity computation, avoiding the combinato- rial explosion of direct pairwise comparisons between full verb-object labels, a limitation faced by LLM-derived base- lines. Together, these components create a flexible and gen- eralizable evaluation protocol that aligns more closely with human judgment and accommodates the growing range of HOI models, including VLMs and MLLMs. Our main con- tributions are as follows: • A novel evaluation metric for HOI detection that enables graded matching based on semantic similarity, going be- yond the binary matches used in existing metrics. • A flexible scoring framework that supports both struc- tured HOI class outputs and class-free open-vocabulary predictions, making it compatible with existing and emerging models, including VLMs and MLLMs. • A scalable decomposition-based design by assessing verb and object similarity separately, avoiding exhaustive HOI pairwise comparisons and supporting significantly larger HOI vocabularies than existing benchmarks. Our user study shows that the proposed metric achieves 85.73% agreement with the average human rating, surpass- ing the 78.61% average agreement among human annota- tors, highlighting its alignment with human judgment. 2. Related Work HOI Benchmarks The field of HOI detection research has been studied across different settings through bench- marks including HICO-DET [6], V-COCO [28] and SWIG- HOI [50]. While recent benchmarks expand the label space up to 10k HOI pairs [50], they still rely on predefined vo- cabularies and exact-match evaluation. Even for zero-shot evaluation, HICO-DET is commonly used by holding out a subset of HOI classes during training [27], allowing eval- uation on unseen interactions. However, this setting still requires all HOI classes to be predefined prior to inference, and the number of unseen classes is typically limited (e.g., around 120). The current approach limits their flexibility in truly open-world scenarios where models may produce di- verse or semantically equivalent interaction descriptions. A concurrent work reformulates HOI detection as a multiple- choice, multiple-answer task to enable evaluation across VLMs and HOI detection methods [19]. Our approach in- stead focuses on open-vocabulary evaluation, allowing flex- 2 ible assessment of diverse predictions. HOI Detection Methods Standard HOI detection methods localize human–object pairs and identify their interactions from a predefined set of HOI classes [32, 39, 46, 58]. While effective within closed label sets, these methods struggle to generalize to unseen interactions. To address this limita- tion, zero-shot HOI approaches leverage pretrained vision- language models (VLMs) like CLIP [41] to tap into ex- ternal knowledge for novel HOI classes [4, 17, 26, 34, 36]. Open-vocabulary HOI methods go further, aiming to handle a larger and more fine-grained interaction space. CMD-SE [21] for example enhances HOI class representa- tions using GPT-generated body-part descriptions. Despite these advances, state-of-the-art methods still rely on prede- fined HOI categories during training and inference, limit- ing their flexibility in open-world scenarios. Meanwhile, VLMs and MLLMs exhibit the capacity to describe inter- actions beyond fixed vocabularies. However, existing HOI benchmarks and evaluation protocols are not suitable to as- sess such open-ended predictions, highlighting the need for a new evaluation metric for open-vocabulary HOI detection. Evaluation Metrics for HOI The standard metric for HOI detection is mean Average Precision (mAP), which evalu- ates exact matches over a fixed set of HOI categories. While effective in closed-set settings, mAP’s exact matching is not suitable for open-vocabulary HOI, where semantically valid predictions may not exactly match labeled ground truth, leading to misclassification. To address this, several alterna- tive scoring strategies have emerged across vision-language tasks. CLIP-based similarity scoring leverages joint vision- language embeddings to measure the alignment between predicted and reference descriptions. BERT-based similar- ity [43] and language model plausibility scores [44] further capture semantic alignment using contextualized textual embeddings or likelihood estimates from pretrained lan- guage models. Incorporating external linguistic resources such as WordNet [35] also enables structured semantic rea- soning. WordNet supports quantifying similarity through distance between sets of cognitive synonyms that share a common meaning, so called synsets, that are structured in directed acyclic graphs (DAGs). However, these metrics are based on either lexical similarity or visual similarity, and are not designed to capture HOI similarity, which involves both semantic and visual components. Building on these insights, we propose a new evaluation metric for HOI de- tection that incorporates semantic similarity, enabling more robust evaluation in open-vocabulary settings. 3. Methods We propose a semantic similarity-based evaluation frame- work to address key limitations of existing HOI metrics, such as their reliance on exact label matching and inflexibil- ity to open-vocabulary predictions. Our approach reformu- lates HOI evaluation as a graded matching problem, where predicted and ground-truth HOI classes are compared using semantic similarity scores rather than exact matching. To support open-vocabulary reasoning, we map verb and object labels from HOI datasets to their corresponding WordNet synsets [35]. This disambiguation step mitigates lexical am- biguity, ensuring that semantic comparisons reflect intended meanings. While this is applicable to any HOI benchmark, we illustrate it on HICO-DET [7], due to its comprehensive and precise annotations. High-annotation completeness is crucial in open-vocabulary evaluation. If valid interactions are missing, semantically correct predictions may be mis- classified as false positives. We then compute fine-grained verb-to-verb and object- to-object similarity scores using averaged ratings from mul- tiple LLMs. These verb and object similarity scores need to be computed only once per benchmark dataset to build a similarity lookup table and subsequent evaluations sim- ply query the table. This makes our framework highly ef- ficient and reusable. For object labels, we compare against a large neighborhood of synsets in the WordNet hierarchy. For verb labels, due to the shallow and fragmented struc- ture of the verb taxonomy in WordNet, we score each verb synset against a manually curated subset of approximately 7,150 HOI-related synsets (from a total of 13,000). Our approach supports both structured HOI class outputs and open-vocabulary verb–object pairs, offering a flexible eval- uation protocol that accommodates a wide range of mod- els, including existing HOI detection methods and VLMs or MLLMs. By supporting the full set of verb and object synsets in WordNet (i.e., approximately 126,000 lemmas) our framework enables open-vocabulary evaluation across a wider range of verb–object combinations. 3.1. Similarity Between Verbs and Objects Let each predicted Human-Object Interaction (HOI) be rep- resented as p = (b p h ,b p o ,v p ,o p ), where b p h and b p o represent bounding boxes for the human and object, v p and o p refer predicted verb and object labels. Similarly, the ground truth for this human-object pair is defined as g = (b g h ,b g o ,v g ,o g ). For bounding box matching, we follow the existing evalu- ation protocol [11, 51]. Please refer to the supplementary material for more details. Next, we define a semantic similarity score between each predicted HOI and each ground-truth HOI that passes the bounding box matching step. Let M = (p,g) | match box (p,g) = 1 be the set of all predicted ground- truth HOI pairs with matching bounding boxes. For each pair (p,g)∈M, p = (v p ,o p ) and g = (v g ,o g ) are the pre- dicted and ground-truth verb and object labels. We define instance similarity as: sim(p,g) = f (sim v (v p ,v g ), sim o (o p ,o g )),(1) 3 Closed model prediction hold motorcycle ride motorcycle straddle motorcycle Open Vocabulary prediction grab motorcycle drive moped sit motorcycle synset Matching & sense disambiguition LLM agreement based pairwise similarity GT HOI classes: hold motorcycle ride motorcycle sit on motorcycle Prematched synsets hold ridesit moto. hold10.50.4/ ride0.510.6/ stra.0.40.60.75/ moto.///1 with confidence SHOE-PR-Curve per class SHOE-F1 score per class SHOE mF1 SHOE mAP grab0.750.50.5/ driv.0.50.750.6/ sit0.50.61/ mop.///0.75 Our SHOE TP – FP Bar 100% TP hold motorcycle 100% TP ride motorcycle 75% TP straddle motorcycle 75% TP grab motorcycle 100% TP drive moped 75% TP sit motorcycle without confidence Figure 2. SHOE Framework Overview. SHOE evaluates closed and open-vocabulary HOI predictions splitting by verb and object mapping to WordNet synsets and computing LLM agreement based pairwise similarity with ground-truth interactions. Predictions get matched to the ground truth based on the highest pair similarity. SHOE mAP is calculated if confidence score is available. where sim v (v p ,v g ) ∈ [0, 1] and sim o (o p ,o g ) ∈ [0, 1] denote the verb and object similarity scores, respectively. The function f(·,·) combines these scores into an over- all instance-level similarity.We explore several aggre- gation strategies, including weighted mean, geometric mean (f geom (v,o) = √ vo), and minimum (f min (v,o) = min(v, o)). In our model evaluation, we use the standard arithmetic mean, f arith (v,o) = wv + (1− w), with w = 1 2 . Instance-wise Matching Score We now define a soft matching score per individual ground-truth HOI, based on the most similar prediction. This allows us to compute soft true positives, false positives, and false negatives at the in- stance level while accounting for semantic similarity. Let G = g 1 ,g 2 ,...,g m be the set of all ground-truth HOIs, and P =p 1 ,p 2 ,...,p n be the set of predicted HOIs. For each ground-truth HOI g j ∈ G , we compute an instance- wise matching score t j , t j = max p∈P sim(p,g j ), p g j = arg max p∈P sim(p,g j ) (2) where t j ∈ [0, 1] reflects how well the prediction set P cov- ers ground-truth instance g j and p g j denotes the matched prediction for g j . We collect all such matched predictions into a setP G =p g j | g j ∈G. LetC =c 1 ,c 2 ,...,c k be the set of all unique GT HOI classes in the dataset, where c = (v,o). For each predicted HOI p i ∈ P , we compute a prediction matching score s i , s i = max g∈C sim(p i ,g), c s i = arg max g∈C sim(p i ,g) (3) where s i measures similarity between the predicted p i and its most similar ground-truth class c s i in the evaluation set. 3.2. Dataset-level Aggregation Standard evaluation metrics such as mAP rely on confi- dence scores to rank predictions. For VLMs, however, no native HOI confidence score exists. Recent studies have shown that prompted confidence scores from VLMs and MLLMs may not reliably reflect true uncertainty, especially in open-ended tasks [9, 60]. Alternatively the token proba- bility can be used as a proxy for an HOI confidence score but it reflects only models token-generation dynamics rather than the correctness of the predicted HOI. To account for this, we provide two evaluation modes: category (1) a confidence-based mode that aligns with mAP-style ranked evaluation (token probabilities are used for VLMs), and category (2) a confidence-free mode that evaluates all predictions equally. This design allows for both compatibility with conventional benchmarks and ro- bustness to potential miscalibration. Figure 2 shows the workflow of the two aggregation strategies. We begin by computing soft precision, recall, and F1 scores per HOI class, based on the instance-wise matching score. This al- lows evaluation that aligns with standard mAP-style metrics while supporting semantic soft matching. The confidence score rank is denoted with i. Here i denotes the prefix of the ranked prediction list up to position i if confidence scores are available. If no confidence scores are used, i is dropped and the calculation is done using all predictions of the class. G c be the set of ground-truth instances of class c ∈ C. For each class c∈C, we define True Positives (TP) as: TP (i) c = X g j ∈G (i) c t j (4) False Negatives (FN): FN (i) c = X g j ∈G (i) c (1− t j )(5) 4 False Positives (FP): FP (i) c = X p k ∈P (i) G 1− t k + X p k /∈P (i) G 1(6) The first term captures predictions matched to ground-truth instances within G (i) c , but not exact matching (t j ̸= 1). The second term includes unmatched predictions. Soft Precision: Precision (i) c = TP (i) c TP (i) c + FP (i) c (7) Soft Recall: Recall (i) c = TP (i) c TP (i) c + FN (i) c (8) The F1-Score can be calculated for every model. How- ever, for models of category (1) a set confidence score rank i needs to be chosen to allow for fair comparison. The Soft F1 Score is defined as: F1 (i) c = 2· Precision (i) c · Recall (i) c Precision (i) c + Recall (i) c (9) These scores can be aggregated into the final mean soft F1 score over all GT HOI classes. Mean Soft F1 Score: mF1 = 1 |C| X c∈C F1 (i) c (10) Additionally for models of category (2) the Soft Aver- age Precision can be calculated from the Area under the Precision-Recall-Curve. Soft Average Precision (Soft-AP): Soft-AP c = X i Recall (i) c − Recall (i−1) c · Precision (i) c (11) Similar to the F1 score these Soft-APs can be aggregated into the mean over all classes. Soft mean Average Precision (Soft-mAP) is defined as: Soft-mAP = 1 |C| X c∈C Soft-AP c (12) 4. Experiments 4.1. Experiment Setting Evaluation Dataset We build our evaluation on top of the widely used HICO-DET dataset [7], which contains 600 predefined HOI classes, each defined by a verb–object pair. While HICO-DET provides a foundation for HOI detection, its label space is limited to a small and fixed set of 117 verbs and 80 objects. To support open-vocabulary evaluation, we map each verb and object in HICO-DET to its correspond- ing WordNet synset. This allows us to expand the HOI label space by including semantically related concepts in the WordNet hierarchy (e.g., hypernyms, hyponyms, and sibling nodes). For example, the verb “hold” in HICO- DET can be connected to related verbs such as “clench,” “cling,” or “grasp.” Similarly, the object “bicycle” can be linked to related concepts like “vehicle” or “mountain bike.” This WordNet-based vocabulary expansion produces a sig- nificantly broader set of candidate HOI classes while main- taining semantic alignment with the original dataset. Evaluation Metrics The standard evaluation metric for hu- man–object interaction (HOI) detection is mAP, which as- sesses performance based on spatial alignment of predicted and ground-truth human and object bounding boxes and ex- act matching of verb and object class labels. This metric is used across benchmark datasets such as HICO-DET [7] and SWIG-HOI [49]. In this work, we proposed a new se- mantically aware evaluation metric SHOE that relaxes the binary matching constraint in mAP by introducing graded similarity between predicted and ground-truth HOI classes, resulting in a soft mAP score or SHOE-score that captures partial semantic matches. Implementation Details To compute the pairwise verb–verb and object–object similarities that underlie our metric, we begin by mapping the verbs and objects in HICO-DET to their corresponding synsets in WordNet. Using the WordNet graph structure, we expand each label to include semantically related nodes, such as parent and child synsets for objects, and a manually annotated set of 7,150 HOI verb candidates, resulting in a significantly broader HOI label vocabulary. We then use LLMs to com- pute similarity scores between all candidates and ground truth verbs and objects in this expanded label space. Each LLM rated the similarity between labels on a five-point scale, from completely dissimilar (0) to synonymous (4), using the glosses of the corresponding synsets as reference. To handle the computational load of exhaustively com- paring 7k candidate verbs against 117 GT verbs from HICO-DET (≈ 850K comparisons), we use Qwen3- 32B [56] to perform initial exhaustive pairwise compar- isons. This yields a sparse similarity matrix, with most scores being zero due to semantic dissimilarity. For ob- ject pairs and the subset of verb pairs with non-zero similar- ity, we collect additional judgments using four more LLMs, DeepSeek-V3-0324 [29], Llama-4-Maverick-17B [47], Yi- 1.5-34B-Chat [57], and Gemini-2.5-pro [10], to improve ro- bustness through cross-model agreement. We average these model outputs to produce similarity scores that serve as the basis for evaluating semantic alignment between predicted and ground truth HOI labels. To assess consistency across models, we compute Pearson correlation between their sim- ilarity scores (see Figure 3), which quantifies the degree of linear score alignment between pairs of models. We find that the object similarity ratings of LLMs (lower 5 DeepSeek- V3-0324 Gemini- 2.5-Pro Llama-4- Maverick-17b Qwen-3-32B Yi-1.5- 34B-Chat DeepSeek- V3-0324 Gemini- 2.5-Pro Llama-4- Maverick-17b Qwen-3-32B Yi-1.5- 34B-Chat 0.670.590.620.65 0.610.570.540.54 0.720.660.500.54 0.610.830.690.56 0.640.750.670.84 Pearson Correlation verbs (upper triangle) | objects (lower triangle) 0.50 0.55 0.60 0.65 0.70 0.75 Correlation Figure 3. Pearson correlation between LLMs for verb (upper tri- angle) and object (lower triangle) similarity ratings. triangle) demonstrate better agreement between models than the verb similarity ratings (upper triangle).The strongest object correlation occurs between Qwen3-32B and Yi-1.5-34B-Chat (r = 0.84), while verb correlations vary more widely, ranging from 0.50 to 0.72, highlighting how the broader semantic range and polysemy of actions complicate verb similarity scoring. 4.2. Validation of Pairwise Verb Synset Filtering To verify that using Qwen3-32B to filter verb synset pairs did not unintentionally exclude semantically meaningful verb pairs, we conducted a control experiment. We ran- domly sampled 100k verb synset pairs from the pool of excluded pairs (Qwen3-32B similarity rating = 0) and re- evaluated them using the other LLMs included in SHOE. Across models, disagreement with Qwen’s zero-similarity judgments remained minimal: 1.318 % Llama 4 Maverick- 17B, 0.903 % DeepSeek-V3, 1.105 % Yi-1.5-34B-Chat, and 0.245 % Gemini 2.5 Pro. This shows that the filtering step preserves relevant similarity coverage while substan- tially reducing computational cost. 4.3. User Study We conducted a user study with five annotators to evaluate how well our HOI similarity metric reflects human judg- ments. The study involved 500 HOI pair comparisons, each consisting of a ground-truth interaction from the HICO- DET dataset and an artificially generated prediction. The ground-truth pairs were selected to reflect the class distri- bution in HICO-DET. Using our SHOE similarity metric, we generated predicted HOI interactions that spanned di- verse semantic relationships, grouped into three categories: (1) same verb, different object; (2) different verb, same ob- ject; and (3) different verb and different object. For each category, we sampled predicted HOI pairs at four distinct A1A2A3A4A5 A1 A2 A3 A4 A5 1.000.610.630.650.64 0.611.000.530.650.66 0.630.531.000.590.61 0.650.650.591.000.72 0.640.660.610.721.00 Inter-Annotator Spearman Correlation 0.50 0.55 0.60 0.65 0.70 0.75 Spearman Figure 4. Pairwise Spearman rank correlation (ρ) between anno- tators (A1–A5) on HOI similarity ratings. Higher values indicate stronger alignment in their relative rankings of interaction pairs. similarity levels, based on their SHOE scores (ranging from 1 to 4), with an approximately equal number of examples per level. Annotators were shown each HOI pair along with WordNet glosses for all involved synsets and were asked to rate the similarity between the two on a 5-point scale: 0 (completely dissimilar), 1 (somewhat related but clearly distinct), 2 (related but easily distinguishable), 3 (very sim- ilar with subtle differences), and 4 (interchangeable). Our study shows moderate to strong agreement between annotators in Figure 4. The Spearman’s ρ values ranged between 0.531 and 0.716, with most values above 0.6, a threshold commonly considered indicative of moderate to strong agreement [1]. Additionally we compute the average agreement score between annotators. For each pair of anno- tators, we compute agreement scores over the set of items they both rated. Given a ground-truth rating r i and a com- parison rating r j for the same item, the agreement score is defined as: Agreement(r i ,r j ) = 1− |r i − r j | 4 (13) The pairwise agreement is then calculated as the mean of these scores over all shared items. Finally, the overall agree- ment score is computed as the average of all pairwise agree- ment values across annotator pairs. The average agreement among annotators is 78.61%. 4.4. Evaluation of Similarity Metrics Based on our user study, we evaluated how closely differ- ent HOI similarity metrics align with human judgment by comparing their scores to average human ratings, as sum- marized in Table 3. The average agreement is computed using the same formula defined in Equation 13. Our pro- posed SHOE metric, using the standard arithmetic mean, achieved the highest agreement with annotators (85.73%), surpassing the average agreement among human annota- tors (78.61%). Alternative aggregation strategies, such as 6 0255075100125150 V = O 10 2 10 3 10 4 10 5 10 6 10 7 10 8 10 9 Number of comparisons (a) Growth comparison Brute-force comparisons (V × O) 2 Our comparisons V 2 + O 2 HICO-DETSHOE- expanded 10 3 10 4 10 5 10 6 10 7 Predictable HOI labels (sim > 0) 600 38,084,752 (b) Vocabulary size expansion Figure 5. (a) Comparison of the number of pairwise similarity computations required by brute-force HOI evaluation versus our approach. The brute-force method grows as (V× O) 2 , while our method grows asV 2 +O 2 . (b) SHOE expands the 600 HOI classes from HICO-DET to over 38M semantically related HOI labels. the geometric mean or taking the minimum verb or ob- ject SHOE score, yield slightly lower agreement but remain competitive. In contrast, the instance-based direct-matching score used in mAP has the lowest agreement with human ratings (38.90%), confirming that rigid exact matching is insufficient for open-vocabulary HOI evaluation. We also evaluated a set of LLMs, including DeepSeek- V3-0324 [29], Llama-4-Maverick-17B [47], Yi-1.5-34B- Chat [57], and Gemini-2.5-pro [10], by prompting them to directly rate the similarity between full HOI pairs. These models show high alignment with human judgments, with agreement scores ranging from 77.52% to 83.34%. By con- trast, our method, decomposing HOI similarity into inde- pendent verb and object similarities, achieves better align- ment and is significantly more scalable. Instead of scoring every possible verb-object pair combination (which grows quadratically with the number of HOI classes), we only need to compute pairwise similarities among verbs and among objects (see Figure 5 a). This design drastically increases the HOI vocabulary, expanding the 600 discrete HICO-DET labels into over 38M semantically related HOIs (see Figure 5 b) while maintaining computational efficiency. These similarity scores have to be computed only a single time per dataset to build a similarity lookup table, and sub- sequent evaluations simply query the table. CLIP is known for its strength in aligning vision and lan- guage, however, it performs poorly in HOI pair similarity comparisons, achieving only 59.11% agreement with hu- man annotations. Classical WordNet-based graph distance metrics, such as Wu–Palmer (WUP), Leacock–Chodorow (LCH), and Path similarity, also perform poorly, with agreement scores ranging from 52% to 57%. This sug- gests that although WordNet’s taxonomy encodes human- understandable lexical relationships, its hierarchical dis- tances are insufficient to capture the nuanced semantics re- quired for HOI similarity. Similarly, sentence embedding CategoryMetricAgreement (%) SHOE Standard85.73 Geometric Mean84.29 Minimum84.01 LLMs DeepSeek-V3-032483.34 Qwen3-32B82.03 Llama-4-Mav-17B82.10 Yi-1.5-34B-Chat81.55 Gemini-2.5-Pro77.52 WordNet WUP / LCH / Path57.09 / 55.69 / 52.47 Embeddings CLIP-ViT-B (gloss/pair)59.11 / 58.15 BGE (gloss/pair)57.41 / 56.37 SBERT (gloss/pair)54.09 / 52.86 mAP direct-match38.90 Table 1. Agreement between HOI similarity metrics and average human judgments collected via a user study. Each metric scores the similarity of human-object interaction (HOI) pairs. For meth- ods that compute cosine similarity, they either use embeddings of HOI phrases (pair) or embeddings of their WordNet glosses (gloss). WordNet-based metrics are semantic graph distances be- tween verb-object synset pairs. models such as SentenceBERT [42] and BGE [59] yield comparably low agreement scores. These models, while ef- fective for general semantic similarity tasks, appear to lack the fine-grained compositional understanding needed to dis- tinguish subtle variations in human-object interactions. 4.5. Hyperparameter Tuning To determine how strongly verb similarity versus object similarity should contribute to the final HOI similarity score, we conduct a hyperparameter ablation minimizing the Mean Absolute Error (MAE) over the weighting fac- tor w in the arithmetic mean (see Section 3.1). Let each example belong to a semantic category c(i) ∈ 1,...,C, and let v i , o i and y i denote the verb and object similarities and human-annotated similarity score associated with pre- diction i. For each category c, we compute the MAE of a weighted arithmetic mean wv i + (1− w)o i as MAE c (w) = 1 N c X i:c(i)=c wv i + (1− w)o i − y i . (14) The optimal weight w ⋆ c for category c is the value of w that minimizes the MAE. The results of the hyperparam- eter tuning are shown in Figure 6. We observe that Cate- gory 1 (same verb, different object) favors the object term, yielding a low optimum of w ⋆ 1 = 0.267; Category 2 (differ- ent verb, same object) relies more heavily on the verb term, 7 0.00.20.40.60.81.0 w 0.4 0.6 0.8 1.0 1.2 MAE MAE based hyperparameter tuning of w category category 1 category 2 category 3 full Figure 6. Optimizing the arithmetic-mean weight w via MAE for each of the three categories in our user study. Category 1 (same verb, different object), Category 2 (different verb, same object), Category 3 (both different) with w ⋆ 2 = 0.733; and Category 3 (both different) slightly emphasizes verbs, giving w ⋆ 3 = 0.640. The overall agree- ment increases to 87.63% by category based tuning. As the user study is relatively small in size (∼166 pairs per cate- gory), we set w = 0.5 in all experiments. 4.6. Evaluated Models We compare our proposed SHOE mAP with the standard mAP on HICO-DET across four settings: default, zero- shot rare first unseen combination (RF-UC), and open- vocabulary and VLMs with DETR [5] aided object detec- tion (see Table 2). The models span recent state-of-the-art HOI detectors, including LAIN [14], CMMP [22], ADA- CM [20], and HOLA [18], EZ-HOI [17] under default and zero-shot configurations, as well as CMD-SE [21] and THID [50] for open-vocabulary generalization and state-of- the-art VLMs like Qwen2.5-32B-VL [2], InternVL3-38B [8], and GPT-4.1 [38].Existing VLMs exhibit limited grounding capabilities and are unable to predict bounding boxes reliably. Therefore, we first predict human and ob- ject bounding boxes using DETR. In the model prompt we provide the DETR detections and task the models to predict the interaction verb. We use the verb token probabilities as a proxy for confidence score. As shown in Table 2, SHOE mAP produces slightly higher scores than standard mAP in the default and zero- shot settings by rewarding semantically related predictions that differ from the ground truth, thus reducing the false negative rate and better reflecting human understanding of interaction similarity. HOLA (ViT-L) achieves the high- est SHOE-score with 39.92 in the default setting, while LAIN (ViT-B) reaches a 35.37 score in the zero-shot setting. The open-vocabulary models show lower overall mAP and SHOE mAP scores among all evaluated methods. THID reaches the highest SHOE mAP in this category with 22.04. The results of VLMs with DETR highlight the limita- Method Type ModelmAP SHOE mAP Default LAIN (ViT-B)35.3036.58 CMMP (ViT-L)37.7739.06 ADA-CM (ViT-L) 38.4039.72 HOLA (ViT-L)39.05 39.92 Zero-shot RF-UC CMMP (ViT-B)32.1832.95 EZ-HOI (ViT-B)33.1333.83 HOLA (ViT-B)34.1935.26 LAIN (ViT-B)34.60 35.37 Open- Vocabulary CMD-SE18.2019.03 THID22.01 22.04 DETR + VLMs GPT-4.149.5061.67 InternVL3-38B42.0058.03 Qwen2.5-VL-32B34.83 66.03 Table 2. Performance comparison across different HOI models and evaluation metrics on HICO-DET, including our proposed SHOE mAP and standard mAP. For VLMs, token probabilities are used as a proxy for the confidence score. tions of strictly instance-based metrics like mAP. All mod- els achieve significantly higher SHOE mAP than standard mAP. GPT-4.1 exhibits the highest standard mAP (49.50) outperforming other VLMs. Without being trained on the dataset, it most accurately reproduces the exact HOI la- bels defined in HICO-DET. Qwen2.5-VL-32B has the low- est standard mAP among the three VLMs showing lower exact-match accuracy. However, it achieves the highest SHOE-mAP (66.03) of all tested models, indicating strong semantic and visual understanding of HOIs. This confirms that SHOE reveals meaningful distinctions that mAP fails to capture and underlines the HOI capabilities of VLMs. SHOE mAP offers a more nuanced evaluation by incor- porating semantic similarity beyond exact verb-object pair matching, while still preserving compatibility with standard mAP scoring for exact matches. 5. Conclusion We introduced SHOE, a new evaluation metric for HOI detection that captures semantic similarity between verb- object pairs by decomposing interactions into their verb and object components. By leveraging LLMs to compute pair- wise similarities, SHOE enables a fine-grained assessment of model predictions, especially under open-vocabulary condition. Our user study demonstrates that SHOE aligns more closely with human judgment than existing metrics. The modular structure of SHOE also allows for efficient scaling with larger HOI vocabulary. We hope SHOE will serve as a practical evaluation metric for future HOI mod- els, going beyond the limits of exact class matching. 8 References [1] Haldun Akoglu. User’s guide to correlation coefficients. Turkish Journal of Emergency Medicine, 18(3):91–93, 2018. 6 [2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 1, 8, 2 [3] Olivier Bodenreider. The unified medical language system (umls): integrating biomedical terminology. Nucleic Acids Research, 32(Database issue):D267–D270, 2004. 7 [4] Yichao Cao, Qingfei Tang, Xiu Su, Song Chen, Shan You, Xiaobo Lu, and Chang Xu. Detecting any human-object interaction relationship: Universal hoi detector with spatial prompt learning on foundation models. Advances in Neural Information Processing Systems, 36, 2024. 3 [5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, pages 213–229. Springer, 2020. 8, 1 [6] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. In 2018 ieee winter conference on applications of computer vision (wacv), pages 381–389. IEEE, 2018. 1, 2, 6 [7] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 2018. 2, 3, 5 [8] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 8, 2 [9] Prateek Chhikara. Mind the confidence gap: Overconfi- dence, calibration, and distractor effects in large language models. arXiv preprint arXiv:2502.11028, 2025. 4 [10] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. 5, 7, 1, 2 [11] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303–338, 2009. 3 [12] Joseph L Fleiss and Jacob Cohen.The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological mea- surement, 33(3):613–619, 1973. 4 [13] Michael A Goodrich and Alan C Schultz. Human–robot in- teraction: a survey. Foundations and trends® in human– computer interaction, 1(3):203–275, 2008. 1 [14] Sanghyun Kim, Deunsol Jung, and Minsu Cho. Locality- aware zero-shot human-object interaction detection. In Pro- ceedings of the Computer Vision and Pattern Recognition Conference, pages 20190–20200, 2025. 8, 2 [15] Yu Kong and Yun Fu. Human action recognition and predic- tion: A survey. International Journal of Computer Vision, 130(5):1366–1401, 2022. 1 [16] Klaus Krippendorff. Content analysis: An introduction to its methodology. Sage publications, 2018. 4 [17] Qinqian Lei, Bo Wang, and Robby T. Tan. Ez-hoi: Vlm adaptation via guided prompt learning for zero-shot hoi de- tection. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 3, 8, 2 [18] Qinqian Lei, Bo Wang, and Tan Robby T. Hola: Zero-shot hoi detection with low-rank decomposed vlm feature adapta- tion. In Proceedings of the IEEE/CVF international confer- ence on computer vision, 2025. 8, 1, 2 [19] Qinqian Lei, Bo Wang, and Robby T. Tan.Crosshoi- bench: A unified benchmark for hoi evaluation across vision- language models and hoi-specific methods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), 2026. 2 [20] Ting Lei, Fabian Caba, Qingchao Chen, Hailin Jin, Yuxin Peng, and Yang Liu. Efficient adaptive human-object interac- tion detection with concept-guided memory. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 6480–6490, 2023. 8, 2 [21] Ting Lei, Shaofeng Yin, and Yang Liu. Exploring the po- tential of large foundation models for open-vocabulary hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16657–16667, 2024. 1, 3, 8, 2 [22] Ting Lei, Shaofeng Yin, Yuxin Peng, and Yang Liu. Explor- ing conditional multi-modal prompts for zero-shot hoi de- tection. In European Conference on Computer Vision, pages 1–19. Springer, 2024. 8, 2 [23] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. 2 [24] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In In- ternational Conference on Machine Learning, pages 12888– 12900. PMLR, 2022. 1 [25] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 1 [26] Liulei Li, Jianan Wei, Wenguan Wang, and Yi Yang. Neural- logic human-object interaction detection. Advances in Neu- ral Information Processing Systems, 36, 2024. 3 [27] Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 20123–20132, 2022. 2 [28] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ́ ar, and C Lawrence 9 Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014. 2, 6 [29] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 5, 7, 1 [30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1 [31] Yuansen Liu, Haiming Tang, Jinlong Peng, Jiangning Zhang, Xiaozhong Ji, Qingdong He, Donghao Luo, Zhenye Gan, Junwei Zhu, Yunhang Shen, Chaoyou Fu, Chengjie Wang, Xiaobin Hu, and Shuicheng YAN. Human-MME: A holistic evaluation benchmark for human-centric multimodal large language models. In The Fourteenth International Confer- ence on Learning Representations, 2026. 1 [32] Jinguo Luo, Weihong Ren, Weibo Jiang, Xi’ai Chen, Qiang Wang, Zhi Han, and Honghai Liu. Discovering syntactic interaction clues for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 28212–28222, 2024. 3 [33] Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman. When thinking drifts: Evidential grounding for robust video reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. 1 [34] Yunyao Mao, Jiajun Deng, Wengang Zhou, Li Li, Yao Fang, and Houqiang Li. Clip4hoi: Towards adapting clip for practi- cal zero-shot hoi detection. Advances in Neural Information Processing Systems, 36, 2024. 3 [35] George Miller and Christiane Fellbaum. WordNet: An Elec- tronic Lexical Database. The MIT Press, 1998. 2, 3 [36] Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23507–23517, 2023. 3 [37] Keane Ong, Wei Dai, Carol Li, Dewei Feng, Hengzhi Li, Jingyao Wu, Jiaee Cheong, Rui Mao, Gianmarco Mengaldo, Erik Cambria, and Paul Pu Liang. Human behavior atlas: Benchmarking unified psychological and social behavior un- derstanding. In The Fourteenth International Conference on Learning Representations, 2026. 1 [38] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anad- kat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Bal- com, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett- Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fo- tis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Ja- son Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Cur- rier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Sim ́ on Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogi- neni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, An- drew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Mal- facini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob Mc- Grew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David M ́ ely, Ashvin Nair, Rei- ichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Gi- ambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Fil- ipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rim- bach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sas- try, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Fe- lipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer ́ on Uribe, Andrea Val- lone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welin- der, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren 10 Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. 8, 2 [39] Jeeseung Park, Jin-Woo Park, and Jong-Seok Lee. Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17152–17162, 2023. 3 [40] Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. Grounded situation recognition. In European Conference on Computer Vision, pages 314–332. Springer, 2020. 6 [41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 3 [42] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Asso- ciation for Computational Linguistics, 2019. 7 [43] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sen- tence embeddings using Siamese BERT-networks. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 3982–3992, 2019. 3 [44] Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pages 2699–2712, 2020. 3 [45] Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Eledath, Gerard Medioni, and Leonid Si- gal. Energy-based learning for scene graph generation. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 13936–13945, 2021. 1 [46] Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query-based pairwise human-object interaction detec- tion with image-wide contextual information. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10410–10419, 2021. 3 [47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ́ e Lacroix, Baptiste Rozi ` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 5, 7, 1 [48] David Wan, Han Wang, Ziyang Wang, Elias Stengel- Eskin, Hyunji Lee, and Mohit Bansal. Multimodal fact- level attribution for verifiable reasoning.arXiv preprint arXiv:2602.11509, 2026. 1 [49] Suchen Wang, Kim-Hui Yap, Henghui Ding, Jiyan Wu, Jun- song Yuan, and Yap-Peng Tan. Discovering human interac- tions with large-vocabulary objects via query and multi-scale detection. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 13455–13464, 2021. 2, 5, 6 [50] Suchen Wang, Yueqi Duan, Henghui Ding, Yap-Peng Tan, Kim-Hui Yap, and Junsong Yuan.Learning transferable human-object interaction detector with natural language su- pervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 939–948, 2022. 2, 8, 1 [51] Tiancai Wang, Rao Muhammad Anwer, Muhammad Haris Khan, Fahad Shahbaz Khan, Yanwei Pang, Ling Shao, and Jorma Laaksonen. Deep contextual attention for human- object interaction detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 3 [52] Mingrui Wu, Yuqi Liu, Jiayi Ji, Xiaoshuai Sun, and Ron- grong Ji. Toward open-set human object interaction detec- tion. Proceedings of the AAAI Conference on Artificial Intel- ligence, 38(6):6066–6073, 2024. 1 [53] Xianghui Xie, Bharat Lal Bhatnagar, and Gerard Pons-Moll. Chore: Contact, human and object reconstruction from a sin- gle rgb image. In European Conference on Computer Vision, pages 125–145. Springer, 2022. 1 [54] Yiteng Xu, Peishan Cong, Yichen Yao, Runnan Chen, Yue- nan Hou, Xinge Zhu, Xuming He, Jingyi Yu, and Yuexin Ma. Human-centric scene understanding for 3d large-scale scenarios. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20349–20359, 2023. 1 [55] Zihui Xue, Mi Luo, and Kristen Grauman. Seeing the arrow of time in large multimodal models. In The Thirty-ninth An- nual Conference on Neural Information Processing Systems, 2025. 1 [56] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. 5, 1 [57] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024. 5, 7, 1 [58] Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Spatially conditioned graphs for detecting human-object in- teractions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13319–13327, 2021. 3 [59] Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment any- thing, 2023. 7 [60] Kaitlyn Zhou, Jena D Hwang, Xiang Ren, and Maarten Sap. Relying on the unreliable: The impact of language models’ reluctance to express uncertainty. arXiv preprint arXiv:2401.06730, 2024. 4 11 Supplementary Material 1. LLM Ablation for SHOE-Metric To assess the sensitivity of SHOE to the specific set of LLMs (Qwen3-32B [56], DeepSeek-V3-0324 [29], Llama- 4-Maverick-17B [47], Yi-1.5-34B-Chat [57], and Gemini- 2.5-pro [10]) used to compute the final similarity score and whether a smaller or differently composed ensemble could achieve comparable alignment with human judgments, we evaluated all possible subsets of the five LLMs used in our ensemble, ranging from single-model configurations to the full five-model setup (Table 1). Across all configurations, agreement stays within a narrow 0.84–0.86 range. Single models already achieve strong alignment, though perfor- mance varies (e.g., DeepSeek-V3 is highest at 86.46 %; Maverick lowest at 84.11 %). Adding a second model pro- vides only marginal benefit: most two-model combinations match the performance of the stronger individual model, in- dicating limited complementary information at this scale. Larger ensembles (3–4 models) modestly reduce variance but do not substantially increase the average agreement. Although the gains in average agreement are modest, we retain the ensemble SHOE score as it may reduce the influ- ence of model-specific biases. Our user study did not show strong trends of any of the included models but is limited to 500 pairs of HOIs. For datasets that focus on more spe- cialized domains, individual model biases are more likely to surface, and the ensemble may provide a more balanced estimate. Overall, the ablation shows that SHOE is largely insensitive to the exact choice of LLMs. As long as models of comparable capability are included, the resulting similar- ity scores remain stable. As LLMs evolve, SHOE can adopt newer models without changing its characteristic behavior. 2. Open-Vocabulary Evaluation of HOI Models We compare Vision-Language Models (VLMs) with state- of-the-art HOI methods in Table 2. As VLMs cannot reli- ably provide detection bounding boxes or confidence scores for HOI class prediction, we use the off-the-shelf object de- tection, DETR [5], commonly adopted in two-stage HOI de- tection methods, and VLMs’ token probabilities for mAP- style ranked evaluation. However, not all VLMs provide token probabilities. Additionally, using DETR as an initial object detector omits the VLMs’ capabilities to detect hu- man, object and interaction, as it only has to find the interac- tion for the already predefined human-object pair. To extend our study, we report results for the confidence-free evalua- tion mode (i.e., category (2) confidence-free mode, see Sec. 3.2 in the main paper) for all models. We add Gemini-2.5- flash [10] and LLaVa-Onevision-72b [30] in this compari- son as token-wise probabilities are not needed for this evalu- ation setting. We present the GT miss rate (false negatives), prediction miss rate (false positives), and SHOE mF1 for all the models included in our evaluation. On the single-HOI HICO-DET [6] subset (see Table 2). We define: • GT Miss Rate (FN %): The percentage of ground-truth HOI instances that are not matched by any prediction (false negatives): GT Miss Rate = |FN| |GT| × 100 where|GT| is the number of ground-truth HOI instances, and|FN| is the number of ground-truth instances with no matched prediction (e.g., under the chosen IoU and simi- larity threshold). • Prediction Miss Rate (FP %): The percentage of pre- dicted HOI instances that do not match any ground-truth instance (false positives): Prediction Miss Rate = |FP| |Pred| × 100 where|Pred| is the total number of predictions, and|FP| is the number of predictions with no matched ground-truth. For models that provided confidence scores with their pre- dictions, we set the confidence score rank to i = 0.5. We report that HOLA [18] (ViT-L) achieves the highest SHOE mF1 score (52.21) among existing fully supervised meth- ods, while maintaining low miss rates for both ground- truth and predictions. HOLA (ViT-B) performs best in the zero-shot RF-UC setting, with a SHOE mF1 score of 49.07. Open-vocabulary models such as CMD-SE [21] and THID [50] have higher miss rates and lower mF1 scores, highlighting the challenge of open-world generalization. These findings are consistent with the model ratings ob- served on the full HICO-DET dataset. Notably, the VLMs and MLLMs perform comparably to or even outperform the existing HOI detection methods, despite lacking HOI- specific supervision, and exhibiting high GT and prediction miss rates. This is consistent with our observation that these models tend to make fewer predictions per ground-truth in- stance than standard HOI methods. The strong overall per- formance suggests that VLMs and MLLMs produce seman- tically plausible interaction predictions and generalize inter- action concepts via language grounding. 1 #DSG.Mv.Qw.YiAgree.% 11000086.46 10100084.56 10010084.11 10001085.60 10000184.87 Avg.85.12±0.92 21100086.46 21010086.46 21001086.46 21000186.46 20110084.56 20101084.56 20100184.56 20011084.58 20010184.15 20001185.60 Avg.85.39±0.99 #DSG.Mv.Qw.YiAgree.% 31110085.55 31101085.76 31100186.12 31011085.40 31010185.52 31001185.93 30111084.71 30110184.53 30101184.88 30011185.12 Avg.85.35±0.53 41111085.67 41110185.86 41101186.40 41011185.91 40111185.24 Avg.85.82±0.42 5 (SHOE)1111185.73 Table 1. Combined LLM ablation: left = sizes 1-2, right = sizes 3-5. Agreement scores for LLMs with human annotation. 1 = model included, 0 = excluded. # = number of models included. LLMs in order: DeepSeek-V3-0324, Gemini-2.5-pro, Llama-4-Maverick-17B, Qwen3-32B, Yi-1.5-34B-Chat. Method Type Model GT Miss Rate (FN %) Prediction Miss Rate (FP %) SHOE mF1 Default LAIN [14] (ViT-B)21.0437.7745.80 CMMP [22] (ViT-L)20.0637.6450.47 ADA-CM [20] (ViT-L)19.1836.2151.93 HOLA [18] (ViT-L)19.4535.8352.21 Zero-shot RF-UC CMMP [22] (ViT-B)21.1539.1344.84 EZ-HOI [17] (ViT-B)20.4538.1046.94 HOLA [18] (ViT-B)20.0037.2949.07 LAIN [14] (ViT-B)20.8338.2046.30 Open- Vocabulary CMD-SE [21]36.9360.6737.07 THID [50]47.8178.8628.71 VLMs & MLLMs Qwen2.5-32B-VL [2]30.4860.7554.40 InternVL3-38B [8]55.0986.1040.04 LLaVa-Onevision-72B [23]32.0464.1354.50 GPT-4.1 [38]27.8655.1053.46 Gemini-2.5-flash [10]34.3959.0551.16 Table 2. Performance comparison across different HOI models on the single HOI HICO-DET [6] subset (575 classes) using GT Miss Rate, Prediction Miss Rate and SHOE mF1 scores on HICO-DET, along with GT miss rate (false negatives) and prediction miss rate (false positives). confidence score rank i = 0.5 was chosen for all models that provide confidence scores to make them comparable to the VLM and MLLM methods. 2 3. Hardware Requirements and Runtime In the following we provide an overview of the hardware used and the runtime required to compute the object and verb SHOE-similarity matrices on HICO-DET. These ma- trices have to be computed only once per dataset which en- ables all subsequent experiments to reuse them without any additional model inference. This section is meant as a prac- tical reference for researchers designing new datasets, illus- trating the one-time computational cost required to generate verb–verb and object–object similarity matrices. Our setup combined an on-premise server with 4× NVIDIA RTX 6000 Ada (48 GB) GPUs, cloud infer- ence via the Lambda service for selected models, and the Gemini-2.5-Pro API. We first evaluated the full set of ≈850k verb–verb pairs with Qwen3-32B, which required ≈6 days on the RTX 6000 Ada system. From these, we ex- tracted the≈120k non-zero similarity pairs and ran them on DeepSeek-V3-0324, Llama-4-Maverick-17B, and Yi-1.5- 34B-Chat, each completing in ≈30 h. For the 40k ob- ject–object pairs, all models required≈6 h each. DeepSeek and Llama were executed on Lambda cloud instances with runtimes matching the on-premise hardware, while Gemini- 2.5-Pro completed both verb and object evaluations under batched API requests within ≈ 28h. To reduce wall-clock time, several runs were parallelized across cloud machines. The total cost of API usage and cloud inference was≈ 200$ for the full HICO-DET similarity evaluation. 4. Synset Matching Procedure Open-vocabulary predictions must first be mapped to their corresponding WordNet [35] synsets before they can be evaluated with the SHOE metric. This requires convert- ing the model outputs into consistent lexical forms before matching them to WordNet. Predicted verbs that are not in base form are lemmatized (e.g., ”riding” to ”ride”), and ob- ject names containing underscores are split into their sep- arate words. Each normalized lemma is then queried in WordNet to retrieve all associated synsets (i.e., possible senses). For multi-word objects such as “dining table”, we query each component word separately and aggregate the valid synsets as a candidate pool. During evaluation, the synset from this pool with the highest semantic alignment to the ground-truth synset is selected, and this best-matching sense is used for the SHOE similarity computation. 5. User Study 5.1. User Study Setup and Examples To better illustrate the design of our user study, we provide a representative example of the annotation interface and rat- ing process. In each task, annotators were presented with a pair of HOI labels, along with the corresponding Word- Figure 1. Example of the user study interface. Annotators are shown two HOI interactions along with their WordNet glosses and asked to rate their semantic similarity on a 5-point scale. Net synsets and their glosses (definitions) for both the verb and object components. Annotators were asked to assess the semantic similarity between the two HOIs based on the information provided. Figure 1 shows an example of the annotation interface used in our study. The two HOIs are displayed along with their WordNet glosses to help anno- tators understand the meaning of each interaction. Annota- tors then assign a similarity score on a 5-point scale ranging from 0 (completely dissimilar) to 4 (interchangeable). 5.2. Inter-Annotator Score-Distribution We analyzed the overall score distribution across annota- tors to examine how consistently they used the 0–4 rat- ing scale (Figure 2). All annotators used the full range of scores, though each showed distinct rating tendencies. For example, A4 assigned score 0 more frequently than oth- ers, while A5 used score 4 most often. A2 and A3 tended to assign scores clustered around the middle of the scale. Although the evaluation criteria were consistent across an- 3 050100150200 Count 0 1 2 3 4 Score 53 68 94 141 144 22 53 149 188 88 44 67 112 108 169 109 96 86 183 26 46 36 95 138 185 Score Distribution by Annotator Annotator A1 A2 A3 A4 A5 Figure 2. Per-annotator distribution of HOI similarity scores rang- ing from 0 (no similarity) to 4 (high similarity), showing individual rating tendencies across the annotation set. notators, their scoring patterns suggest varying degrees of strictness. To better understand how annotators varied in their as- signments depending on the overall consensus of an item, we grouped examples by their majority (most frequent) score and visualized the distribution of all annotator votes for each group (Figure 3). Items with a majority score of 0 or 4 showed strong agreement, with most annotators assign- ing the same score. In contrast, examples with mid-range majority scores (e.g., 2) showed more varied responses across neighboring scores, reflecting greater subjectivity in these cases. 5.3. Inter-Annotator Agreement Metrics For the assessment of inter-annotator reliability we em- ployed Quadratic Weighted Kappa (QWK) and Krippen- dorff’s alpha that account for the ordinal structure of the data: Quadratic Weighted Kappa. Quadratic Weighted Kappa (QWK) [12] measures agreement between two annotators, assigning partial credit for near matches. It is defined as: κ = 1− P i,j w ij O ij P i,j w ij E ij , where O ij is the observed count of examples where rater 01234 Majority Score 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of Votes Distribution of Annotator Votes by Majority Score Assigned Score 01234 Figure 3. Distribution of Annotator Votes by Majority Scores. Each bar shows the number of times a given annotator (A1–A5) assigned each score from 0 to 4. The numeric labels on top of each bar indicate the raw count. A assigned score i and rater B assigned score j, and E ij is the expected count of such pairings under random chance. The weight matrix w ij penalizes disagreements quadrati- cally: w ij = (i− j) 2 (k− 1) 2 , with k = 5 being the number of discrete rating levels. Figure 4 shows a moderate level of agreement overall, with stronger consistency between pairs such as A1 & A5 and A2 & A5, and slightly weaker alignment between A3 & A4. The average QWK score across all annotator pairs was 0.590, indicating moderate agreement. Krippendorff’s Alpha (Ordinal). Krippendorff’s Al- pha [16] generalizes agreement measurement to multiple annotators and missing values. For ordinal labels, it pe- nalizes disagreement based on the normalized distance be- tween scores: α = 1− D o D e , where D o is the observed disagreement, and D e is the expected disagreement under chance. The pairwise dis- agreement between scores i and j is calculated as: δ(i,j) 2 = i− j k− 1 2 . The Krippendorff’s alpha for ordinal ratings was 0.566, supporting the finding of moderate consistency among an- notators. These results suggest that while annotators may assign slightly different scores, they generally agree on the relative severity or quality of the examples. This indicates 4 A1A2A3A4A5 A1 A2 A3 A4 A5 1.000.620.630.580.65 0.621.000.550.540.64 0.630.551.000.510.61 0.580.540.511.000.56 0.650.640.610.561.00 Quadratic Weighted Kappa Between Annotators 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 QWK Figure 4. Pairwise Quadratic Weighted Kappa (QWK) between annotators. All values are between 0.51 and 0.65, indicating mod- erate agreement. Diagonal entries represent perfect self-agreement (κ = 1.00). that the annotation process is reliable, as the ordinal struc- ture of the labels is largely preserved across annotators. 6. SHOE-Metric 6.1. Verb and Object based LLM agreement with human annotators The two settings in Table 3 show that all LLMs achieve strong agreement with human annotators if either object or verb is changed while the other is the same as the GT. Object mismatches show slightly higher agreement over all LLMs than verb mismatches. DeepSeek-V3-0324 shows the highest agreement in both categories. Our SHOE score, which aggregates model predictions through majority vot- ing, demonstrates competitive agreement in both settings and reliability for different types of prediction mismatches. 6.2. User Study with Images Our primary user study was conducted without images in order to collect semantic similarity judgments that gener- alize across different scenes, not tied to any specific visual instance. Including images risks biasing annotators toward incidental scene details, such as lighting, occlusion, or un- usual body poses, that have nothing to do with the semantic relationship between the interactions and are visual noise that HOI models are meant to filter out. However, as SHOE is ultimately applied to image-conditioned HOI predictions, we conducted a second user study to verify that semantic judgments remain stable when images are actually present. The annotators evaluated 500 HOI pairs, each accompanied by an image from the HICO-DET dataset corresponding to the ground-truth HOI class in the pair. They were tasked to decide if they wanted to update their original annotation Metric ModelVerb Agreement (%) Object Agreement (%) Standard SHOE78.8280.09 DeepSeek-V3-032479.8981.21 Qwen3-32B78.8681.41 Yi-1.5-34B-Chat79.0180.61 Llama-4-Maverick-17B77.9280.99 Gemini-2.5-pro78.3481.34 Table 3. Average LLM agreement with human annotator ratings on HOI pairs where either the verb or object differs from the ground truth while the other is identical. or keep it the same with the image present. Across all 500 HOI pairs, only 7.65% of ratings were updated (95% CI: 6.29–9.00%). The magnitude of these changes was small, only a single point on the 0-4 point scale (≈1.14 [95% CI 1.11, 1.17]). This demonstrates that the semantic judgments collected without images remain consistent even when an- notators are later presented with visual context further sup- porting the reliability of our metric. 6.3. Statistical Basis for Participant and Annotation Counts Our user study evaluates 500 HOI pairs through ratings from five independent annotators. To verify that this sample size is statistically adequate, we compute the Spearman cor- relation between model scores and human ratings. We ob- tain ρ = 0.799 with a 95% Fisher confidence interval (CI) of [0.765, 0.829]. This interval represents the range of plausi- ble values for the true correlation if the study were repeated many times with the same sample size. The interval is nar- row and strongly positive. This supports that 500 items provide a robust estimate of the true correlation. We also performed a permutation test with 10,000 resamples, which tests if such a high correlation could occur under random ratings. The result p < 0.001 indicates that the observed correlation is extremely unlikely to arise by chance. To assess whether five annotators per item are sufficient, we estimate the within-item variability (σ ≈ 0.761) and compute the standard error of the mean rating (SE = σ √ k ). Five raters achieved SE ≈ 0.34. This SE implies that the mean rating would vary by 0.34 of a point across differ- ent groups of annotators, which is small relative to the full rating range (0-4) and sufficient to distinguish meaningful differences between items. This shows that five annotators are enough to provide reliable similarity scores. 5 6.4. SHOE-algorithms To make our work more reproducible we describe the im- plementation of the two SHOE-scoring algorithms. Our main SHOE-mAP-score algorithm is shown in Algorithm 0. For each image, predictions are filtered by a confidence threshold and matched to ground-truth instances based on the common IoU > 0.5 requirement on the human and ob- ject bounding boxes. Predictions count as soft true posi- tives when they match spatially and have highest seman- tic similarity of all predictions for that GT. Predictions that do not match any ground truth get penalized through false positive penalties. For each unmatched prediction, the GT class with the highest similarity score is found. A FP is as- signed to that class if the similarity score is larger than a certain threshold. The algorithm sorts predictions by confi- dence before it calculates true positives and false positives using soft labels for each class. Precision and recall get computed at each threshold before obtaining Average Preci- sion (AP) through the calculation of area under the resulting precision-recall curve. The final SHOE mAP is the mean of AP scores across all classes. The procedure for computing the SHOE mF1 score for HOI predictions using soft matching based on semantic similarity is described in Algorithm 0. If models provide confidence scores only predictions larger than a chosen threshold are included in the calculation (s ≥ τ ). For each image the spatial alignment of each prediction is verified through bounding box IoU and the semantic similarity be- tween predicted verb-object pairs and ground truth pairs is computed with our provided SHOE similarity maps. Pre- dictions get matched identically to the SHOE-mAP-score. In the final aggregation step precision, recall and F1 scores for each class are calculated based on the accumulated FP, TP and FN before averaging over all classes for the mean F1 score (mF1). 7. Future Work While large VLMs show strong semantic reasoning for HOI prediction, they currently rely on an external detector such as DETR due to their limited detection capabilities. VLMs require substantially more parameters (≈1.5B - ≈400B) compared to standard HOI detectors (≈40M), making them impractical for real-time deployment and HOI detection from videos. Their prompt sensitivity and non-deterministic outputs further complicate consistent predictions and inte- gration into downstream systems. This highlights the need to develop true open-vocabulary, real-time HOI predictors that combine strong localization with true semantic under- standing. On the dataset side, existing HOI benchmarks often fo- cus on relatively small subsets of interactions and do not exhaustively annotate all visible HOIs within each image often using broad action categories like ”cooking”, ”work- ing”, or ”cleaning” but miss lower-level HOIs like ”cut- ting”, ”holding”, ”brushing”. As the field moves toward open-vocabulary HOI recognition, it will be essential to establish clear annotation guidelines and to develop more comprehensive datasets. 8. Benchmarking Datasets We conducted our primary evaluation on HICO-DET [6], as it is widely used in the HOI community. Most HOI models provide pretrained checkpoints on HICO-DET mak- ing it suitable for comparison between standard, zero-shot and open-vocabulary HOI models. SWIG-HOI [49] is com- monly used for open-vocabulary classification as it expands the label space to 1,000 object categories and 406 actions. However, its annotations inherit the scene-level abstraction of SWiG [40], which causes many fine-grained HOIs to be missed or collapsed into coarse descriptions. Fig. 5 shows a qualitative comparison between 3 common scenar- ios in both HICO-DET and SWIG-HOI dataset. Specific interactions such as eat, wash, or cut are annotated within HICO-DET. However, SWIG-HOI includes broader verbs like cook or clean as their ground-truth HOI classes, be- cause it is derived from general scene understanding. Moreover, SWIG-HOI includes only a single human an- notation per image, even when multiple people interact with objects (see Fig. 5 second SWIG-HOI example ”cook veg- etable/knife”) where two people cutting vegetables are vis- ible in the image but only 1 person and the correspond- ing HOI is annotated. This is acceptable for closed-set classifiers with fixed verb–object categories but becomes problematic in open-vocabulary evaluation with no prede- fined categories. In such settings, a model may correctly predict a valid but unannotated human–object interaction, which is then incorrectly counted as a false positive pe- nalizing category-free predictors. SWIG-HOI’s scene-level annotation style and incomplete human coverage makes it less suitable for open-vocabulary HOI evaluation. For this reason, we chose HICO-DET for our quantitative open- vocabulary evaluation. V-COCO [28] is an older benchmark that offers sub- stantially less interaction categories than the newer HICO- DET (26 actions vs 117 actions). Many HOI models do not provide checkpoints for V-COCO and focus on eval- uating on HICO-DET or SWIG-HOI. To demonstrate that the SHOE metric transfers seamlessly across benchmarks we report some exemplary results on this dataset (Table 4). The trends of the evaluated LLMs mirror those observed on HICO-DET. GPT-4.1 shows strongest performance for the standard mAP score while Qwen2.5-VL-32B achieves the highest SHOE mAP on V-COCO.. 6 ModelmAP SHOE-mAP GPT-4.150.4757.82 Qwen2.5-VL-32B41.1264.77 InternVL3-38B46.3450.79 Table 4. Comparison of mAP and SHOE-mAP on V-COCO for three common VLMs. VLMs token probabilities are used as a proxy for the confidence score and DETR is used for object detec- tion. 9. Qualitative results To better understand the results of Table 2, we present qual- itative examples of GPT-4.1’s predictions and their align- ment with HICO-DET ground truth based on our instance- level SHOE score (see Figure 6). The examples show that GPT-4.1 often generates semantically plausible outputs that remain close in meaning to the intended action, even when the exact ground-truth interaction is missed. For example, the ground-truth action “pet a giraffe” re- ceives a high similarity score of 0.775 when matched with the prediction “touch a giraffe,” which is semantically plau- sible given the close relation between these actions. Simi- larly, “wash a train” is matched with “clean a train” with a similarity score of 0.85. These examples highlight the abil- ity of the VLMs and MLLMs to perform robust conceptual reasoning and demonstrate that our SHOE score can cap- ture a broad range of semantically aligned open-vocabulary interactions. 10. Limitations Our proposed SHOE-Metric relies on WordNet sense dis- ambiguation and LLM similarity scoring, both of which can introduce biases into the metric. As WordNet senses were developed by linguists primarily from English corpora it may miss coverage in domain-specific or culturally diverse verb senses. Similarly, LLM judgments can reflect biases present in their training data. As existing HOI benchmarks focus on common general interactions, we did not find any apparent biases for these datasets during testing. For build- ing highly domain-specific HOI datasets with low WordNet coverage (e.g., medical or laboratory settings) we suggest using definitions from common ontologies in the field such as UMLS [3] and reducing risks of domain biases by con- ducting an expert user study, similar to the methodology adopted in this work. 7 Algorithm 1: SHOE mAP Evaluation for HOI Predic- tions 1: Load ground truth setG =(v,o,b h ,b o ) 2: Load predictions p = (v ′ ,o ′ ,b ′ h ,b ′ o ,s)∈P 3: Load similarity maps S v , S o 4: Initialize per-class lists S k , L k , and GT counts N GT [k] = 0 5: for each image I do 6: G I ← ground truth for I 7: P I ← predictions for I 8:Initialize matched flags forP I 9: for each (v,o,b h ,b o )∈G I do 10:k ← class ID of (v,o) 11: N GT [k] += 1 12: M← predictions inP I where IoU(b h ,b ′ h )≥ θ and IoU(b o ,b ′ o )≥ θ and prediction not already matched 13:ifM̸=∅ then 14:Select p ∗ = arg max p∈M 1 2 (S v (v,v ′ ) + S o (o,o ′ )) 15:Let p ∗ = (v ′∗ ,o ′∗ ,b ′∗ h ,b ′∗ o ,s ∗ ), where p ∗ ∈M⊆P 16:Compute similarity score σ = 1 2 (S v (v,v ′∗ ) + S o (o,o ′∗ )) 17:Append confidence and similarity score (s ∗ ,σ) toS k ,L k 18:Mark p ∗ as matched 19:else 20:Append empty pair (0.0, 0.0) toS k ,L k 21:end if 22: end for 23: for each unmatched p = (v ′ ,o ′ ,b ′ h ,b ′ o ,s)∈P I do 24:Find (v ∗ ,o ∗ )∈G I maximizing similarity 25:σ = 1 2 (S v (v ∗ ,v ′ ) + S o (o ∗ ,o ′ )) 26:if σ ≥ δ then 27:k ∗ ← class ID of (v ∗ ,o ∗ ) 28:Append (s, 0.0) toS k ∗ ,L k ∗ 29:end if 30: end for 31: end for 32: for each class k = 1...K do 33:SortS k ,L k by descending score 34:Let ε be a small constant (e.g., 10 −8 ) to avoid division by zero 35:Compute cumulative true positives: TP i = P i j=1 L k [j] 36:Compute cumulative false positives: FP i = i−TP i 37:Prec i = TP i /(TP i + FP i + ε) 38:Rec i = TP i /(N GT [k] + ε) 39:Compute AP k as area under the PR-curve 40: end for 41: Report mAP = 1 K P K k=1 AP k Algorithm 2: SHOE mF1 Evaluation for HOI Predic- tions 1: Load ground truth setG =(v,o,b h ,b o ) 2: Load predictions p = (v ′ ,o ′ ,b ′ h ,b ′ o ,s)∈P with confidence score rank s≥ τ 3: Load similarity maps S v , S o 4: Initialize TP = FP = FN = 0 per class 5: for each image I do 6: G I ← ground truth for I 7: P I ← predictions for I 8:Initialize matched flags forP I 9: for each (v,o,b h ,b o )∈G I do 10: M← predictions inP I where IoU(b h ,b ′ h )≥ θ and IoU(b o ,b ′ o )≥ θ and prediction not already matched 11:ifM̸=∅ then 12:Select p ∗ = arg max p∈M 1 2 (S v (v,v ′ ) + S o (o,o ′ )) 13:Let p ∗ = (v ′∗ ,o ′∗ ,b ′∗ h ,b ′∗ o ,s ∗ ), where p ∗ ∈M⊆P 14:Compute similarity score σ = 1 2 (S v (v,v ′∗ ) + S o (o,o ′∗ )) 15:Accumulate TP for class (v,o) as σ 16:Accumulate FP for class (v,o) as 1− σ 17:Mark p ∗ as matched 18:else 19:Accumulate FN for class (v,o) as 1.0 20:end if 21: end for 22: for each unmatched p = (v ′ ,o ′ ,b ′ h ,b ′ o )∈P I do 23:Find best (v,o)∈G I maximizing similarity 24:σ = 1 2 (S v (v,v ′ ) + S o (o,o ′ )) 25:if σ ≥ δ then 26:Accumulate FP for class (v,o) as 1.0 27:end if 28: end for 29: end for 30: for each class (v,o) do 31:Let ε be a small constant (e.g., 10 −8 ) to avoid division by zero 32:Compute precision = TP TP+FP+ε 33:Compute recall = TP TP+FN+ε 34:Compute F1 = 2·precision·recall precision+recall+ε 35: end for 36: Report mean F1 over all classes 8 (a) HICO-DET(b) SWIG-HOI Object(s): Apple Interactions: eat, hold Object(s): Apple Interactions: lick Object(s): Banana, Knife Interactions: hold, cut with, wield Object(s): Knife, Vegetable Interactions: cook Object(s): Knife Interactions: hold, wash Object(s): Knife Interactions: clean Figure 5. Qualitative comparison of interaction annotations. HICO-DET captures fine-grained, specific interactions (e.g., ”eat”, ”wash”, ”cut with”), whereas SWIG-HOI primarily provides broad scene descriptions (e.g., ”cook”, ”clean”). 9 GT: pet a/an giraffe Synsets: verb=pet.v.01, obj=giraffe.n.01 � matched prediction: Prediction: touch a/an giraffe Similarity: verb=0.550, obj=1.000, sim total=0.775 � unmatched prediction Prediction: feed a/an giraffe Most similar class: feed a/an giraffe with sim total=1.000 GT: fly a/an kite Synsets: verb=fly.v.03, obj=kite.n.03 � matched prediction: Prediction: fly a/an kite Similarity: verb=1.000, obj=1.000, sim total=1.000 GT: pull a/an kite Synsets: verb=pull.v.01, obj=kite.n.03 � matched prediction: Prediction: hold a/an kite Similarity: verb=0.350, obj=1.000, sim total=0.675 GT: cut a/an cake Synsets: verb=cut.v.01, obj=cake.n.03 � matched prediction: Prediction: cut a/an cake Similarity: verb=1.000, obj=1.000, sim total=1.000 � unmatched prediction Prediction: hold a/an knife Most similar class: hold a/an knife with sim total=1.000 GT: hold a/an motorcycle Synsets: verb=hold.v.02, obj=motorcycle.n.01 � matched prediction: Prediction: hold a/an motorcycle handlebar Similarity: verb=1.000, obj=1.000, sim total=1.000 GT: ride a/an motorcycle Synsets: verb=ride.v.10, obj=motorcycle.n.01 � matched prediction: Prediction: ride a/an motorcycle Similarity: verb=1.000, obj=1.000, sim total=1.000 GT: sit_on a/an motorcycle Synsets: verb=sit.v.01, obj=motorcycle.n.01 � matched prediction: Prediction: ride a/an motorcycle Similarity: verb=0.400, obj=1.000, sim total=0.700 GT: wash a/an train Synsets: verb=wash.v.03, obj=train.n.01 � matched prediction: Prediction: clean a/an train Similarity: verb=0.700, obj=1.000, sim total=0.850 GT: flush a/an toilet Synsets: verb=flush.v.05, obj=toilet.n.02 � matched prediction: Prediction: touch a/an toilet Similarity: verb=0.100, obj=1.000, sim total=0.550 Figure 6. Qualitative results for GPT4.1 on HICO-DET single HOI subset. 10