Paper deep dive
Gaze Target Estimation Anywhere with Concepts
Xu Cao, Houze Yang, Vipin Gunda, Zhongyi Zhou, Tianyu Xu, Adarsh Kowdle, Inki Kim, James M. Rehg
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Estimating human gaze targets from images in-the-wild is an important and formidable task. Existing approaches primarily employ brittle, multi-stage pipelines that require explicit inputs, like head bounding boxes and human pose, in order to identify the subject of gaze analysis. As a result, detection errors can cascade and lead to failure. Moreover, these prior works lack the flexibility of specifying the gaze analysis task via natural language prompting, an approach which has been shown to have significant benefits in convenience and scalability for other image analysis tasks. To overcome these limitations, we introduce the Promptable Gaze Target Estimation (PGE) task, a new end-to-end, concept-driven paradigm for gaze analysis. PGE conditions gaze prediction on flexible user text or visual prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject for gaze analysis. This approach integrates subject localization with gaze estimation, and eliminates the rigid dependency on intermediate analysis stages. We develop a scalable data engine to generate Gaze-Co (Gaze Estimation with Concepts), a dataset and benchmark of 120K high-quality, prompt-annotated image pairs. We also propose GazeAnywhere, the first model designed for PGE. GazeAnywhere uses a transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation. GazeAnywhere achieves state-of-the-art performance on multiple PGE benchmarks, setting a strong baseline for this new problem even on a difficult out-of-domain, real-world clinical dataset. GazeAnywhere is open-sourced in this http URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.11367v1
- Canonical: https://arxiv.org/abs/2608.11367v1
Trouble viewing inline? Open PDF directly →
Full Text
88,795 characters extracted from source content.
Expand or collapse full text
Gaze Target Estimation Anywhere with Concepts Xu Cao 1 , Houze Yang 1 * , Vipin Gunda 1 , Zhongyi Zhou 2 , Tianyu Xu 2 , Adarsh Kowdle 2 , Inki Kim 1 , James M. Rehg 1† 1 University of Illinois Urbana-Champaign 2 Google xucao2,jrehg@illinois.edu Abstract Estimating human gaze targets from images in-the-wild is an important and formidable task. Existing approaches primar- ily employ brittle, multi-stage pipelines that require explicit inputs, like head bounding boxes and human pose, in order to identify the subject of gaze analysis. As a result, detec- tion errors can cascade and lead to failure. Moreover, these prior works lack the flexibility of specifying the gaze analy- sis task via natural language prompting, an approach which has been shown to have significant benefits in convenience and scalability for other image analysis tasks. To overcome these limitations, we introduce the Promptable Gaze Target Estimation (PGE) task, a new end-to-end, concept-driven paradigm for gaze analysis. PGE conditions gaze prediction on flexible user text or visual prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject for gaze analysis. This approach integrates subject localization with gaze estimation, and eliminates the rigid dependency on intermediate analysis stages. We de- velop a scalable data engine to generate Gaze-Co (Gaze Estimation with Concepts), a dataset and benchmark of 120K high-quality, prompt-annotated image pairs. We also propose GazeAnywhere, the first model designed for PGE. GazeAnywhere uses a transformer-based detector to fuse features from frozen encoders and simultaneously solves sub- ject localization, in/out-of-frame presence, and gaze target heatmap estimation. GazeAnywhere achieves state-of-the- art performance on multiple PGE benchmarks, setting a strong baseline for this new problem even on a difficult out- of-domain, real-world clinical dataset. GazeAnywhere is open-sourced in github.com/IrohXu/GazeAnywhere. 1. Introduction Human gaze is a fundamental non-verbal cue, conveying a wealth of social and cognitive information [28,51]. It * Co-first author † Corresponding author short light brown hair and a red shirt child Box: [x, y, w, h] short light brown hair and a red shirt child Wrong due to incorrect subject detection ! Head Detector Gaze Model Image Encoder Text Encoder Detector Gaze Tracker GazeAnywhere Figure 1. Gaze target estimation in in-the-wild environments. Prior methods such as Sharingan and ViTGaze have to rely on an Open- Vocabulary Detector (OVD) to produce auxiliary head boxes via dynamic human prompts, introducing a sequential dependency that becomes a major bottleneck. is integral to human interaction, used to initiate social con- tact, signal attention and interest, manage conversational turn-taking, and regulate intimacy. As a direct proxy for cognition, gaze can also reveal a person’s intentions, pref- erences, and emotional states [54,62]. Consequently, its study has attracted significant interest across diverse fields, including psychology, human-computer interaction, and clin- ical research on conditions such as autism spectrum disorder (ASD) [1, 2, 22]. Despite its importance, accurately estimating a person’s gaze target in unconstrained, “in-the-wild” settings via im- age analysis remains a formidable challenge. Current ap- proaches typically require explicit prior information (such as human head and face bounding boxes, pose estimations, and depth) not only for training but also as critical inputs during inference [58,60]. Consequently, the most common architecture is a multi-stage pipeline where intermediate in- puts are generated by a series of pre-processing steps prior to gaze estimation (Figure 1). This process typically requires first detecting and tracking a person and then precisely local- izing their head or face with a bounding box before the gaze direction can be computed [12,79]. This sequential depen- dency creates a critical bottleneck, as inaccuracies in these initial stages (common in crowded scenes, poor lighting, or with challenging cases like detecting children’s faces) can arXiv:2608.11367v1 [cs.CV] 11 Aug 2026 cascade, leading to a failure of the entire system. To overcome this limitation and provide greater conve- nience and flexibility in specifying gaze analysis tasks, we propose a paradigm shift toward an end-to-end, concept- based framework for gaze target estimation. This approach is inspired by recent advancements in vision foundation mod- els, such as the Open-vocabulary Detectors (OVDs) [25,81], and the Segment Anything Model (SAM) series [9], which demonstrate remarkable abilities to detect and segment ob- jects based on semantic-level text or visual concept prompts rather than explicit localization cues via bounding boxes. We extend this core idea to the domain of human gaze un- derstanding, designing a promptable model that can directly identify the gaze target of a specified person within an image. By conditioning on a concept of the subject (e.g., “the boy in the red shirt”), our approach eliminates the dependencies on intermediate head bounding boxes or pose keypoints. This allows the model to infer gaze targets from a semantic under- standing of the scene in a user-friendly end-to-end fashion, paving the way for more robust and versatile systems. Our main contributions can be summarized as: (1) We define the Promptable Gaze Target Estimation (PGE) task, which extends the traditional gaze target es- timation problem to an unconstrained, text promptable end-to-end paradigm. (2) We design a scalable data engine to generate 120K high quality PGE training annotations consisting of subject text to gaze alignment data pairs. (3)We introduce GazeAnywhere, the first promptable concept-driven gaze target estimation model. Our model achieves state-of-the-art performance in several benchmarks including an out-of-domain private dataset for autism children’s gaze target estimation. 2. Related Work Human Gaze Target Estimation. Interpreting gaze is a cru- cial component of human behavior understanding [21,77]. This has motivated the "gaze-following" task, introduced by datasets like GazeFollow [54,55], VideoAttentionTar- get [15], GOO [64] and ChildPlay [61], which requires a model to predict the scene location a person is looking at. Dominant approaches have thus far employed multi-branch, fusion-based architectures. These models separately process explicit cues such as head position [58,60,72,73], pose [4], text-based directions [46,69], facial expressions [37], and depth [29,33,45,65], subsequently combining these fea- tures to predict gaze points [11,30,32,41,45,68,76,80]. In addition, existing end-to-end models cannot specify the subject person during inference either [18,66,68]. While effective, these strategies are dependent on the availability and accuracy of these intermediate representations. Concur- rently, other studies have expanded the task’s scope, such as multi-view gaze target estimation [47] and GazeHOI [62] for open-vocabulary targets. Despite these advances, a com- mon limitation still persists: a dependency on auxiliary in- formation, such as precise head bounding boxes or pose estimations. This reliance poses significant challenges in realistic settings, where such priors are often unreliable or unavailable. Promptable and Interactive Object Detection. Recent advances in promptable and open-vocabulary perception are enabling models to generalize beyond fixed label sets, espe- cially the referring expression comprehension tasks [8,31, 49,50,74,78]. OVD methods, for instance, leverage large- scale vision-language encoders like CLIP to detect arbitrary concepts specified by text at inference time, even for cate- gories unseen during training [25,42,48,83]. In parallel, interactive segmentation frameworks demonstrate how mod- els can respond to flexible text and visual prompts [9,36]. Together, these advances suggest the new paradigm: visual perception systems can be effectively guided by conceptual cues, rather than rigidly predefined supervision. Building on this paradigm, we formulate gaze target estimation as a concept-conditioned reasoning task, where both the subject and their attended region are inferred from semantic prompts rather than explicit localization inputs. Vision Foundation Models. Vision Foundation Models (VFMs) have become a dominant approach in computer vision, entailing a significant shift of models trained on massive web-scale datasets. VFMs primarily include two branches: (1) weakly-supervised models like CLIP [52], SigLIP [67,75], and MetaCLIP [6,16,70], which learn powerful representations from image-text pairs using con- trastive losses, and (2) self-supervised learning (SSL) models like DINO series [34,34,59], which learn robust visual fea- tures from unlabeled images. These powerful, pre-trained encoders now serve as general-purpose backbones for a wide range of downstream tasks. This trend has also influenced gaze estimation, where systems like ViTGaze [60] and Gaze- LLE [58] have successfully adapted VFM architectures and leveraged their pre-trained features to improve performance, demonstrating the value of these models for fine-grained, human-centric tasks. 3. Method 3.1. Promptable Gaze Target Estimation We define the PGE task as estimating the gaze target location of a specific subject within an image or video, identified by a user-provided prompt. This prompt can be of two types: text prompting via a short, natural language query; and visual one by prompting a spatial coordinate, such as the center point of a head bounding box. In text prompting, our goal is to support any simple, visually-groundable noun phrase as a text prompt. However, this introduces intrinsic GazeAnywhere Visual Encoder Visual concept: People at [0.39, 0.18] Text Encoder Projection 푾 푽 Projection 푾 푻 Gaze Tracker [x, y, w, h] Head Tracker In Presence Predictor ቄ Text concept: brown hair with bangs in a blue and white striped cardigan girl or 휙 푉 휙 푇 휓 Detector 푍 푉 푍 푇 Head Token Presence Token [CLS] Token [EOS] Token Figure 2. An overview of the GazeAnywhere end-to-end framework for PGE. The model uses frozen visual (DINOv3) and text (dino.txt) encoders to provide features to a trainable transformer-based detector. This detector utilizes multiple decoders to simultaneously predict a subject’s head bounding box , in-frame presence , and gaze target. GazeAnywhere is conditioned on flexible user prompts, such as a natural language text description or visual (coordinate-based) cues. ambiguity (e.g., "the person in the back"). To mitigate this and enable unambiguous identification, we structure text prompts around four main categories. A user can combine descriptions from these categories to specify a subject: (1) Appearance: Noun phrases describing a person, consisting of an identity (e.g., woman, man, child) and optional modifiers (e.g., hair type/color, clothes, glasses, hat). (2) Location: The subject’s spatial position in the image (e.g., center, left, top-right). (3) Pose: The subject’s static posture. (4) Action: Verb phrases describing what the subject is doing. More formally, given an input RGB imageI ∈ R 3×H×W and a promptP(either textTor a visual cue), the goal of PGE is to produce a gaze heatmap ˆ H ∈ R H out ×W out . Each element ˆ H(i,j)represents the probability that the subject specified byPis gazing at the spatial location(i,j). Un- like classic gaze estimation methods, PGE demands that the model solve the task end-to-end, directly linking a flexible, high-level query to a final gaze heatmap. This formulation is substantially more challenging as it precludes the use of aux- iliary inputs common in traditional pipelines (e.g., subject bounding boxes, pose keypoints, or depth maps). Our model must implicitly learn to perform subject identification, local- ization, and gaze estimation jointly with text input, rather than relying on the explicit outputs of separate, specialized models like open-vocabulary detectors or pose estimators. 3.2. GazeAnywhere Architecture Figure 2 shows the overall architecture of GazeAnywhere. The model consists of a frozen image encoder, a frozen text encoder to proceed visual modality and text modality. A transformer-based detector is used to learn joint repre- sentations and map text prompt into the main gaze target estimation and auxiliary tasks. Image Encoder. We use a frozen ViT, denotedφ V (·), as our image encoder to extract general visual features. Consistent with PGE’s problem definition, we do not em- ploy any auxiliary models for dedicated depth or pose feature extraction. The image encoder processes an in- put imageI ∈ R 3×H×W by dividing it into a sequence ofN V patch tokens, to which a learnable[CLS]token is prepended. The resulting output sequence fromφ V is φ V (I) = [c,s 1 ,s 2 ,...,s N V ] ∈ R (1+N V )×D V . Here,D V is the embedding dimension,c ∈ R D V is the final[CLS] token embedding, ands i ∈ R D V is the output embedding for the i-th patch. Text Encoder. We employ a frozen text encoder,φ T (·), which consists of a series of transformer blocks and a final linear layer. The linear layer maps the output[EOS]token’s feature to the image embedding space. To prepare the input, a tokenizer first converts the textTinto a sequence of token IDs. These IDs are then mapped to initial text embeddings T E via an embedding layer, and the sequence is padded to a fixed context length,L T . The encoderφ T processes this em- bedding sequenceT E , producing the final output sequence: φ T (T E ) = [t 1 ,t 2 ,...,t N T ,t eos ,...,t pad ] ∈ R L T ×D T . Here,D T is the output embedding dimension,t i ∈ R D T is the output embedding for thei-th content token, and t eos ,t pad ∈ R D T are the special tokens representing the end of the input and padding, respectively. Projection Layers. The image encoderφ V and text en- coderφ T output features with potentially different dimen- sions,D V andD T , respectively. To map these features into a unified, shared space, we introduce two trainable linear projection layers,W V andW T . These layers project the high-dimensional features into a common, lower-dimension D, where D < min(D V ,D T ): Z V = W V · φ V (I)(1) Z T = W T · φ T (T E )(2) Here,W V andW T are the learnable projection matrices, and the operation·denotes a per-token linear transformation. This process results in a sequence of visual tokensZ V ∈ R (1+N V )×D and text tokensZ T ∈ R L T ×D , which now share the same embedding dimension. Task-Specific Embeddings. Beyond the primary cross- modal feature alignment, we introduce two specialized, learn- able embeddings to explicitly model key sub-problems: a head token and a gaze presence token. 1.Head Token: This is a learnable embedding designed to explicitly predict the head localization of the prompted subject, serving as the image-text alignment objective in our task. It is initialized using the embedding of the text [EOS] token. 2.Target Presence Token: This token is introduced to address the in/out-of-frame gaze target boolean predic- tion objective. The rationale for this is that the in/out decision relies on global contextual cues from the entire image, which conflicts with the inherently local nature of the target localization objective. Forcing a single query or mechanism to handle both can be counterproductive. Therefore, we decouple the localization and in/out pre- diction tasks. This dedicated, learnable global token is responsible for the in/out prediction and is initialized using the embedding of the visual [CLS] token. Detector Transformer. After extracting and projecting the visual and text features, we introduce a Detector Trans- former,ψ(·), to fuse these representations and refine them for the gaze target estimation task.ψ(·)apply the same Transfomrer block in DINOv3 [59]. The input toψis a sin- gle sequenceFconstructed by concatenating the projected features and our specialized task tokens. First, we define the head tokent h ∈ R D and target presence tokent p ∈ R D . These are formed by combining the projected global tokens (c ′ from vision,t ′ eos from text) with dedicated learnable embeddings, E presence : t h = t ′ eos (3) t p = c ′ + E presence (4) Lets ′ = [s ′ 1 ,...,s ′ N V ] be the sequence of projected visual patch tokens fromZ V (excludingc ′ ) andt ′ = [t ′ 1 ,...,t ′ N T ] be the projected text content tokens fromZ T (excludingt ′ eos and padding). The full input sequenceFis then assembled as: F = [t h , t ′ , s ′ ,t p ]∈ R (N T +N V +2)×D (5) We inject positional information by adding 1D sinusoidal position embeddings to the text tokenst ′ and 2D sinusoidal position embeddings to the visual tokenss ′ [20]. The De- tector transformerψis a stack ofkstandard transformer blocks;kis a hyperparameter ablated in our experiments. ψprocessesFand outputs a refined sequence of the same dimension, ψ(F)∈ R (N T +N V +2)×D . Specific tokens from this output are then passed to dedicated decoders. Decoders. The Detector transformerψoutputs a refined sequence of tokens. We attach three distinct prediction heads to specific tokens from this sequence to produce the final outputs. •Gaze Tracker (Heatmap Decoder): The refined visual patch tokensˆs∈ R N V ×D are first re-assembled from their 1D sequence form back into a 2D spatial grid. This feature map is then fed through a convolutional decoder, consist- ing of two transposed convolutional layers, which upsam- ples the features to the output heatmap ˆ H ∈ R H out ×W out . In our experiments, we set H out = W out = 64. •Head Tracker (Box Decoder): We use the refined head token ˆ t h ∈ R D for an auxiliary head localization task. The token is passed through a 3-layer feed-forward network (FFN) with ReLU activations and a hidden dimension of D. This head regresses a 4-dimensional vector[x,y,w,h] representing the normalized center coordinates, width, and height of the subject’s head box. •Presence Predictor (In/Out Decoder): The refined gaze presence token ˆ t p ∈ R D is used to predict whether the gaze target is in or out of the frame. It is processed by a 2-layer FFN (with one hidden layer of dimensionDand ReLU activation) that outputs a single logit for the binary classification. 3.3. Learning Objective We train our model end-to-end with a joint multi-task objec- tive. The total lossL total is a weighted linear combination of three loss terms: one for the gaze heatmap, one for the gaze presence, and one for the auxiliary head localization task. L total =L gaze +L presence +L head (6) The gaze heatmap lossL gaze is a pixel-wise binary cross- entropy (BCE) loss. The supervisory target is a heatmap Y, constructed by placing a 2D Gaussian (σ = 3) at the ground-truth gaze target location. Let ˆ Ybe the predicted heatmap. The loss is defined as: L gaze =− 1 N N X p=1 [y p log(ˆy p ) + (1− y p ) log(1− ˆy p )] (7) whereN = H out × W out is the total number of pixels, andy p andˆy p are the ground-truth and predicted values for a single pixel p, respectively. The gaze presence lossL presence is a Focal Loss supervised with a binary labelY presence ∈ 0, 1. Let ˆ Y presence ∈ [0, 1] be the model’s predicted probability that the target is present (Y presence = 1). The loss is defined as: L presence =L focal (Y presence , ˆ Y presence )(8) where the hyperparameter of the focal loss is the default value from [40]. The subject head bounding box lossL head is a linear com- bination of theL 1 loss and the generalized IoU loss. which is widely used by object detection tasks. It defined as: L head = λ l 1 ||b− ˆ b|| 1 + λ iou L iou (b, ˆ b)(9) whereband ˆ bis the ground truth head box and predicted head box.L IoU is the GIoU loss [8,57].λ l 1 andλ iou are head object detection hyperparameters. We followed DETR [8] and OWLViT [48] to set λ l 1 = 5 and λ iou = 2. 4. Gaze with Concept (Gaze-Co) Dataset Training GazeAnywhere for the PGE task requires a large and diverse dataset annotated with concepts, a resource that no existing gaze dataset provides. To address this, we devel- oped a scalable data engine that generates annotations via a human-in-the-loop feedback process. This engine worked in tandem with two human annotators (co-authors) to per- form several key functions: aligning heterogeneous annota- tions, filtering low-quality frames, generating concise con- cept phrases, and facilitating human verification. After three rounds of iteration, we created Gaze-Co, the first large-scale dataset for PGE, containing 120K samples sourced from the training set of GazeFollow, VisualAttentionTarget (VAT) and ChildPlay. To establish a comprehensive benchmark, we also converted the test sets of these well-known gaze datasets to the PGE format, creating GazeFollow-Concept, VAT-Concept, and ChildPlay-Concept. We further conducted experiments on a private, Institutional Review Board (IRB)- approved, out-of-domain (OOD) evaluation set with several frames in 40 child social communication (Child-SC) videos. 4.1. Data Engine Figure 3 illustrates the workflow of the data engine. We can divide the process into three stages: (1) data alignment and filter; (2) concept generation; (3) verification. Data Alignment and Filter. The source datasets differ in coordinate conventions, split policies, and metadata. We therefore adopt a unified schema with explicit pixel coor- dinates for the head box(x min ,y min ,x max ,y max ), and a normalized gaze point(g x /W,g y /H). Then, to ensure reli- able person-scale learning signals, we apply geometric and sharpness filters. Annotations are retained only if the head box width≥30 px, height≥40 px, area≥2500 px 2 , and the box-to-image ratio∈ [0.008, 0.3], with sufficient Tenengrad focus. These thresholds remove extremely small, oversized, or blurry instances while preserving diverse valid samples. Concept Generation. For each retained annotation, we pro- duce a short, lowercase concept phrase comprising attribute, Data Pool Auxiliary Anno Filter DataMLLM API Human Verifier MLLM Verifier short dark hair, glasses, and a light blue plaid shirt man Long brown hair, and a dark blue shirt woman pass pass fail fail Data Model HumanLabel Batch Label Figure 3. Overview of the GazeAnywhere data engine. position, action, and pose, together with a coarse count of visible people. Concept generation is executed with a pro- duction Vision Language Model (VLM) accessed through API (Gemini 2.5 Pro [17,26]), using batch processing with fixed prompts and rates. The attribute phase privileges stable visual cues (hair, glasses, beard, colors, and patterns) and the final token is constrained to one of man, woman, boy, girl, infant, child as an apparent (perceived) age/sex presentation label, used solely as a visual category cue rather than a veri- fied identity attribute; when indeterminate, we write “adult” or “child. The position uses brief canvas references (e.g., “bottom left corner”). Action and pose are explicitly non- overlapping: action describes ongoing interaction or motion with object or direction when visible, while pose captures static body configuration and facing direction. When a field is indeterminate, we write “none.” Verification. We adopt an Multi-modal Large Language Model (MLLM)-first, human-in-the-loop verification work- flow. The Gemini 2.5 Pro reviews all generated concepts and flags each as pass or fail. Human annotators then spot-check a random subset of the MLLM passed cases and evaluate the batch success rate. During review, both the MLLM and human annotators check whether each concept correctly matches its designated head box (consistency), whether all four fields are present and non-conflicting (completeness), and whether the text contains no sensitive or identifying in- formation (privacy). If the human verifier finds the batch success rate is low, the data engine will return to the con- cept generation stage. The human verifier then adjusts the prompts and rules and re-runs the concept generation and verification until the observed error rate is kept low (≤ 1%). For the private Child-SC dataset with IRB restriction, all concept annotations are generated manually by authorized human annotators without being sent to the MLLM. 4.2. Gaze-Co Dataset and Benchmarks Gaze-Co is the first large-scale dataset for promptable gaze target estimation, unifying GazeFollow, VAT, and ChildPlay under a shared schema with concept-level annotations for both training and evaluation. Training Data. The Gaze-Co 120K training set contains about 120K images from the official training splits of the Gaze ModelHead Detection#Total ParamTime per sample[ms]↓ GazeFollow-ConceptVAT-ConceptChildPlay-ConceptChild-SC (OOD) AUC↑Avg L2↓Min L2↓L2↓AP↑L2↓AP↑L2↓AP↑ ViTGaze [60] Ground-Truth--0.9550.0970.0450.0980.8790.1130.905-- GroundingDINO-B [42]255M1160.9130.1920.1160.2320.7920.1700.8630.1390.795 LLMDet-L [25]365M3390.8970.1890.1130.2350.7630.1580.8690.1650.825 OWLv2-L [49]460M1790.9270.1700.0940.2410.7490.1510.8620.1680.781 RexSeek [31]3B11600.9450.1190.0630.1230.8410.1170.9030.1360.818 Sharingan [63] Ground-Truth--0.9440.1140.0590.1090.8520.1180.854-- GroundingDINO-B [42]338M1190.8600.2590.1780.3520.7200.2160.8880.3490.771 LLMDet-L [25]449M3420.8650.2410.1600.2810.7850.2130.8740.3470.832 OWLv2-L [49]544M1820.8670.2550.1730.2910.7450.1850.8930.2630.847 RexSeek [31]3B11590.8880.2070.1420.2730.6010.1780.8630.1660.810 Gaze-LLE [58] Ground-Truth--0.9610.0990.0450.1010.8750.1130.912-- GroundingDINO-B [42]539M1450.9250.1650.1060.2340.7800.1520.8630.1720.856 LLMDet-L [25]650M3680.9120.1680.1090.2300.7930.1450.8750.1750.855 OWLv2-L [49]745M2080.9410.1460.0890.2290.7920.1270.8930.1610.830 RexSeek [31]3B11830.9540.1080.0540.1210.8610.1190.9140.1720.846 GazeAnywhere-CLIP-L430M350.9530.1050.0560.1370.8740.1040.9150.1460.868 GazeAnywhere-DINOv3-L870M960.9580.0990.0500.1230.8790.0980.9060.0900.902 Table 1. PGE results on four datasets. The input is the text prompt of the subject person’s appearance, position, action and pose. For baseline methods, the OVD is used to extract the bounding box with the input prompt and then feed the bounding box to the gaze models. Latency is compared with inference running speed per image in batch size = 1. three source datasets. Each record includes the target head box, normalized gaze point, in/out-of-frame label, and a com- pact concept phrase (attribute, position, action, and pose). All samples pass image quality filters, ensuring diverse, valid instances across viewpoint, poses, scales, and interaction contexts. Benchmark Settings. The benchmark uses the official test splits of GazeFollow, VAT, and ChildPlay, each converted into the Gaze-Co format. We evaluate concept-conditioned gaze prediction under three settings: (i) in-domain testing on the test sets; and (i) OOD evaluation on the Child-SC dataset, a developmental sample of children’s gaze behavior collected under an IRB-approved study (see the appendix for the dataset description). Every text prompt in the test set has been human-verified rather than spot-checked, ensuring ac- curacy and consistency. Each model receives the image, with the concept text added or altered according to the test setting. This setup provides a consistent framework for comparing models under controlled concepts-based conditions. 4.3. Metrics We evaluate models using heatmap Area Under the Curve (AUC) in GazeFollow and pixelwise L2 in all. For heatmap AUC, the predicted heatmap is treated as a confidence map to compute an ROC curve against the binary gaze target map. Pixelwise L2 measures the Euclidean distance be- tween the heatmap peak and the ground-truth gaze point. For GazeFollow-Concept, each image includes multiple gaze an- notations directed at the same target person, so we addition- ally report Avg L2 (distance to the mean of all targets) and Min L2 (distance to the nearest target). For VAT-Concept, ChildPlay-Concept, and Child-SC (the IRB-approved OOD set), annotations include binary in/out labels relative to the target region; thus, we report pixelwise L2, and average pre- cision (AP) to jointly evaluate localization and in/out binary classification. 5. Experiments We evaluate GazeAnywhere on the PGE task, comparing its text-prompting capabilities against State-of-the-Art (SOTA) two-stage pipelines that integrate OVDs for head/human detection with a separate gaze modeling stage. We also present a series of ablation studies demonstrating the impor- tance of the frozen encoders, validating our loss design, and analyzing the differences between visual and text prompt- ing. Finally, we demonstrate a real-world application, the “AnyGaze Agent,” a system that integrates GazeAnywhere with an Augmented Reality (AR) device and a MLLM. 5.1. Implementation Details All models are trained for 25 epochs using the Adam opti- mizer and a cosine learning rate scheduler with an initial rate of 1e-3 and batch size 128, followed by an additional 5 epochs with a reduced learning rate of 1e-5. All training experiments are conducted with 4 NVIDIA H100 GPUs. The inference is running with 1 NVIDIA L40S GPU. We adopt the DigiLens ARGO smartglass as the AR platform to deploy AnyGaze Agent for real world experiments. More details are shown in the appendix. 5.2. Main Results Table 1 compares GazeAnywhere against strong two-stage baselines, which we created by pairing three SOTA gaze methods Gaze-LLE [58], Sharingan [63], ViTGaze [60] with three leading OVDs for human detection [25,42,49]. De- ModelParam GazeFollow-ConceptVAT-Concept Avg L2↓Min L2↓L2↓AP↑ Qwen3-VL-8B [3]8B0.2010.1370.2860.651 Gemini 2.5 Flash [17]-0.2160.1560.2920.661 GazeAnywhere-DINOv3-L870M0.0990.0500.1230.879 Table 2. Compare GazeAnywhere with SOTA VLMs. Prompt type GazeFollow-ConceptVAT-Concept AUC↑Avg L2↓Min L2↓AUC↑L2↓AP↑ No prompting0.9440.1440.0900.8750.2100.796 visual prompting0.9580.1000.0500.9140.1310.894 text prompting (appearance)0.9520.1130.0620.9040.1530.840 text prompting (position)0.9530.1240.0730.8930.1880.826 text prompting (action)0.9490.1290.0780.8970.1800.839 text prompting (pose)0.9520.1210.0700.9090.1630.859 text prompting (all)0.9580.0990.0500.9280.1230.879 Table 3. Comparison of different prompt strategies. tails of these baselines are shown in the appendix. The encoders of GazeAnywhere can be CLIP-L [52] or DINOv3- L [59] with dino.txt [34]. On the PGE text-prompting task, GazeAnywhere achieves SOTA performance on all metrics across the three public datasets, as well as on our challenging OOD private dataset from a real-world assessment setting in which children’s social communication skills are quantified by experts. 5.3. Ablation Study We use GazeAnywhere-DINOv3-L for all following up abla- tion experiments in GazeFollow-Concept and VAT-Concept. More results are shown in Appendix Sec 13. Compare GazeAnywhere with SOTA VLMs. To demon- strate the utility of GazeAnywhere in PGE, we evaluate the 0-shot performance of SOTA VLM on gaze point prediction. Table 2 shows the comparison of GazeAnywhere, Qwen3- VL-8B and Gemini 2.5 Flash. GazeAnywhere surpass all of them, highlighting the importance of building specific model for PGE. PGE with Different Prompting. GazeAnywhere supports both visual (coordinate-based text) and text (natural lan- guage) prompts, as illustrated in Figure 2. In Table 3, we compare the performance of these different strategies. We find that text-based prompting achieves performance on par with visual prompting. Furthermore, our decomposition analysis of text prompt composition reveals that the sub- ject’s appearance and pose description are the most critical components for the PGE task. Loss ablations. We conducted an ablation study (Table 4) on our objective function’s components: gaze heatmap, pres- ence, and head losses. The essential gaze heatmap loss was always active, while we trained models removing the pres- ence loss, the head loss, and both. Results indicate the pres- ence loss only supports the auxiliary in/out prediction, not help gaze estimation. The head loss, however, improves both L gaze L presence L head GazeFollow-ConceptVAT-Concept AUC↑Avg L2↓Min L2↓AUC↑L2↓AP↑ ✓✗0.9560.1020.0520.9250.135- ✓✗0.9550.1030.0550.9240.1360.863 ✓✗✓0.9580.0990.0510.9240.128- ✓0.9580.0990.0500.9280.1230.879 Table 4. Ablation experiment on loss selection. EncoderParam GazeFollow-ConceptVAT-Concept AUC↑Avg L2↓Min L2↓AUC↑L2↓AP↑ CLIP-B [52]152M0.9420.1230.0700.9080.1450.837 CLIP-L [52]430M0.9530.1050.0560.9130.1370.874 SigLIP2-B [67]379M0.9490.1150.0640.9040.1480.860 SigLIP2-L [67]886M0.9530.1050.0560.9100.1470.858 DINOv3-L [34, 59]870M0.9580.0990.0500.9280.1230.879 MetaCLIP2-H [16]1.9B0.9510.1090.0590.9120.1500.857 Table 5. Comparison of different encoders for GazeAnywhere. the gaze target estimation and the target presence prediction. Comparison of Different Encoders. We conduct an ab- lation on the encoder backbone, comparing CLIP [52], SigLIP 2 [67], MetaCLIP 2 [16] and DINOv3 [59] (with dino.txt [34]). In all experiments, the encoders were frozen, with only the projection layer, transformer detector, and de- coder heads being fine-tuned. As shown in Table 5, the DINOv3-based model achieves the best performance on nearly all metrics, highlighting its superior visual-text align- ment and understanding for PGE. 5.4. Visualization Figure 4 showcases qualitative gaze estimation results from GazeAnywhere. The input, displayed in the black boxes, is a text prompt describing only the subject’s appearance, such as "light brown hair and a blue striped shirt boy". The visual- izations demonstrate that GazeAnywhere performs robustly not only in simple scenarios with 2-3 people but also in com- plex, crowded scenes with four or more individuals. Notably, the final two examples, "long black high ponytail hair and a pink shirt girl" and "short blonde hair wearing a dark blue and light gray shirt boy", are from an OOD Child-SC video dataset. The model’s successful performance on this unseen data highlights its generalization and robustness. 5.5. GazeAnywhere as Agent in AR Previous gaze estimation models ignore the real-world appli- cation experiment. Inspired by recent tool-enhanced MLLM workflows [9,71], we developed the GazeAnywhere Agent (Figure 5). This system uses a central MLLM (Gemini 2.5 [17]) that leverages GazeAnywhere as a specialized tool to solve advanced user queries, such as, "How many gaze shifts does this girl with the white dress present?". The work- flow captures User Audio and Environment Images from an AR Glass. MLLM calls Whisper v3 [53] to transcribe the audio, followed by query reasoning and prompt rephrasing. The agent converts the high-level query into a low-level text long dark hair and glasses in a purple long-sleeved shirt woman light brown hair and a blue striped shirt boy short dark hair in a black top and red pants woman short dark hair and a white sleeveless top and black pants woman short dark wavy hair and a light- colored dress woman brown wavy hair and a grey shirt childshort dark hair and a light blue t-shirt boy short brown hair wearing a green and yellow shirt and blue jeans boy long black high ponytail hair and a pink shirt girl short blonde hair wearing a dark blue and light gray shirt boy Figure 4. Visualization of GazeAnywhere’s gaze target estimation results from several datasets. The input is the text prompt (shown in the black box) describing only the subject person’s appearance and the image. GazeAnywhere detect the subject’s head and track the gaze target. More qualitative comparison is shown in the appendix. AR Glass Environment Images User Audio with Query Multimodal LLM The user ask us to track the gaze of the closet girl. There are four people in the scene, the girl is wearing a leopard print top, and she has long blonde hair with a silver headband. She is in the right of the stream image. whisper GazeAnywhere memory Query Reasoning and Tool Calling Post-processing Smart Glasses Screen Agent Analysis Reasoning Figure 5. Workflow of MLLM-powered GazeAnywhere Agent. MLLMGaze Shift MAE/min↓Eye Contact MAE/min↓ Gemini 2.5 Flash (Raw)9.24714.763 Gemini 2.5 Flash (AnyGaze Agent)2.3374.756 Gemini 2.5 Pro (Raw)6.06310.268 Gemini 2.5 Pro (AnyGaze Agent)2.5466.672 Table 6. Comparison of GazeAnywhere Agent and no agent frame- work. prompt (e.g., "girl with white dress") , calls the GazeAny- where tool to generate gaze tracing, and then uses its VLM function to analyze the post-processed video, providing the user with the required analysis. We collected 10 real-world videos with rich gaze move- ment using the DigiLens ARGO AR glass to test the agent’s performance. The evaluation focused on two tasks: gaze shift calculation and eye contact calculation with other social partners. Using the Mean Absolute Error (MAE) per-minute, the GazeAnywhere Agent demonstrated significantly better performance than a raw, single MLLM solution. Results of GazeAnywhere Agent experiment is showed in Table 6. 6. Discussion The practical applications of human gaze target estimation are diverse and impactful. In healthcare, for instance, this technology can significantly enhance the analysis of non- verbal communication behaviors which are implicated in the diagnosis and treatment of developmental conditions such as autism [56]. In order for AI models to be used in clinical applications, they must be sufficiently robust and easy to use by nonexperts. This works takes a significant step in that direction for the task of gaze assessment. Our concept- based approach, which allows subjects to be identified by their attributes in natural language, is a first step towards the flexible and convenient specification of a broad set of behavioral analysis tasks. In addition, by creating a unified end-to-end learnable architecture we increase robustness by eliminating brittle stage-wise approaches to identifying the subjects of gaze analysis. Our approach is beneficial even in comparison to using state-of-the-art OVD models to identify subjects, e.g. the SOTA OVD OWLv2 has only a 70% detection accurate rate in Child-SC for the child head and face detection tasks. 7. Conclusion We present GazeAnywhere, a system that enables interactive human gaze target estimation using flexible, open-vocabulary text prompts to identify the subject. Our principal contribu- tions include introducing the novel Promptable Gaze Target Estimation (PGE) task and Gaze-Co benchmark, proposing a tailored transformer-based detector and learning objective, and developing a human-and-AI-in-the-loop data engine to adapt existing datasets. GazeAnywhere achieves state-of- the-art results in Gaze-Co benchmark, and its robustness is further validated on a challenging out-of-domain (OOD) dataset of child social communication videos. We believe GazeAnywhere and the Gaze-Co benchmark represent im- portant milestones, paving the way for future research and applications in social AI and human behavior understanding. Acknowledgments Portions of this work were supported in part by NIH R01 MH114999, the CIFAR Child and Brain Development pro- gram, and the Health Care Engineering Systems Center at University of Illinois Urbana-Champaign. Gemini API used in the project is supported by Google. This work also used Delta at the National Center for Supercomputing Applica- tions (NCSA) through allocation CIS251391 from the Ad- vanced Cyberinfrastructure Coordination Ecosystem: Ser- vices & Support (ACCESS) program [5], which is supported by U.S. National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. References [1]Henny Admoni and Brian Scassellati. Social eye gaze in human-robot interaction: a review. Journal of Human-Robot Interaction, 6(1):25–63, 2017. 1 [2]Michael Argyle, Mark Cook, and Duncan Cramer. Gaze and mutual gaze. The British Journal of Psychiatry, 165(6): 848–850, 1994. 1 [3]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5- vl technical report. arXiv preprint arXiv:2502.13923, 2025. 7 [4]Jun Bao, Buyu Liu, and Jun Yu. Escnet: Gaze target detec- tion with the understanding of 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14126–14135, 2022. 2 [5]Timothy J Boerner, Stephen Deems, Thomas R Furlani, Shel- ley L Knuth, and John Towns. Access: Advancing innovation: Nsf’s advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and experience in advanced research computing 2023: Computing for the common good, pages 173–176. 2023. 9 [6]Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the net- work. arXiv preprint arXiv:2504.13181, 2025. 2 [7] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017. 1 [8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, pages 213–229. Springer, 2020. 2, 5, 1 [9]Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoub- hik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025. 2, 7 [10]Wenhe Chen, Hui Xu, Chao Zhu, Xiaoli Liu, Yinghua Lu, Caixia Zheng, and Jun Kong. Gaze estimation via the joint modeling of multiple cues. IEEE Transactions on Circuits and Systems for Video Technology, 32(3):1390–1402, 2021. 1 [11]Yihua Cheng and Feng Lu. Gaze estimation using transformer. In 2022 26th International Conference on Pattern Recognition (ICPR), pages 3341–3347. IEEE, 2022. 2 [12]Yihua Cheng, Haofei Wang, Yiwei Bao, and Feng Lu. Appearance-based gaze estimation with deep learning: A review and benchmark. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 46(12):7509–7528, 2024. 1 [13] Eunji Chong, Katha Chanda, Zhefan Ye, Audrey Souther- land, Nataniel Ruiz, Rebecca M Jones, Agata Rozga, and James M Rehg. Detecting gaze towards eyes in natural social interactions and its use in child assessment. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(3):1–20, 2017. 1 [14]Eunji Chong, Nataniel Ruiz, Yongxin Wang, Yun Zhang, Agata Rozga, and James M Rehg. Connecting gaze, scene, and attention: Generalized attention estimation via joint mod- eling of gaze and scene saliency. In Proceedings of the Euro- pean conference on computer vision (ECCV), pages 383–398, 2018. 1 [15]Eunji Chong, Yongxin Wang, Nataniel Ruiz, and James M Rehg. Detecting attended visual targets in video. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5396–5406, 2020. 2, 1 [16]Yung-Sung Chuang, Yang Li, Dong Wang, Ching-Feng Yeh, Kehan Lyu, Ramya Raghavendra, James Glass, Lifei Huang, Jason Weston, Luke Zettlemoyer, et al. Meta clip 2: A world- wide scaling recipe. arXiv preprint arXiv:2507.22062, 2025. 2, 7 [17]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long con- text, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. 5, 7 [18] Ryan Anthony Jalova de Belen, Gelareh Mohammadi, and Arcot Sowmya.Gazedetr: Gaze detection using disen- tangled head and gaze representations.arXiv preprint arXiv:2508.12966, 2025. 2, 1 [19]Bardia Doosti, Ching-Hui Chen, Raviteja Vemulapalli, Xuhui Jia, Yukun Zhu, and Bradley Green. Boosting image-based mutual gaze detection using pseudo 3d gaze. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1273– 1281, 2021. 1 [20]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020. 4 [21]Nathan J Emery. The eyes have it: the neuroethology, function and evolution of social gaze. Neuroscience & biobehavioral reviews, 24(6):581–604, 2000. 2 [22] Terje Falck-Ytter, Elisabeth Fernell, Åsa Lundholm Hedvall, Claes Von Hofsten, and Christopher Gillberg. Gaze per- formance in children with autism spectrum disorder when observing communicative actions. Journal of autism and developmental disorders, 42(10):2236–2245, 2012. 1 [23]Lifeng Fan, Yixin Chen, Ping Wei, Wenguan Wang, and Song- Chun Zhu. Inferring shared attention in social scene videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6460–6468, 2018. 1 [24]Yi Fang, Jiapeng Tang, Wang Shen, Wei Shen, Xiao Gu, Li Song, and Guangtao Zhai. Dual attention guided gaze target detection in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11390–11399, 2021. 1 [25]Shenghao Fu, Qize Yang, Qijie Mo, Junkai Yan, Xihan Wei, Jingke Meng, Xiaohua Xie, and Wei-Shi Zheng. Llmdet: Learning strong open-vocabulary object detectors under the supervision of large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14987–14997, 2025. 2, 6, 3 [26]Google DeepMind and Google. Gemini api and model card.https://ai.google.dev/gemini-api, 2025. Models: Gemini 2.5 Pro. 5 [27]Anshul Gupta, Samy Tafasca, and Jean-Marc Odobez. A modular multimodal architecture for gaze target prediction: Application to privacy-sensitive settings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5041–5050, 2022. 1 [28] John M Henderson. Human gaze control during real-world scene perception. Trends in cognitive sciences, 7(11):498– 504, 2003. 1 [29]Nora Horanyi, Linfang Zheng, Eunji Chong, Aleš Leonardis, and Hyung Jin Chang. Where are they looking in the 3d space?In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2678–2687, 2023. 2 [30]Zhengxi Hu, Kunxu Zhao, Bohan Zhou, Hang Guo, Shichao Wu, Yuxue Yang, and Jingtai Liu. Gaze target estimation in- spired by interactive attention. IEEE Transactions on Circuits and Systems for Video Technology, 32(12):8524–8536, 2022. 2 [31]Qing Jiang, Lin Wu, Zhaoyang Zeng, Tianhe Ren, Yuda Xiong, Yihao Chen, Liu Qin, and Lei Zhang. Referring to any person. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21667–21678, 2025. 2, 6, 3 [32] Tianlei Jin, Zheyuan Lin, Shiqiang Zhu, Wen Wang, and Shunda Hu. Multi-person gaze-following with numerical coordinate regression. In 2021 16th IEEE International Con- ference on Automatic Face and Gesture Recognition (FG 2021), pages 01–08. IEEE, 2021. 2 [33]Tianlei Jin, Qizhi Yu, Shiqiang Zhu, Zheyuan Lin, Jie Ren, Yuanhai Zhou, and Wei Song. Depth-aware gaze-following via auxiliary networks for robotics. Engineering Applications of Artificial Intelligence, 113:104924, 2022. 2 [34]Cijo Jose, Théo Moutakanni, Dahyun Kang, Federico Baldas- sarre, Timothée Darcet, Hu Xu, Daniel Li, Marc Szafraniec, Michaël Ramamonjisoa, Maxime Oquab, et al. Dinov2 meets text: A unified framework for image-and pixel-level vision- language alignment. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24905–24916, 2025. 2, 7 [35]Petr Kellnhofer, Adria Recasens, Simon Stent, Wojciech Matusik, and Antonio Torralba. Gaze360: Physically un- constrained gaze estimation in the wild. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6912–6921, 2019. 1 [36]Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9579–9589, 2024. 2 [37] Yantao Lai, Rong Quan, Dong Liang, and Jie Qin. Clipgaze: Zero-shot goal-directed scanpath prediction using clip. In ICASSP 2025-2025 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025. 2 [38]Susan R Leekam, Emma Hunnisett, and Chris Moore. Tar- gets and cues: Gaze-following in children with autism. The Journal of Child Psychology and Psychiatry and Allied Disci- plines, 39(7):951–962, 1998. 1 [39]Dongze Lian, Zehao Yu, and Shenghua Gao. Believe it or not, we know what you are looking at! In Asian Conference on Computer Vision, pages 35–50. Springer, 2018. 1 [40] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Pro- ceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 5 [41]Zhi-Yi Lin, Jouh Yeong Chew, Jan van Gemert, and Xucong Zhang. Gazehta: End-to-end gaze target detection with head- target association. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 9447–9454. IEEE, 2025. 2 [42]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European con- ference on computer vision, pages 38–55. Springer, 2024. 2, 6, 3 [43] Manuel J Marín-Jiménez, Andrew Zisserman, and Vittorio Ferrari. " here’s looking at you, kid." detecting people looking at each other in videos. 2011. 1 [44]Manuel J Marin-Jimenez, Vicky Kalogeiton, Pablo Medina- Suarez, and Andrew Zisserman. Laeo-net: revisiting peo- ple looking at each other in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3477–3485, 2019. 1 [45]Qiaomu Miao, Minh Hoai, and Dimitris Samaras. Patch-level gaze distribution prediction for gaze following. In Proceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 880–889, 2023. 2 [46]Qiaomu Miao, Alexandros Graikos, Jingwei Zhang, Sounak Mondal, Minh Hoai, and Dimitris Samaras. Diffusion-refined vqa annotations for semi-supervised gaze following. In European Conference on Computer Vision, pages 439–457. Springer, 2024. 2 [47] QiaomuMiao,VivekRajuGolani,JingyiXu, Progga Paromita Dutta, Minh Hoai, and Dimitris Samaras. Multi-view gaze target estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5371–5381, 2025. 2 [48]Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In European conference on computer vision, pages 728–755. Springer, 2022. 2, 5 [49]Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scal- ing open-vocabulary object detection. Advances in Neural Information Processing Systems, 36:72983–73007, 2023. 2, 6, 3 [50]Kai Niu, Yanyi Liu, Yuzhou Long, Yan Huang, Liang Wang, and Yanning Zhang. An overview of text-based person search: Recent advances and future directions. IEEE Transactions on Circuits and Systems for Video Technology, 34(9):7803–7819, 2024. 2 [51]Yuko Okumura, Yasuhiro Kanakogi, Takayuki Kanda, Hiroshi Ishiguro, and Shoji Itakura. The power of human gaze on infant learning. Cognition, 128(2):127–133, 2013. 1 [52]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 2, 7 [53] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recog- nition via large-scale weak supervision. In International con- ference on machine learning, pages 28492–28518. PMLR, 2023. 7 [54] Adria Recasens, Aditya Khosla, Carl Vondrick, and Antonio Torralba. Where are they looking?Advances in neural information processing systems, 28, 2015. 1, 2 [55]Adria Recasens, Carl Vondrick, Aditya Khosla, and Antonio Torralba. Following gaze in video. In Proceedings of the IEEE International Conference on Computer Vision, pages 1435–1443, 2017. 2, 1 [56]James M Rehg, Agata Rozga, Gregory D Abowd, and Matthew S Goodwin. Behavioral imaging and autism. IEEE Pervasive Computing, 13(2):84–87, 2014. 8 [57] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized in- tersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658–666, 2019. 5 [58] Fiona Ryan, Ajay Bati, Sangmin Lee, Daniel Bolya, Judy Hoffman, and James M Rehg. Gaze-lle: Gaze target estima- tion via large-scale learned encoders. 2025. 1, 2, 6, 3 [59]Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Di- nov3. arXiv preprint arXiv:2508.10104, 2025. 2, 4, 7 [60]Yuehao Song, Xinggang Wang, Jingfeng Yao, Wenyu Liu, Jinglin Zhang, and Xiangmin Xu. Vitgaze: gaze following with interaction features in vision transformers. Visual Intelli- gence, 2(1):1–15, 2024. 1, 2, 6, 3 [61] Samy Tafasca, Anshul Gupta, and Jean-Marc Odobez. Child- play: A new benchmark for understanding children’s gaze behaviour. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20935–20946, 2023. 2, 1 [62] Samy Tafasca, Anshul Gupta, Victor Bros, and Jean-Marc Odobez. Toward semantic gaze target detection. Advances in Neural Information Processing Systems, 37:121422–121448, 2024. 1, 2 [63]Samy Tafasca, Anshul Gupta, and Jean-Marc Odobez. Sharingan: A transformer architecture for multi-person gaze following. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2008–2017, 2024. 6, 3 [64] Henri Tomas, Marcus Reyes, Raimarc Dionido, Mark Ty, Jonric Mirando, Joel Casimiro, Rowel Atienza, and Richard Guinto. Goo: A dataset for gaze object prediction in retail environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3125– 3133, 2021. 2 [65] Francesco Tonini, Cigdem Beyan, and Elisa Ricci. Multi- modal across domains gaze target detection. In Proceedings of the 2022 International Conference on Multimodal Interac- tion, pages 420–431, 2022. 2, 1 [66]Francesco Tonini, Nicola Dall’Asen, Cigdem Beyan, and Elisa Ricci. Object-aware gaze target detection. In Proceed- ings of the IEEE/CVF international conference on computer vision, pages 21860–21869, 2023. 2, 1 [67]Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. 2, 7 [68]Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, and Wei Shen. End-to-end human-gaze-target detection with transformers. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2192–2200. IEEE, 2022. 2, 1 [69] Jun Wang, Hao Ruan, Mingjie Wang, Chuanghui Zhang, Huachun Li, and Jun Zhou. Gazeclip: Towards enhanc- ing gaze estimation via text guidance.arXiv preprint arXiv:2401.00260, 2023. 2 [70]Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Rus- sell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309.16671, 2023. 2 [71]Bufang Yang, Lilin Xu, Liekang Zeng, Kaiwei Liu, Siyang Jiang, Wenrui Lu, Hongkai Chen, Xiaofan Jiang, Guoliang Xing, and Zhenyu Yan. Contextagent: Context-aware proac- tive llm agents with open-world sensory perceptions. arXiv preprint arXiv:2505.14668, 2025. 7 [72] Yaokun Yang and Feng Lu. Gaze target detection based on head-local-global coordination. In European Conference on Computer Vision, pages 305–322. Springer, 2024. 2 [73]Yaokun Yang, Yihan Yin, and Feng Lu. Gaze target detection by merging human attention and activity cues. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6585– 6593, 2024. 2 [74]Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre- training for open-world detection. Advances in Neural Infor- mation Processing Systems, 35:9125–9138, 2022. 2 [75] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 2 [76]Mingfang Zhang, Yunfei Liu, and Feng Lu. Gazeonce: Real-time multi-person gaze estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4197–4206, 2022. 2 [77]Ruohan Zhang, Akanksha Saran, Bo Liu, Yifeng Zhu, Sihang Guo, Scott Niekum, Dana Ballard, and Mary Hayhoe. Hu- man gaze assisted artificial intelligence: A review. In IJCAI: Proceedings of the Conference, page 4951, 2020. 2 [78]Shizhou Zhang, De Cheng, Wenlong Luo, Yinghui Xing, Duo Long, Hao Li, Kai Niu, Guoqiang Liang, and Yanning Zhang. Text-based person search in full images via semantic-driven proposal generation. In Proceedings of the 4th International Workshop on Human-centric Multimedia Analysis, pages 5– 14, 2023. 2 [79]Wulue Zhang, Jianbin Xiong, Xiangjun Dong, Qi Wang, and Weikun Dai. Tcnet: Gaze estimation based on temporal body-head-eyes correlation in dynamic scenes. IEEE Sensors Journal, 2025. 1 [80] Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. Appearance-based gaze estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4511–4520, 2015. 2 [81]Zhixiong Zhang, Shuangrui Ding, Xiaoyi Dong, Songxin He, Jianfan Lin, Junsong Tang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Sec: Advancing complex video object segmentation via progressive concept construction. arXiv preprint arXiv:2507.15852, 2025. 2 [82]Hao Zhao, Ming Lu, Anbang Yao, Yurong Chen, and Li Zhang. Learning to draw sight lines. International Journal of Computer Vision, 128(5):1076–1100, 2020. 1 [83] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähen- bühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European conference on computer vision, pages 350–368. Springer, 2022. 2 Gaze Target Estimation Anywhere with Concepts Supplementary Material 8. Further Discussions & Social Impact 8.1. Toward End-to-end Gaze Target Estimation The evolution of human gaze estimation shows a clear trend: a move away from complex auxiliary features like pose and depth towards streamlined, head box-only inputs [10, 14,19,23,24,27,35,39,44,82]. This simplification has spurred the development of end-to-end, OpenPose [7]-like, and DETR [8]-like bottom-up approaches that can detect all head box-gaze pairs within a scene [18,65,66,68]. However, a critical limitation persists. These methods lack identity association; they can find the gaze of everyone but cannot identify the gaze of a specific person. This necessitates separate modules or post-processing to link a detected gaze to a particular individual. Thus, the cascaded detection error will still exist. Our work, GazeAnywhere, directly addresses this gap. We propose the first text-promptable pipeline that simultaneously resolves human identification and gaze target estimation, enabling targeted queries for a specific person’s gaze. 8.2. Future Application Joint attention, the capability of following another person’s head turn and gaze direction, typically emerges in children with Autism Spectrum Disorder (ASD) years later than in typically developing children [38]. Previous research has demonstrated the strong potential of gaze target estimation models to capture these atypical joint attention behaviors, offering a promising avenue for the early screening and detection of ASD [13,43]. The concept-based prompting flexibility of the GazeAnywhere model offers a significant evolution in this domain. In future clinical and home-based settings, this model could be deployed to continuously and non-invasively track a child’s gaze behavior. Critically, GazeAnywhere and GazeAnywhere Agent can be used by pediatricians or even the patient’s parents simply by describ- ing the patient’s appearance or location in the prompt (e.g., "the child in the blue shirt"), thereby omitting the compli- cated and labor-intensive process of manually drawing head bounding boxes for annotation. This simplified usage of- fers the benefit of longitudinal tracking outside of a clinical setting, enabling earlier intervention and more comprehen- sive developmental monitoring. In addition, user can query GazeAnywhere Agent to let MLLM post-process the target tracking video and provide high-level gaze behavior infor- mation like gaze shift. 9. GazeAnywhere Agent In this section, we introduce the GazeAnywhere Agent, a vi- sual agentic framework designed to process natural-language gaze estimation and post-analysis requests. Figure 6 illus- trate the workflow of the agent. The system dynamically queries a MLLM to orchestrate specific tools. The initial version of the agent integrates two primary models as the tool: Whisper-large-v3 for audio-to-text conversion and our proposed GazeAnywhere model for PGE target prediction. Given an input image or video and a user request via audio , the MLLM acts as a planner and controller. It first converts the user’s audio to text, analyzes the scene context, devises a step-by-step plan, and subsequently invokes the GazeAnywhere model. After each action, the agent receives visual feedback by visualizing the gaze target within the scene. This feedback is stored in memory, enabling the agent to revise its plan and determine the next steps for analysis. This pipeline handles queries far more complex than simple noun phrases, facilitating a deeper understanding of human gaze behavior in video streams. 10. Dataset & Benchmark 10.1. Training Set The Gaze-Co training set contains 119,525 samples in to- tal. Each record includes the target head bounding box, normalized gaze point, an in/out-of-frame label, and a com- pact concept phrase (attribute, position, action, and pose). The training data are constructed from three published gaze datasets after applying the image-quality filters, MLLM- based concept generation, and human in-loop MLLM verifi- cation described in the main text. In terms of source datasets, 69.6% (83,148 samples) come from GazeFollow [54,55], 19.6% (23,481 samples) from VideoAttentionTarget [15], and 10.8% (12,896 samples) from ChildPlay [61] (see Fig. 7a). For the apparent subject category, 51.0% (60,983 sam- ples) are labeled as man, 32.2% (38,508) as woman, 8.2% (9,773) as boy, 6.1% (7,337) as girl, 1.6% (1,916) as child (unspecified gender), and 0.8% (1,008) as infant (unspeci- fied gender) (Fig. 7b). These labels reflect perceived visual categories rather than verified identity attributes. Regarding gaze location, 13.9% (16,671 samples) of annotations are out-of-frame, while 86.1% (102,854) fall within the image (Fig. 7c). AR Glass Environment Images & Video User Audio with Query ... Help me analyze the gaze shift of the girl closed to me... AnyGaze Agent Context-aware Planning 1. Call whisper-large-v3 to extract the audio-caption. 2. The user ask us to analyze the gaze shift of the closet girl. There are four people in the scene, the girl is wearing a leopard print top, and she has long blonde hair with a silver headband. She is in the right of the stream images. Call AnyGaze to visualize the target with input prompt: appearance: blonde hair with a silver headband girl, position: right of the image. 3. Analysis with the visualized video stream. Smart Glasses Screen Gaze shift: 2 whisper Query AnyGaze Agent AnyGaze Agent AnyGaze Agent AnyGaze Agent GazeAnywhere Analysis Figure 6. Step-by-step explanation of how GazeAnywhere Agent works. (a) Source dataset distribution(b) Apparent subject category distribution(c) In-frame vs out-of-frame gaze distribution Figure 7. Training-set statistics of the Gaze-Co dataset: (a) proportion of each source dataset, (b) distribution of apparent subject categories, and (c) proportion of in-frame vs out-of-frame gaze annotations. 10.2. Concept-based In-domain Test Set We derive three concept-augmented test splits by converting the official test splits of GazeFollow, VAT, and ChildPlay into our unified PGE schema (image, head box, normal- ized gaze point, in/out-of-frame label, and concept phrase). After applying the same image-quality filters as in the train- ing set, we obtain GazeFollow-Concept, VAT-Concept, and ChildPlay-Concept. To guarantee a high quality benchmark for both baselines and our model evaluation, all concept annotations are human verified instead of using MLLM. GazeFollow-Concept. After filtering, GazeFollow-Concept contains 2,436 (image, head box) records. In terms of ap- parent subject category, 49.1% (1,197 samples) are labeled as man, 31.7% (772) as woman, 10.3% (250) as boy, 5.6% (136) as girl, 2.1% (50) as child (unspecified gender), and 1.3% (31) as infant (unspecified gender). All annotations in this split correspond to in-frame gaze targets (100%, 2,436 samples). In the dataset, each (image, head box) record is associated with multiple human gaze point annotations from the original GazeFollow dataset, which motivates the addi- tional Avg L2 and Min L2 metrics used in the main text: Avg L2 is defined as the distance between the predicted gaze point and the mean of all human annotations, and Min L2 as the distance to the nearest human-annotated gaze point. VAT-Concept. VAT-Concept contains 5,301 records. For ap- parent subject categories, 45.9% (2,435 samples) are labeled as man, 43.8% (2,324) as woman, 5.8% (310) as boy, 0.4% (20) as girl, 0.1% (5) as child (unspecified gender), and 3.9% (207) as infant (unspecified gender). Regarding gaze loca- tion, 35.5% (1,884 samples) of annotations are out-of-frame, while 64.5% (3,417) are in-frame. ChildPlay-Concept. ChildPlay-Concept contains 1,238 records. In terms of apparent subject category, 8.5% (105 samples) are labeled as man, 32.9% (407) as woman, 36.8% (455) as boy, 12.0% (148) as girl, 4.4% (55) as child (unspec- ified gender), and 5.5% (68) as infant (unspecified gender). For gaze location, 14.7% (182 samples) of annotations are out-of-frame, while 85.3% (1,056) are in-frame. Across all splits, the apparent subject categories reflect per- ceived visual attributes rather than verified identity labels. 10.3. Concept-based Out-of-domain Test set For out-of-domain evaluation, we utilize Child–Social Com- munication (Child-SC), a private dataset protected by IRB. It captures natural interactions between children and clinicians, where the clinician guides the child’s attention across vari- ous targets using toys, thus eliciting frequent and structured gaze shifts. The dataset comprises 326 video clips from 40 children, sampled at 5 fps, yielding a total of 151,533 images. Due to privacy regulations, these images cannot be processed by cloud-based MLLM; consequently, all target- person concepts were manually annotated, strictly adhering to the style and protocols of our MLLM-generated concepts. 11. Baseline Details 11.1. Open-Vocabulary Detector (OVD) As baselines, we use the OVD models to locate the target person described by a text prompt. This step supports our main task: to predict the point of view of the subject. Each OVD model takes an image and a prompt, matches text to visual regions in a shared vision–language space, and scores candidate boxes by text–image similarity. It outputs the highest-confidence bounding box for the prompted person, which we use as the subject-person localization. We also compared with the SOTA open-vocabulary human detection model RexSeek [31], which is a 3B foundation model in referring expression comprehension task. GroundingDINO-B. GroundingDINO-B[42]isa Transformer-based detector featuring a dual-encoder single-decoder architecture that deeply fuses image and text features. It employs a language-guided query selection module to initialize object queries based on the input prompt. This mechanism produces a series of refined candidate boxes associated with prediction scores. From these outputs, we identify the target person by selecting the box with the highest confidence score for the referring phrase. LLMDet-L. LLMDet-L [25] enhances open-vocabulary detection through multimodal co-training, where a large lan- guage model generates detailed captions to enrich feature alignment during training. At test time, with the LLM re- moved, the detector takes the image and prompt to generate multiple region candidates. It evaluates these regions by matching them against the text embedding, enabling us to filter the results and retrieve the top-ranked bounding box as the localized subject. OWLv2-L.OWLv2-L [49] scales up the OWL-ViT archi- tecture using a massive self-training strategy on over one billion weakly supervised examples. It utilizes a Vision Transformer backbone to directly predict bounding boxes and text-alignment scores from image tokens. When queried with the target person’s description, the model outputs a collection of detected objects with their semantic similarity scores, from which we select the best-matching candidate to localize the person. 11.2. Gaze Model Following the localization step, we evaluate several gaze- following models to predict the target person’s point of re- gard. These models accept the full scene image and the localized person region as input. They output a 2D gaze heatmap (probability distribution), and we extract the coor- dinates of the peak value from the heatmap to represent the final predicted gaze location. ViTGaze ViTGaze [60] is a single-modality gaze- following model that predicts a person’s gaze target using RGB information only. Given the full image and the tar- get person’s head bounding box, it employs a pre-trained ViT to extract human–scene interaction cues directly from self-attention maps, eliminating the need for extra modali- ties. The model outputs a 2D gaze heatmap along with an in/out-of-frame score for evaluation. SharinganSharingan [63] introduces a transformer-based architecture designed to capture global gaze interactions. It represents the target person via a Person Gaze Token, constructed by fusing head-crop features with normalized head-box coordinates. This token is processed with scene tokens by a ViT encoder to model human–scene dependen- cies. The model outputs a 2D gaze heatmap representing the spatial probability of the gaze target and an in/out-of-frame score. Gaze-LLEGaze-LLE [58] is a streamlined estimator built on a frozen, large-scale DINOv2 encoder, departing from traditional multi-branch head/scene architectures. Given the full image and the target person’s head bounding box, it encodes the head location as a positional prompt injected into the scene features, using a lightweight transformer decoder to model head–scene relations. The model predicts a 2D gaze heatmap along with an in-/out-of-frame score. 12. Experimental Protocol 12.1. AR Device for GazeAnywhere Agent We use DigiLens ARGO in the experiment to capture video data in real-world settings (Fig. 8). Its 48 MP camera records high-resolution video with autofocus, optical and electronic Figure 8. DigiLens ARGO AR glasses used for video and audio capture and on-device feedback in the GazeAnywhere Agent. stabilization, 4×4 pixel binning, and strong low-light support. For audio, a five-microphone beamforming array is designed to pick up the wearer’s voice in noisy environments and provides spatial recordings suitable for analysis. 12.2. Implementation Details of GazeAnywhere- DINOv3-L The deployed version of GazeAnywhere-DINOv3-L con- sists of a detector transformer with 3 layers and a dimension ofD = 256. Both the visual and text prompts are trained jointly. For visual prompting, we apply diverse augmentation techniques during training, including head/body bounding box jittering, color jittering, random resizing and cropping, horizontal flipping, rotation, and masking of scene patches. For text prompting, as the subject position text information is fixed, we limit visual augmentation to random scene patch masking and apply text augmentation with reordering appear- ance, location, pose, and action attributes. During training, the input resolution is 512× 512. 13. More Results 13.1. Impact of Frozen Encoder. A key design choice for GazeAnywhere is to keep the im- age and text encoders frozen. We validate this approach in Table 7, which compares the default frozen model against one where the DINOv3 image encoder or the text encoder are fine-tuned. Unfreezing image or text encoders leads to a clear drop in performance. This demonstrates that DINOv3’s pre-trained features are highly robust and generalizable for the PGE task, and that fine-tuning may lead to overfitting or harmful feature drift. 13.2. Impact of Detector Dimension. We study the impact of the Detector transformer’s layer di- mensionDin Table 8. The results indicate that performance plateaus atD = 128. We observed no significant perfor- mance gain from increasingDfurther, and thus selected VisualTextTrainable Param GazeFollow-ConceptVAT-Concept AUC↑Avg L2↓Min L2↓AUC↑L2↓AP↑ ✗870.2M0.9310.1500.0950.8860.2120.823 ✓✗332.1M0.9430.1190.0670.8860.2060.813 ✗✓541.7M0.9520.1300.0780.8740.2010.825 ✓3.6M0.9580.0990.0500.9280.1230.879 Table 7. Comparison of the encoder frozen strategies. D of ψ(·) GazeFollow-ConceptVAT-Concept AUC↑Avg L2↓Min L2↓AUC↑L2↓AP↑ 640.9530.1150.0620.9150.1440.871 1280.9600.1000.0500.9280.1160.875 2560.9580.0990.0500.9280.1230.879 5120.9590.1040.0540.9170.1220.875 Table 8. Ablation experiment on the selection of the Detector Transformer dimension. Layer Num of ψ(·) GazeFollow-ConceptVAT-Concept AUC↑Avg L2↓Min L2↓AUC↑L2↓AP↑ 10.9490.1260.0760.8820.1790.846 20.9570.1040.0540.9200.1230.871 30.9580.0990.0500.9280.1230.879 40.9570.1000.0510.9280.1200.858 50.9590.0950.0470.9290.1210.888 Table 9. Ablation experiment on the selection of the transformer layer number in the Detector. D = 256as it provides the best trade-off between accuracy and computational cost. 13.3. Ablation on Detector’s Transformer Layer Number We conduct another ablation study to explore the layer num- ber of transformer blocks in detector transformers. Results are shown in Table 9. After increasing the layer number to 3, the model shows stable performance. 14. Qualitative Analysis In Figure 9, we qualitatively compare GazeAnywhere with the current state-of-the-art model, Gaze-LLE. Although Gaze-LLE performs well in sparse scenes with only one or two individuals, its performance degrades noticeably as crowd density increases. As shown in Figure 9, the upstream OVD module becomes unreliable in these complex settings and typically fails in two ways. First, it may localize the wrong person, causing Gaze-LLE to estimate gaze for an incorrect target. Second, it may produce an overly large bounding box that covers multiple people; even if the true target is included, Gaze-LLE cannot reliably disambiguate whom to condition on. These examples expose a key limi- tation of two-stage gaze estimation pipelines in real-world social scenes. Ground Truth Any Gaze Gaze - LLE Detic GroundingDINO - B LLMDet - L OWLv2 - L Prompt blond hair and a grey t-shirt man balding man with a beard in a light blue shirt man short black hair and a plaid shirt an curly brown hair and a sleeveless white dress woman Figure 9. Qualitative comparison of gaze-target localization conditioned on appearance prompts. Each column corresponds to a different sample, and each row shows predictions from a different method. Our method produces sharper and more accurate heatmaps around the true gaze targets. 15. Related Prompts For reproducibility, we include the exact natural-language prompts used to query the MLLM in our pipeline. These prompts support three major components: the concept- generation data engine, the MLLM-only gaze prediction baseline, and the GazeAnywhere Agent for video-based so- cial gaze analysis. Unless otherwise noted, the prompts are shown verbatim as used in our batch API calls. 15.1. Data Engine This section summarizes the prompts used by the data en- gine to construct concept-level annotations for each subject person. The attribute prompt (Fig. 10) instructs the MLLM to produce a compact description of appearance, position, action, pose, and people count for the person marked by the green head box. The concept verification prompt (Fig. 11) then asks the MLLM to check, field by field, whether a candidate concept matches the image and to return JSON flags for attribute, position, action, pose, and an overall pass/fail decision. Together with spot-checks from human annotators, these prompts implement the MLLM component of our human-in- the-loop data engine. 15.2. MLLM Baseline Here we provide the prompt used for the MLLM-only gaze target prediction baselines, Gemini-2.5-flash (Fig. 12) and Qwen3-VL-8b (Fig. 13). Given an image and a textual concept description, the model is asked to predict an gaze in/out-frame flag and a normalized 2D gaze target point, and to return the answer in a strict JSON format. 15.3. GazeAnywhere Agent This section lists the prompts used to compare gaze-target analysis with an MLLM alone versus an MLLM assisted by the GazeAnywhere Agent on smart-glasses recordings. The raw-video prompt (Fig. 14) presents the model with the original AR recording and asks it to infer social gaze behavior directly from the unannotated video. The GazeAnywhere-agent prompt (Fig. 15) uses the same video but with GazeAnywhere overlays (subject head box, gaze point, and out-of-frame indications), and instructs the model to count gaze shifts to social partners and overall gaze shifts. 16. Notations We present the description all the notations in our paper in the last two pages. Concept Generation Prompt TASK Return a description for the person with a green bounding box in head: The description is a natural, concise attribute phrase (<30 words in total). STYLE & CONTENT - all lowercase - avoid generic words: person, people, adult; avoid starting with a/an/the - prefer stable visible attributes, in this order: hair style and color / hat > glasses / beard > top garment color and pattern > pant / dress garment color and pattern. - the LAST word of attributes MUST be one of: man, woman, boy, girl, infant, child - prefer short and clear location description, such as bottom left corner. - action: describe ongoing interaction or movement; make it specific by adding target/object/direction when visible. keep “what is being done” here. if unclear, write "none". - pose: describe static body configuration and facing direction; keep “how the body is” here (orientation, posture, limb arrangement). if unclear, write "none". - keep action and pose distinct and non-overlapping. - also provide an approximate count of people visible in the scene; report a single integer when feasible; if indeterminate, write "none". You output format is: <attribute> attributes of the human </attribute> <position> position of the human in the camera </position> <action> action of the human </action> <pose> pose of the human </pose> <count> estimated number of people in the scene </count> Figure 10. Concept generation prompt used for generating concept phrases for the target person. Concept Verification Prompt TASK You see an image with one green bounding box on a human head and a candidate description from Prompt A, with: <attribute>, <position>, <action>, <pose>, <count>. Check whether the first four fields match the target human and the scene. Ignore <count>. CHECKING RULES attribute must describe the same person in the green box. hair / hat / glasses / beard / clothes must match. last word must be one of: man, woman, boy, girl, infant, child. label as correct only if all above are satisfied. position must match the boxed human location (e.g., top left, bottom center, center right). label as correct only if consistent with boxed human position. action must be a visible ongoing movement or interaction of this person. if not clearly visible, the correct value should be "none". label as correct only if supported by the image. pose must describe static body configuration and facing direction (orientation, posture, limb arrangement). must be distinct from action; if unclear, should be "none". label as correct only if supported by the image and distinct from action. OVERALL overall is "pass" only if all four checks are "correct". otherwise overall is "fail". OUTPUT FORMAT Output only a single JSON object with exactly these keys and values: "attribute_check": "correct" or "incorrect" "position_check": "correct" or "incorrect" "action_check": "correct" or "incorrect" "pose_check": "correct" or "incorrect" "overall": "pass" or "fail" Figure 11. Concept verification prompt used to check attribute, position, action, and pose consistency for each subject. Gaze Target Prediction Prompt of Gemini 2.5 Flash You are given an image, where the top-left corner is (0, 0) and the bottom-right corner is (1, 1). Coordinates are normalized by the image width and height. You are also given a description of the subject person in the image: Attribute: attribute Location: position Action: action Pose: pose Based on this description and the image, perform the following tasks: 1. In-frame gaze flag Indicate whether the subject person is looking at a target inside the image frame. Output 1 if the gaze target lies within the image frame. Output 0 if the subject person is looking outside the image frame. 2. Gaze target point Predict the gaze target of the subject person as a point (x, y)∈ [0, 1], with exactly three decimal places for both x and y. The values must be normalized by the image width and height. Output format Return only a valid JSON object, with no extra text, in the following format: "in_frame_gaze": 0, "gaze_target": "x": 0.X, "y": 0.X in_frame_gaze must be either 0 or 1. x and y must be numbers in [0,1] with three decimal places. Figure 12. Gaze target prediction prompt used for in-frame flagging and point estimation on Gemini-2.5 baseline Gaze Target Prediction Prompt of Qwen3-VL. You are given an image, where the top-left corner is (0.000 , 0.000), the bottom-right corner is (1.000, 1.000). The pixel point in the image is normalized to 0.000 to 1.000. All values are rounded by 3. Here is the description of the subject person Question Based on the description of a subject person in the image, perform the following task: (1) Indicate whether the subject person is looking at a target inside the image frame. Output 1 if the gaze target lies within the image frame. Output 0 if the subject person is looking outside the image frame. Provide the inside prediction between the <inside> and </inside> tags. (2) Predict the gaze target of the subject person as a point (x, y) x and y are in [0.000, 1.000]. If looking outside, randomly give values. Provide the x of point between the <x> and </x> tags, y of point between the <y> and </y> tags. Figure 13. Gaze target prediction prompt used for in-frame flagging and point estimation on Qwen3-VL-8B. Gaze Shift Analysis Prompt on Single MLLM Solution This is a short video for human gaze target understanding. Can you give me an analysis of these tasks"? The subject child is: short black hair white dress girl 1. "Gaze shift to social partner": The number of gaze shifts to the nearby person happened. 2. "Total gaze shift count": The number of gaze shifts happened. (change the gaze target to another object or out-of-frame) Hint: Gaze shifting is the coordinated movement of the eyes and head to look at a new target. Per gaze shift means changing the gaze target from one object/human to another object/human You should analyze the video and provide the answers to the above tasks. Figure 14. Gaze shift analysis prompt used for counting gaze shifts and eye contact events on Single MLLM. Gaze Shift Analysis Prompt on GazeAnywhere Agent This is a short video for human gaze target understanding. The green bounding box is the detected subject child. If the bounding box’s color becomes blue, it indicates the subject is looking out of the frame. The green point is the child’s gaze target, and we overlap it with the raw video. Can you give me an analysis of these tasks"? The subject child is: short black hair white dress girl 1. "Gaze shift to social partner": The number of gaze shifts to the nearby person happened. 2. "Total gaze shift count": The number of gaze shifts happened. (change the gaze target to another object or out-of-frame) Hint: Gaze shifting is the coordinated movement of the eyes and head to look at a new target. Our green point in the frame can indicate the target location. So you should infer if the target is changed. Per gaze shift means changing the gaze target from one object/human to another object/human You should analyze the video and provide the answers to the above tasks. Figure 15. Gaze shift analysis prompt used for counting gaze shifts and eye contact events on GazeAnywhere agent. Data and Indices HHeight of input image WWidth of input image I ∈ R 3×H×W Input RGB image PPrompt TText H out Height of output image W out Width of output image ˆ H ∈ R H out ×W out Gaze heatmap Embeddings and Image Encodings φ V (·)Image encoder N V Number of patch tokens D V Visual embedding dimension [CLS]Classification token c∈ R D v [CLS] token embedding s i ∈ R D V Visual output embedding token Embeddings and Text Encodings φ T (·)Text encoder [EOS]End of sentence token T E Initial text embeddings L T Fixed context length D T Text embedding dimension t eos End of sentence token t pad Padding token t i ∈ R D T Text embedding token Projection Layers W V ∈ R D V ×D Trainable visual linear projection layer W T ∈ R D T ×D Trainable test linear projection layer DProjected dimension Z V ∈ R (1+N V )×D Projected visual tokens Z T ∈ R L T ×D Projected text tokens Detector Transformer ψ(·)Detector transformer t h ∈ R D Head token t p ∈ R D Target presence token c ′ Projected global visual tokens t ′ eos Projected global text tokens E head Learnable head embeddings E presence Learnable presence embeddings s ′ ∈ R N V ×D Projected visual patch tokens from Z V (excluding c ′ ) t ′ ∈ R (L T −2)×D Projected text patch tokens from Z T (excluding t ′ eos and padding) F ∈ R (N T +N V +2)×D Full input sequence F ψ(F)∈ R (N T +N V +2)×D Output refined sequence of detector transformer Decoder ˆs∈ N V × DRefined visual patch tokens ˆ t h ∈ R D Refined head tokens xnormalized center x coordinate of head tracker ynormalized center y coordinate of head tracker wnormalized width of head tracker hnormalized height of head tracker ˆ t p ∈ R D Refined predict tokens Learning Objective L total Total loss L gaze Gaze heatmap BCE loss σStandard deviation of 2D Gaussian ˆ YPredicted heatmap NTotal number of pixels pSingle pixel on the heatmap y p Ground-truth of p ˆy p Predicted values of p L presence Target presence focal loss Y presence ∈ 0, 1Ground truth target presence ˆ Y presence ∈ [0, 1]Predicted target presence L focal Focal loss L head Head box loss L 1 Mean absolute error L IoU GIoU loss bGround truth head box ˆ bPredicted truth head box λ l 1 head object detection hyperparameter λ IoU head object detection hyperparameter Data Engine x min x coordinate of top-left corner of the head box y min y coordinate of top-left corner of the head box x max x coordinate of bottom-right corner of the head box y max y coordinate of bottom-right corner of the head box g x x coordinate of ground truth gaze point g y y coordinate of ground truth gaze point