Paper deep dive

GRAZE: Grounded Refinement and Motion-Aware Zero-Shot Event Localization

Syed Ahsan Masud Zaidi, Lior Shamir, William Hsu, Scott Dietrich, Talha Zaidi

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 43

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 4/2/2026, 11:50:23 PM

Summary

GRAZE is a training-free, zero-shot pipeline for localizing the First Point of Contact (FPOC) in untrimmed American football practice videos. It utilizes Grounding DINO for candidate discovery, a motion-aware scoring strategy for ranking, and SAM2 for pixel-level contact verification, effectively decoupling detection confidence from physical contact evidence.

Entities (4)

GRAZE · systempipeline · 100%Grounding DINO · model · 100%SAM2 · model · 100%First Point of Contact (FPOC) · event · 95%

Relation Signals (3)

GRAZE → uses → Grounding DINO

confidence 100% · GRAZE uses Grounding DINO to discover candidate player-dummy interactions

GRAZE → uses → SAM2

confidence 100% · uses SAM2 as an explicit pixel-level verifier of contact

GRAZE → localizes → First Point of Contact (FPOC)

confidence 95% · GRAZE, a training-free pipeline for FPOC localization

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:American football practice generates video at scale, yet the interaction of interest occupies only a brief window of each long, untrimmed clip. Reliable biomechanical analysis, therefore, depends on spatiotemporal localization that identifies both the interacting entities and the onset of contact. We study First Point of Contact (FPOC), defined as the first frame in which a player physically touches a tackle dummy, in unconstrained practice footage with camera motion, clutter, multiple similarly equipped athletes, and rapid pose changes around impact. We present GRAZE, a training-free pipeline for FPOC localization that requires no labeled tackle-contact examples. GRAZE uses Grounding DINO to discover candidate player-dummy interactions, refines them with motion-aware temporal reasoning, and uses SAM2 as an explicit pixel-level verifier of contact rather than relying on detection confidence alone. This separation between candidate discovery and contact confirmation makes the approach robust to cluttered scenes and unstable grounding near impact. On 738 tackle-practice videos, GRAZE produces valid outputs for 97.4% of clips and localizes FPOC within $\pm$ 10 frames on 77.5% of all clips and within $\pm$ 20 frames on 82.7% of all clips. These results show that frame-accurate contact onset localization in real-world practice footage is feasible without task-specific training.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

43,004 characters extracted from source content.

Expand or collapse full text

GRAZE: Grounded Refinement and Motion-Aware Zero-Shot Event Localization Syed Ahsan Masud Zaidi1 Lior Shamir1 William Hsu1 Scott Dietrich2 Talha Zaidi1 1Kansas State University, Manhattan, KS, USA 2Albright College, Reading, PA, USA ahsanzaidi@ksu.edu, lshamir@ksu.edu, bhsu@ksu.edu sdietrich@albright.edu, tzaidi@ksu.edu Abstract American football practice generates video at scale, yet the interaction of interest occupies only a brief window of each long, untrimmed clip. Reliable biomechanical analysis, therefore, depends on spatiotemporal localization that identifies both the interacting entities and the onset of contact. We study First Point of Contact (FPOC), defined as the first frame in which a player physically touches a tackle dummy, in unconstrained practice footage with camera motion, clutter, multiple similarly equipped athletes, and rapid pose changes around impact. We present GRAZE, a training-free pipeline for FPOC localization that requires no labeled tackle-contact examples. GRAZE uses Grounding DINO to discover candidate player-dummy interactions, refines them with motion-aware temporal reasoning, and uses SAM2 as an explicit pixel-level verifier of contact rather than relying on detection confidence alone. This separation between candidate discovery and contact confirmation makes the approach robust to cluttered scenes and unstable grounding near impact. On 738738 tackle-practice videos, GRAZE produces valid outputs for 97.4%97.4\% of clips and localizes FPOC within ±10± 10 frames on 77.5%77.5\% of all clips and within ±20± 20 frames on 82.7%82.7\% of all clips. These results show that frame-accurate contact onset localization in real-world practice footage is feasible without task-specific training. 1 Introduction Practice video recordings in contact sports like American football provide useful information on player technique and can help improve the performance of early-career athletes. However, the unstructured nature of raw footage remains a barrier to automated analysis. Raw practice clips are long, multi-scene, and contain both spatial and temporal noise, mixing warm-up drills, recovery periods, and coaching instructions around only a few seconds of biomechanically relevant action. Coaches, athletic trainers, and biomechanists require that action window indexed accurately and reproducibly. While spatio-temporal datasets such as AVA [7] have formalized atomic action localization in broadcast footage, they provide no mechanism for contact onset detection in unconstrained monocular recordings. In this paper, we study First Point of Contact (FPOC) localization in tackle-practice video, defined as the first frame at which a player makes physical contact with a padded tackle dummy. FPOC is the natural temporal anchor for contact-centered analysis: strike-zone assessment, postural alignment at impact, and kinematic profiling all require knowing exactly when contact began. An incorrect FPOC shifts pose measurements to pre- or post-contact postures, making rubric-based scoring unreliable. Solving this requires both identifying the correct player-dummy pair in a cluttered scene and determining the precise frame of contact. Achieving this from raw practice footage is hard for reasons standard sports-video benchmarks do not capture. Cameras are handheld or field-mounted, panning and jittering throughout. Backgrounds shift abruptly between repetitions. Multiple athletes in similar gear share the frame, and some stand between the camera and the active player, creating distractors that score highly under any appearance-based detector. Standard bounding boxes fail in both directions: they can overlap the dummy without physical contact, and at the true moment of contact, occlusion can eliminate that overlap entirely. Representative failure conditions are shown in Figure 1. Figure 1: Representative recording challenges in tackle-practice videos. (a) Variable scene composition with multiple players, no visible tackle dummy, and a non-sprinting actor under unstable camera motion. (b) Multiple players with mutual occlusion near the contact window. (c) Player and dummy partially or fully absent due to abrupt panning. (d) Inverted camera orientation with motion blur and multi-player overlap onto the dummy. The fundamental difficulty is the gap between pair discovery and contact confirmation. Detection confidence reflects appearance similarity to a text prompt, not physical intersection. What FPOC localization requires is pixel-level evidence that the two objects overlap at the earliest frame where that overlap occurs, and a strategy that stays robust when grounding is unstable across consecutive frames. We address FPOC localization with a training-free pipeline requiring no task-specific fine-tuning and no labeled examples from this domain. The central insight is to treat SAM2 [21] not as a passive segmentation backend but as an explicit pixel-level contact verifier: given candidate player-dummy pairs, we propagate masks with SAM2 and use their intersection as direct contact evidence, decoupled from grounding confidence. This enables a ranked multi-candidate fallback in which all grounding candidates across prompt sets, temporal positions, and detection thresholds are evaluated until SAM2 mask overlap independently confirms contact. Candidates are generated using Grounding DINO [12] with hierarchical prompt scheduling and progressive threshold relaxation, then ranked by a geometric motion signal combining displacement magnitude with cosine similarity between the player’s motion vector and the approach vector toward the dummy. Because grounding succeeds most reliably at mid-contact, when both objects are simultaneously salient, the grounding frame is systematically biased later than the true onset. A two-phase backward refinement using sequential backtracking and binary search corrects this bias and recovers the earliest consistent frame. This paper makes three contributions. First, we introduce a training-free spatiotemporal localization pipeline for FPOC detection in unconstrained tackle-practice footage, to our knowledge the first system targeting frame-accurate contact onset localization in this domain at scale. Second, we propose using SAM2 mask propagation as an independent pixel-level contact verifier, decoupling detection confidence from contact evidence and enabling a multi-candidate fallback that persists until mask overlap confirms contact. Third, we introduce a geometric directional motion scoring strategy based on cosine similarity between the player’s motion vector and the dummy approach vector, combined with displacement magnitude, to stabilize candidate ranking without any learned classifier or tracker. 1.1 Related Work Sports video analysis has advanced rapidly, but largely under conditions that practice footage does not satisfy. Broadcast cameras track the action deliberately with controlled lighting, and benchmarks like SoccerNet [6, 3] and action spotting formulations [2] have pushed temporal precision toward single-frame accuracy in that setting. Strong backbones such as I3D [1] and SlowFast [4] were designed and evaluated on curated datasets that assume a clean, actor-centered field of view, and survey work has traced how heavily broadcast-era gains depend on those assumptions [23]. American football practice looks almost nothing like this. Formation recognition and play-tagging methods exist for the sport [17, 18, 16], and more recent work has begun treating tackle practice as a domain in its own right [24], but these contributions target clip-level labels and strategic structure, leaving frame-accurate contact onset unaddressed. Tackle safety has attracted a parallel line of work. Nafi et al. [14] classify clips by risk level using 3D convolutional networks, demonstrating that a useful injury screening signal exists in practice footage, but clip-level labels do not identify when contact begins. A follow-on effort [15] uses Cascade R-CNN to segment individual players against cluttered backgrounds. Image-level segmentation is temporally inert, however: it provides no mechanism for maintaining identity across frames or for deciding when two actors first intersect. Spatial and temporal problems are handled separately throughout this body of work; our setting requires both to be solved together. Because visual appearance varies substantially across sessions and equipment configurations, training a task-specific detector is impractical. Contrastive image, language pretraining [20] demonstrated that aligning vision and language at scale yields representations that transfer to unseen categories, motivating a generation of open-vocabulary detectors. GLIP [9] unified grounding and detection through language-image pretraining; OWL-ViT [13] showed that a contrastively trained Vision Transformer achieves competitive open-set detection without task-specific heads; and Grounding DINO [12] fused a DINO-style detector with cross-modal attention for strong open-vocabulary performance. Phrase-level grounding is particularly suited here because detailed descriptions of player posture and equipment help separate the active tackler from bystanders. The limitation shared by all these methods is that per-frame confidence measures co-occurrence, not contact. Video segmentation can close this gap if used correctly. Self-supervised pretraining [22] has shown that temporal masked autoencoding yields representations that generalize broadly, and SAM [8] established promptable segmentation at image scale. SAM2 [21] extends this to video via a streaming memory architecture. We depart from the standard use pattern, where SAM2 is a backend that returns masks for downstream use, by treating mask intersection itself as a contact detection signal. When a propagated player mask begins to overlap a separately propagated dummy mask, that is direct geometric evidence of contact, wholly independent of detection confidence. Supervised temporal action localization methods such as BMN [11] and ActionFormer [25] achieve strong results on curated benchmarks but require frame-level annotations unavailable for contact onset in practice footage. The closest annotation-free relatives are zero-shot temporal methods T3AL [10] and ZEETAD [19], and natural-language moment retrieval [5]. All produce temporal segments associated with action categorie, a useful output for recognition but the wrong granularity for contact onset. At 30 fps, a half-second segment window spans 15 frames, and whether a measurement lands on pre-contact posture or mid-contact deformation depends on which of those frames is selected. More fundamentally, these methods respond to what an action looks like; FPOC is determined by whether two specific objects physically intersect, a question no appearance-based segment predictor is designed to answer. 2 Methodology 2.1 Problem Formulation Let =Itt=1TV=\I_t\_t=1^T be an untrimmed video of T frames. The scene contains at least one player P approaching a stationary tackle dummy D. Given V, we seek two temporal quantities: the First Frame with Both Objects (tFFBOt_FFBO), the earliest frame in which both P and D are simultaneously visible, and the First Point of Contact (tFPOCt_FPOC), the exact frame at which the player first physically intersects the dummy. We also produce per frame pixel masks ℳt(P)M^(P)_t and ℳt(D)M^(D)_t over the event window [tFFBO,tend][t_FFBO,t_end], where tendt_end is defined in Eq. 10. The pipeline uses no task-specific fine-tuning and no labeled examples from this domain. 2.2 Pipeline Overview Figure 2: Four-phase pipeline. (1) Grounding: Grounding DINO searches multiple temporal positions with hierarchical prompts and progressive threshold relaxation, retaining all valid player-dummy candidates. (2) Validation: Each candidate is scored by temporal consistency, displacement magnitude, and directional approach toward the dummy; candidates are ranked before SAM2 evaluation. (3) Refinement and Segmentation: Backward refinement recovers tFFBOt_FFBO; SAM2 propagates player and dummy masks from that frame. (4) Contact Verification: FPOC is the first frame where the propagated masks overlap. If no overlap is found, the next ranked candidate is evaluated. Figure 2 summarizes the four-phase approach. Grounding proposes candidate player-dummy bounding-box pairs, which a motion analysis module ranks by geometric plausibility. Backward refinement then corrects the temporal bias that arises when grounding latches onto a mid-contact frame rather than the true event onset. SAM2 verifies contact at the pixel level. Because grounding confidence and physical contact are distinct quantities, we treat them separately, i.e., a candidate is accepted only when its propagated masks produce measurable pixel overlap, regardless of how confidently it was detected. 2.3 Hierarchical Grounding and Candidate Generation Player appearance varies across practice sessions depending on attire, equipment, and lighting conditions. A single prompt cannot cover this range reliably. We use a three-level hierarchy =Pgear,Pnogear,PgenericP=\P_gear,P_nogear,P_generic\, ordered from most to least specific. For instance, PgearP_gear describes a helmeted player sprinting in a forward lean, while PgenericP_generic describes only a person running toward a red object. The dummy prompt is fixed across all three levels, as the tackle dummy has a consistent physical description. Each player-dummy prompt pair is submitted jointly to Grounding DINO [12] as a single phrase-grounding query, grounding both noun phrases in one forward pass. We probe six temporal positions across the video, searching a local offset window around each at three progressively relaxed box-confidence thresholds. Rather than returning on the first success, we collect every valid candidate across all positions, thresholds, and prompt levels. Figure 3 illustrates this search. This exhaustive collection matters in practice because grounding quality and contact quality do not correlate monotonically; a strong detection response at mid-contact may track the wrong player, while a weaker detection at an earlier frame can still confirm the correct onset through mask intersection. Figure 3: Temporal localization of first contact using multiframe grounding and backward refinement. (1) Multi-frame grounding queries the untrimmed clip at a set of normalized time positions; in this example, a valid player-dummy pair is first detected at position 0.9, defining a tackle event window. (2) Backward refinement then steps backward from the grounding frame to find FFBO (the first frame in which both the player and dummy are simultaneously present). (3) Starting from FFBO, SAM2 propagates masks forward and identifies FPOC as the earliest frame where the propagated masks provide evidence of player-dummy contact. Detection classification. Raw detections di\d_i\ carry bounding boxes bi∈ℝ4b_i ^4, confidence scores sis_i, and phrase labels lil_i. Each detection is assigned to the player or dummy class based on label keywords and geometry. Let d=dummy, pad, bag, tackle, training, blockingK_d=\dummy, pad, bag, tackle, training, blocking\ and p=player, person, athlete, helmet, football, running, sprintK_p=\player, person, athlete, helmet, football, running, sprint\. Let AR(di)=hi/wiAR(d_i)=h_i/w_i denote the height-to-width aspect ratio, τarea=0.01⋅HW _area=0.01· HW a minimum area threshold, and edge(di)edge(d_i) an indicator that the detection center lies within 15% of the frame width from either horizontal edge or within 10%10\% from the top: kind(di)=dummyif (AR(di)>2.0)∨(li∈d)playerif li∈p∧area(di)>τarea∧¬edge(di)kind(d_i)= casesdummy&if (AR(d_i)>2.0) (l_i _d)\\[4.0pt] player&if l_i _p\\ & if \ area(d_i)> _area\\ & if \ (d_i) cases (1) where edge(di)edge(d_i) is true when the detection center lies within the frame boundary margins. Dummies with AR(di)<0.8AR(d_i)<0.8 are rejected as horizontal objects, since a practice dummy is always upright. Players are not filtered by aspect ratio: a player mid-tackle may be nearly horizontal. Among valid dummy detections, the best candidate D∗D^* is selected by a weighted score over confidence sds_d, normalized distance distc(d)dist_c(d) from the image center, and verticality: D∗=argmaxd∈dummy[ D^*= _d _dummy [ 0.4sd+0.3(1−distc(d)) 04\,s_d+3 (1-dist_c(d) ) (2) +0.3min(AR(d)3.0, 1)] +3\, \! ( AR(d)3.0,1 ) ] The top three player candidates ranked by proximity to D∗D^* are forwarded to the validation stage. 2.4 Temporal Validation and Motion Scoring A detection at a single frame can be a false positive arising from motion blur, partial occlusion, or a background athlete passing near the dummy. Each candidate at primary frame t0t_0 is matched against detections in 14 neighboring frames ℱval=t0+δF_val=\t_0+δ\ via a composite confidence score over normalized IoU, center displacement, and area similarity. Let τIoU=0.20 _IoU=0.20 and τdist=200 _dist=200 px. The size ratio is sr(d0,dv)=min⁡(A0,Av)/max⁡(A0,Av)sr(d_0,d_v)= (A_0,A_v)/ (A_0,A_v), where A denotes box area: conf(d0,dv)= (d_0,d_v)= 0.55IoU(d0,dv)τIoU+0.30(1−dist(d0,dv)τdist) 055\, IoU(d_0,d_v) _IoU+30 (1- dist(d_0,d_v) _dist ) (3) +0.15sr(d0,dv) +15\;sr(d_0,d_v) The mean of conf(d0,dv)conf(d_0,d_v) across all matched frames defines the consistency score cconsc_cons. A player candidate must match in at least 3 of the 14 frames; a dummy candidate requires at least 2. Let Q denote the set of matched player detections across ℱvalF_val, with cmc_m the center of match m∈m and c0c_0 the center at t0t_0. Temporal persistence alone does not separate the active tackler from a stationary bystander who appears consistently near the dummy. We therefore compute two geometric signals from the matched player centers across ℱvalF_val. The displacement score measures how much the player has moved across the validation window, normalized by a reference distance of 200 pixels: mdisp(P)=min⁡(1||∑m∈∥c0−cm∥2200, 1)m_disp(P)= \! ( 1|Q| _m c_0-c_m _2200,\;1 ) (4) The directional approach score measures whether the observed motion points toward the dummy. Let c¯past c_past be the mean center of all matched detections with frame_idx<t0frame\_idx<t_0. The unit motion vector is v^motion=(c0−c¯past)/∥c0−c¯past∥ v_motion=(c_0- c_past)/ c_0- c_past and the unit approach vector is v^to-dummy=(cD∗−c0)/∥cD∗−c0∥ v_to-dummy=(c_D^*-c_0)/ c_D^*-c_0 , where cD∗c_D^* is the dummy center. Their cosine similarity is scaled to [0,1][0,1]: mdir(P,D)=⟨v^motion,v^to-dummy⟩+12m_dir(P,D)= v_motion,\, v_to-dummy +12 (5) Candidates with mdisp<0.08m_disp<0.08 or mdir<0.30m_dir<0.30 are discarded. The surviving candidates are ranked by: confoverall=0.3ccons+0.3mdisp+0.4mdirconf_overall=0.3\,c_cons+0.3\,m_disp+0.4\,m_dir (6) SAM2 evaluates candidates in descending order of confoverallconf_overall, stopping at the first that produces mask overlap. 2.5 Two-Phase Backward Refinement Grounding is most reliable near mid-contact, when both objects are large and simultaneously salient. As a result, tgt_g tends to fall later than the true event start tFFBOt_FFBO. Two phases correct this. In Phase 1, the pipeline steps backward from tgt_g one frame at a time, re-running detection with the reference boxes as spatial anchors and tolerating at most one consecutive miss before stopping. This recovers frames where both objects remain jointly detectable without over-extending into frames that precede the drill. In Phase 2, exponential offsets 5,10,20,50\5,10,20,50\ frames back from the current boundary are probed. When both objects are found at a probe frame, binary search localizes the exact earliest consistent frame. The result is tFFBOt_FFBO, and boxes re-extracted at that frame serve as the SAM2 seed prompts. 2.6 SAM2 Contact Verification and FPOC SAM2 [21] is initialized at tFFBOt_FFBO with the refined bounding boxes and propagates separate binary masks for the player (i=Pi=P) and dummy (i=Di=D) forward through the remaining frames: ℳt(i)=SAM2(,tFFBO,b(i)),i∈P,DM^(i)_t=SAM2\! (V,\,t_FFBO,\,b^(i) ), i∈\P,D\ (7) where ℳt(i)∈0,1H×WM^(i)_t∈\0,1\^H× W is the binary mask for object i at frame t, and b(i)b^(i) is the corresponding seed box. Contact at each frame is quantified as the pixel-wise logical intersection of the two propagated masks: overlapt=∑x,yℳt(P)(x,y)∧ℳt(D)(x,y)overlap_t= _x,yM^(P)_t(x,y)\; \;M^(D)_t(x,y) (8) The FPOC is the earliest frame where overlap reaches at least one pixel (τoverlap=1 _overlap=1): tFPOC=min⁡t|overlapt≥τoverlapt_FPOC= \t\; |\;overlap_t≥ _overlap \ (9) If no such frame exists for the current candidate, it is rejected and the next in the ranked list is evaluated. This is the mechanism driving the multi-candidate fallback: contact evidence, not detection confidence, decides which candidate is accepted. In untrimmed practice footage, there is no consistent visual marker that defines when a tackle ends. Some drills stop immediately after contact, while others continue as the player drives through contact and keeps moving with the dummy. Since our goal is onset localization, we avoid estimating an end time from mask overlap and instead output a compact, event-centered window that includes the approach and the initial collision. We use a fixed post-contact tail. Let LpostL_post be the number of frames retained after contact. We define tend=min⁡(T,tFPOC+Lpost),t_end= \! (T,\;t_FPOC+L_post ), (10) and output masks and frames over =[tFFBO,tend]W=[\,t_FFBO,\,t_end\,]. Unless stated otherwise, we use Lpost=20L_post=20 frames. A result is accepted unless both confoverall<0.25conf_overall<0.25 and mdir<0.20m_dir<0.20 hold simultaneously, which filters genuinely degenerate detections while preserving valid FPOC estimates with modest overall scores. Clips where no candidate confirms contact are flagged for manual review. 3 Experiments 3.1 Dataset We evaluated on 738 untrimmed monocular videos recorded at multiple American football training facilities using consumer-grade smartphones and fixed-mount cameras at 30 fps. Recording conditions vary substantially in background, viewing angle, distance to the drill, and illumination. The player population spans youth athletes through adults; attire ranges from full football uniform to casual clothing. This variability in the dataset helps keep the dataset natural and generalizes to represent real-world practice footage. The Tackle dummies used in the dataset are upright rectangular padded targets. Clips were recorded in both indoor and outdoor settings, with the majority outdoors. No clips were excluded for quality, and no domain-specific fine-tuning was applied to any component. Ground-truth FPOC labels were obtained via frame-by-frame manual annotation: the annotator marked the first frame in which the player physically collides with and moves the dummy (not merely the first frame of visual overlap/occlusion). We estimate an annotation uncertainty of approximately ±3± 3 frames. Out of 738 videos, 681 include a verified ground-truth FPOC label; the remaining 57 were recorded without a simultaneous annotation pass. We define the evaluable subset as the intersection between the ground-truth pool (681 clips) and the set of clips for which a given method successfully produces a valid segmentation/tracking output. Consequently, in the ablation study the evaluable subset size varies by setting (Table 2). Our full system segments 719 clips, yielding an evaluable subset of 666. 3.2 Evaluation Protocol FPOC localization has an asymmetric cost structure: early predictions can be trimmed in post-processing, whereas late ones shift downstream pose measurements into post-contact postures. We report tolerance-window accuracy: prediction t t is correct if |t^−t∗|≤ϵ| t-t^*|≤ε frames, for ϵ∈5,10,15,20ε∈\5,10,15,20\ frames. End-to-end accuracy fixes the denominator at all 738 videos, penalizing any clip the system fails to process. Conditional precision restricts the denominator to the evaluable subset, isolating per-clip localization quality from coverage. Our ablation isolates the contribution of each pipeline component, grounding breadth, motion filtering, and backward refinement, all of which are specific to the contact-onset problem. 3.3 Baselines We construct three internal ablation baselines by selectively enabling pipeline components (Table 1). All four configurations share identical SAM2 mask propagation and overlap-based FPOC detection. The ablation exclusively targets the grounding and candidate-selection stages upstream of SAM2, keeping the contact-verification step constant and ensuring clean attribution of performance differences. Table 1: Ablation component configuration. Component SOLE TRACE MARS GRAZE (ours) Single prompt + fixed threshold ✓ ✓ ✓ ✗ Multi-prompt search ✗ ✗ ✗ ✓ Temporal validation ✗ ✓ ✗ ✓ Motion scoring ✗ ✗ ✓ ✓ Backward refinement ✗ ✓ ✗ ✓ Multi-candidate retry ✗ ✗ ✗ ✓ Baseline B1: SOLE (Single-prompt Open-vocabulary Localization Estimator) uses a single fixed prompt and a fixed box-confidence threshold, with no temporal validation, motion scoring, or backward refinement. It represents the minimum viable grounding front-end feeding into the shared SAM2 contact verifier. Baseline B2: TRACE (Temporal vAlidation with refinement and Consistency Estimation) adds multi-frame temporal consistency validation and two-phase backward FFBO refinement to SOLE, but omits motion scoring. Crucially, temporal persistence alone cannot separate the active tackler from a stationary bystander who appears consistently near the dummy, nor from a player still repositioning at drill onset who moves laterally rather than forward. Without a directional motion filter, TRACE rejects these valid players at the validation stage, collapsing coverage to 46.9%, confirming that temporal consistency requires directional motion scoring to be discriminative rather than merely restrictive. Baseline B3: MARS (Motion-Aware Region Selection) adds player motion scoring to SOLE while omitting temporal validation and backward refinement. Its coverage (91.9%) and conditional precision (85.7% at ϵ=20ε=20) are nearly identical to SOLE (92.0%, 85.8%), showing that motion scoring in isolation changes candidate ranking upstream but does not improve per-clip localization accuracy. Its benefit emerges in the full system, where it interacts with multi-prompt progressive search and multi-candidate SAM2 retry to suppress distractors before they reach the segmentation stage. GRAZE is the complete pipeline: hierarchical multi-prompt progressive search, temporal validation, directional motion scoring, two-phase backward refinement, and ranked multi-candidate SAM2 retry. 4 Results We evaluated GRAZE on 738 untrimmed practice videos, of which 681 carry frame-accurate ground-truth FPOC annotations. We report three metrics. Coverage is the fraction of videos for which the pipeline produces any output. End-to-end accuracy treats all 738 clips as the denominator, penalizing videos for which the system produces no output. Conditional precision restricts the denominator to the evaluable subset, clips where both a system output and a ground-truth label exist, isolating per-clip localization quality from coverage effects. Of GRAZE’s 719 outputs, 666 overlap with the 681 GT-labeled clips; the remaining 53 segmented clips lack annotations, and 15 GT-labeled clips fell into manual review. A representative segmentation is shown in Figure 4. Figure 4: Representative tackle-event frames. Columns show (a) tFFBOt_FFBO (first frame with both objects visible), (b) tFPOCt_FPOC (contact onset), and (c) a post-contact frame near tendt_end. Rows show the raw frame, the dummy mask, the player mask, and the composite overlay. 4.1 Coverage Figure 5: Ablation study across three performance axes. (a) Segmentation coverage on all 738 videos. Solid bars show correctly segmented clips; hatched bars show clips routed to manual review; the dashed boundary marks the evaluable subset. (b) End-to-end FPOC accuracy at four frame-error tolerances over all 738 clips. Dotted lines mark the best-baseline value per tolerance. (c) Conditional FPOC precision over the evaluable subset. At ±5± 5 frames SOLE (80.3%) and MARS (80.4%) marginally lead GRAZE (79.1%); the full-system advantage emerges and widens from ±10± 10 frames onward. Coverage results appear in Figure 5(a) and the top rows of Table 2. GRAZE produces valid outputs on 719 of 738 videos (97.4%), routing only 19 clips to manual review. SOLE (grounding and SAM2 only) and MARS (SOLE plus motion scoring) each process around 92% of the dataset; the roughly 40-clip gap relative to GRAZE is closed by multi-prompt progressive search and multi-candidate SAM2 retry, neither of which is present in either baseline. TRACE (SOLE plus temporal validation and backward refinement) is the outlier at 46.9%. The failure is not localization error but coverage collapse. Without a directional motion gate, temporal validation becomes unreliable in crowded scenes: detections can shift between nearby athletes across validation frames, lowering match consistency below the acceptance threshold and causing otherwise valid candidates to be rejected. TRACE then routes these clips to manual review rather than passing a candidate to SAM2. Directional scoring prevents this by requiring candidates to show clear approach motion toward the dummy, making the validation stage genuinely discriminative. 4.2 Localization Accuracy Table 2 summarizes results at all tolerance levels. On the full 738-clip measure, GRAZE correctly localizes FPOC in 71.4% of videos within ± 5 frames, 77.5% within ± 10 frames, and 82.7% within ± 20 frames. Margins over SOLE and MARS grow with tolerance, 3.4, 6.9, and 10.1 points at the three thresholds. Because backward refinement primarily rescues clips where the initial grounding anchor is temporally displaced from the true onset, the benefit is most visible at wider tolerances. Conditional precision at ϵ=20ε=20 frames reaches 91.6% for GRAZE, against 85.7-85.8% for SOLE and MARS. The advantage grows from 2.5 points at ϵ=10ε=10 frames to 5.8 points at ϵ=20ε=20, consistent with refinement having its largest effect on clips with the biggest temporal offset. The one reversal is at ϵ<5ε<5 frames, where MARS reaches 80.4% and SOLE 80.3%, against GRAZE’s 79.1%. This is a genuine trade-off: backward refinement occasionally steps one to two frames past the true onset, shifting a small number of near-perfect predictions into the adjacent error band. The payoff is a halved catastrophic-error rate, GRAZE’s |err|≥20|err|≥ 20 frame rate is 8.4%, versus 14.2% and 14.3% for SOLE and MARS. For downstream biomechanical measurement, a two-frame overshoot is far less damaging than a twenty-frame miss, so the tail reduction is the more consequential result. Table 2: FPOC localization results on 738 videos (681 with ground-truth annotations). Frame error is |t^FPOC−tFPOC∗|| t_FPOC-t^*_FPOC|. End-to-end accuracy uses all 738 clips as the denominator. Conditional precision uses the evaluable subset (system output ∩ GT-labeled pool). †GRAZE segments 719 clips; 666 carry ground-truth labels (53 segmented clips lack annotations; 15 GT-labeled clips fell into manual review). Metric SOLE TRACE MARS GRAZE (ours) Coverage Correctly segmented 679 346 678 719 Coverage (%) 92.0 46.9 91.9 97.4 Evaluable clips† 625 323 624 666 End-to-end accuracy (%): all 738 videos |err|<5|err|<5 f 68.0 30.2 68.0 71.4 |err|<10|err|<10 f 70.6 31.6 70.6 77.5 |err|<15|err|<15 f 71.8 32.2 71.7 80.8 |err|<20|err|<20 f 72.6 33.1 72.5 82.7 Conditional precision (%): evaluable clips |err|<5|err|<5 f 80.3 69.0 80.4 79.1 |err|<10|err|<10 f 83.4 72.1 83.5 85.9 |err|<15|err|<15 f 84.8 73.7 84.8 89.5 |err|<20|err|<20 f 85.8 75.5 85.7 91.6 Error tail (conditional) |err|≥20|err|≥ 20 f (%) 14.2 24.5 14.3 8.4 Figure 4 shows a representative successful example from the evaluable set. SAM2 maintains a stable dummy mask across the contact window despite partial occlusion during impact, and the player mask deforms through the sprint-to-contact transition without identity loss. The predicted tFPOCt_FPOC corresponds to the earliest frame of nonzero mask intersection, matching the annotator-defined onset. The most common error in correctly processed clips is approach-from-behind geometry: in the final stride, the player’s image-plane silhouette overlaps the dummy before physical contact occurs, typically placing the prediction two to four frames early. This directly accounts for the shortfall at ϵ<5ε<5 frames and is a fundamental limitation of 2D mask overlap as a contact proxy. Among the 19 unprocessed clips, two patterns account for nearly all failures. Off-camera approaches produce no simultaneous player-dummy view, making grounding impossible regardless of prompt or threshold. In concurrent-player scenes two athletes approach within the same short window; neither clears the directional motion threshold independently, so both are discarded and the clip is flagged for review. 5 Conclusion We presented GRAZE, a training-free pipeline for frame-accurate contact onset localization in unconstrained American football practice footage. The central contribution is a reframing of SAM2 mask propagation: rather than treating it as a passive segmentation backend, GRAZE uses it as an independent pixel-level contact verifier wholly decoupled from grounding confidence. This separation enables a ranked multi-candidate fallback strategy in which all hypotheses generated across prompt hierarchies, temporal positions, and detection thresholds are tried in quality order until mask intersection independently confirms contact. Applied to 738 unconstrained practice videos without any domain-specific fine-tuning, GRAZE achieves 97.4% coverage and conditional FPOC precision of 91.6% within ± 20 frames and 85.9% within ± 10 frames. The ablation isolates the contribution of each stage: multi-prompt progressive search closes the coverage gap from 92% to 97.4%; directional motion scoring is what makes temporal validation discriminative rather than merely restrictive, as removing it (TRACE) collapses coverage to 46.9% by discarding valid players who fail persistence tests designed for stationary objects; and two-phase backward refinement halves the catastrophic-error rate relative to the strongest single-component baselines SOLE and MARS. The 2.6% of unresolved clips expose two structural limits. Off-camera approaches produce no simultaneous player-dummy view, making grounding geometrically impossible regardless of prompt quality. Concurrent multi-player scenes expose the limits of per-candidate directional scoring, which has no mechanism to compare relative approach trajectories across athletes. Future work will address both through a temporal multi-object tracker that maintains identity continuity across frames, replacing per-frame grounding disambiguation. Replacing image-plane mask overlap with depth-aware contact verification is a natural extension that would tighten the FPOC criterion for approach-from-behind trajectories, while preserving the core deployment constraint of zero domain-specific annotation at any pipeline stage. References [1] J. Carreira and A. Zisserman (2018) Quo vadis, action recognition? a new model and the kinetics dataset. External Links: 1705.07750, Link Cited by: §1.1. [2] A. Cioppa, A. Deliège, S. Giancola, B. Ghanem, M. V. Droogenbroeck, R. Gade, and T. B. Moeslund (2020) A context-aware loss function for action spotting in soccer videos. External Links: 1912.01326, Link Cited by: §1.1. [3] A. Deliège, A. Cioppa, S. Giancola, M. J. Seikavandi, J. V. Dueholm, K. Nasrollahi, B. Ghanem, T. B. Moeslund, and M. V. Droogenbroeck (2021) SoccerNet-v2: a dataset and benchmarks for holistic understanding of broadcast soccer videos. External Links: 2011.13367, Link Cited by: §1.1. [4] C. Feichtenhofer, H. Fan, J. Malik, and K. He (2019) SlowFast networks for video recognition. External Links: 1812.03982, Link Cited by: §1.1. [5] J. Gao, C. Sun, Z. Yang, and R. Nevatia (2017) TALL: temporal activity localization via language query. External Links: 1705.02101, Link Cited by: §1.1. [6] S. Giancola, M. Amine, T. Dghaily, and B. Ghanem (2018-06) SoccerNet: a scalable dataset for action spotting in soccer videos. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), p. 1792–179210. External Links: Link, Document Cited by: §1.1. [7] C. Gu, C. Sun, D. A. Ross, C. Vondrick, C. Pantofaru, Y. Li, S. Vijayanarasimhan, G. Toderici, S. Ricco, R. Sukthankar, C. Schmid, and J. Malik (2018) AVA: a video dataset of spatio-temporally localized atomic visual actions. External Links: 1705.08421, Link Cited by: §1. [8] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, P. Dollár, and R. Girshick (2023) Segment anything. External Links: 2304.02643, Link Cited by: §1.1. [9] L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J. Hwang, K. Chang, and J. Gao (2022) Grounded language-image pre-training. External Links: 2112.03857, Link Cited by: §1.1. [10] B. Liberatori, A. Conti, P. Rota, Y. Wang, and E. Ricci (2024) Test-time zero-shot temporal action localization. External Links: 2404.05426, Link Cited by: §1.1. [11] T. Lin, X. Liu, X. Li, E. Ding, and S. Wen (2019) BMN: boundary-matching network for temporal action proposal generation. External Links: 1907.09702, Link Cited by: §1.1. [12] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang (2024) Grounding dino: marrying dino with grounded pre-training for open-set object detection. External Links: 2303.05499, Link Cited by: §1.1, §1, §2.3. [13] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Mahendran, A. Arnab, M. Dehghani, Z. Shen, X. Wang, X. Zhai, T. Kipf, and N. Houlsby (2022) Simple open-vocabulary object detection with vision transformers. External Links: 2205.06230, Link Cited by: §1.1. [14] N. M. Nafi, S. Dietrich, and W. Hsu (2022-07) Risky tackle detection from american football practice videos using 3d convolutional networks. In Proceedings of the 18th International Conference on Machine Learning and Data Mining (MLDM), USA. Cited by: §1.1. [15] N. M. Nafi, A. Rediger, S. Dietrich, and W. Hsu (2023) Relevant instance segmentation in american football practice images to aid risky tackle detection. In 2023 International Conference on Machine Learning and Applications (ICMLA), Vol. , p. 725–729. External Links: Document Cited by: §1.1. [16] S. Nayab, S. R. Chohan, A. Jameel, S. R. Shah, S. A. M. Zaidi, A. N. Jha, and K. Siddique (2025) Advancing remote and continuous cardiovascular patient monitoring through a novel and resource-efficient iot-driven framework. External Links: 2505.03409, Link Cited by: §1.1. [17] J. Newman, J. Lin, L. Dah-Jye, and J. Liu (2021-01) Automatic annotation of american football video footage for game strategy analysis. Electronic Imaging 2021, p. 303–1. External Links: Document Cited by: §1.1. [18] J. Newman, A. Sumsion, S. Torrie, and D. Lee (2023-02) Automated pre-play analysis of american football formations using deep learning. Electronics 12 (3). External Links: Document, ISSN 2079-9292, Link Cited by: §1.1. [19] T. Phan, K. Vo, D. Le, G. Doretto, D. Adjeroh, and N. Le (2023) ZEETAD: adapting pretrained vision-language model for zero-shot end-to-end temporal action detection. External Links: 2311.00729, Link Cited by: §1.1. [20] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021) Learning transferable visual models from natural language supervision. External Links: 2103.00020, Link Cited by: §1.1. [21] N. Ravi, V. Gabeur, Y. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C. Wu, R. Girshick, P. Dollár, and C. Feichtenhofer (2024) SAM 2: segment anything in images and videos. External Links: 2408.00714, Link Cited by: §1.1, §1, §2.6. [22] Z. Tong, Y. Song, J. Wang, and L. Wang (2022) VideoMAE: masked autoencoders are data-efficient learners for self-supervised video pre-training. External Links: 2203.12602, Link Cited by: §1.1. [23] F. Wu, Q. Wang, J. Bian, H. Xiong, N. Ding, F. Lu, J. Cheng, and D. Dou (2022) A survey on video action recognition in sports: datasets, methods and applications. External Links: 2206.01038, Link Cited by: §1.1. [24] S. A. M. Zaidi, W. Hsu, and S. Dietrich (2025) ViTs for action classification in videos: an approach to risky tackle detection in American football. SSRN Electronic Journal. Note: Available at SSRN: https://ssrn.com/abstract=5611720. 10.2139/ssrn.5611720 Cited by: §1.1. [25] C. Zhang, J. Wu, and Y. Li (2022) ActionFormer: localizing moments of actions with transformers. External Links: 2202.07925, Link Cited by: §1.1.

Instant research discovery