Paper deep dive

Content Based Video Narration of Gameplay with Vision Language Models

Mathew Varghese

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 43

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 8/17/2026, 4:49:23 AM

Summary

The paper presents a content-based video narration system that generates esports-style commentary for arbitrary gameplay recordings using a general-purpose Vision-Language Model (VLM) and Text-to-Speech (TTS). The system employs three key mechanisms: temporal mosaic packing (arranging 9 frames into a 3x3 image to reduce API costs and capture motion), context-conditioned prompting (using recent narrations to suppress repetition), and duration-conditioned generation with elastic alignment (ensuring audio fits segment slots). It supports both cloud TTS and local on-device TTS (Voxtral on Apple Silicon), requiring no game-specific instrumentation or training.

Entities (8)

Age of Empires II · game · 95%Context-Conditioned Prompting · technique · 95%Temporal Mosaic Packing · technique · 95%Vision-Language Model (VLM) · model · 95%Apple Silicon · hardware · 90%Elastic Alignment · technique · 90%GPT-4o · model · 90%Voxtral TTS · model · 90%

Relation Signals (6)

System → requires → no game-specific instrumentation

confidence 95% · with no game-specific instrumentation, no engine telemetry, and no task-specific training

Context-Conditioned Prompting → suppresses → repetition

confidence 95% · suppressing the repetition that dominates per-segment captioning of static scenes

Elastic Alignment → ensures → frame-accurate muxing

confidence 90% · giving frame-accurate muxing without a forced aligner

Temporal Mosaic Packing → reduces → image payload cost

confidence 90% · the mosaic reduces per-minute image payloads by 9x

Voxtral TTS → runson → Apple Silicon

confidence 90% · a 6-bit quantized 4B-parameter on-device TTS model on Apple silicon

GPT-4o → usedfor → narration generation

confidence 90% · The vision stage uses OpenAI GPT-4o

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Live game commentary is scarce: it exists for professional esports broadcasts and almost nowhere else. We present a content-based video narration system that produces spoken, esports-style commentary for arbitrary gameplay recordings using a general-purpose vision-language model (VLM) and a text-to-speech back end, with no game-specific instrumentation, no engine telemetry, and no task-specific training. Three mechanisms carry the system. Temporal mosaic packing arranges nine uniformly sampled frames into a single 3x3 image, letting an image-native VLM reason about motion while consuming one image payload per segment instead of nine. Context-conditioned prompting replays the K most recent narrations as assistant-role history, suppressing the repetition that dominates per-segment captioning of static scenes. Duration-conditioned generation and elastic alignment constrain narration length in the prompt, then time-scale or symmetrically pad the synthesized audio so each utterance fills its segment slot exactly, giving frame-accurate muxing without a forced aligner. The implementation supports either cloud TTS or a 6-bit quantized 4B-parameter on-device TTS model on Apple silicon, making the speech stage fully local. We report a qualitative case study on real-time strategy footage, a cost model showing the mosaic reduces per-minute image payloads by 9x, and a candid account of observed failure modes - hallucinated game state, resolution loss from mosaicking, and prosody artifacts from time-scaling. We release the system as a reproducible baseline, with an evaluation protocol for the quantitative study a full version will report.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

39,515 characters extracted from source content.

Expand or collapse full text

Content-Based Video Narration of Gameplay with Vision–Language Models Mathew Varghese Affiliation: University of Washington Seattle, WA, USA mathewvarghesemanu@gmail.com Abstract Live game commentary is scarce: it exists for professional esports broadcasts and almost nowhere else. We present a content-based video narration system that produces spoken, esports-style commentary for arbitrary gameplay recordings using a general-purpose vision–language model (VLM) and a text-to-speech back end, with no game-specific instrumentation, no engine telemetry, and no task-specific training. Three mechanisms carry the system. Temporal mosaic packing arranges nine uniformly sampled frames into a single 3×33× 3 image, letting an image-native VLM reason about motion while consuming one image payload per segment instead of nine. Context-conditioned prompting replays the K most recent narrations as assistant-role history, suppressing the repetition that dominates per-segment captioning of static scenes. Duration-conditioned generation and elastic alignment constrain narration length in the prompt, then time-scale or symmetrically pad the synthesized audio so each utterance fills its segment slot exactly, giving frame-accurate muxing without a forced aligner. The implementation supports either cloud TTS or a 6-bit quantized 4B-parameter on-device TTS model on Apple silicon, making the speech stage fully local. We report a qualitative case study on real-time strategy footage, a cost model showing the mosaic reduces per-minute image payloads by 9×9×, and a candid account of observed failure modes: hallucinated game state, resolution loss from mosaicking, and prosody artifacts from time-scaling. We release the system as a reproducible baseline, with an evaluation protocol for the quantitative study a full version will report. Index Terms: game commentary, vision–language models, video captioning, text-to-speech, esports, accessibility, multimodal generation † Work conducted independently; affiliation is listed for identification only and does not imply institutional sponsorship or review.† Implementation available at https://github.com/mathewvarghesemanu/Content-based-video-narration-using-deep-learning I Introduction Commentary is part of how games are consumed. A professional esports broadcast pairs the video feed with one or two human casters who name what is on screen, attach stakes to it, and modulate their delivery to match the tension of the moment. That layer is what turns a screen recording into something watchable by someone who is not the player. It is also almost entirely absent outside the professional tier: the overwhelming majority of gameplay video (amateur uploads, training footage, replays, playtest captures, accessibility-oriented recordings) ships silent or with only game audio. The obvious reason is cost. Human commentary does not scale to the volume of gameplay video produced, and the classical automation route does not help, because it is not general. Prior systems that generate game commentary are typically wired into a specific title: they read structured game state from an engine, a replay file, or a telemetry API, and map that state to language with templates or a trained generator [3, 2]. Such systems produce accurate, well-grounded text, and they are useless for the next game, because the state schema changes. Modern vision–language models suggest a different trade. A general-purpose VLM consumes pixels, which every game emits, and produces language, which is what commentary is made of. Accuracy is traded away (the model does not know the rules, cannot read a resource counter reliably, and will invent plausible detail), but generality is bought: the same pipeline runs on a real-time strategy replay, a first-person shooter clip, and a driving game without a line of game-specific code. For the large space of gameplay video that has no commentary at all, that trade is often the correct one. This paper works through what it actually takes to make that trade concrete. The naive construction (caption every frame, speak every caption) fails on three separate axes, and each failure has a cheap fix that we adopt as a contribution. 1. Cost and temporality. Image-native VLMs charge per image and, when fed frames independently, see no motion. We pack a window of nine uniformly sampled frames into one 3×33× 3 mosaic image (Section I-B), so a single request carries the temporal evidence of a whole segment at one image’s cost. This follows the image-grid observation of Kim et al. [5], applied here under a hard latency and rate-limit budget. 2. Repetition. Consecutive segments of gameplay look alike, and a stateless captioner responds with near-identical text, which is intolerable in speech where the listener cannot skim. We condition each request on the last K generated narrations, injected as assistant-role turns (Section I-C), and instruct the model not to repeat itself. 3. Synchronization. Narration must fit the segment it describes, but generative text has unbounded length and TTS duration is not known until synthesis. We constrain length in the prompt with an explicit word and duration budget, then close the residual gap with elastic alignment: time-scaling long clips and symmetrically padding short ones (Section I-F), yielding exact slot occupancy without forced alignment. We further show that the speech stage need not be a cloud dependency: a 6-bit quantized 4B-parameter TTS model runs locally on consumer Apple silicon and drops into the pipeline behind the same interface as the hosted TTS service, which matters for cost, for offline use, and for footage that a user does not want to upload. Our contributions are: (i) a complete, reproducible, training-free pipeline for spoken content-based gameplay narration, with cloud and on-device speech back ends; (i) the three mechanisms above, each isolated and described well enough to reimplement; (i) an analytical cost model for mosaic packing and a qualitative case study on strategy-game footage; and (iv) an explicit account of failure modes plus a pre-registered evaluation protocol for the quantitative study we intend to report in the full-length version of this work. We position this paper as a preprint baseline: the system and its mechanisms are complete and reproducible, the human-subject evaluation is not yet run, and we state which is which. I Related Work I-A Video captioning and dense event description Describing video in natural language is a long-standing task, from clip-level captioning on MSR-VTT [15] to dense event captioning, which localizes and describes multiple overlapping events in untrimmed video [6, 18, 16]. Gameplay narration is dense captioning with two extra constraints that the benchmark task does not impose: the output must be continuous (every second of video gets audio, not just interesting ones) and it must be speakable within a fixed time budget. Our elastic alignment stage exists precisely because the second constraint has no analogue in text-output captioning. I-B Sports and esports commentary generation Sports broadcasting is the closest applied neighbour. SoccerNet-Caption introduced dense video captioning for soccer broadcasts [10], and GOAL added knowledge-grounded commentary generation [12]. On the games side, Ishigaki et al. generate racing-game commentary from vision, language, and structured data [3], and earlier work explores commentary and explanation for game-playing agents [2]. The distinguishing assumption in nearly all of this work is privileged access to structured state: event logs, tracking data, or engine telemetry. We deliberately give that up. Our system sees only what a viewer sees, which is what allows it to be pointed at an arbitrary title. I-C Vision–language models for video Instruction-tuned VLMs [8, 7, 1, 11] made general visual description practical without task-specific training, and a line of work adapts them to video by sampling and projecting frame features [9, 17, 16]. Most relevant to us, IG-VLM [5] shows that simply arranging sampled frames into an image grid lets an image VLM perform competitively on video question answering. We adopt that representation and report on its practical costs in a production-shaped setting: the resolution loss it inflicts on 1080p source frames is, in our experience, the dominant accuracy limitation of the whole pipeline (Section VII). I-D Speech synthesis Neural TTS is mature enough to be treated as a component [13, 4], and recent large TTS models produce expressive, controllable speech [14]. Our interest is narrower and systems-oriented: whether the speech stage can be made local. We show it can, using a 6-bit quantized 4B model executed through an Apple-silicon inference runtime, at quality we found acceptable for commentary and with no per-request cost. I-E Accessibility Automatic audio description for blind and low-vision audiences is an established accessibility practice for film and television, and games remain poorly served by it. Continuous narration of gameplay is structurally the same artifact as audio description, produced under a different framing; we return to this in Section VIII. I System I-A Overview and notation The input is a video V of duration D seconds. We partition it into N=⌊D/T⌋N= D/T contiguous segments of T seconds. Each segment i receives exactly one narration utterance uiu_i, synthesized to an audio clip aia_i whose duration is forced to a target τ. The output is V with its audio track replaced by ⨁iai _ia_i. Frames are sampled uniformly at 9/T9/T frames per second, so that each segment contributes nine frames, exactly the capacity of a 3×33× 3 mosaic. This coupling of the sampling rate to the mosaic geometry is deliberate: it makes T the single knob that trades temporal resolution against request count. Fig. 1 shows the full pipeline. Input video V (D s) Uniform sampling 9/T9/T fps → frames Temporal mosaic 3×33× 3 pack, window W VLM request + history ℋKH_K + prompt P⁡(τ)P(τ) Narration uiu_i (word-budgeted) TTS back end cloud or on-device Elastic alignment speed-up / pad to τ Concatenate + mux → narrated video historyui−K..i−1u_i-K..i-1 Fig. 1: Pipeline. Blue: video decoding. Orange: language generation. Green: speech and assembly. The dashed edge is the context-conditioning loop that suppresses cross-segment repetition. I-B Temporal mosaic packing For segment i we take a window W of consecutive sampled frames centred on the segment (with clamping at the sequence boundaries), and composite them into a single image on a 3×33× 3 lattice in raster order, so that reading order matches time order: Mi[x+w⋅c,y+h⋅r]=f9i+3r+c[x,y],r,c∈0,1,2,M_i[x+w\!·\!c,\;y+h\!·\!r]=f_9i+3r+c[x,y], r,c∈\0,1,2\, where w×hw× h is the source frame size. The composite is JPEG-encoded, base64-embedded, and sent as one image in the request. The motivation is economic as much as representational. A hosted VLM prices and rate-limits per image; the deployment we developed against imposed a hard daily request cap, which for a naive per-frame pipeline is exhausted by a few minutes of footage. Let R be the images consumed per minute of video. Per-frame captioning at the same temporal resolution gives Rframe=540/TR_frame=540/T images per minute, whereas mosaic packing gives Rmosaic=60/TR_mosaic=60/T, a 9×9× reduction, independent of T. At our default T=9T=9 s this is 6.76.7 images per minute rather than 6060, which is the difference between narrating an hour of footage inside a daily cap and not. The representational argument is that the mosaic exposes change. A single frame cannot show that a unit moved; nine tiles in reading order can, and the model does describe motion and escalation given them (Section V). The cost, which we do not minimize, is spatial: nine 1080p tiles form a 5760×32405760× 3240 composite that the API downsamples aggressively, so fine detail (resource counters, health bars, minimap icons, small text) is effectively destroyed, while large overlay text (alerts, banners) generally survives. This asymmetry is the root cause of most content errors we observed (Section V). I-C Context-conditioned prompting Gameplay is visually autocorrelated: adjacent segments of a base-building phase or a positional standoff look nearly identical. A stateless captioner therefore returns nearly identical text, and repetition that is merely dull in subtitles is unbearable in speech. We condition generation on recent output. Let ℋK=⟨ui−K,…,ui−1⟩H_K= u_i-K,…,u_i-1 be the last K narrations. These are inserted into the request as assistant-role turns preceding the current user turn, so the model perceives them as its own prior utterances rather than as instructions, and the task prompt closes with an explicit directive to maintain context and avoid repeating previous messages. We use K=5K=5 by default; larger K increases prompt tokens linearly and, in our observation, yields diminishing novelty benefit once the window covers the visible scene phase. This mechanism does double duty. Besides suppressing repetition, it produces narrative continuity: because the model can see that it already said an attack was building, it tends to describe the next segment as escalation or resolution rather than as a fresh scene. That is a large part of what makes the output sound like commentary rather than like a sequence of captions. I-D Duration-conditioned generation Each utterance must be speakable in τ seconds. We set τ=2Tτ=2T, i.e. narration audio runs at twice the wall-clock rate of segment progression, which reflects the design choice that commentary is continuous over the video rather than sparse; τ is a free parameter and setting τ=Tτ=T yields exactly real-time narration. The target is injected into the prompt by template substitution, together with a hard word budget and a style directive: ⬇ Quickly analyze the attached game screenshot. Identify the most prominent features: key units, any ongoing combat, notable buildings, and current resources without going into math. Provide a brief description focusing only on these critical aspects, suitable for a quick audio_duration-second overview less than 40 words like realtime e-sports commentary with emotional prompts. Don’t mention the name of the game or the info that this is a screenshot. Keep in mind the context and try not to repeat the previous messages. Target audience is esports fans. Three parts of this prompt are load-bearing and were each added in response to an observed failure. The word budget (<40<40 words) bounds synthesis duration enough that the alignment stage rarely has to apply aggressive time-scaling. Suppressing the meta-frame (“don’t mention that this is a screenshot”) removes a persistent artifact in which the model narrated the medium rather than the match, the single most immersion-breaking failure we encountered. Naming the audience and register (“esports fans”, “emotional prompts”) is what moves the output from neutral description into commentary voice, and it is a substantial part of the perceived quality difference. We also cap generation with a token limit as a second line of defence, and we detect refusal-shaped responses (a leading apology) and retry, up to three attempts, since a spoken refusal in the middle of a narration track is a total failure of the artifact. I-E Speech synthesis with interchangeable back ends The TTS stage is behind a two-function interface (generate_tts(text, path)), with two implementations selected by configuration: • Cloud. A hosted high-definition TTS model (tts-1-hd) with a selectable voice. Highest quality and lowest engineering cost; incurs per-character charges and requires uploading the narration text. • On-device. A 4B-parameter Voxtral TTS model quantized to 6 bits, executed on Apple silicon through the MLX array framework, exposing 19 voices across several languages. Audio is generated as 24 kHz float samples, concatenated across streaming chunks, and encoded to MP3. No API key, no network, no per-request cost. That the on-device path is viable changes the deployment envelope. The VLM stage still requires a hosted model, but the stage that scales with narration volume, and that would dominate cost on a long video, runs locally. It also removes a privacy objection for unreleased or personal footage, at least for the text-to-audio half of the pipeline. I-F Elastic temporal alignment Synthesized clip durations are not controllable a priori: the same 35-word narration may render to 8 or 13 seconds depending on voice and phonetic content. We therefore force each clip to exactly τ after the fact. Given clip aia_i of duration did_i: ai′=speedup⁡(ai,di/τ),di>τ,sil⁡(τ−di2)⊕ai⊕sil⁡(τ−di2),di≤τ,a_i = casesspeedup\! (a_i,\ d_i/τ ),&d_i>τ,\\[2.0pt] sil\! ( τ-d_i2 ) a_i \! ( τ-d_i2 ),&d_i≤τ, cases where speedupspeedup is phase-vocoder time-scaling that preserves pitch and sil⁡(t)sil(t) is t seconds of silence. Because every ai′a_i has duration exactly τ, concatenation places utterance i at a known offset iτiτ and the final mux is a stream copy of the video with the new audio track: no forced aligner, no drift accumulation, no re-encode of the video. The symmetric padding is a small but consequential detail: centring the utterance in its slot means the speech onset lags the segment boundary rather than colliding with it, which reads as a caster reacting to what just happened instead of pre-empting it. The failure mode of this stage is compression on the other side: when the model overshoots its word budget, ratios above roughly 1.31.3 produce audibly rushed, chipmunked delivery. Tightening the word budget is a better fix than tolerating the artifact. I-G Algorithm Algorithm 1 Content-based gameplay narration 1: video V, segment length T, history K, target τ=2Tτ=2T 2: F←SampleFrames(V, 9/T)F← SampleFrames(V,\ 9/T) 3: ℋ←⟨⟩H← \, ; U←⟨⟩U← \, 4: for i=0i=0 to ⌊|F|/9⌋−1 |F|/9 -1 do 5: Mi←Mosaic3x3(F,window centred at i)M_i← Mosaic3x3(F,\ window centred at i) 6: for attempt=1attempt=1 to 33 do 7: u←VLM(P⁡(τ),Mi,last K of ℋ)u← VLM(P(τ),\ M_i,\ last K of H) 8: if u valid and not refusal-shaped then break 9: end if 10: end for 11: append u to ℋH and to U 12: end for 13: for each ui∈Uu_i∈ U do 14: ai←TTS(ui)a_i← TTS(u_i) ⊳ cloud or on-device 15: ai′←FitToDuration(ai,τ)a_i ← FitToDuration(a_i,\ τ) 16: end for 17: return Mux(V,⨁iai′) Mux(V,\ _ia_i ) IV Implementation The system is implemented in Python. Frame extraction and final muxing use FFmpeg; mosaic composition uses Pillow; audio time-scaling, padding, and concatenation use pydub. Every parameter in Table I is exposed in a single YAML file, so a run is fully described by that file plus the input video. Models. The vision stage uses OpenAI GPT-4o, called through the chat completions HTTP endpoint with the mosaic base64-inlined as a single image and a 300-token completion cap. An earlier iteration of the system used gpt-4-vision-preview; the pipeline is otherwise unchanged, which is weak evidence that it is not tied to a particular model generation. The cloud speech stage uses OpenAI tts-1-hd with the nova voice. The on-device speech stage uses a 6-bit quantized 4B-parameter Voxtral TTS checkpoint executed through MLX with soundfile for audio I/O. None of the three mechanisms in Section I depends on this choice. Mosaic packing requires only that the model accept an image; context conditioning requires only a chat-style role-tagged history; duration conditioning is prompt text. Any instruction-following VLM with an image input and any TTS with a file output can be substituted, and we name the specific models here because the qualitative results in Section V are not reproducible without them, not because the design assumes them. TABLE I: Configuration surface and defaults Parameter Default Effect segment_duration T 9 s sampling rate, request count vision.window_size 9 frames per mosaic vision.history_size K 5 repetition suppression vision.max_tokens 300 bound on utterance length vision.prompt §I-D register, budget, grounding tts.method on-device cloud vs. local speech target duration τ 2T2T narration density Two engineering details matter for reproducibility. Frame extraction is idempotent (it is skipped when frames already exist) because decoding dominates wall-clock time on long inputs and iterating on prompts should not pay for it repeatedly. Previous narration transcripts are archived with a timestamp rather than overwritten, since prompt iteration is the main development loop and comparing successive transcripts is how one evaluates a prompt change. V Qualitative Case Study We ran the pipeline on a 60-second, 1920×10801920× 1080, 60 fps capture of an Age of Empires I: Definitive Edition match (Castle Age, eight players, an active raid in progress), with T=9T=9 s, K=5K=5, τ=18τ=18 s, and mosaic window W=9W=9, using GPT-4o for narration. Table I gives the verbatim output of the first three segments, unedited. TABLE I: Verbatim narration, RTS footage, T=9T=9 s Seg Generated narration 0 “Red alert! Villagers under heavy attack by Lorraine’s forces! Castle stands strong at the heart, surrounded by buzzing farms. Resources are holding, but pressure is mounting. Will they withstand this relentless assault?” 1 “Amidst the chaos, Lorraine and Burgundy strike! Cavalry charges fiercely! Villagers scurry, farms buzz with productivity. The castle stands resolute, resources steady. Can this defense hold strong? The tension’s palpable!” 2 “Tension peaks! Cavalry units clash fiercely near the crucial farms! Villagers hustle, castles and buildings stand firm. Resources hold, but Burgundy and Lorraine’s aggressive push intensifies! Defense must rally, or the momentum could shift dramatically!” Several properties are visible in this sample and are, in our experience, representative. Entity grounding works at the coarse level. Villagers, cavalry, farms, and the castle are correctly identified from appearance alone. The opponent names (Lorraine, Burgundy) are also correct, but their provenance is worth separating: the footage carries an in-game alert overlay reading “You are being attacked by 5 Lorraine” and “7 Burgundy”, so the model is reading large overlay text rather than recognizing civilizations visually. Both capabilities are useful and neither required game-specific configuration, which is the premise of the approach, but only the first is visual recognition, and a paper that conflated them would overstate what the VLM is doing. Register transfer works. The output is unmistakably in commentary voice: second-person stakes, rhetorical questions, escalation vocabulary. This comes entirely from the audience-and-register clause of the prompt. Continuity emerges from the history mechanism. The three segments form an arc (assault begins, defenders hold, tension peaks) rather than three independent descriptions of a similar scene. Nothing in the system models the match state; the arc is a side effect of the model seeing its own prior utterances. Lexical repetition is nonetheless still visible (“farms”, “resources hold”, “stands strong” recur), which tells us K=5K=5 history attenuates but does not eliminate the problem. Confidence outruns evidence. “Resources are holding” and “resources steady” appear in all three segments. The source frames in fact carry an exact, legible resource bar (1125 wood, 657 food, 400 gold, 894 stone, 35/70 population); after 3×3× spatial reduction in the mosaic and further downsampling by the API, those digits are a few pixels tall and unreadable. The model is therefore emitting a plausible commentary phrase, not a reading of game state, and the information it is bluffing about was present in the input and discarded by our own representation. This is the cleanest available illustration of the cost side of mosaic packing, and it is not hedged: a listener could not identify it as invention. We treat it as the central open problem of the approach and discuss it below. Prompt–content mismatch is tolerated. The prompt used for this run was left over from an earlier experiment and explicitly named a popular first-person shooter title, while the footage was real-time strategy. The model ignored the stated title and described what it actually saw, in genre-appropriate vocabulary. This is accidental but useful evidence for generality (pixels dominate the prompt’s genre hint), and it simultaneously shows that the prompt cannot be relied upon to steer the model into a specific title’s domain vocabulary when that is what one wants. V-A Cost At T=9T=9 s the pipeline issues 6.76.7 VLM requests and 6.76.7 TTS utterances per minute of video, against 6060 VLM images per minute for per-frame captioning at identical temporal resolution. With the on-device speech back end the TTS component of marginal cost is zero and the pipeline’s monetary cost is exactly the VLM requests. VI Proposed Evaluation Protocol This preprint reports the system and a qualitative case study; it does not report a human-subject evaluation. We state the protocol we intend to run so that the claims of the full version are pre-registered rather than retrofitted. Corpus. 30 clips of 60–120 s spanning three genres (RTS, FPS, racing), 10 per genre, drawn from publicly available gameplay recordings. Conditions. (C1) full system; (C2) no history (K=0K=0), isolating context conditioning; (C3) per-frame captioning instead of mosaic, at matched temporal resolution and 9×9× the request cost, isolating mosaic packing; (C4) human-written commentary on a subset, as a ceiling; (C5) original game audio only, as a floor. Subjective measures. Mean opinion score on a 5-point scale for naturalness, informativeness, excitement/appropriateness of register, and synchronization, collected from participants who play the relevant genre, with each clip rated by at least five raters and inter-rater agreement reported. Objective measures. (i) Factual grounding rate: fraction of verifiable claims per utterance that are correct, annotated against the video by two independent annotators. This is the metric that directly targets the hallucination failure above. (i) Repetition: mean pairwise ROUGE-L and embedding cosine similarity between consecutive utterances, which should separate C1 from C2 if context conditioning does what we claim. (i) Alignment error: distribution of |di−τ||d_i-τ| before correction, and the fraction of clips requiring a time-scale ratio above 1.3. (iv) Cost and latency: requests, tokens, and wall-clock seconds per minute of video, cloud versus on-device speech. Accessibility study. A separate small-n study with blind and low-vision participants, evaluating the output as audio description rather than as entertainment, since the two use cases weight informativeness and excitement very differently and we expect the current prompt to be tuned wrongly for the former. VII Limitations and Failure Modes Hallucinated game state. The model asserts quantities it cannot see. Because the register is confident by construction, these assertions are indistinguishable from grounded ones to a listener. Any use of this system where a viewer might act on the commentary as information (coaching, analysis, accessibility) requires this to be fixed, not merely noted. Two directions apply: prompt-level hedging that permits the model to omit rather than invent, and hybrid grounding that supplies verified values for it to use, which we develop under the next item since the two problems share a cause. Resolution loss from mosaicking. Nine 1080p tiles are downsampled to a degree that destroys HUD text and small sprites. This is a direct trade against the 9×9× cost saving, and it is the mechanism behind the previous failure. We believe the strongest available fix is region-of-interest decomposition, and we state it here as the first item of future work because it attacks the hallucination and the resolution loss with one mechanism. Game interfaces are not visually uniform: they are spatially structured, and the structure is stable across an entire session. In the strategy footage of Section V the screen decomposes into a resource and population bar (top-left), an objectives panel (top-right), an alert and chat overlay (upper-left), a scoreboard (lower-right), a selected-unit and production panel (bottom-centre), and a minimap (bottom-right), leaving the playfield as the only region where the 3×33× 3 mosaic’s motion evidence actually matters. Each non-playfield region is small, fixed in position, and information-dense in exactly the way that survives poorly under downsampling. This suggests a two-stream request rather than a single flattened image. The playfield is mosaicked as now, at low resolution, because what is being asked of it is coarse and temporal: what moved, what is fighting, where the pressure is. The HUD regions are cropped at native resolution and handled separately and cheaply: numeric fields (resources, population, score, timers) by OCR, the minimap by colour-blob analysis giving contact locations and rough force distribution, and alert and chat overlays by direct text extraction, which the case study shows already works. The extracted values are then injected into the prompt as a structured state block, and the prompt is constrained to use those numbers and to say nothing quantitative that does not appear in them. This converts the model’s job from guessing at state to narrating supplied state, which is where LLMs are reliable. Two properties make this attractive beyond accuracy. The HUD readers are cheap (OCR on a handful of small crops costs no VLM tokens), so the approach does not give back the 9×9× saving that motivated mosaicking. And the cost of generality is bounded and honest: region coordinates are per-title configuration, roughly a dozen rectangles in a YAML file, produced once per game and, for titles with stable interfaces, reusable across patches. That is a far weaker game-specific dependency than the engine or telemetry access that prior commentary systems assume (Section I), and it degrades gracefully: with no region file the system behaves exactly as it does today. Auto-detecting HUD regions by observing which screen areas remain static across many frames while the playfield changes is a plausible way to remove even that step, and we have not attempted it. No audio input. The system never listens to the game. Gunfire, ability sounds, and voice lines carry event information that the visual channel often lacks, and ignoring them is a significant missed signal. Fixed segmentation. Segments are uniform, so a decisive teamfight and an idle economy phase receive equal narration budget. Event-driven segmentation (detect salience, then allocate narration) is the obvious improvement and would change the architecture more than any other item on this list. Batch, not live. The pipeline is offline. Live use requires streaming frame capture, latency-bounded generation, and incremental muxing; the per-request round trip is the binding constraint. Single voice, no interplay. Real broadcasts use play-by-play and colour commentary in dialogue. We emit one voice; the multi-voice extension is straightforward with the existing back ends and untried. Prosody artifacts. Time-scaling beyond roughly 1.3×1.3× is audible. Tighter word budgets, or duration-aware regeneration, are preferable to post-hoc compression. No quantitative evaluation yet. As stated in Section VI. The claims in this paper about repetition suppression and register transfer are qualitative observations from development, not measured effects. VIII Ethical Considerations Accessibility. The most defensible use of this system is audio description of games for blind and low-vision players and viewers, a population that commercial games serve poorly. That use case demands accuracy over excitement, which inverts the current prompt’s priorities, and it makes the hallucination problem a safety-relevant defect rather than a quality one. Labour. Automatic commentary touches the work of human casters. We note that the realistic near-term application is footage that would otherwise have no commentary at all (amateur uploads, replays, archives) rather than professional broadcasts, where the quality gap is wide and the social function of a known caster is not substitutable. We do not think this observation dissolves the concern; we think it bounds it. Provenance. Generated commentary should be labelled as such. Confident synthetic narration over real footage is a plausible vector for misrepresenting what happened in a match, and the system as built provides no watermark or disclosure mechanism. Content and rights. Gameplay footage is subject to the rights of both the capturing player and the game publisher, and the cloud paths transmit frames and text to third-party services. The on-device speech back end reduces but does not remove that exposure. IX Conclusion We presented a training-free system that narrates arbitrary gameplay video with spoken, esports-style commentary using a general vision–language model and an interchangeable speech back end, one of which runs entirely on-device. Three mechanisms carry the system: temporal mosaic packing, which buys motion evidence at a ninth of the image cost; context-conditioned prompting, which converts independent captions into a continuous narrative; and duration-conditioned generation with elastic alignment, which makes speech fit video exactly without a forced aligner. A qualitative case study on strategy-game footage shows correct coarse entity grounding, successful register transfer, and emergent continuity, alongside confident hallucination of game state that our own mosaic had discarded. We identify that as the central unsolved problem, and region-of-interest decomposition (reading HUD panels, minimap, and alert overlays at native resolution while the playfield stays mosaicked) as the first target of future work. We have specified the evaluation protocol for the quantitative study rather than reporting partial numbers, and we release the implementation as a reproducible baseline for a task that, despite abundant gameplay video, has almost no open tooling. Acknowledgments The author thanks the open-source maintainers of FFmpeg, Pillow, pydub, and MLX, without which this system would have been substantially more work than it was. References [1] J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. (2022) Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), p. 23716–23736. Cited by: §I-C. [2] B. Harrison, C. Purdy, and M. O. Riedl (2017) Toward automated story generation with markov chain monte carlo methods and deep neural networks. In Proc. AAAI Conf. Artificial Intelligence and Interactive Digital Entertainment (AIIDE) Workshops, Cited by: §I, §I-B. [3] T. Ishigaki, G. Topić, Y. Hamazono, H. Noji, I. Kobayashi, Y. Miyao, and H. Takamura (2021) Generating racing game commentary from vision, language, and structured data. In Proc. 14th Int. Conf. Natural Language Generation (INLG), p. 265–275. Cited by: §I, §I-B. [4] J. Kim, J. Kong, and J. Son (2021) Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In Proc. Int. Conf. Machine Learning (ICML), p. 5530–5540. Cited by: §I-D. [5] W. Kim, C. Choi, W. Lee, and W. Rhee (2024) An image grid can be worth a video: zero-shot video question answering using a VLM. arXiv preprint arXiv:2403.18406. Cited by: item 1, §I-C. [6] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles (2017) Dense-captioning events in videos. In Proc. IEEE Int. Conf. Computer Vision (ICCV), p. 706–715. Cited by: §I-A. [7] J. Li, D. Li, S. Savarese, and S. Hoi (2023) BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proc. Int. Conf. Machine Learning (ICML), p. 19730–19742. Cited by: §I-C. [8] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §I-C. [9] M. Maaz, H. Rasheed, S. Khan, and F. S. Khan (2024) Video-ChatGPT: towards detailed video understanding via large vision and language models. In Proc. 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §I-C. [10] H. Mkhallati, A. Cioppa, S. Giancola, B. Ghanem, and M. Van Droogenbroeck (2023) SoccerNet-Caption: dense video captioning for soccer broadcasts commentaries. In Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition Workshops (CVPRW), p. 5073–5084. Cited by: §I-B. [11] OpenAI (2023) GPT-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §I-C. [12] J. Qi, J. Yu, T. Tu, K. Gao, Y. Xu, X. Guan, X. Wang, Y. Dong, B. Xu, L. Hou, J. Li, J. Tang, W. Qi, Y. Sun, and B. Xu (2023) GOAL: a challenging knowledge-grounded video captioning benchmark for real-time soccer commentary generation. In Proc. ACM Int. Conf. Information and Knowledge Management (CIKM), p. 5391–5395. Cited by: §I-B. [13] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan, et al. (2018) Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. In Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), p. 4779–4783. Cited by: §I-D. [14] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei (2023) Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111. Cited by: §I-D. [15] J. Xu, T. Mei, T. Yao, and Y. Rui (2016) MSR-VTT: a large video description dataset for bridging video and language. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), p. 5288–5296. Cited by: §I-A. [16] A. Yang, A. Nagrani, P. H. Seo, A. Miech, J. Sivic, I. Laptev, and C. Schmid (2023) Vid2Seq: large-scale pretraining of a visual language model for dense video captioning. In Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), p. 10714–10726. Cited by: §I-A, §I-C. [17] H. Zhang, X. Li, and L. Bing (2023) Video-LLaMA: an instruction-tuned audio-visual language model for video understanding. In Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, p. 543–553. Cited by: §I-C. [18] L. Zhou, Y. Zhou, J. J. Corso, R. Socher, and C. Xiong (2018) End-to-end dense video captioning with masked transformer. In Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), p. 8739–8748. Cited by: §I-A.

Instant research discovery