Paper deep dive
Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioning
Duy Tran Thanh, Thien-Phuc Doan, Long Nguyen-Vu, Ngo Tan Vu Khanh
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/3/2026, 2:28:24 AM
Summary
The paper introduces 'Adjudicated Captioning', an inference-time multi-agent framework for strict zero-shot image captioning (ZIC) that improves upon the IFCap baseline. It addresses the limitation of single-checkpoint alignment scoring by inserting a frozen Cross-Attention Verifier to re-rank retrievals and attaching learned Beam Rerankers (TriFuse and MemAttend) at the output beam. These components are trained self-supervised via Borda-consensus distillation without paired image-caption labels, achieving a CIDEr score of 117.6 on COCO Karpathy, significantly outperforming previous strict-ZIC methods.
Entities (12)
Relation Signals (10)
Adjudicated Captioning → achievesscoreon → COCO Karpathy
confidence 98% · reaches CIDEr 117.6 ... on COCO Karpathy
Adjudicated Captioning → improves → IFCap
confidence 95% · reaches CIDEr 117.6 ... up from 108.0 ... for IFCap
TriFuse → istype → Multilayer Perceptron
confidence 95% · TriFuse, a multilayer perceptron
MemAttend → istype → Transformer
confidence 95% · MemAttend, a memory-attended transformer
Adjudicated Captioning → uses → Cross-Attention Verifier
confidence 93% · insert a frozen Cross-Attention Verifier that re-ranks the top-9 retrievals to top-5
Adjudicated Captioning → uses → Retrieval Encoder
confidence 92% · install a stronger frozen Retrieval Encoder at the input
Adjudicated Captioning → uses → MemAttend
confidence 90% · pairing TriFuse ... with MemAttend
Adjudicated Captioning → uses → TriFuse
confidence 90% · attach a learned Reranker pairing TriFuse
Cross-Attention Verifier → isinstantiatedby →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Zero-shot image captioning (ZIC) describes images without paired image-caption supervision during captioner training, relying on text-only corpora and frozen pretrained image-text scorers. Existing retrieval-augmented methods score image-text alignment once, at retrieval, then commit the captioner's autoregressive beam under language-model probability alone, leaving the decoder without further visual grounding feedback. Progress has stalled, with no method improving on the strict-regime best since 2024. We propose Adjudicated Captioning, an inference-time multi-agent framework that restores grounding feedback at multiple checkpoints over an unchanged IFCap captioner. First, we install a stronger frozen Retrieval Encoder at the input. Second, between retrieval and decoding we insert a frozen Cross-Attention Verifier that re-ranks the top-9 retrievals to top-5. Third, at the output beam we attach a learned Reranker pairing TriFuse, a multilayer perceptron, with MemAttend, a memory-attended transformer, the pipeline's only learned components; both are trained self-supervised by Borda-consensus distillation across the three frozen scorers, using no paired image-caption labels and no reference captions. Under the inductive headline protocol, with rerankers fit on the disjoint COCO Karpathy validation beam and applied frozen to test, the framework reaches CIDEr 117.6 and SPICE 21.9 on COCO Karpathy, up from 108.0 and 20.3 for IFCap, a +9.6 CIDEr gain, and +7.7 above NES, the strongest synthetic-image-augmented method at 109.9, without retraining the captioner. A training-free fixed-fusion baseline reaches 115.8 CIDEr, so +7.8 of the +9.6 gain comes from the non-learned architectural intervention and the remaining +1.8 from the learned rerankers. The same recipe transfers off-COCO without captioner retraining: +8.1 CIDEr on Flickr30k Karpathy and +5.7 on NoCaps overall.
Tags
Links
- Source: https://arxiv.org/abs/2607.28986v1
- Canonical: https://arxiv.org/abs/2607.28986v1
Trouble viewing inline? Open PDF directly →
Full Text
94,063 characters extracted from source content.
Expand or collapse full text
Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioning Duy Tran Thanh*, Thien-Phuc Doan, Long Nguyen-Vu, Ngo Tan Vu Khanh *Corresponding author. Duy Tran Thanh is a Senior AI Engineer with AI Platform OneNexus, OneMount (e-mail: duy.tran10@onemount.com).Thien-Phuc Doan is with the School of Electronic Engineering, Soongsil University, Seoul 06978, South Korea (e-mail: phucdt@ssu.ac.kr).Long Nguyen-Vu is with the AI Laboratory, MoAdata, Seoul, South Korea (e-mail: longnv@moadata.ai).Ngo Tan Vu Khanh is with the University of Economics Ho Chi Minh City (UEH), Ho Chi Minh City, Vietnam (e-mail: khanhntv@ueh.edu.vn). Abstract Zero-shot image captioning aims to describe images in natural language without paired image–caption supervision during Captioner training, relying instead on text-only corpora and frozen pretrained image–text scorers. Existing retrieval-augmented methods score image–text alignment exactly once, at the initial retrieval step, and then commit the Captioner’s autoregressive beam under language-model probability alone, leaving the decoder without further visual grounding feedback. Progress in this regime has consequently stalled, with no method improving on the strict-regime best since 20242024. We propose Adjudicated Captioning, an inference-time multi-agent framework that restores grounding feedback at multiple checkpoints over an unchanged IFCap Captioner and crosses the plateau without paired image–caption supervision. Our contributions are threefold. First, we install a stronger frozen Retrieval Encoder at the input. Second, between retrieval and decoding we insert a frozen Cross-Attention Verifier that re-ranks the top-99 retrievals to top-55, producing a Verifier-adjudicated retrieval pool. Third, at the output beam we attach a learned Reranker pairing TriFuse, a multilayer perceptron, with MemAttend, a memory-attended transformer, the pipeline’s only learned components; both are trained self-supervised by Borda-consensus distillation across the three frozen scorers, consuming no paired image–caption labels and no reference captions. To our knowledge these are the first learned Beam Rerankers proposed under the captioner-text-only regime. Under the inductive headline protocol, with Rerankers fit on the disjoint COCO Karpathy validation beam and applied frozen to test, the framework reaches CIDEr (Consensus-based Image Description Evaluation) 117.6117.6 and SPICE (Semantic Propositional Image Caption Evaluation) 21.921.9 on COCO Karpathy, up from 108.0108.0 and 20.320.3 for IFCap, a +9.6+9.6 CIDEr gain, and +7.7+7.7 above Negative Entity Suppression (NES), the strongest synthetic-image-augmented method at 109.9109.9, without retraining the Captioner. A training-free fixed-fusion baseline reaches 115.8115.8 CIDEr, so +7.8+7.8 of the +9.6+9.6 gain is attributable to the non-learned architectural intervention at the retrieval and verification checkpoints, and the remaining +1.8+1.8 to the learned Rerankers. The same recipe transfers off-COCO without Captioner retraining: +8.1+8.1 CIDEr on Flickr30k Karpathy and +5.7+5.7 on NoCaps overall. A Verifier-substrate swap to a non-COCO-fine-tuned matcher still yields 114.0114.0 CIDEr, +6.0+6.0 over IFCap, showing the gain is not solely conditional on COCO-specific Verifier tuning. Index Terms: Beam re-ranking, BLIP, CLIP, cross-modal alignment, retrieval-augmented generation, self-supervised learning, vision–language models, zero-shot image captioning. Retrieval ViT-B/32✓Captioner “A snowboarder and a snowboarder are posing…” CIDEr 108.0108.0PriorRetrieval ViT-bigG/14✓NEWVerifier BLIP-ITM cross-attn✓NEWCaptioner Learned Reranker TriFuse ++ MemAttend✓NEW “Two men are snowboarding down a snowy hill.” CIDEr 117.6117.6+9.6+9.6Ours Figure 1: From single-checkpoint to multi-checkpoint alignment scoring. Prior strict-ZIC pipelines (top) score image–text alignment at exactly one place: the retrieval-time cosine (✓). Our pipeline (bottom) adds three novel alignment-scoring components (NEW ✓): a stronger Retrieval encoder, a Cross-Attention Verifier, and a learned Beam Reranker. The Captioner is left untouched. The three new heads are trained self-supervised over the three frozen scorers, with no paired image–caption labels. Result: +9.6+9.6 CIDEr on COCO Karpathy (117.6117.6 vs. 108.0108.0). I Introduction Retrieval-augmented image captioning has converged on a single design rule: image–text alignment is scored exactly once, at retrieval, and is not consulted again. ClipCap [1] fixed the recipe. A small mapping network turns a CLIP image embedding into a soft prefix for a frozen GPT-2 [2]; under the strict text-only zero-shot image captioning (ZIC) regime the mapping network sees no image during training and bridges the modality gap at inference using CLIP cosines. Subsequent work, ranging from CapDec [3] and DeCap [4] to ViECap [5], MeaCap [6], and IFCap [7], has refined this recipe in different places, but every method commits the Captioner’s beam under language-model probability after retrieval, without any further image–text alignment check. Progress under this rule has been uneven, and it has stalled outright since 20242024. On COCO Karpathy [8, 9], CIDEr111Metric acronyms used throughout: CIDEr, Consensus-based Image Description Evaluation; SPICE, Semantic Propositional Image Caption Evaluation; BLEU, Bilingual Evaluation Understudy; METEOR, Metric for Evaluation of Translation with Explicit ORdering; ROUGE-L, Recall-Oriented Understudy for Gisting Evaluation, longest-common-subsequence variant. improved from 92.992.9 with ViECap to 95.495.4 with MeaCap, a +2.5+2.5 step, and then to 108.0108.0 with IFCap, a larger +12.6+12.6 step delivered by two simultaneous changes, image-like retrieval and frequency-based entity filtering, rather than by any single new component. Since IFCap, no method operating inside the strict regime has, to our knowledge, improved on 108.0108.0. The one entry above it, Negative Entity Suppression (NES) [10] at 109.9109.9, gains +1.9+1.9 and does so only by departing from the regime and training on Stable-Diffusion-generated images alongside the text corpus. Successive strict-ZIC methods have each introduced a new filter between retrieval and decoding, including entity-aware masking in ViECap, frequency-based entity filtering in IFCap, and attention-level suppression in NES; every one of these is a further constraint on a single retrieval step rather than a new alignment-scoring event. We attribute the plateau to a structural property of the pipeline that has remained unchallenged since retrieval-augmented captioning was introduced, rather than to a limit of language modelling. Image–text alignment is consulted exactly once, at the initial CLIP-cosine retrieval, after which the decoder operates on its own. The Captioner’s beam is selected by language-model probability and is therefore biased toward fluent outputs that need not be fully grounded; the retrieval pool itself remains noisy because a dual-encoder contrastive model orders out-of-distribution images only approximately. No prior strict-ZIC system inserts a stronger image–text scorer at any later point in the pipeline. Building on this observation, we propose a strict-ZIC framework that scores image–text alignment at multiple alignment checkpoints rather than only at retrieval, as illustrated in Figure 2. Concretely, we make three architectural changes on top of an unchanged Captioner. First, we replace IFCap’s OpenAI CLIP ViT-B/32 [11] dual encoder with a scaling-law-stronger OpenCLIP ViT-bigG/14 [12] for top-99 retrieval against the 566566K-caption COCO Karpathy training corpus, which is the largest single-component lift in our ablation. We refer to the two by their architecture names, ViT-B/32 and ViT-bigG/14, throughout. Second, we insert a frozen Cross-Attention Verifier between retrieval and decoding to re-rank the top-99 down to top-55; instantiated with the BLIP image–text matching head [13], the Verifier’s rank correlation with the Retrieval Encoder cosine is only ρ≈0.4ρ\!≈\!0.4 on IFCap retrievals, providing partly orthogonal information that the dual encoder cannot supply. Third, at the Captioner’s output we attach two complementary self-supervised learned Beam Rerankers. TriFuse is a 113113-parameter multilayer perceptron (MLP) that fuses three frozen-scorer signals per beam candidate: the GPT-2 log-probability, the Retrieval Encoder cosine, and the Verifier’s matching score. MemAttend is a 17.417.4K-parameter memory-augmented transformer encoder that lets candidates cross-attend to the Verifier-adjudicated retrieval pool. Both Rerankers are trained by listwise Borda-consensus distillation across the three frozen scorers; no ground-truth captions and no paired image–caption supervision are used at any point, so the strict text-only training regime is preserved end-to-end. The Captioner itself, namely IFCap’s released mapping network and GPT-2 decoder, is never retrained or fine-tuned. We adopt the operational definition used by every prior strict-ZIC method: the regime constrains Captioner training, not the externally pretrained frozen scorers consumed at inference (Sec. I-D). To avoid ambiguity we use the more precise label captioner-text-only ZIC under frozen paired-pretrained scorers where it matters. Within this regime, our inductive headline, with TriFuse and MemAttend trained on the disjoint COCO Karpathy validation beam-2020 dump and applied frozen to the test split, reaches CIDEr 117.6117.6 / SPICE 21.921.9 / BLEU-44 32.832.8 / METEOR 27.927.9 / ROUGE-L 55.555.5, +9.6+9.6 CIDEr over the previous strict-ZIC state of the art, IFCap, and +7.7+7.7 over the synthetic-image-augmented NES. A transductive variant, trained on the test images’ frozen-scorer signals (no reference captions used), differs by at most 0.10.1 on any primary metric, and not at all on CIDEr, and is reported as a strict upper bound, not the headline. A training-free fixed-fusion baseline reaches 115.8115.8 CIDEr without any learned head, isolating the architectural intervention from the learned step. The same inference-only recipe transfers off-COCO without Captioner retraining: +8.1+8.1 CIDEr on Flickr30k Karpathy [14, 15] and +5.7+5.7 on NoCaps [16] overall, with consistent in-/near-/out-domain gains (Table VI, benchmark-transductive). A reference-free CLIPScore (no ground-truth captions consumed) and a reference-proxy Caption Hallucination Assessment with Image Relevance (CHAIR) check corroborate the gain, the latter reducing sentence-level reference-proxy hallucination from 13.28%13.28\% to 7.84%7.84\%, roughly 41%41\% relative. In summary, our contributions are as follows: • We articulate and empirically validate an architectural hypothesis for the strict-ZIC plateau, namely that image–text alignment is scored at a single alignment checkpoint, and propose a remedy: place strong, frozen image–text scorers at multiple alignment checkpoints, including a Cross-Attention Verifier between retrieval and decoding and a learned Reranker at the Captioner’s output. • We introduce two complementary, fully self-supervised Beam Rerankers: TriFuse, a 113113-parameter MLP, and MemAttend, a 17.417.4K-parameter memory-augmented transformer encoder. Both are trained by Borda-consensus distillation across three frozen scorers, with no paired image–caption supervision. To our knowledge these are the first learned Beam Rerankers proposed for the captioner-text-only ZIC regime; supervised captioning has used learned beam rerankers (e.g. MBR-style decoding [17], ListNet rank objectives [18]), but those use paired image–caption labels that this regime forbids. • Under the captioner-text-only ZIC regime with frozen paired-pretrained scorers, our pipeline reaches CIDEr 117.6117.6 and SPICE 21.921.9 on COCO Karpathy (inductive headline), surpassing IFCap by +9.6+9.6 and NES by +7.7+7.7 CIDEr without retraining or fine-tuning the Captioner. The same inference-only recipe transfers off-COCO: +8.1+8.1 CIDEr on Flickr30k Karpathy and +5.7+5.7 CIDEr on NoCaps overall. I Related Work I-A Strict zero-shot image captioning Strict-ZIC methods can be sorted by where they apply image conditioning. ZeroCap [19] and MAGIC [20] apply it during decoding, rescoring partial sequences against a CLIP score at each generation step. CapDec [3] pushes the burden upstream by perturbing text features with isotropic Gaussian noise at training time, so the mapping network sees an image-like distribution at inference. DeCap [4] avoids the prefix altogether and decodes directly in CLIP space. ViECap [5] prepends an entity-aware hard prompt, MeaCap [6] prepends a memory bank of retrieved captions, and IFCap [7] combines image-like noise injection with a frequency-based entity filter on the retrieval pool. The lineage shares one structural commitment: a CLIP-style scorer at retrieval is the only image-conditioning event, after which the Captioner generates without further alignment checks. IFCap reaches CIDEr 108.0108.0 on COCO Karpathy, the previous strict-text-only best on this benchmark. Paired-supervised foundation models such as BLIP-2 [21], InstructBLIP [22], and LLaVA [23] train on hundreds of millions of paired images and captions and reach far higher CIDEr, with BLIP-2 at 145.8145.8; they lie outside the strict-text-only regime, and we report them in a separate block of Table I. We build atop IFCap’s released checkpoint without modifying its weights, and our entire intervention is inference-time. I-B Synthetic-image-augmented zero-shot captioning A second line replaces text-only training with synthetic image–caption pairs generated from training captions. SynTIC [24] renders an image per training caption with Stable Diffusion [25] and trains on those synthetic pairs, reaching 101.1101.1 CIDEr. PCM-Net [26] repairs the rendering quality with a patch-wise cross-modal mixup and reaches 103.8103.8. NES [10], the strongest entry at 109.9109.9, layers attention-level negative-entity suppression on the same synthetic-image retrieval pipeline. These methods pay the price of image generation at training time and modify the Captioner; our work does neither. The pipeline we describe is an inference-time intervention on the released IFCap checkpoint, and outperforms NES by +7.7+7.7 CIDEr on COCO Karpathy while remaining captioner-text-only with frozen paired-pretrained scorers. I-C Image–text alignment scoring and reranking Hessel et al. [27] introduced CLIPScore as a reference-free image–text alignment metric, since used to rerank caption beams in supervised captioning. We use it in its ViT-bigG/14 incarnation [12] at the output checkpoint of a strict-ZIC pipeline. An output-checkpoint substrate sweep we ran (CLIP variant held against the rest of the pipeline) confirms monotone scaling: CIDEr rises from 111.7111.7 with OpenAI CLIP ViT-B/32 to 114.4114.4 with ViT-bigG/14. The full pipeline, with the stronger Retrieval Encoder, the Verifier, the accompanying entity-filter threshold rescale from K=5K=5 over L=9L=9 retrievals to K=3K=3 over L=5L=5 that keeps the admission fraction fixed once the Verifier shrinks the pool (Sec. I-B), and the learned TriFuse+MemAttend Reranker, reaches 117.6117.6. Cross-encoder reranking over a dual-encoder candidate set is standard in document retrieval [28]; listwise learning-to-rank objectives [18] and minimum-Bayes-risk decoding [17] provide the closest parallels for our beam-arbitration step, and ensemble distillation [29] motivates our Borda-consensus training target. The cross-encoder rerank has been applied at training time in image captioning: NES [10] uses synthetic-image retrieval as a second-opinion at training time, and PCM-Net [26] uses a patch-wise mixup to repair the second-opinion. To our knowledge, no prior strict-ZIC method inserts a frozen cross-attention image–text matcher, specifically the BLIP-ITM head [13], at inference time as a second-stage rerank, and no prior method places a separate learned rerank stage at the Captioner’s output beam. We empirically demonstrate that placing scorers at both alignment checkpoints crosses the strict-ZIC plateau by +9.6+9.6 CIDEr without retraining the Captioner. I-D Operational definition of the strict-text-only regime The term “strict zero-shot image captioning” is sometimes interpreted as “no paired image–text data anywhere in the system,” an interpretation that no existing method, ours or prior, satisfies. We follow the operational definition used by the strict-ZIC family of DeCap [4], ViECap [5], MeaCap [6], IFCap [7], and NES [10]: the constraint applies to the Captioner training, not to externally pretrained frozen scorers. The Captioner sees a text corpus alone; the frozen image–text scorers used at inference, namely CLIP variants and the BLIP-ITM head, were pretrained on paired image–text data and are used as-is at evaluation time. Under this definition, prior strict-ZIC work already uses OpenAI CLIP ViT-B/32, trained on 400400M web image–text pairs, at retrieval, and ClipCap-style mapping is itself defined relative to a frozen pretrained CLIP. Our pipeline uses two such scorers. The first is OpenCLIP ViT-bigG/14 [12] pretrained on LAION-2B image–text pairs. The second is the BLIP image–text matching head [13] fine-tuned on COCO image–text pairs. The ViT-bigG/14 pretraining set does not enumerate which images were paired with which text, but the LAION-22B distribution overlaps the COCO domain; the BLIP-ITM head was explicitly fine-tuned on COCO captions. Two clarifications are in order. First, no COCO Karpathy training/validation/test caption is used as a label during our pipeline; retrieval reads the COCO training caption corpus as a text-only index, exactly as IFCap does. Second, BLIP-ITM’s explicit COCO fine-tuning is not the same as CLIP’s broad web-scale pretraining, and we do not claim equivalence; we retain BLIP-ITM as a frozen scorer because it provides a stronger cross-attention image–text matcher than any dual encoder, and we separately report the Flickr30k-fine-tuned BLIP variant (Table V) so a reader can judge how much of the COCO gain is conditional on that fine-tuning. It is a stronger image–text matcher than CLIP, and our gain decomposes as “stronger scorer at retrieval” plus “stronger scorer at output” rather than “access to COCO supervision elsewhere”. Our COCO Karpathy result should be read as a ZIC result conditional on access to a COCO-tuned frozen Verifier; we report Flickr30k Karpathy and NoCaps val with the same frozen Verifier in Sec. V-F to show that the gain persists when the Captioner has never been trained on Flickr30k or NoCaps captions. We report a Verifier-substrate ablation in our Verifier-substrate analysis: swapping the COCO-fine-tuned BLIP-ITM head for the Flickr30k-fine-tuned variant of the same architecture [13] still yields a +6.0+6.0 CIDEr lift over IFCap, confirming that the architectural intervention is not an artefact of the Verifier’s COCO exposure. I Method imageIIRetrieval Encoder frozen dual encoderVerifier frozen cross-attention matcherCaptioner + prompt ℛ5⋆R _5TriFuse ++ MemAttend + memory ℛ5⋆R _5Stage 1Stage 2Stage 3Stage 4 y^(I) y(I): “A man riding a motorcycle on a dirt road on a hill side.” Memory banktrain=566C_train\!=\!566Kcaption corpus “A man on a motorcycle …” “Dog catching a frisbee …” “A bowl of fruit …” … 566566K captionsindex lookup top-9 ℛ9R_9: ∙ “A man riding a motorbike …” ∙ “A motorcycle on a hillside …” ∙ (7 more) top-5 ℛ5⋆R _5 (Verifier softmax): ∙ 0.940.94 “A man on a motorcycle …” ∙ 0.910.91 “Motorcycle, dirt road …” ∙ (3 more) beam-20 20Y_20 (LM-ranked): y1y_1: “A man on a motorcycle on a road” y2y_2: “A man riding a motorbike …” ⋮ y20y_20: (17 more) ranking score r^y r_y: βz(TriFuse)+(1−β)z(MemAttend)β\,z(TriFuse)\!+\!(1-β)\,z(MemAttend) β=0.75β=0.75, take argmaxy _y frozen pretrained modeltrained in this paper (self-supervised) Figure 2: Full inference pipeline. The four stages run left-to-right; each agent box carries its role and any side-channel input it consumes, and the concrete output of each stage is shown in the dashed card below. The yellow Memory bank cylinder above Stage 1 represents the external corpus of train=566C_train\!=\!566K training captions, with sample caption cards shown alongside, from which the Retrieval Encoder draws. Stage 1, Retrieval: a frozen Retrieval Encoder encodes I; image–text cosine over the precomputed corpus index returns the top-99 candidates ℛ9R_9. Stage 2, Verification: a frozen Cross-Attention Verifier re-ranks the 99 candidates by image–text matching softmax and keeps the top-55, the Verifier-adjudicated retrieval pool ℛ5⋆R _5. Stage 3, Captioning: the unmodified Captioner consumes the image feature and ℛ5⋆R _5 as soft prefix and hard prompt at K=3K=3, emitting a 2020-wide beam 20Y_20. Stage 4, Reranking, the only trained stage: TriFuse (MLP) and MemAttend (transformer with ℛ5⋆R _5 as memory tokens) jointly score the beam; their z-normalised convex combination at β=0.75β=0.75 selects the final caption y y. TriFuse and MemAttend are trained self-supervised by Borda-consensus distillation across the three frozen scorers, namely the LM log-probability, the Retrieval cosine, and the Verifier matching score, with no paired image–caption labels. Only the upstream pretrained scorers, loaded frozen as indicated by the snowflake, ever saw image–text pairs. Existing captioner-text-only ZIC pipelines face two architectural weaknesses. First, image–text alignment is scored at exactly one place, the initial CLIP-cosine retrieval, so the retrieval pool is dual-encoder-noisy. Second, the Captioner’s beam is selected by language-model probability alone and is therefore biased toward fluent-but-not-fully-grounded outputs. We address both by scoring image–text alignment at multiple alignment checkpoints. Our framework integrates three components on top of an unchanged IFCap Captioner. At Stage 1 we install a stronger frozen Retrieval Encoder. At Stage 2 we insert a frozen Cross-Attention Verifier between retrieval and decoding. At Stage 4 we attach a learned Beam Reranker that fuses three frozen-scorer signals into a single ranking score. The section proceeds in four subsections. Sec. I-A fixes notation and reviews the IFCap baseline. Sec. I-B introduces the triadic alignment pipeline as four equations. Sec. I-C introduces TriFuse, the MLP Reranker. Sec. I-D introduces MemAttend, the memory-augmented transformer Reranker. TriFuse and MemAttend are the only learned modules in the framework. They are trained under the protocols defined in Sec. IV-D: the headline inductive protocol fits the heads on the disjoint COCO Karpathy validation beam dump and applies them frozen to test; the transductive variant fits on test-image beam features using only frozen-scorer pseudo-labels and is reported as an upper bound. The overall flow is shown in Figure 2. I-A Notation and the IFCap baseline Let train=c1,…,cNC_train=\c_1,…,c_N\ be a text-only training corpus of N≈566,747N≈ 566,747 captions, drawn from the COCO Karpathy training split, which contains about 113113K images with 5 reference captions each. Let CLIPimg(⋅)CLIP_img(·) and CLIPtxt(⋅)CLIP_txt(·) denote OpenAI’s CLIP ViT-B/32 image and text encoders, and let c=CLIPtxt(c)/∥⋅∥t_c=CLIP_txt(c)/\|·\| denote the L2-normalised CLIP-text feature of caption c. For a test image I, let ı^=CLIPimg(I)/∥⋅∥ =CLIP_img(I)/\|·\|. IFCap [7] trains a small mapping network MθM_θ on the text corpus alone, using CLIP-text features as a proxy for the image features it will see at inference. Concretely, IFCap perturbs each ct_c with isotropic Gaussian noise of variance σ2=0.016σ^2=0.016 to mimic the distribution of ı , and trains Mθ(~c,r1,…,rk)M_θ( t_c,\t_r_1,…,t_r_k\) to predict the caption c given k=5k=5 nearest-text retrievals. At inference, the noise-injected proxy ~c t_c is replaced by the actual image feature ı . A retrieval pool ℛL(I)R_L(I) of size L=9L=9 is built by ranking trainC_train by cosine to ı , and the Captioner consumes ℛ5(I)⊂ℛ9(I)R_5(I) _9(I) in two ways: as a soft prefix [30] through MθM_θ, and as a hard prompt through a frequency-based entity filter that admits a noun if it appears in ≥K=5≥ K=5 of the 99 retrieved captions. On COCO Karpathy test, this pipeline reaches CIDEr 108.0108.0. I-B Triadic alignment pipeline We replace the single-checkpoint alignment scoring of IFCap with a four-stage pipeline: Stage 1 is a stronger dual-encoder retrieval, Stage 2 is a Cross-Attention Verifier, Stage 3 is the unchanged IFCap Captioner, and Stage 4 is a learned Beam Reranker. We describe each stage in turn, immediately following the equation that defines it. Stage 1: Retrieval. For test image I, we retrieve the nine closest training captions in the cosine space of a strong frozen dual encoder: ℛ9(I)=top-9c∈trainϕimg(I)⋅ϕtxt(c).R_9(I)\;=\; top-9_c _train~~ _img(I)· _txt(c). (1) IFCap retrieves with OpenAI CLIP ViT-B/32 [11], a 151151M-parameter dual encoder trained on 400400M web image–text pairs. In this work we instantiate the Retrieval Encoder with ViT-bigG/14 [12], an approximately 2.52.5B-parameter encoder trained on LAION-22B [31]; the image encoder, the text encoder, and the corpus index all come from this model. The dual-encoder retrieval index has not been revisited as foundation-CLIP scaling laws improved, and our ablation in Section V-G shows this single substitution is the largest single-component lift in the table, contributing +3.6+3.6 CIDEr. For the remainder of the paper we refer to the Stage-1 model as the Retrieval Encoder when context is clear. Stage 2: Cross-Attention Verification. We re-score the 99 candidates with a frozen cross-attention image–text matcher, instantiated here with the BLIP-ITM head [13], and keep the top 55: ℛ5⋆(I)=top-5c∈ℛ9(I)sV(I,c),R _5(I)\;=\; top-5_c _9(I)~~s_V(I,c), (2) where sV(I,c)∈(0,1)s_V(I,c)∈(0,1) is the Verifier’s frozen softmax matching score. Even with a stronger dual encoder at Stage 1, ranks remain imperfect: dual encoders score independent embeddings without any cross-attention. We measure the rank correlation between the Retrieval Encoder cosine and the Verifier score on the top-99 across all 5,0005,000 COCO Karpathy test images and obtain Spearman ρ≈0.4ρ\!≈\!0.4. The two scorers disagree on which retrieval best matches the image roughly 60%60\% of the time, and the Verifier ranking is the one downstream captioning cares about, since the cross-attention matcher was trained discriminatively for image–text alignment. We refer to the Stage-2 model as the Verifier for the remainder of the paper. Threshold rescaling. IFCap’s entity filter admits a noun if it appears in ≥K≥ K of the retrieved captions, with default K=5K=5 over L=9L=9 captions, an admission fraction of 5/9≈56%5/9≈ 56\%. After the Verifier rerank shrinks the pool from 99 to 55, the same K=5K=5 would require unanimous agreement and collapse the prompt to near-empty; we rescale to K=3K=3 over L=5L=5, an admission fraction of 3/5=60%3/5=60\%. Of the available integer thresholds at L=5L=5, K=3K=3 is the closest to the original fraction, since K=2K=2 would admit 40%40\%. The combination of Verifier rerank and threshold rescale is mutually enabling: neither works in isolation, as shown in rows 3–5 of Table VII. Stage 3: IFCap Captioning (unchanged). Run the unmodified IFCap Captioner on the verified pool to emit a wide beam: 20(I)=IFCap-beam20(ı^,ℛ5⋆(I)).Y_20(I)\;=\;IFCap-beam_20 ( ,\;R _5(I) ). (3) Here ı is the ViT-B/32 image feature that IFCap’s released mapping network expects, and IFCap-beam20IFCap-beam_20 runs IFCap with the rescaled entity-filter K=3K=3 and beam width 2020. The Captioner is unchanged; only its inputs are. Stage 4: Learned Beam Rerank. Pick the final caption by maximising the learned Beam Reranker TriFuseϕTriFuse_φ, or its convex combination with MemAttend described in Sec. I-D, over the 2020 candidates: y^(I)=argmaxy∈20(I)TriFuseϕ(y(I)). y(I)\;=\; _y _20(I)~TriFuse_φ\! (f_y(I) ). (4) Each candidate is summarised by a 33-d feature vector that fuses the three frozen scorers, the language model’s log-probability, the Retrieval Encoder’s cosine, and the Verifier’s matching score, into a per-candidate fingerprint: y(I)=[z(ℓy)⏟LM logprob,z(sRy)⏟Retrieval cosine,z(sV(I,y))⏟Verifier]∈ℝ3,f_y(I)\;=\; [\, z( _y)_LM logprob,\ z(s_R^\,y)_Retrieval cosine,\ z(s_V(I,y))_Verifier\, ] ^3, (5) with ℓy=logpGPT-2(y∣ℛ5⋆(I)) _y\!=\! p_GPT-2(y _5(I)), sRy=cos(ϕimg(I),ϕtxt(y))s_R^\,y\!=\! ( _img(I),\, _txt(y)), and z(⋅)z(·) per-image z-normalisation across the 2020-wide beam, so the three signals share a common scale. The IFCap Captioner’s beam is otherwise selected by language-model probability alone, which ranks fluency rather than visual matching; we empirically observe that a frozen image–text scorer applied at the output picks a different caption roughly 35%35\% of the time, and CIDEr improves whenever it does. TriFuseϕTriFuse_φ is the only learned module in the pipeline; we describe it next. I-C TriFuse: minimal MLP head image I prompt: top-55 Verifier-adjudicated retrievals ℛ5⋆(I)R _5(I) ∙ “A man on a motorcycle …” ∙ “Motorcycle on a hill …” (3 more) ViT-B/32mapping net MθM_θentity filter (K=3K=3)GPT-2 decoderCaptioner beam-2020 candidates 20(I)=y1,y2,…,y20Y_20(I)=\y_1,y_2,…,y_20\GPT-2 outputGPT-2 LM Retrieval Encoder Verifier yℓy _ysRys_R^\,ysybs b_yz-normalise per image (across 2020-wide beam)feature vector y(I)∈ℝ3f_y(I)\!∈\!R^3concatenated z-scores of the three scorersInference pathway (frozen)TriFuse Reranker Linear lift 3→83\!→\!8GELUnon-linearityLinear hidden 8→88\!→\!8GELUnon-linearityLinear project 8→18\!→\!1r^y r_yargmax over the 2020-wide beamselects final caption y^(I) y(I)y(I)f_y(I)Self-supervised training (no paired labels)Per-image ranksrank of y under each frozen scorer (0–1919)Borda consensusaverage inverted ranks → softmax pseudo-labelListwise cross-entropy losstrains ϕφ to match the consensus pseudo-label∇ℒ updates ϕφ Figure 3: TriFuse architecture and training (two-panel layout). Left panel, inference pathway ( all frozen): the unmodified Captioner is shown as a vertical stack of its four sub-components: ViT-B/32, mapping network MθM_θ, entity filter at K=3K=3, and GPT-2 decoder. The image I feeds into ViT-B/32; the prompt ℛ5⋆(I)R _5(I) feeds into MθM_θ and the entity filter; GPT-2 emits the 2020-wide beam 20(I)Y_20(I). For each candidate y, three frozen scorers, namely the GPT-2 LM, the ViT-bigG/14 Retrieval Encoder, and the Verifier, emit a scalar (ℓy _y, sRys_R^\,y, sybs b_y), which are z-normalised across the beam and concatenated into y(I)∈ℝ3f_y(I)\!∈\!R^3. Right panel, TriFuse Reranker ( blue, the only trainable module): Linear→ → → → , 113113 parameters; the argmax over the beam selects the final caption y y. Bottom (dashed blue): self-supervised training. The pseudo-label per candidate is the Borda-consensus score across the three scorers’ per-image ranks, softmaxed to a target distribution, and TriFuse is trained by listwise cross-entropy. The dashed orange arrow shows where ∇ℒ updates ϕφ. No paired image-caption labels are used. Architecture. TriFuse is a minimal MLP with 33 input features per beam candidate, two hidden layers of width h=8h=8, GELU activation [32], and a scalar output: TriFuseϕ() _φ(f) =W3GELU(W2GELU(W1+1)+2)+b3, =W_3\,GELU\! (W_2\,GELU(W_1f+b_1)+b_2 )+b_3, (6) W1 W_1 ∈ℝ8×3,W2∈ℝ8×8,W3∈ℝ1×8. ^8× 3,\ \ W_2 ^8× 8,\ \ W_3 ^1× 8. (7) Trainable parameters: |ϕ|=113|φ|=113, namely 3232 in the input layer, 7272 in the hidden layer, and 99 in the output layer. Inference is O(K|ϕ|)O(K|φ|) per image with K=20K=20, which is sub-millisecond on a single GPU. Self-supervised training. TriFuse is trained under the protocols of Sec. IV-D, using only frozen-scorer pseudo-labels (no reference captions). The headline inductive protocol fits on the disjoint COCO Karpathy validation beam-2020 dump; the transductive protocol fits on the test beam-2020 dump and is reported as an upper bound. For each image I and each candidate y∈20(I)y _20(I), we compute the rank rs(y∣I)∈0,…,K−1r_s(y I)∈\0,…,K-1\ of y within the beam under each frozen scorer s∈=LM,R,Vs =\LM,R,V\. The pseudo-label is the per-image softmax of the Borda-consensus score [33]: qy(I)=1||∑s∈(K−1−rs(y∣I)),q_y(I)= 1|S| _s (K-1-r_s(y I)), (8) q~y(I)=expqy(I)∑y′expqy′(I). q_y(I)= q_y(I) _y q_y (I). (9) The training loss is the listwise softmax cross-entropy [18]: ℒ(ϕ)=−1N∑I,yq~y(I)logσy(TriFuseϕ(y(I))),L(φ)=- 1N _I,\,y q_y(I)\, \, _y\! (TriFuse_φ(f_y(I)) ), (10) where σy(⋅) _y(·) is the per-image softmax over the K-wide beam. We train for 100100 epochs with Adam [34] at learning rate 10−210^-2. No paired image–caption supervision is used: the pseudo-labels are derived entirely from the frozen scorers’ outputs on the Captioner’s own beam, so TriFuse training does not consume any data outside the strict-ZIC text-only regime. Training takes under one minute on a single GPU. Why TriFuse beats a fixed-α linear z-mix. A linear mix αz(ℓ)+(1−α)z(sR)α\,z( )+(1-α)\,z(s_R) uses two signals weighted by a single global hyperparameter and is constrained to lie in their span. TriFuse uses three signals, adding the Cross-Attention Verifier’s matching score on (I,y)(I,y) that the linear mix discards, and learns a non-linear combination conditioned on per-image normalised feature interactions. Empirically, TriFuse beats the best fixed-α linear mix on every primary metric. The Borda-consensus pseudo-label provides a stable training target even though no candidate has a ground-truth label: the y that all three frozen scorers agree on tends to be a high-CIDEr caption, and learning to predict consensus generalises to the per-image argmax. I-D MemAttend: memory-augmented transformer head Memory tokensBeam tokens (top-2020 candidates)MEM1MEM_1MEM2MEM_2…MEM5…MEM_5BEAM1BEAM_1BEAM2BEAM_2…BEAM20BEAM_20jM∈ℝ5f^M_j\!∈\!R^5 (Verifier, len, rank)yB∈ℝ5f^B_y\!∈\!R^5 (LM, Retrieval, Verifier, rank, len)M=5M\!=\!5K=20K\!=\!20Linear projection WinW_in + role embeddingembedded tokens t(0)∈ℝ32h^(0)_t ^32, t∈1,…,25\;t∈\1,…,25\Encoder Layer 1 (4-head self-attn ++ FFN)Encoder Layer 2 (same)hidden states t(L)∈ℝ32h^(L)_t ^32 (BEAM tokens go to head)Output head: linear → score per BEAM tokenMemAttendψ(I,y)∈ℝMemAttend_ψ(I,y) for y∈20y _20Per-image z-normalise; βz(TriFuse)+(1−β)z(MemAttend)β\,z(TriFuse)\!+\!(1\!-\!β)\,z(MemAttend), β=0.75\;β\!=\!0.75TriFuse(yB)ϕ_φ(f^B_y)MLP, see Fig. 3y^(I)=argmaxy∈20[βz(TriFuseϕ)+(1−β)z(MemAttendψ)] y(I)= _y _20\, [β\,z(TriFuse_φ)+(1\!-\!β)\,z(MemAttend_ψ) ]Attention readout: BEAM (query) →\!→\! MEM (key)12345MEMjy1y_1yty_tyKy_KBEAM yty_t (query)dark == high attn weight orange row == one yty_t exampleinside encodertrainable (|ψ|=17.4|ψ|\!=\!17.4K params):WinW_in, ∙r_ , Encoder 1//2, WoutW_out, boutb_outfrozen: all upstream scorers + Captioner Figure 4: MemAttend architecture (Memory-Attended Reranker). Top: 55 memory tokens MEMjMEM_j (orange, scalars from the Stage-2 Verifier-adjudicated retrievals) and 2020 beam tokens BEAMyBEAM_y (gray, scalars from the three frozen scorers). Stack ( blue, the only trainable parameters): a linear projection Win∈ℝ32×5W_in\!∈\!R^32× 5 with two role embeddings (MEM vs. BEAM) lifts each token to dmodel=32d_model=32; two transformer encoder layers (44-head self-attention, FFN 32→64→3232\!→\!64\!→\!32, GELU); an output head emits a scalar per BEAM token. |ψ|=17.4|ψ|\!=\!17.4K total. Ensemble (red): MemAttend scores are per-image z-normalised and convex-combined with the parallel TriFuse head (Fig. 3) at β=0.75β=0.75 to pick the final caption y y. Bottom-left: cross-attention pattern over the K×MK×M token grid; each BEAM token attends to all MEM tokens (and to other BEAM tokens), implementing an explicit working-memory readout. Bottom-right: trainability legend. Trained self-supervised by Borda-consensus distillation across the three frozen scorers; no paired image-caption labels. TriFuse’s per-candidate MLP cannot directly model interactions between beam candidates or between beam candidates and the Stage-22 retrieval pool. We propose a complementary head with that capacity: MemAttend, a memory-augmented cross-scorer transformer [35] that frames the beam-rerank step as conditioning on an explicit working memory of the Verifier-adjudicated retrievals. Token layout. Each test image gives rise to a sequence of M+K=5+20=25M+K=5+20=25 tokens: [MEM1,…,MEM5]⏟retrieval memory∥[BEAM1,…,BEAM20]⏟Captioner beam. [\,MEM_1,…,MEM_5\,]_retrieval memory\ \|\ [\,BEAM_1,…,BEAM_20\,]_Captioner beam. (11) Each beam token BEAMyBEAM_y carries the same 55-d feature vector TriFuse would see, namely the LM log-probability, the Retrieval Encoder’s cosine, the Verifier’s matching score on (I,y)(I,y), the beam rank, and the length. Each memory token MEMjMEM_j carries an analogous 55-d feature vector for the j-th Verifier-verified retrieval, namely its Verifier matching score, retrieval rank, and length, with zero-padding to dimension 55. All features are z-normalised per image. A learned role embedding ∈ℝ2×dr ^2× d distinguishes memory tokens from beam tokens; it is added after the input projection so the model can tell the two token classes apart. Architecture. MemAttend is a 22-layer transformer encoder with 44 heads, hidden dimension dmodel=32d_model=32, feed-forward network (FFN) expansion factor 22, GELU activation [32], no dropout, and a per-token scalar score head: (0)=Wint+role(t),(ℓ+1)=TFLayer(ℓ)((ℓ)),h^(0)=W_in\,f_t+r_role(t), ^( +1)=TFLayer^( )(h^( )), (12) MemAttendψ(I,y)=Wouty(L)+bout,MemAttend_ψ(I,y)=W_out\,h^(L)_y+b_out, (13) where only beam tokens are scored at the output. Trainable parameters: |ψ|=17,377|ψ|=17,377, namely 192192 in the input projection, 6464 in the two role embeddings, 17,08817,088 across the two encoder layers, and 3333 in the output head. Added to TriFuse’s |ϕ|=113|φ|=113, the two learned heads hold 113+17,377=17,490113+17,377=17,490 trainable parameters in total. Every other component of the pipeline is frozen, so this is the complete trainable-parameter budget of the method. Training uses the same Borda-consensus pseudo-label as TriFuse, with listwise softmax cross-entropy, Adam [34] at learning rate 3×10−33×10^-3, and 200200 epochs. No paired image–caption supervision. Why MemAttend is not a strict superset of TriFuse. The transformer’s cross-attention lets a beam candidate’s score depend on the entire memory and beam context, not just its own three frozen-scorer signals. In practice, MemAttend’s higher capacity drives its training loss substantially below TriFuse’s, 1.271.27 versus 1.441.44 over the same Borda target, but its CIDEr lands 0.20.2 below TriFuse’s: the transformer fits the imperfect pseudo-label more tightly and overfits, while TriFuse’s tighter inductive bias generalises slightly better on its own. Because the two heads’ top-11 picks agree on only 7575–82%82\% of test images, they are partly orthogonal. We therefore use a fixed convex combination of their z-normalised scores, y^=argmaxy[ y\;=\; _y\, [ βz(TriFuseϕ(y)) β\,z\! (TriFuse_φ(f_y) ) (14) + + (1−β)z(MemAttendψ(y,ℳ))], (1-β)\,z\! (MemAttend_ψ(f_y,M) )\, ], which lifts every primary metric over either head alone. MemAttend contributes most where the consensus signal is ambiguous, that is on low-margin beam candidates; TriFuse contributes most where the consensus is sharp. We discuss the per-benchmark behaviour of β in our cross-domain experiments. IV Experimental Setup IV-A Benchmarks and metrics We evaluate on three benchmarks. COCO Karpathy test [8, 9] contains 5,0005,000 unique images and 25,01025,010 reference captions under the IFCap convention; it serves as the primary in-domain benchmark, and the Captioner was trained on the COCO Karpathy training caption corpus alone, with no paired image–caption supervision. Flickr30k Karpathy test [14, 15] contains 1,0001,000 images and 5,0005,000 references, and serves as a cross-domain benchmark for the COCO-trained pipeline. NoCaps val [16] contains 4,5004,500 images and tests novel-object transfer, split into in-, near-, and out-domain partitions. Metrics are BLEU-44 [36], METEOR [37], ROUGE-L [38], CIDEr [39], and SPICE [40] from the standard COCO Captions evaluation suite [41]; we report all metrics on a ×100× 100 scale throughout. SPICE requires Java 11; the runtime is documented in our release. IV-B Hardware and software Headline experiments run on a single NVIDIA L4 GPU with 2424 GB of memory on a GCP instance running CUDA 12 and PyTorch 2.4.12.4.1, matching IFCap’s pinned configuration. Cross-platform reproduction uses a single AMD Instinct MI355X with the gfx950 architecture, ROCm 7.2, and PyTorch 2.8.02.8.0+rocm6.46.4. The two platforms reproduce CIDEr and SPICE to the one-decimal precision reported throughout this paper; byte-exact JSON identity holds on the L4 but not on the MI355X, because ROCm/CUDA non-determinism is amplified by the gfx942-on-gfx950 kernel override. All frozen scorer checkpoints are exact pinned versions: the OpenCLIP [12] ViT-bigG/14 backbone pretrained on LAION-2B; the BLIP-ITM image–text matching head [13] fine-tuned on COCO captions, used as the Stage-2 Verifier; OpenAI CLIP ViT-B/32 [11] for the Captioner’s image encoder, so the released IFCap [7] mapping network is consumed unchanged; and GPT-2 small [2] with 124M parameters for the Captioner’s language model. The IFCap Captioner checkpoint is the released in-domain COCO model from the official IFCap repository, used unmodified. Random seed is 4242 throughout. IV-C Baseline reproduction Before measuring our pipeline we re-ran IFCap’s released checkpoint, as shown in Table I. Every COCO and NoCaps metric reproduces the published value exactly, and the COCO in-domain generated-caption file is byte-identical to the authors’ released JSON. Flickr30k CIDEr reproduces at 64.564.5 against the published 64.464.4, a 0.10.1 gap on the ×100× 100 scale that sits at the level of the determinism drift reported in Sec. V-H. Except where noted, comparisons in this paper are versus this reproduction rather than paper-quoted numbers. Benchmark Split Metric Paper Ours COCO Karp. in-domain CIDEr 108.0108.0 108.0108.0 COCO Karp. in-domain SPICE 20.320.3 20.320.3 Flickr30k Karp. in-domain CIDEr 64.464.4 64.564.5 NoCaps val overall CIDEr 74.074.0 74.074.0 NoCaps val in/near/out CIDEr 70.1/72.5/72.170.1/72.5/72.1 70.1/72.5/72.170.1/72.5/72.1 TABLE I: Reproduction of IFCap from the released checkpoint. All entries are ×100× 100, at the one-decimal precision at which IFCap publishes them. Every row matches the published value except Flickr30k CIDEr, which lands 0.10.1 higher, at the run-to-run determinism level quantified in Sec. V-H. IV-D Evaluation protocols for the learned Reranker The TriFuse and MemAttend Rerankers introduce a learned component over the Captioner’s beam. Because the choice of which images supply the training features for these heads affects how the result should be interpreted, we report three protocols on the same pipeline: 1. Training-free fixed fusion, the inductive baseline with no learned step. The Stage-44 ranker is a fixed-α linear z-mix of LM log-prob and ViT-bigG/14 CLIPScore with α=0.48α=0.48, selected by Borda-consensus pseudo-label loss on a held-out beam dump and never on reference captions. No parameters are fit on test-set features; this is our strictest non-transductive baseline. 2. Inductive learned Reranker, headline protocol. TriFuse and MemAttend are trained on beam-2020 features extracted from the COCO Karpathy validation split, which contains 5,0005,000 images disjoint from the test split, using the same Borda-consensus pseudo-label, then applied unchanged to the Karpathy test beam-2020 features. No test-set features enter training. The choice of α, β, hidden width, and epoch count is fixed once on validation by pseudo-label loss rather than by reference CIDEr. 3. Transductive learned Reranker, upper-bound protocol. TriFuse and MemAttend are trained on beam-2020 features extracted from the Karpathy test split, with the same Borda-consensus pseudo-label, then applied to the same images. No reference captions are used as labels; the training signal is the frozen scorers’ rankings on the Captioner’s own beam, computed at inference time. This is a form of unsupervised test-time adaptation: parameters are fit to test inputs but no test labels are seen. We report this only as an upper bound; the inductive protocol (item 2) is our headline. IV-E Hyperparameter selection criterion The learned-component hyperparameters, namely the ensemble weight β=0.75β=0.75, the fixed-fusion weight α=0.48α=0.48, the TriFuse hidden width h=8h=8, the MemAttend hidden dimension dmodel=32d_model=32 with 22 encoder layers, and the BLIP top-k=5k=5, were selected by minimising Borda-consensus pseudo-label loss on a held-out beam dump, not by reference-metric CIDEr. The entity-filter threshold K=3K=3 was fixed algebraically rather than swept: the BLIP rerank shrinks the entity pool from L=9L=9 to L=5L=5, and K=3K=3 admits 3/5=60%3/5=60\%, the closest available integer threshold to IFCap’s original 5/9≈56%5/9≈ 56\%. We do not perform random restart sweeps; the seed is fixed at 4242. IV-F Computational budget A one-time preprocessing step encodes all N=566,747N=566,747 COCO Karpathy training captions with OpenCLIP ViT-bigG/14 and caches the L2-normalised 12801280-d embeddings on disk at about 33 GB. The encode cost is approximately 55 GPU-hours on a single NVIDIA L4 with 2424 GB of memory, batch 128128, FP3232; wall-clock time on faster GPUs such as the A100100, H100100, or MI300300 is several times lower. The cache is reusable across all subsequent runs of the pipeline. Per test image, the additional cost over vanilla IFCap inference breaks down as follows. Stage 1, the ViT-bigG/14 image encode and top-99 cosine over the corpus, costs approximately 0.050.05 s. Stage 2, the BLIP-ITM rerank pass over 99 retrievals, costs around 1.01.0 s. Stage 4 input features, namely one ViT-bigG/14 image encode, 2020 text encodes, and 2020 BLIP-ITM forwards, cost roughly 1.41.4 s. TriFuse’s own forward is sub-millisecond. The total is about 2.52.5 s per image on top of vanilla IFCap inference, on a single GPU, with no Captioner retraining. TriFuse training is a one-time fit of under one minute on the 55K-image beam-2020 feature dump; MemAttend training takes approximately 55 minutes over 200200 epochs. V Experiments V-A Implementation details We use the released IFCap Captioner without retraining. The only inference-time changes are the following. (i) One-time, we encode all 566,747566,747 COCO Karpathy training captions with the frozen Stage-1 Retrieval Encoder and cache the L2-normalised 12801280-d embeddings. (i) For each test image we encode it with the Retrieval Encoder and retrieve the top-99 training captions by cosine. (i) The frozen Verifier re-ranks the 99 candidates and keeps the top-55. (iv) The unmodified IFCap Captioner runs with entity-filter K=3K=3 and beam width 2020 to dump a 2020-wide beam. (v) For each beam candidate y we compute three per-candidate features, the LM log-probability ℓy _y, the Retrieval Encoder’s cosine sRys_R^\,y, and the Verifier’s matching score sV(I,y)s_V(I,y), and z-normalise per image. (vi) TriFuse is an MLP of 113113 parameters with two hidden layers of width h=8h=8 and GELU [32], trained for 100100 epochs with Adam [34] at learning rate 10−210^-2 using listwise softmax cross-entropy against the Borda-consensus pseudo-label; MemAttend is a 22-layer transformer encoder of 17.417.4K parameters, trained for 200200 epochs with Adam [34] at learning rate 3×10−33×10^-3 against the same target. (vii) We take the convex combination βz(TriFuse)+(1−β)z(MemAttend)β\,z(TriFuse)+(1-β)\,z(MemAttend) at β=0.75β=0.75 and use its argmax per image as the final caption. No Captioner retraining and no weight modification of any frozen model. Inference cost. The one-time corpus encode is approximately 55 GPU-hours on a single NVIDIA L4 (matching the budget reported in Sec. IV). Per test image, the additional cost over vanilla IFCap is about 0.050.05 s for Retrieval Encoder lookup, approximately 1.01.0 s for Verifier rerank, around 1.41.4 s for beam-feature extraction, and sub-millisecond for the TriFuse and MemAttend forwards, for a total of roughly 2.52.5 s per image. TriFuse training is a one-time fit of under one minute; MemAttend training takes about 55 minutes. Both heads are fit on the 5,0005,000-image beam-2020 feature dump. Hardware. See Sec. IV for the canonical hardware specification: headline experiments use a single NVIDIA L4 (2424 GB, CUDA 12, PyTorch 2.4.12.4.1, IFCap’s pinned configuration); cross-platform reproduction uses a single AMD Instinct MI355X (gfx950, ROCm 7.27.2, PyTorch 2.8.02.8.0+rocm6.46.4). All reported numbers reproduce to the one-decimal CIDEr/SPICE precision used throughout on either platform. V-B Datasets and metrics Datasets, metrics, reproduction, and protocols follow Sec. IV. Reference-free grounding is additionally measured by ViT-bigG/14 CLIPScore, and hallucination by CHAIR [42] computed against the 55 reference captions per test image. V-C Baselines We compare against three families: (i) strict text-only ZIC, including ZeroCap [19], MAGIC [20], CapDec [3], DeCap [4], ViECap [5], MeaCap [6], and IFCap [7]; (i) strict ZIC with training-time augmentation by synthetic images, namely SynTIC [24], PCM-Net [26], and NES [10]; and (i) paired-supervised models reported for context, namely ClipCap [1], BLIP [13], and BLIP-2 [21]. We also include a foundation multimodal large language model (MLLM), Qwen2.5-VL-3B [43], fed our retrievals as in-context exemplars. Following template practice, we keep these regimes in separate table blocks. V-D Main results: COCO Karpathy Table I reports the full metric set, with all comparators grouped by training regime. Our framework reaches CIDEr 117.6117.6 / SPICE 21.921.9, the best strict-text-only result on this benchmark to our knowledge, surpassing the previous strict-ZIC state of the art, IFCap (CIDEr 108.0108.0), by +9.6+9.6 and the strongest synthetic-image-augmented method NES (CIDEr 109.9109.9) by +7.7+7.7 without any Captioner retraining. SPICE 21.921.9 further surpasses ClipCap’s 21.121.1, a paired-supervised baseline trained on full COCO image–caption pairs. Every reference-based metric improves simultaneously over IFCap; no metric is traded for another. The TriFuse+MemAttend ensemble contributes +1.8+1.8 CIDEr over a fixed-α z-mix that has access to the same input scorers; these are the only learned components in the pipeline. Figure 5: CIDEr on COCO Karpathy test, grouped by training regime. Progress in the captioner-text-only ZIC regime has plateaued near IFCap’s 108.0108.0; the only entry to surpass it, NES at 109.9109.9, does so by stepping outside the regime via Stable-Diffusion-generated synthetic images. Our framework reaches 117.6117.6 without retraining the Captioner and without any paired image–caption supervision, exceeding even the paired-supervised baseline ClipCap (113.1113.1). Numbers from Table I. Method Venue Encoder Train data B-4 MET R-L CIDEr SPICE Captioner-text-only ZIC; frozen paired-pretrained scorers allowed; no retraining-time image augmentation: ZeroCap [19] CVPR’22 ViT-B/32 GPT-2 prior — 15.4 — 34.5 9.2 MAGIC [20] arXiv’22 ViT-B/32 text only 12.9 17.4 — 49.3 11.3 CapDec [3] EMNLP’22 ViT-B/32 text only 26.4 25.1 — 91.8 11.9 DeCap [4] ICLR’23 ViT-B/32 text only 24.7 25.0 — 91.2 18.7 ViECap [5] ICCV’23 ViT-B/32 text only 27.2 24.8 — 92.9 18.2 MeaCap [6] CVPR’24 ViT-B/32 text only 27.2 25.3 — 95.4 19.0 IFCap [7] EMNLP’24 ViT-B/32 text only 30.7 26.6 53.8 108.0 20.3 Captioner-text-only ZIC with synthetic-image training augmentation: SynTIC [24] AAAI’24 ViT-B/32 text + SD synth img 29.9 25.8 53.2 101.1 19.3 PCM-Net [26] ECCV’24 ViT-B/32 text + SD synth img 31.5 25.9 53.9 103.8 19.7 NES [10] AAAI’26 ViT-B/32 text + SD synth img 30.8 26.8 — 109.9 20.6 Supervised image captioning (paired image–text training; reported for context, not regime-comparable): ClipCap [1] arXiv’21 ViT-B/32 paired 33.5 27.5 — 113.1 21.1 BLIP [13] ICML’22 ViT-B/16 14M paired 39.7 — — 133.3 — BLIP-2 [21] ICML’23 ViT-G 129M paired 43.7 — — 145.8 — Qwen2.5-VL-3B [43] + our retrievals arXiv’25 Qwen-ViT-L MLLM paired pretrain 21.8 28.1 50.6 77.1 23.3 Ours: captioner-text-only ZIC under frozen paired-pretrained scorers; inference-time intervention on IFCap’s released checkpoint: Ours (ViT-bigG/14 in + BLIP Verifier + TriFuse++MemAttend out, inductive) — ViT-bigG/14 captioner text only; frozen paired scorers 32.832.8 27.927.9 55.555.5 117.6117.6 21.921.9 TABLE I: COCO Karpathy test. CIDEr and SPICE reported ×100× 100 following prior convention. Our 117.6117.6 CIDEr is the best captioner-text-only ZIC result on this benchmark under frozen paired-pretrained scorers, surpassing the previous strict-ZIC state of the art, IFCap, by +9.6+9.6 and the strongest synthetic-image-augmented method NES by +7.7+7.7 without any Captioner retraining. The “Ours” row reports the inductive protocol (Rerankers fit on the disjoint COCO Karpathy validation beam dump and applied frozen to test); the transductive variant differs by at most 0.10.1 on any metric, and not at all on CIDEr, and is reported in Table IV; the only learned components are TriFuse (113113 parameters, <1<1 min fit) and MemAttend (17.417.4K parameters, ≈5≈5 min fit), both fit self-supervised by Borda-consensus distillation across three frozen scorers. SPICE 21.921.9 surpasses ClipCap (21.121.1), a paired-supervised baseline. The IFCap row reports our reproduction of the released checkpoint (Table I), which is what every comparison in this paper is made against (Sec. IV), so it also carries a ROUGE-L value that the original paper does not report. In the train-data column, SD denotes Stable Diffusion. “Encoder” refers to the dominant frozen image–text scorer used at retrieval and output; the Captioner itself is IFCap’s released ViT-B/32 checkpoint, unmodified, in every row of this paper. Supervised methods are listed for context to bound how far the strict-ZIC regime is from the paired-data ceiling—not as direct comparators. Figure 6: Qualitative comparison on COCO Karpathy test. Eight test images on which Ours (blue) corrects a salient mistake in the IFCap baseline (red). IFCap’s failures are typical of single-encoder, single-checkpoint retrieval: under-counting (top-left, “a” snowboarder vs. two), wrong subject (banana vendors → “smiling woman”), missing relations (pitcher’s mound mis-localised as “running across a field”), and category confusion (giraffe + zebra → “giraffe and other animals in a zoo”). Ours produces captions that match the ground-truth semantics more closely, consistent with the −41%-41\% CHAIRs reduction in Table I. Hallucination (CHAIR proxy). The +0.083+0.083 CLIPScore lift could in principle come either from genuinely more accurate captions or from style preferences of the Retrieval Encoder substrate at the output checkpoint. We disambiguate with a closed-vocabulary hallucination metric (Table I), the reference-proxy variant of CHAIR [42]: for each prediction, count MS-COCO-class objects mentioned in the predicted caption that do not appear in any of the 55 reference captions for that image (since instance segmentation annotations are not used by our pipeline at inference). Both CHAIRs, which fires when a sentence has at least one hallucinated object, and CHAIRi, the instance-level hallucinated-to-mentioned ratio, drop by around 41%41\% relative against the IFCap baseline. The largest per-class drops are on classes whose presence is hardest to infer from ViT-B/32 retrieval alone: person (48→1748\!→\!17), sink (45→1445\!→\!14), and TV (28→≤1028\!→\!≤ 10). The reference-proxy reduction is consistent with our pipeline writing fewer COCO-class nouns that no reference annotator mentioned; whether the gain holds against instance-mask CHAIR remains to be verified. IFCap baseline Ours (TriFuse) rel. Δ CHAIRs (↓ , sent-level) 13.28%13.28\% 7.84%7.84\% −41%-41\% CHAIRi (↓ , inst-level) 9.61%9.61\% 5.61%5.61\% −42%-42\% # objects mentioned (55K imgs) 74747474 72577257 −3%-3\% # hallucinated objects 718718 407 −%-43\% TABLE I: Reference-proxy CHAIR hallucination metrics. We use the 55 COCO references for each test image as the ground-truth-object proxy in lieu of instance-segmentation annotations: a standard variant when instance masks are unavailable, conservative since an object truly in the image but absent from all references is marked hallucinated. The conservatism is symmetric across both rows. Our pipeline drops both sentence- and instance-level reference-proxy CHAIR rates by approximately 41%41\%. Reference-proxy CHAIR is not equivalent to instance-mask CHAIR; an object truly in the image but absent from all five references is counted as hallucinated, and the gap should be read as “fewer COCO-class nouns that no reference annotator wrote down” rather than “fewer objects truly absent from the image”. We do not therefore claim objects actually present; an instance-mask CHAIR run is queued (Sec. VI). V-E Evaluation-protocol comparison and Verifier substrate Following Sec. IV-D we report all three protocols. The inductive variant, in which TriFuse++MemAttend are trained on the disjoint COCO Karpathy validation beam dump and applied frozen to the test split, matches the transductive default to within determinism noise on every primary metric (Table IV). We therefore take the inductive variant as the headline, since it does not rely on parametric adaptation to test inputs; the transductive number is reported as a strict upper bound. The training-free fixed-α z-mix isolates the contribution of the architectural intervention (stronger Retrieval Encoder ++ Verifier rerank ++ threshold rescale) from the learned step. Protocol B-4 MET R-L CIDEr SPICE Training-free fixed-α 32.332.3 27.727.7 55.255.2 115.8115.8 21.621.6 Inductive (headline) 32.832.8 27.927.9 55.555.5 117.6117.6 21.921.9 Transductive (upper bound) 32.932.9 27.927.9 55.655.6 117.6117.6 22.022.0 TABLE IV: Protocol comparison on COCO Karpathy, all metrics ×100× 100. Training-free uses no learned head. Inductive trains TriFuse++MemAttend on the disjoint Karpathy validation beam-2020 dump (no test features) and applies the heads frozen to test. Transductive trains the heads on the test beam-2020 dump using only frozen-scorer pseudo-labels (no reference captions). The inductive and transductive variants are identical on CIDEr and differ by at most 0.10.1 on any other metric, at or below the determinism floor of Sec. V-H; we headline the inductive result. Verifier substrate. Table V swaps the Verifier checkpoint while leaving the rest of the pipeline (Retrieval Encoder, Captioner, Reranker) unchanged. Replacing the COCO-fine-tuned BLIP-ITM Verifier with the Flickr30k-fine-tuned variant of the same architecture lifts CIDEr to 114.0114.0, +6.0+6.0 over IFCap; the residual +3.6+3.6 CIDEr between 114.0114.0 and the 117.6117.6 headline is conditional on COCO Verifier tuning, while the larger share of the lift is attributable to the architectural intervention. The Flickr30k-Verifier row is therefore the architecturally cleaner comparison for COCO, since its training distribution does not overlap COCO. All rows use the inductive headline protocol; exact checkpoints are listed in the caption. Verifier checkpoint B-4 MET R-L CIDEr SPICE IFCap baseline, no Verifier 30.730.7 26.626.6 53.853.8 108.0108.0 20.320.3 BLIP-ITM Flickr30k-fine-tuned [13] 31.731.7 27.227.2 54.754.7 114.0114.0 21.121.1 BLIP-ITM COCO-fine-tuned (headline) 32.832.8 27.927.9 55.555.5 117.6117.6 21.921.9 TABLE V: Verifier-substrate ablation, all metrics ×100× 100. Stage-2 Cross-Attention Verifier swapped between two BLIP-ITM checkpoints; Stages 1, 3, 4 and all hyperparameters are held fixed at the inductive headline protocol. The COCO and Flickr30k-fine-tuned variants of the BLIP-ITM Large head are both drawn from the official Salesforce LAVIS release [44]; the Flickr30k variant’s training distribution does not overlap COCO. The +6.0+6.0 CIDEr lift achieved without any COCO Verifier exposure indicates that the architectural intervention is the larger contributor; a residual +3.6+3.6 CIDEr is conditional on COCO fine-tuning of the Verifier. V-F Cross-domain transfer: Flickr30k and NoCaps Figure 7: Cross-domain CIDEr lifts from the same inference-only recipe, no Captioner retraining: +8.1+8.1 on Flickr30k Karpathy and +5.7+5.7 on NoCaps overall, with consistent gains across in-, near-, and out-domain NoCaps splits. Out-domain, where COCO-trained Captioners typically degrade fastest, gains +5.9+5.9, matching the near-domain lift and exceeding the +5.4+5.4 on in-domain. Numbers from Table VI. The same inference-only recipe transfers off-COCO without Captioner retraining. Per-benchmark transductive Rerankers (TriFuse and MemAttend re-fit on the target benchmark’s beam-2020 dump, same Borda-consensus pseudo-label) give the following gains over IFCap. Benchmark IFCap Ours Δ COCO Karpathy test CIDEr / SPICE 108.0108.0 / 20.320.3 117.6117.6 / 21.921.9 +9.6+9.6 Flickr30k Karpathy test CIDEr / SPICE 64.464.4 / 17.017.0 72.572.5 / 18.018.0 +8.1+8.1 NoCaps val (COCO-trained) in-domain CIDEr / SPICE 70.170.1 / 11.211.2 75.575.5 / 11.611.6 +5.4+5.4 near-domain CIDEr / SPICE 72.572.5 / 10.910.9 78.478.4 / 11.411.4 +5.9+5.9 out-domain CIDEr / SPICE 72.172.1 / 9.6 09.6 78.078.0 / 9.9 09.9 +5.9+5.9 overall CIDEr / SPICE 74.074.0 / 10.510.5 79.779.7 / 10.910.9 +5.7+5.7 TABLE VI: Cross-domain transfer of the same recipe (benchmark-transductive). The Stage 11 + Stage 22 + Stage 44 stack lifts CIDEr by +5+5 to +10+10 on every benchmark. The Flickr30k IFCap baseline is the published 64.464.4 rather than our 64.564.5 reproduction (Table I); measured against the reproduction the lift is +8.0+8.0, a 0.10.1 difference at the determinism floor of Sec. V-H. No retraining or fine-tuning of the Captioner. Rerankers are re-fit per benchmark on that benchmark’s beam-2020 dump using only frozen-scorer Borda-consensus pseudo-labels (no paired image–caption supervision is consumed); this is benchmark-transductive adaptation, not fully frozen cross-domain transfer. NoCaps overall CIDEr exceeds the maximum of the three subset CIDErs because CIDEr is a per-corpus term-frequency-inverse-document-frequency (TF-IDF) weighted metric, not a weighted average of subsets; the same pattern 74.0>max(70.1,72.5,72.1)74.0> (70.1,72.5,72.1) is present in the IFCap baseline and reflects how the NoCaps evaluator aggregates references. Figure 8: Qualitative comparison on cross-domain benchmarks: Flickr30k (top row) and NoCaps (bottom row). Same colour scheme as Fig. 6. The same inference-only recipe lifts caption quality off-COCO: corrected counts (Flickr30k row, “three young boys”, not “a boy and three children”), corrected actions (“playing tug-of-war” rather than “standing around a stick”), and corrected NoCaps out-of-domain entities (“library filled with bookshelves” rather than “books stacked on each other”; “mother and baby kangaroo grazing” rather than “two kangaroos in a grassy area”). The Captioner is unchanged across all rows. V-G Factorial ablation Table VII decomposes the +9.6+9.6 CIDEr lift across the three architectural changes: stronger Retrieval Encoder at Stage 11, Verifier rerank at Stage 22, and learned Beam Reranker at Stage 44. Configuration B-4 MET R-L CIDEr SPICE 1 IFCap baseline (ViT-B/32, K=5K=5) 30.730.7 26.626.6 53.853.8 108.0108.0 20.320.3 2 ViT-bigG/14 input (drop-in, K=5K=5) 31.531.5 26.926.9 54.554.5 111.6111.6 20.820.8 3 ViT-B/32, K=3K=3 only (neg. ctrl.) 28.128.1 26.826.8 53.053.0 103.7103.7 21.221.2 4 ViT-B/32, BLIP only (neg. ctrl.) 30.630.6 26.326.3 53.553.5 105.7105.7 19.819.8 5 ViT-B/32, BLIP ++ K=3K=3 31.131.1 27.027.0 54.354.3 110.2110.2 20.820.8 6 ViT-bigG/14, BLIP ++ K=3K=3 32.132.1 27.127.1 54.854.8 113.1113.1 21.121.1 7 ++ ViT-bigG/14 out (linear z-mix, α=0.48α=0.48) 32.332.3 27.727.7 55.255.2 115.8115.8 21.621.6 8 ++ TriFuse++MemAttend ensemble (Ours, inductive) 32.832.8 27.927.9 55.555.5 117.6117.6 21.921.9 TABLE VII: Factorial ablation on top of IFCap’s released Captioner, no retraining; all metrics ×100× 100. Row 2 isolates that swapping ViT-B/32 → ViT-bigG/14 at retrieval input alone is the largest single-component lift (+3.6+3.6 CIDEr). Rows 3, 4 isolate that BLIP rerank and entity-threshold rescale are mutually enabling: each alone falls below the baseline, and only their combination in row 5 exceeds it. Rows 5 vs. 6 show the input-endpoint substrate effect: +2.9+2.9 CIDEr from ViT-B/32 → ViT-bigG/14 at retrieval. Row 7 adds ViT-bigG/14 output rerank with a fixed-α linear mix (+2.7+2.7). Row 8 (Ours) replaces the linear mix with the learned TriFuse++MemAttend ensemble, lifting every metric simultaneously (+1.8+1.8 CIDEr, +0.3+0.3 SPICE). V-H Robustness checks Determinism. Across reruns of the same configuration on identical hardware we observe around 0.10.1 CIDEr of drift on the ×100× 100 scale, attributable to GPU non-determinism in the GPT-2 forward. The +9.6+9.6 CIDEr lift over baseline is roughly 96×96× this floor. Verification through IFCap’s official script. We verify rows 5 and 6 of Table VII (110.2110.2 and 113.1113.1) by running IFCap’s own evaluation script on our reranked retrieval pools at K=3K=3. Both numbers reproduce exactly. Row 7 is the row-66 Captioner output with a downstream ViT-bigG/14 rerank applied to the dumped 2020-wide beam at α=0.48α=0.48. Row 8 (Ours, TriFuse) operates on the same beam dump; only the final scoring head changes, and the Captioner outputs are byte-identical to row 7. No paired data, no reference captions in TriFuse/MemAttend training. Under both protocols the training signal is the frozen scorers’ rankings on the Captioner’s own beam, never the COCO reference captions. The Borda-consensus pseudo-label is a deterministic function of (ℓy,sRy,sV(I,y))\( _y,s_R^\,y,s_V(I,y))\ over each image’s beam, and the Captioner’s beam itself is generated from the test (or validation) image alone. Under the inductive headline protocol the training loop iterates over the disjoint COCO Karpathy validation 5,0005,000-image beam dump; under the transductive upper-bound protocol it iterates over the test 5,0005,000-image beam dump. References enter only at the final evaluation step in either case. Retrieving from training-side captions is not leakage. Our pipeline retrieves at inference from trainC_train (566,747566,747 captions) drawn from the COCO Karpathy training split. We address the natural concern with five points. (i) The Captioner is trained on trainC_train as text only; it never sees any image during training. (i) Test images and the captions in trainC_train come from disjoint image splits; the 566566K captions correspond to the 113113K Karpathy training images, which do not overlap with the 5,0005,000 test images. (i) The approximately 2525K reference captions for the test images (used only by the evaluation toolkit) are held out and never enter the retrieval index. (iv) Our pipeline does not output retrieved captions verbatim; the Captioner generates new text conditioned on the retrievals as soft prefix and hard prompt, and the final answer is a beam decode from GPT-2, not a retrieval look-up. (v) Retrieve-at-inference is the defining structure of strict-ZIC: ViECap, MeaCap, IFCap, and NES all retrieve from training-side captions at inference. The remaining concern, that the corpus and the test references share writers’ style, is directly answered by our reference-free CLIPScore lift (+0.083+0.083) and CHAIR-proxy reduction (−41%-41\% relative): by metrics that do not consume references, our captions are still measurably better-grounded. CLIP–BLIP rank correlation. We measure the rank correlation between CLIP cosine and BLIP-ITM softmax over the top-99 retrievals across all 5,0005,000 test images. Mean Spearman ρ=0.41±0.18ρ=0.41± 0.18 (std across images). The two scorers are positively but loosely correlated, supporting the architectural claim in Sec. I-B: BLIP-ITM provides non-redundant signal over CLIP cosine, so cascading them is informative rather than tautological. Negative result: contrastive decoding does not lift further. A natural follow-up is to use BLIP’s bottom-44 captions as negative evidence in contrastive decoding [45]: ℓcd=(1+λ)ℓpos−λℓneg _cd=(1+λ) _pos-λ\, _neg. We sweep λ∈0,0.1,0.3λ∈\0,0.1,0.3\ and observe monotone CIDEr decay across 110.2110.2, 109.6109.6, and 108.4108.4. The negative branch is uniformly harmful at this scale. We attribute this to the negative pool being already-low-quality CLIP retrievals: they carry weak distributional support for the wrong caption, so subtracting their logits does not sharpen the next-token distribution toward the truth, and instead noises the LM with semantically related distractors. We report this so future work knows to pick negatives from a stronger source if it wants to chain contrastive decoding onto this pipeline. VI Discussion Why alignment-checkpoint scoring works. The pipeline architecture introduced here can be read as a simple principle: when a retrieval-augmented generator scores image–text alignment at a single point and lets the decoder fill in the rest by language-model probability, the alignment signal that survives to the output is bottlenecked by whichever scorer was used at that single point. Adding a second, partly orthogonal frozen scorer at the output beam recovers signal that was lost between retrieval and decoding. The same observation explains why which scorers are placed at the two checkpoints matters monotonically as established in the alignment-substrate sweep: scaling-law-stronger CLIP variants help at both ends in additive magnitudes, and replacing CLIP with a cross-attention matcher (BLIP-ITM) provides information that no dual-encoder of any scale can supply. The architectural claim is therefore independent of any particular scorer choice; it predicts that future stronger frozen scorers will move the pipeline further. What the learned Reranker adds, and what it does not. The training-free fixed-fusion baseline reaches CIDEr 115.8115.8 (+7.8+7.8 over IFCap), so most of the +9.6+9.6 CIDEr lift over IFCap is explained by “where alignment is scored” rather than by “what the learned head learns”. The TriFuse+MemAttend ensemble adds a further +1.8+1.8 CIDEr (115.8→117.6115.8→ 117.6). Our headline reports the inductive variant, in which the Rerankers are trained on the disjoint COCO Karpathy validation beam dump and applied frozen to test; the transductive variant differs by at most 0.10.1 on any primary metric, and not at all on CIDEr, and is reported only as an upper bound. The learned Reranker therefore contributes its +1.8+1.8 CIDEr without parametric adaptation to the test inputs. The training-free 115.8115.8 remains a lower bound on what the architectural intervention alone buys without any learned head whatsoever. Connection to retrieval-augmented generation more broadly. Alignment-checkpoint scoring is not specific to captioning. Any retrieval-augmented generator (open-domain question answering (QA) with a retriever and a generator, dialogue systems with a knowledge corpus, code completion with a retrieval index) scores relevance at retrieval and then trusts the generator’s autoregressive beam. The pattern proposed here (a second, cross-attention-capable scorer between retrieval and generation, plus a self-supervised learned Reranker at the generation output trained by frozen-scorer consensus) transfers directly. We leave a study of the technique on open-domain QA to future work. Limitations. • Retrieval-corpus exposure. The retrieval index reads the COCO training caption corpus, whose writers’ style and object distribution overlap with the COCO test references. Reference-free CLIPScore at the output checkpoint and the CHAIR-proxy hallucination check both improve, indicating the gain is not just COCO-style mimicry; a retrieval-corpus swap to a non-COCO source such as C33M [46] or LAION-COCO [31] would be the most direct test, but it remains to be performed. • Verifier provenance. The headline uses the COCO-fine-tuned variant of the Verifier [13]. The Verifier-substrate ablation in our Verifier-substrate analysis shows the architectural intervention still lifts CIDEr by +6.0+6.0 over IFCap when we swap in the Flickr30k-fine-tuned variant of the Verifier, but the residual +3.6+3.6 CIDEr from COCO-tuning persists. A broader Verifier sweep over BLIP-2 [21], SigLIP [47], and OpenCLIP cross-encoder variants would map the dependence more completely. • NoCaps cross-benchmark inductive transfer. Table VI reports cross-benchmark inductive transfer on Flickr30k but only benchmark-transductive on NoCaps; the NoCaps beam-feature dump used for our earlier cross-domain run was not re-produced on the hardware used for this round of experiments. The Flickr30k cross-benchmark-inductive result (matching the per-benchmark refit within 0.10.1 CIDEr) strongly suggests NoCaps would behave the same way; a NoCaps cross-benchmark inductive row is the most natural follow-up. • Captioner family. All experiments are on top of IFCap’s released checkpoint. The architectural prediction is that any strict-ZIC Captioner that emits a beam can be capped with an alignment-scoring stack; we have not yet tested this on a Captioner from outside the ClipCap-prefix family. • Hyperparameter selection. α,β,K,hα,β,K,h, layer counts were selected by Borda-consensus pseudo-label loss (no reference-metric tuning), as stated in Sec. IV-E. We cannot rule out that a reference-tuned selection would yield a slightly higher CIDEr. • Single-seed evaluation. We report a single seed (4242). Determinism noise on identical hardware is about 0.10.1 CIDEr on the ×100× 100 scale, measured across reruns. The inductive and transductive protocols sit at or below that floor, being identical on CIDEr and differing by at most 0.10.1 on any other metric in Table IV, which is why we treat them as indistinguishable rather than ranked. The differences that carry the paper’s claims are far above the floor, namely +7.8+7.8 CIDEr for the architectural intervention and +1.8+1.8 for the learned Rerankers. A multi-seed Cohen’s d over TriFuse/MemAttend training initialisations would nonetheless tighten the statistical claim. • Evaluation-metric coverage. The headline numbers use the n-gram-based COCO suite (BLEU-44, METEOR, ROUGE-L, CIDEr, SPICE). These metrics are known to under-weight semantic accuracy and over-weight surface phrasing, so a high CIDEr does not by itself certify that a system describes a scene faithfully. A complete 2026 diagnostic should additionally report a human-aligned learned metric such as Polos [48], an open-vocabulary hallucination metric such as ALOHa [49], and an exhaustive-description score such as CAPability [50]. We use frozen-third-party CLIPScore and a CHAIR-proxy hallucination probe as lightweight grounding checks; the broader sweep, including reward-corrected fine-tuning baselines [51], remains future work. VII Conclusion We argued that the captioner-text-only zero-shot image captioning plateau (ViECap 92.9→92.9→ MeaCap 95.4→95.4→ IFCap 108.0→108.0→ NES 109.9109.9 on COCO Karpathy CIDEr) is not a fundamental ceiling but an architectural artefact: every prior method scores image–text alignment with a single dual-encoder, namely ViT-B/32, at the retrieval input and leaves the Captioner’s beam structurally bottlenecked by that scorer. We addressed this with two changes. First, we strengthen scoring at both alignment checkpoints, swapping ViT-B/32 for an OpenCLIP ViT-bigG/14 at retrieval and inserting a frozen BLIP-ITM Cross-Attention Verifier in between. Second, we introduced two complementary self-supervised Beam Rerankers: TriFuse, a 113113-parameter MLP, and MemAttend, a 17.417.4K-parameter memory-augmented transformer encoder. Both are trained by Borda-consensus distillation across three frozen scorers, with no paired image–caption supervision and no reference captions in the loop. Our headline reports the inductive variant, in which the Rerankers are trained on the disjoint COCO Karpathy validation beam dump and applied frozen to test; a transductive variant fit on test-image beam features (no test labels) matches the inductive numbers to within determinism noise and is reported only as an upper bound. The training-free fixed-fusion baseline reaches 115.8115.8 CIDEr without any learned step, isolating the architectural contribution. The full pipeline reaches CIDEr 117.6117.6 / SPICE 21.921.9 on COCO Karpathy under the inductive headline protocol: +9.6+9.6 CIDEr over IFCap and +7.7+7.7 over the synthetic-image-augmented NES. It also exceeds the paired-supervised ClipCap baseline on CIDEr (117.6117.6 against 113.1113.1), METEOR (27.927.9 against 27.527.5), and SPICE (21.921.9 against 21.121.1), while trailing ClipCap by 0.70.7 on BLEU-44 (32.832.8 against 33.533.5). The same recipe transfers off-COCO without Captioner retraining: +8.1+8.1 CIDEr on Flickr30k Karpathy and +5.7+5.7 on NoCaps overall, with consistent in/near/out-domain gains. The architectural insight, namely alignment scoring at multiple checkpoints rather than only at retrieval, and the training paradigm, namely consensus distillation across heterogeneous frozen scorers without paired labels, are portable beyond the captioner-text-only ZIC regime: any retrieval-augmented generation pipeline that scores grounding once and decodes elsewhere can be opened up at both ends and capped with a self-supervised learned Reranker. Acknowledgment This work was supported by the University of Economics HCMC, Vietnam. References [1] R. Mokady, A. Hertz, and A. H. Bermano, “Clipcap: Clip prefix for image captioning,” arXiv preprint arXiv:2111.09734, 2021. [2] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019. [3] D. Nukrai, R. Mokady, and A. Globerson, “Text-only training for image captioning using noise-injected clip,” in Findings of the Association for Computational Linguistics: EMNLP 2022, 2022, p. 4055–4063. [4] W. Li, L. Zhu, L. Wen, and Y. Yang, “Decap: Decoding CLIP latents for zero-shot captioning via text-only training,” in The Eleventh International Conference on Learning Representations, 2023. [5] J. Fei, T. Wang, J. Zhang, Z. He, C. Wang, and F. Zheng, “Transferable decoding with visual entities for zero-shot image captioning,” in Proceedings of the IEEE/CVF international conference on computer vision, 2023, p. 3136–3146. [6] Z. Zeng, Y. Xie, H. Zhang, C. Chen, B. Chen, and Z. Wang, “Meacap: Memory-augmented zero-shot image captioning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, p. 14 100–14 110. [7] S. Lee, S.-W. Kim, T. Kim, and D.-J. Kim, “Ifcap: Image-like retrieval and frequency-based entity filtering for zero-shot captioning,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, p. 20 715–20 727. [8] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision. Springer, 2014, p. 740–755. [9] A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, p. 3128–3137. [10] Z. Lu, H. Xu, B. Liu, and K. Wang, “Negative entity suppression for zero-shot captioning with synthetic images,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 9, 2026, p. 7636–7643. [11] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PmLR, 2021, p. 8748–8763. [12] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev, “Reproducible scaling laws for contrastive language-image learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, p. 2818–2829. [13] J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International conference on machine learning. PMLR, 2022, p. 12 888–12 900. [14] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,” Transactions of the association for computational linguistics, vol. 2, p. 67–78, 2014. [15] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik, “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,” in Proceedings of the IEEE international conference on computer vision, 2015, p. 2641–2649. [16] H. Agrawal, K. Desai, Y. Wang, X. Chen, R. Jain, M. Johnson, D. Batra, D. Parikh, S. Lee, and P. Anderson, “Nocaps: Novel object captioning at scale,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, p. 8948–8957. [17] B. Eikema and W. Aziz, “Sampling-based approximations to minimum bayes risk decoding for neural machine translation,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, p. 10 978–10 993. [18] Z. Cao, T. Qin, T.-Y. Liu, M.-F. Tsai, and H. Li, “Learning to rank: from pairwise approach to listwise approach,” in Proceedings of the 24th international conference on Machine learning, 2007, p. 129–136. [19] Y. Tewel, Y. Shalev, I. Schwartz, and L. Wolf, “Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, p. 17 918–17 928. [20] Y. Su, T. Lan, Y. Liu, F. Liu, D. Yogatama, Y. Wang, L. Kong, and N. Collier, “Language models can see: Plugging visual controls in text generation,” arXiv preprint arXiv:2205.02655, 2022. [21] J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning. PMLR, 2023, p. 19 730–19 742. [22] W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning,” Advances in neural information processing systems, vol. 36, p. 49 250–49 267, 2023. [23] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, p. 34 892–34 916, 2023. [24] Z. Liu, J. Liu, and F. Ma, “Improving cross-modal alignment with synthetic pairs for text-only image captioning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 4, 2024, p. 3864–3872. [25] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, p. 10 684–10 695. [26] J. Luo, J. Chen, Y. Li, Y. Pan, J. Feng, H. Chao, and T. Yao, “Unleashing text-to-image diffusion prior for zero-shot image captioning,” in European Conference on Computer Vision. Springer, 2024, p. 237–254. [27] J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y. Choi, “Clipscore: A reference-free evaluation metric for image captioning,” in Proceedings of the 2021 conference on empirical methods in natural language processing, 2021, p. 7514–7528. [28] R. Nogueira and K. Cho, “Passage re-ranking with bert,” arXiv preprint arXiv:1901.04085, 2019. [29] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” 2015. [Online]. Available: https://arxiv.org/abs/1503.02531 [30] X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, p. 4582–4597. [31] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman et al., “Laion-5b: An open large-scale dataset for training next generation image-text models,” Advances in neural information processing systems, vol. 35, p. 25 278–25 294, 2022. [32] D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” arXiv preprint arXiv:1606.08415, 2016. [33] C. Dwork, R. Kumar, M. Naor, and D. Sivakumar, “Rank aggregation methods for the web,” in Proceedings of the 10th international conference on World Wide Web, 2001, p. 613–622. [34] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. [35] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. [36] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, p. 311–318. [37] S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005, p. 65–72. [38] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, p. 74–81. [Online]. Available: https://aclanthology.org/W04-1013/ [39] R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, p. 4566–4575. [40] P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in European conference on computer vision. Springer, 2016, p. 382–398. [41] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick, “Microsoft coco captions: Data collection and evaluation server,” arXiv preprint arXiv:1504.00325, 2015. [42] A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko, “Object hallucination in image captioning,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, p. 4035–4045. [43] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, “Qwen2.5-vl technical report,” 2025. [44] D. Li, J. Li, H. Le, G. Wang, S. Savarese, and S. C. Hoi, “Lavis: A one-stop library for language-vision intelligence,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), 2023, p. 31–41. [45] X. L. Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, T. B. Hashimoto, L. Zettlemoyer, and M. Lewis, “Contrastive decoding: Open-ended text generation as optimization,” in Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers), 2023, p. 12 286–12 312. [46] P. Sharma, N. Ding, S. Goodman, and R. Soricut, “Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, p. 2556–2565. [47] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in Proceedings of the IEEE/CVF international conference on computer vision, 2023, p. 11 975–11 986. [48] Y. Wada, K. Kaneda, D. Saito, and K. Sugiura, “Polos: Multimodal metric learning from human feedback for image captioning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, p. 13 559–13 568. [49] S. Petryk, D. Chan, A. Kachinthaya, H. Zou, J. Canny, J. Gonzalez, and T. Darrell, “Aloha: A new measure for hallucination in captioning models,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), 2024, p. 342–357. [50] Z. Liu, C.-W. Xie, B. Wen, F. Yu, P. Li, B. Zhang, N. Yang, Z. Gao, Y. Zheng, and H. Xie, “Capability: A comprehensive visual caption benchmark for evaluating both correctness and thoroughness,” Advances in Neural Information Processing Systems, vol. 38, 2026. [51] S. Zhao, X. Wang, L. Zhu, and Y. Yang, “Test-time adaptation with clip reward for zero-shot generalization in vision-language models,” in International Conference on Learning Representations, vol. 2024, 2024, p. 3597–3613.