Paper deep dive
When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Prompts
Huichan Seo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/3/2026, 2:28:16 AM
Summary
This paper audits Vision-Language Models (VLMs) acting as judges for image selection on culturally situated prompts. Using the CulturalFrames dataset, the study evaluates Qwen3-VL-4B and SmolVLM2-2.2B models. It finds that the 4B model exhibits significant position bias, often selecting the first image shown, and barely outperforms random choice. The 8B-equivalent performance (implied by comparison or larger context, though specifically 4B and 2.2B are tested) shows less bias. The study highlights that agreement gates (consistency across reorders) can filter out bad decisions for biased judges but may discard good decisions for unbiased ones, necessitating judge-specific auditing.
Entities (7)
Relation Signals (6)
Qwen3-VL-4B → exhibits → position bias
confidence 95% · Qwen picks the first image shown in 49% of calls... reordering changes its choice on 60% of prompts.
CulturalFrames → provides → Human Ratings
confidence 95% · CulturalFrames provides generated images and human ratings that distinguish missing explicit expectations...
SmolVLM2-2.2B → underperforms → Random Choice
confidence 95% · Smol does worse than random: gain −0.073
Agreement Gates → filters → Qwen3-VL-4B Decisions
confidence 90% · For the 4B judge, keeping only decisions that are consistent across orders retains choices that beat random
Qwen3-VL-4B → outperforms → Random Choice
confidence 90% · A 4B-parameter judge barely beats random... gain of 0.039
Qwen3-VL-4B → underperforms → CLIP
confidence 85% · falls short of a CLIP similarity baseline.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-language models (VLMs) increasingly act as judges that pick the best of several generated images, so their choices decide what users see. Such judges are usually validated by score agreement with human ratings, not by the images they return. We audit VLM judges as decision-makers: on 300 culturally situated prompts, we compare the returned image with human ratings the judge never sees and with random choice from the same candidates, and repeat every decision with the candidates reordered. A 4B-parameter judge barely beats random and falls short of a CLIP similarity baseline. It picks the first image shown in 49% of calls (chance: 28%), and reordering changes its choice on 60% of prompts. For this judge, agreement across orders is informative: decisions that survive reordering are much better than random, whereas agreement with a weaker second judge keeps the wrong ones. An 8B judge shows almost no position bias and outperforms CLIP, yet for it the same filter mostly discards good decisions. Agreement helps only when it targets the judge's failure mode, so filters must be re-audited whenever the judge changes. The 4B judge's slight rise in stereotype ratings is no longer detectable after aggregating across orders or with the larger judge.
Tags
Links
- Source: https://arxiv.org/abs/2610.01243v1
- Canonical: https://arxiv.org/abs/2610.01243v1
Trouble viewing inline? Open PDF directly →
Full Text
81,725 characters extracted from source content.
Expand or collapse full text
When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Prompts Huichan Seo Independent Researcher gmlcks00513@gmail.com Code and project page — https://github.com/seochan99/JudgeActs Abstract Vision-language models (VLMs) increasingly act as judges that pick the best of several generated images, so their choices decide what users see. Such judges are usually validated by score agreement with human ratings, not by the images they return. We audit VLM judges as decision-makers: on 300 culturally situated prompts, we compare the returned image with human ratings the judge never sees and with random choice from the same candidates, and repeat every decision with the candidates reordered. A 4B-parameter judge barely beats random and falls short of a CLIP similarity baseline. It picks the first image shown in 49% of calls (chance: 28%), and reordering changes its choice on 60% of prompts. For this judge, agreement across orders is informative: decisions that survive reordering are much better than random, whereas agreement with a weaker second judge keeps the wrong ones. An 8B judge shows almost no position bias and outperforms CLIP, yet for it the same filter mostly discards good decisions. Agreement helps only when it targets the judge’s failure mode, so filters must be re-audited whenever the judge changes. The 4B judge’s slight rise in stereotype ratings is no longer detectable after aggregating across orders or with the larger judge. 1 Introduction A vision-language model (VLM) that selects one of several generated images does more than report a quality score. Its choice determines which candidate a user would receive. The practical question is whether that choice improves the avail- able outputs, not just whether the judge’s scores correlate with human ratings. For culturally situated requests, visual polish and recog- nizable symbols can be poor guides to prompt satisfaction. An image may include a requested object while misrepre- senting the social context around it. A selector can favor that image even when another candidate better satisfies the request. Evaluating this choice requires ratings that did not come from the selector itself. CulturalFrames provides generated images and human rat- ings that distinguish missing explicit expectations, missing implicit expectations, and stereotypes. It also reports weak agreement between automatic metrics and human judgments (Nayak et al. 2025). We use the public CulturalFrames re- lease to study a different decision: choosing which existing image to return. For each prompt, an open-weight VLM sees three or four candidate images and selects one (Figure 3), without gener- ator names, country metadata, or ratings. We compare the selected image’s released ratings with exact random choice and with the best candidate in the same fixed pool, so dif- ferences between policies arise from selection rather than generation. The primary outcome is selection regret, measured from the mean human prompt-alignment rating. It is the gap be- “Chilean family lunch, children eagerly finishing homemade pastel de choclo” Slot ASlot BSlot C Order 1 Order 2 Order 3 judge's pick is best-ratedjudge's pick is not best-rated 0.000.500.83 0.500.830.00 0.830.000.50 Figure 1: Same request, same images, three orders. Each row shows the three candidates in one presentation order, with mean human prompt-alignment ratings. Qwen3-VL-4B picks the image in slot A every time (framed), so its returned image is rated 0.00, 0.50, or 0.83 depending only on order. Images: CulturalFrames (Nayak et al. 2025). arXiv:2610.01243v1 [cs.CV] 1 Oct 2026 tween the highest-rated available candidate and the selected image. Prompt alignment measures request satisfaction, not cultural accuracy alone. We therefore report stereotype and missing-expectation ratings separately. We also measure how often the selected score falls below random expected utility. We call this the below-mean rate (BMR). We make three contributions. • An audit of judges as decision-makers. We score each returned image against held-out human ratings and against random choice from the same candidates, so every com- parison is paired and free of the judge’s own scores. • How position bias changes what users receive. Posi- tion bias in model judges is known; we measure its con- sequence when the judge acts. Both small judges favor early positions, and the same prompt can yield the worst or the best available image depending only on candidate order (Figures 1 and 2). Ablations show that the bias is not caused by our output format or label letters. • An audit of agreement gates across judges. For the 4B judge, keeping only decisions that are consistent across orders retains choices that beat random, whereas agree- ment with a weaker second judge does not. For a larger 8B judge with little position bias, the same gate mostly discards good decisions. A gate must therefore be audited on the prompts it rejects as well as on those it keeps, and re-audited when the judge changes. 2 Related Work Cultural representation and evaluation. CulturalFrames supplies culturally situated prompts, multiple generated im- ages, and ratings of missing expectations and stereotypes (Nayak et al. 2025). Its analysis compares automatic met- rics with human ratings of individual images. Our analysis instead evaluates choices within a candidate set. Seo et al. (2025) study cultural representation in image generation and editing; we study selection among existing outputs. Chang and Chen (2026) train a cultural reward model on preferences derived from CulturalFrames and evaluate pair- wise agreement. This is closely related evidence that cultural image preferences can be learned. We do not propose a com- peting reward model; we audit frozen, prompted selectors and the requests retained by their agreement gates. VLM judges. Zou et al. (2026) show that multimodal judges can favor informative answers without adequately us- ing visual evidence. Park et al. (2026) examine conflicts between textual narratives and perception, and train judges to ground their decisions more reliably. Both concern tasks distinct from our candidate-image selection setting. They motivate checking selected outcomes against independent ratings, but do not establish the mechanism behind the errors measured here. K-Sort Eval uses multi-image VLM rankings, posterior correction, and adaptive comparisons to evaluate visual gen- erators (Li et al. 2026). Our study asks about the selected image within each prompt’s candidate pool, rather than the accuracy or efficiency of a generator leaderboard. We do not claim that comparative image judging or best-of-pool selec- tion is new. Position bias in model judges. Language-model judges are sensitive to the order in which options are presented: pair- wise judges favor a position (Zheng et al. 2023; Wang et al. 2024; Shi et al. 2025), and multiple-choice accuracy shifts with option order (Pezeshkpour and Hruschka 2024); judge verdicts also change when the judging instruction is reworded (Bellibatlu, Raff, and Zhang 2026). Multimodal judges also show position and consistency effects on vision-language benchmarks (Chen et al. 2024), over-score earlier sentences and prefer their own captions when assessing image-text alignment (Saito et al. 2025), and can be swayed by visual manipulations unrelated to quality (Hwang et al. 2025). Pro- posed mitigations include swapping or averaging over orders (Zheng et al. 2023; Wang et al. 2024) and de-biased judging protocols (Asaria, Salomone, and Gandhi 2026). VLM eval- uators are also used to audit generated images at scale, with human validation on a subset (Seo et al. 2026). We measure the same phenomenon when the judge’s output is an action, and ask how it interacts with agreement-based abstention. Judgments used as decision rules. Yao et al. (2026) in- vestigate multimodal reinforcement learning with imperfect rewards, where gains on a proxy can diverge from task perfor- mance. WORLDVIEW examines cultural stereotyping intro- duced during prompt revision (Urman et al. 2026). We isolate a later decision, selecting an existing candidate. The exper- iment does not optimize generators against a reward model or simulate a sequential feedback process. Selective prediction. Risk–coverage analysis evaluates prediction with a reject option (El-Yaniv and Wiener 2010; Geifman and El-Yaniv 2017). We apply it to image selec- tion, using agreement across orders or judges as gates. These gates yield measured operating points, not calibrated con- fidence scores or risk guarantees. Work on vision-language reasoning shows that selective models can abstain unneces- sarily (Srinivasan et al. 2024), and decision-theoretic metrics penalize confident errors more than abstentions (Wu et al. 2026); our gate audit complements these by asking which decisions an agreement rule keeps and which it rejects. 3 Problem and Measures Selection task. Let p denote a prompt and C p = x p1 ,...,x pm p its pool of m p ∈ 3, 4 candidate images, already generated and held fixed. A selection policy π re- ceives the prompt and the images, shown in a given order with labels, and either returns the index π(p)∈1,...,m p of one candidate or abstains. We always record the chosen image’s identity, not its label, so the same image can carry the label “A” in one order and “C” in another. Human reference utility. Each candidate x pi has released prompt-alignment ratings from several annotators, each 0, 0.5, or 1. Its utility h pi ∈ [0, 1] is the mean of these ratings. Judges never see these ratings; they are joined to the chosen image only after selection. Iran “Iranian family gathering with multiple generations in a garden” 0.000.750.75 Poland “Polish family sharing opłatek wafer during Christmas dinner” 0.330.171.00 China “Children lighting colorful lanterns during Mid-Autumn Festival” 0.170.330.75 Brazil “Three generations sharing a cozy Brazilian urban apartment” 0.500.331.00 highest human rating in poolQwen’s pick (original order), below pool meanbar/number: mean human prompt-alignment; tick = pool mean Qwen’s pickQwen’s pick Qwen’s pickQwen’s pick Figure 2: VLM judges can return below-average images when a clearly better candidate exists. Each group is one CulturalFrames prompt (Nayak et al. 2025); numbers are mean human prompt-alignment ratings. Red: Qwen3-VL-4B’s original-order choice, below the pool mean. Green: highest-rated candidate. Prompts follow a fixed rule applied without viewing images (Appendix D); they illustrate the failure mode, not its frequency (Figure 4). Baselines. Choosing uniformly at random has expected utility ̄ h p = 1 m p P m p i=1 h pi , which we compute exactly rather than by simulation. The annotation oracle picks the highest- rated candidate, with utility h ⋆ p = max i h pi ; it is a reference point, not a deployable selector. Outcome measures. For a policy that acts on prompt p and returns image x p,π(p) , with utility h p,π(p) , we report three quantities: regret: R p (π) = h ⋆ p − h p,π(p) ,(1) gain over random: G p (π) = h p,π(p) − ̄ h p ,(2) below-mean event: B p (π) = 1h p,π(p) < ̄ h p . (3) Regret is the primary outcome: zero means the judge returned a best-rated image. Gain is positive when the judge beats random choice on the same pool, and the two are linked by G p = (h ⋆ p − ̄ h p )−R p . Averaging B p over prompts gives the below-mean rate (BMR), a score shortfall relative to random, not a measure of user harm. Random choice has its own below-mean probability, the share of candidates rated below ̄ h p , which we use as its matched comparison. We also report how often the policy returns a best-rated image, counting ties. Every average is taken over the set A π of prompts on which the policy acts, and each policy is compared with random choice on exactly those prompts. Cultural-error outcomes. CulturalFrames also releases, per image, the share of annotators who flag a stereotype, a missing explicit expectation, or a missing implicit expecta- tion. For each of these error rates we report the chosen im- age’s value minus the random-choice average over the same pool, so that negative values mean fewer errors than chance. These outcomes are kept separate from alignment and are not combined into a composite score. A pool is sensitive to an error type when that error rate differs across its candidates, so that selection can change it; we report results on each sensitive subset (Table 2). As a secondary utility we also use overall satisfaction, rescaled from its 1–5 scale to [0, 1]. Presentation orders and agreement gates. Each pool is shown in K = 3 cyclic orders k ∈ 1, 2, 3 (order 1 is canonical, orders 2 and 3 rotate it). An order places the candidates in labeled positions, which we call slots A, B, C, and D; a preference for particular slots regardless of content is position bias. Let π k (p) be the index chosen under order k, and let q p be the share of the three orders that agree on the most frequent choice (1/3, 2/3, or 1). A gate with thresholdτ acts only when q p ≥ τ and then returns that majority choice: τ = 1 is unanimity, τ = 2/3 requires two of three, and τ = 1/3 always acts. The cross-model gate acts only when two judges make the same order-1 choice. For every gate we report coverage|A π |/N over the N = 300 prompts, the outcomes on the prompts it keeps, and, because a gate can look good simply by keeping easy pools, the same outcomes for random choice and for the ungated judge on both the kept and the rejected prompts. Country groups. Each prompt belongs to one of ten coun- try contexts; we report group means of regret and gain with their random baselines, and worst-group regret, the largest group mean. Country labels identify dataset contexts, not homogeneous cultural preferences. (a) Frozen candidate pools CulturalFrames 300 prompts p C p 3–4 candidates per prompt ratings withheld 10 countries Brazil CanadaChileChina Germany IndiaIranJapan Poland South Africa 5 categories Dates of significance Etiquette Family Greetings Religion 4 generators SD-3.5-Large Flux.1-Dev GPT-Image Imagen3 (b) Selection policy cyclic orders AB C B C A C AB π Qwen3-VL-4BSmolVLM2-2.2B one pick per order π(p) 3/3 orders≥2/3 orders both judges agreement gates actabstain agree else (c) Retrospective audit join human ratings by image ID h pi Outcomes Primary: selection regret Paired gain vs. exact random Below-mean rate (BMR) Cultural errors Stereotype Missing explicit Missing implicit Diagnostics Country groups Presentation-slot bias Matched-subset baselines for gates released ratings: withheld from judge, joined only in audit 1 2 3 4 5 6 Figure 3: Audit protocol. (a) Frozen candidate pools from CulturalFrames: culturally situated prompts with three or four images from four generators. (b) Selection: each judge sees only the prompt and the images, in three cyclic orders; agreement gates decide whether to act or abstain. (c) Retrospective audit: released human ratings, never shown to the judge, are joined by image identifier after selection and compared with exact random selection on the same pools. 4 Experimental Setup 4.1 Dataset and Frozen Splits We use the independently released CulturalFrames dataset (Nayak et al. 2025). Image rows are joined to annotation records by exact image identifier, with prompt, country, and generator-suffix checks. We decode every image, check re- peated identifiers and encoded-image content, and require at least three valid, annotated candidates. For repeated prompt text within a country we retain the lexicographically first base identifier before splitting, preventing duplicated re- quests from crossing splits. Missing annotations are never imputed. The provenance manifest pins the public dataset and model revisions. The release contains 3,637 image rows and 3,577 image annotation records, comprising 10,412 individual ratings. 60 image rows have no annotation record (1.6%). Filtering leaves 963 eligible sets. Removing 2 repeated requests within countries leaves 961 unique prompts: 313 three-candidate and 648 four-candidate sets. The main split has 100 three- candidate and 200 four-candidate sets; 11 sets have identical candidate alignment scores. We create a development split of three prompts per coun- try and a disjoint main split of 30 per country, using seed 20261002. Split membership depends only on a deterministic hash of prompt identifiers, without judge outputs or annota- tion values. The equal country allocation defines a balanced- country estimand; it does not reproduce a real deployment distribution. Remaining eligible sets form a reserve, and the main split is not expanded in response to its results. 4.2 Open-Weight Judges The primary judge is Qwen3-VL-4B-Instruct (hereafter Qwen) (Qwen Team 2025); SmolVLM2-2.2B-Instruct (Smol) is the secondary judge (Marafioti et al. 2025; Hug- gingFaceTB 2025). Model cards, revisions, dependency ver- sions, and inference settings are stored with the artifact. We run locally with greedy decoding, a 64-token output limit, and no requested explanation. Qwen images fit an aspect- preserving 448-pixel bounding box. Smol uses its native 384- pixel global view with image tiling disabled to bound mem- ory consumption. The two judges therefore receive different visual token budgets, so differences between them cannot be attributed to model capability alone. Every request interleaves a candidate label with its im- age. The instruction asks for explicit prompt satisfaction, implicit contextual consistency, avoidance of unsupported stereotypes, and visual integrity. It warns against rewarding cultural symbols merely for their number or recognizability. Only the original prompt supplies cultural context. Source model names, category labels, country metadata, and human annotations do not enter the request. For Qwen the output schema is a JSON object with choice and ranking. The ranking must contain every supplied label exactly once, and its first label must equal the choice. Smol receives the same evaluation criteria but a choice-only JSON contract, selected during development because it could not reliably satisfy the full-ranking format. Its adapter can normalize a single explicit label in a one-field structured object; raw JSON and exact-schema validity are tracked separately from recovered decision validity. Rank- ings are not used by any outcome metric. These interface differences limit comparisons of formatting reliability. We record raw text, parsed output, label-to-image mapping, in- put tensor shapes, model revision, elapsed time, and request hashes. Unit tests verify regret extrema, the analytic random expectation, tie behavior, annotation aggregation, and iden- tity preservation under permutation. 4.3 Candidate Order and Invalid Outputs Each prompt receives three distinct cyclic presentations: its canonical identifier order and two deterministic rotations. Canonical order sorts candidate identifiers, which usually places the same generator first; the rotations break this link, so generator effects are assessed over all three orders. The instruction, images, and decoding settings remain fixed. Thus disagreement measures sensitivity to these presentation changes rather than sampled decoding variation or prompt paraphrases. Three rotations do not exhaust all orders, espe- cially for four candidates. The original-order decision is the primary direct policy. No main-run prompt edits are permitted. An unresolved parse failure produces an abstention and remains in the out- put log. Qwen uses strict schema validation; Smol uses its predeclared single-label normalization, with every recovery flagged. A runtime exception permits one retry with identi- cal inputs; invalid text is not repeatedly sampled until valid. Utility and regret are conditional on valid decisions, with coverage relative to the full frozen split reported alongside them. We never assign failed requests a favorable candidate or silently remove their denominator. 4.4 Selection with Abstention The gate thresholds τ ∈ 1/3, 2/3, 1 and the cross-model rule were declared before main inference. A gate acts only when all three order calls parse; ties in the majority are broken by canonical identifier order, so the action never depends on human ratings. Abstention leaves a request unresolved: we do not assume a human fallback or a zero-risk outcome, and we report coverage next to every selective estimate. Because Smol is a weaker judge (Section 5), cross-model results are conditional on that pairing. 4.5 Statistical Analysis The unit of analysis is a prompt with its candidate pool; the three order calls for a prompt are never resampled indepen- dently. We report 95% intervals from 10,000 prompt-level bootstrap resamples (seed 20261002) for means and paired differences, and Wilson intervals for a single policy’s BMR. Intervals condition on the released annotation means. Sec- ondary cultural-error intervals are pointwise, and we addi- tionally report Bonferroni-adjusted intervals across the three error types. Analyses of presentation position, of the prompts that gates reject, and all ablations and the scale extension in Sections 5.5 and 5.6 were added after main inference and are labeled exploratory. Appendix O gives further estimation details. 5 Results 5.1 Direct Selection Table 1 and Figure 4a summarize the direct policies. Qwen returns a valid original-order choice for all 300 prompts. Its mean alignment is 0.770, against 0.730 for random choice and 0.929 for the oracle, a gain of 0.039 (95% CI [0.016, 0.062]). Mean regret is 0.160 [0.136, 0.184]; Qwen returns a best-rated image for 51.0% of prompts and a below-mean image for 36.3%, against 43.1% for random choice (paired difference−0.067, 95% CI [−0.118,−0.016]). Smol does worse than random: gain −0.073 [−0.093, −0.052], regret 0.270, and BMR 60.3%. It rarely follows the output format (Appendix Table 10), and its interface and image resolution differ from Qwen’s, so we treat it as a weak secondary judge rather than as a capability compari- son. Appendix Table 6 shows that the conclusions hold with a below-mean margin of 0.05, with the median in place of the mean, and with overall satisfaction as the utility. 5.2 Presentation Position Both main judges favor early slots (Figure 4c). Qwen picks slot A in 48.8% of its 900 calls, where uniform choice given the pool sizes would give 27.8% (χ 2 3 = 219, p < 10 −10 ); Smol places 93.3% of its choices in slots A or B. Consistent with this, Qwen’s choice changes across the three orders for 59.7% of prompts. Figure 1 shows the extreme case: the judge returns whatever image is shown first. The judge still responds to content. Canonical order places one generator first in most pools, so we pool all three rota- tions, which spreads each generator across slots: Qwen then chooses images from the generator with the highest mean hu- man rating in 57.9% of calls, against 27.2% under uniform choice (Appendix Table 4). Its choices mix a content signal with a strong slot preference. 5.3 What Agreement Gates Keep and Reject A judge that follows position alone picks a different image in each rotation, so it can never be unanimous. Unanimity keeps 121 prompts (40.3%). On them Qwen’s gain over random is 0.141 [0.110, 0.173] and regret 0.074; on the 179 rejected prompts its gain is−0.030 [−0.057,−0.002], below random (Figure 4b). The kept prompts are not easier: their candidates differ more in rating than the rejected ones (mean best-minus- worst range 0.454 versus 0.404), leaving more room for error. The gate only filters prompts; on kept prompts the choice is unchanged. The two-of-three gate is an intermediate operating point: it keeps 80.7% of prompts with gain 0.094 [0.071, 0.118] (Appendix Table 5). Voting over the three orders instead of abstaining, a simple form of order averaging, raises the gain on all prompts from 0.039 to 0.065 [0.043, 0.086]. Agreement between judges behaves differently. Qwen and Smol agree on 59 prompts (19.7%), where Qwen’s gain is −0.060 [−0.111, −0.011]; on the 241 prompts where they disagree it is 0.064 [0.039, 0.087]. Two judges that both prefer early slots can agree for reasons unrelated to content, so this gate keeps the wrong prompts. The result is specific to pairing with a weak judge. Appendix Table 5 lists every gate with its complement. 5.4 Comparison with a CLIP Baseline Would a simpler scorer that cannot be influenced by order do as well? We score each candidate by CLIP prompt–image similarity (Radford et al. 2021; Hessel et al. 2021) and return the highest-scoring image. This CLIP baseline has regret Policyn Cov. (%) Align. Regret [95% CI]Gain over random [95% CI] BMR (%) Best (%) Worst Random300100.0 0.730 0.199 [0.186, 0.213] 0.000 [0.000, 0.000]43.141.9 0.266 Qwen300100.0 0.770 0.160 [0.136, 0.184] 0.039 [0.016, 0.062]36.351.0 0.275 Qwen, majority of 3 orders300100.0 0.795 0.135 [0.114, 0.156] 0.065 [0.043, 0.086]31.754.3 0.186 Qwen, unanimity gate (3/3 orders) 12140.3 0.854 0.074 [0.050, 0.100] 0.141 [0.110, 0.173]16.571.1 0.115 Smol29799.0 0.659 0.270 [0.243, 0.298] −0.073 [−0.093,−0.052]60.329.0 0.400 Cross-model gate5919.7 0.690 0.249 [0.187, 0.316] −0.060 [−0.111,−0.011]54.233.9 0.490 CLIP baseline300100.0 0.799 0.130 [0.109, 0.153] 0.069 [0.047, 0.090]28.055.7– Oracle300100.0 0.929 0.000 [0.000, 0.000] 0.199 [0.186, 0.213]0.0100.0 0.000 Table 1: Selection outcomes on the 300 main prompts (95% prompt-level bootstrap intervals). Cov.: share of prompts on which the policy acts. Align.: mean prompt-alignment rating of returned images. Gain over random is paired with exact random choice on the same prompts. BMR: share of returned images rated below the pool mean. Best: share of best-rated returns, ties included. Worst: highest country-group mean regret. Gated policies are evaluated on the prompts they keep. CLIP is a content-only similarity scorer (Section 5.4). Random QwenSmol 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Mean regret (lower is better) 0.199 0.160 0.270 Oracle = 0 (a) Selection regret −0.10.00.10.2 Gain over random All prompts 3/3 orders agree Orders disagree Judges agree Judges disagree +0.039 (n=300) +0.141 (n=121) −0.030 (n=179) −0.060 (n=59) +0.064 (n=241) Unanimity gate Cross-model gate (b) Gain over random: kept vs. rejected ABCD Presentation slot 0% 10% 20% 30% 40% 50% 60% Share of picks (c) Presentation-slot preference QwenSmolUniform Figure 4: Main results (95% prompt-level bootstrap intervals). (a) Selection regret; lower is better. (b) Gain over random choice on the prompts each gate keeps and on those it rejects. Order unanimity keeps prompts where Qwen beats random and rejects prompts where it falls below random; agreement between the two judges does the reverse. (c) Share of choices by presentation slot; dashed marks show the rate expected under uniform choice given the pool sizes. 0.130 (95% CI [0.109, 0.153]) and gain over random 0.069 [0.047, 0.090], compared with Qwen’s 0.160 and 0.039; the paired difference in gain favors CLIP by 0.030 [0.005, 0.055]. CLIP also selects a below-mean image less often (28.0% ver- sus 36.3%). Yet it raises the stereotype rating relative to ran- dom by 0.026 [0.006, 0.046], as much as Qwen does, while sharply reducing missing explicit expectations (−0.113). An alignment-oriented scorer therefore does not avoid stereo- typed images; this concern is not specific to VLM judges. Qwen agrees with CLIP on 49.0% of prompts (chance 27.8%). Where they agree, Qwen’s gain is 0.099 [0.067, 0.130]; where they disagree it is −0.018 [−0.047, 0.011]. Agreement with an independent, position-invariant scorer thus behaves like order unanimity rather than like agreement with the weaker VLM judge. The CLIP comparison is ex- ploratory (Appendix Table 13). Results do not hinge on noisy annotations. Splitting prompts at the median annotator disagreement, Qwen’s gain is 0.047 [0.014, 0.079] on high-consensus and 0.033 [0.002, 0.062] on low-consensus prompts (Appendix Table 14). The CLIP baseline embeds the raw prompt (no template; all prompts fit the 77-token limit) and each image with the default CLIP preprocessing (resize and center crop to 224 pixels), and returns the image with the highest cosine sim- ilarity; ties follow canonical order. Our main baseline uses ViT-L/14. A smaller ViT-B/32 backbone gives nearly the same gain (0.063 [0.042, 0.085]; difference−0.006 [−0.028, 0.018]), but no stereotype increase (0.003 [−0.016, 0.023]), so the stereotype effect of a similarity scorer depends on the backbone (Appendix Table 15). 5.5 Is the Position Bias an Artifact? The position bias could be an artifact of our interface rather than of the judge. We ran three post-hoc ablations with Qwen on all 300 prompts, leaving the frozen main results unchanged (Figure 5, Appendix Table 11). Output format. The main contract asks for a full ranking whose first entry is the choice, which could prime the first label. With a choice-only contract, Qwen still picks the first slot in 44.8% of calls (uniform 27.8%) and changes its choice Main No ranking Opaque codes 4 orders 0% 20% 40% 60% 49% 45% 52% 52% (a) First slot chosen Main No ranking Opaque codes 4 orders 0.00 0.05 0.10 0.15 0.20 (b) Gain over random all prompts unanimous Figure 5: Post-hoc ablations with Qwen. (a) Share of calls choosing the first-presented image; dashes mark the uniform rate. (b) Gain over random on all prompts and on prompts where all orders agree (95% intervals). No ranking: choice- only output format. Opaque codes: two-character labels in- stead of letters. 4 orders: a fourth rotation for four-image pools. across orders on 54.0% of prompts. Its gain over random is 0.047 [0.024, 0.069], and unanimity keeps 46.0% of prompts with gain 0.123 [0.094, 0.153]. Label identity. Replacing A/B/C/D with opaque two- character codes unrelated to order, Qwen picks the first- presented image in 51.8% of calls (uniform 27.8%) and changes its choice on 62.7% of prompts, with gain 0.021 [−0.001, 0.044]. Full rotation. Adding a fourth rotation for the 200 four- image pools places every image in every slot once. Slot shares are then A 52.5%, B 20.4%, C 14.4%, D 12.8% (uniform 25.0%). Unanimity over four orders keeps 27.5% of these pools with regret 0.095 and gain 0.127 [0.073, 0.178], and voting over the four orders gives gain 0.061 [0.034, 0.087]. In every variant the bias toward the first slot remains and unanimity still keeps prompts on which the judge beats ran- dom. The preference is a property of the judge, not of the letter labels or the ranking format. 5.6 Does Scale Remove the Bias? Is position bias a property of small judges? We ran the larger Qwen3-VL-8B-Instruct on the same 300 prompts and three orders. To fit it in memory we used 8-bit weights, so we also reran the 4B judge with the same quantization as a control. The control agrees with the main 4B run on 94.9% of calls and reproduces its first-slot share (48.2%) and gain (0.044), so neither quantization nor the MLX runtime explains the differences below (Figure 6, Appendix Table 12). At 8B the position bias largely disappears: the first slot is chosen in 27.3% of calls, close to the uniform 27.8% (p = 0.06), although the choice still changes across orders on 43.0% of prompts. The 8B judge is also much better: its gain over random is 0.097 [0.077, 0.117], above the CLIP base- line (0.069; paired difference 0.028 [0.003, 0.054]), and its 4B 4B 8-bit 8B 8-bit 0% 20% 40% 60% 49% 48% 27% (a) First slot chosen 4B 4B 8-bit 8B 8-bit −0.05 0.00 0.05 0.10 0.15 (b) Gain over random all promptsorders agreeorders disagree Figure 6: Scale extension (8B). (a) Share of calls choosing the first slot; dashes mark the uniform rate. (b) Gain over random on all prompts, on prompts where all three orders agree, and on prompts where they do not (95% intervals). The 8-bit 4B run is a control for quantization. stereotype difference is no longer detectable (0.016 [−0.004, 0.036]). The value of the unanimity gate changes with it. For the 4B judge, the prompts unanimity rejects fall below random; for the 8B judge they stay above random (0.068 [0.036, 0.099]), so the gate mostly discards good decisions. Agreement be- tween the 8B and 4B judges shows the same pattern: on the 126 prompts where they disagree, the 8B choice has gain 0.098 and the 4B choice−0.039. A gate is useful only while it filters the failure mode of the judge in front of it, and must be re-audited when the judge changes. 5.7 Cultural-Error Outcomes Figure 7 compares the cultural-error ratings of Qwen’s choices with random choice on the same pools. Qwen lowers missing explicit expectations by 0.056 [−0.084,−0.026] and missing implicit expectations by 0.034 [−0.057, −0.009], and both reductions survive a Bonferroni adjustment across the three error types. It raises the stereotype rating from 0.125 to 0.148 (difference 0.023, pointwise 95% CI [0.002, 0.045]); this interval includes zero after adjustment ([−0.002, 0.051]), so we treat it as exploratory. On the pools where stereo- type ratings differ across candidates, the difference is 0.036 [0.004, 0.068] (Appendix Table 2). An alignment gain there- fore need not carry over to every cultural outcome. Where does the stereotype increase come from? Most of it traces to one generator: Qwen returns SD-3.5-Large images in 41.3% of prompts against 27.6% under uniform choice, and these images carry the highest stereotype ratings (Ap- pendix Table 19). The increase is also larger when several candidates tie for the best alignment (0.038 [0.004, 0.073]) than when one candidate is clearly best (0.014 [−0.012, 0.040]). Aggregating across orders leaves no detectable in- crease: the stereotype difference is 0.007 [−0.013, 0.027] for voting over the three orders and 0.010 [−0.019, 0.039] on −0.10−0.050.000.05 Qwen − matched random (error rate) Stereotype Missing explicit expectation Missing implicit expectation +0.023 −0.056 −0.034 ← fewer errors more errors → 95% CI Bonferroni (3 tests) Figure 7: Cultural-error ratings of Qwen’s choices minus random choice on the same pools; negative values mean fewer errors. Dark bars: pointwise 95% intervals. Light bars: Bonferroni-adjusted intervals across the three error types. Qwen reduces missing expectations and slightly raises stereotype ratings; the stereotype difference does not survive the adjustment. unanimous prompts, whereas on the prompts unanimity re- jects it is 0.032 [0.003, 0.063]. Both aggregated policies still reduce missing explicit expectations (−0.092 and−0.174). 5.8 Country Groups Qwen’s point estimate of gain is positive in 7 of 10 country groups (Appendix Figure 8), ranging from 0.107 in Canada to−0.018 in India. Regret is highest in Japan (0.275, point- wise 95% CI [0.171, 0.387]), where random choice also has the highest regret (0.266). With 30 prompts per group, these intervals are wide and overlap; country labels describe dataset contexts, not cultural preferences, and the analysis is not powered to rank groups. 6 Discussion Evaluate the action, not the score. A judge that ranks images plausibly can still return a poor image when it must act. Comparing the returned image with the best available one and with random choice on the same pool makes the value of the decision visible, and it exposes failures that score correlation hides: here, a strong dependence on where an image appears. Agreement gates need their own audit. Agreement is useful only when it breaks the judge’s failure mode. Agree- ment can retain stable mistakes, and any reduction in regret must be compared with random choice on the kept prompts, and with the judge on the rejected ones. Position bias explains why the two gates we studied diverge. Unanimity across ro- tations demands the same image from different slots, so it filters out position-driven choices; two judges with similar slot preferences can agree for reasons unrelated to content. Order averaging, by voting across rotations, recovers part of the lost value without abstaining. Abstention still leaves a practical question: who handles the rejected request, at what cost, and with what outcome. Is the unanimity result explained by easy prompts? No. Candidates on the prompts that unanimity keeps differ more in rating than those on the rejected prompts, so the kept pools leave more room for error, not less (Section 5); the judge’s gain on them is large, and its gain on the rejected prompts is below zero. What changes with scale? The 8B extension shows that the specific failure is not universal: position bias nearly van- ishes, and with it the gate’s ability to screen out below- random decisions, although order sensitivity itself remains (Section 5.6). The audit itself transfers unchanged and is what reveals the difference. This is the practical lesson: a filter validated for one judge is not validated for the next. What order checks cost. Order checks are cheap relative to their value. On our hardware a single Qwen decision takes a median of 3.5 s and three orders 10.6 s, roughly three times the cost, which buys either the voting gain or the una- nimity filter. Adding a fourth order lowers coverage without lowering regret further (four-order unanimity keeps 27.5% of four-image pools at regret 0.095) (Appendix Figure 14). Order sensitivity is only weakly related to how close the can- didates are: Qwen’s choice changes across orders on 62.8% of prompts in the lowest third of rating range and 53.6% in the highest (Appendix Table 17), so the bias is not confined to near-ties. Alignment and cultural errors are different targets. Both the VLM judge and the CLIP baseline reduce miss- ing cultural expectations, but stereotype ratings move differ- ently: the judge raises them slightly, and so does one CLIP backbone but not another. Aggregating the judge’s decisions across orders leaves no detectable increase. A selector tuned or validated on prompt alignment should therefore be audited separately on stereotype outcomes, and composite scores would hide this trade-off. 7 Limitations and Ethics CulturalFrames covers a limited set of countries, domains, prompts, and generators. Country labels compress substan- tial within-country diversity. Candidate images are existing outputs from a small generator pool, so the results depend on its visual styles and available alternatives. Hiding generator names does not remove recognizable stylistic signatures. Our local input resolution can obscure fine details and is part of the evaluated configuration. Human annotation means are imperfect references. Differ- ent annotators can disagree, small rating panels create ties and noise, and selecting a maximum can exploit annotation noise. The oracle should be interpreted accordingly. Primary bootstrap intervals condition on these means; they are not estimates of uncertainty over all possible human evaluations. Prompts can share semantic templates even after exact-text deduplication, so unmodeled dependence may also narrow intervals. Prompt alignment reflects ordinary request satis- faction and cannot isolate culture-specific value by itself. Missing-expectation and stereotype endpoints provide addi- tional descriptions without an arbitrary composite. The setting simulates selecting a returned image; it mea- sures neither observed user harm nor live sequential editing. We do not treat independent generations as temporal trajecto- ries or claim a feedback-amplification result. Our main judges are small (4B and 2.2B parameters); a single 8B extension, run with 8-bit weights and a matched quantization control, shows that position bias can shrink sharply with scale, and we do not test larger or proprietary models. With 30 prompts per country, country-group intervals are wide, and the study is underpowered to detect moderate group differences. Po- sition, complement-set, content-baseline, ablation, and scale analyses were added after main inference and are exploratory. The CLIP comparison uses two backbones of one similarity model family. Agreement statistics are not epistemic con- fidence, and selective risk is not overall system risk when abstained cases remain unresolved. This is secondary analysis of an independently released public benchmark, with no new participant recruitment or collection of personal data. Derived artifacts retain aggregate scores and candidate identifiers, not annotator identifiers, de- mographics, or free-text comments. The public dataset card does not declare a dataset license; public availability does not establish redistribution permission. We provide newly written code, manifests, and derived outputs; source images and raw human records are not redistributed. Figures 1, 2, and 10 reproduce dataset images with attribution for illus- tration. Applicable institutional requirements for secondary analysis remain an author responsibility. AI assistance. AI coding and writing assistants (OpenAI Codex and Anthropic Claude) assisted with code develop- ment, manuscript drafting and revision, figure code, and draft review. All empirical results are computed from released an- notations and recorded local judge outputs; writing assistance does not supply experimental observations. Authors remain responsible for verifying the analysis, references, and final manuscript. Statement of Contributions. This paper contributes (i) a matched-baseline, order-rotated audit protocol for VLM judges that select from fixed candidate pools, released with a deterministic artifact and an offline browser for every recorded decision (Appendix N); (i) evidence of strong po- sition bias in two small open judges, with post-hoc ablations of the output format and label identity and a scale extension showing that the bias shrinks in a larger judge; and (i) an au- dit of agreement gates on both the prompts they keep and the prompts they reject, compared with a content-only baseline. The paper has no earlier published version; all experiments, analyses, and text are new. 8 Conclusion A 4B VLM judge that chooses which image to return barely beats random choice (0.039), falls short of a CLIP similar- ity scorer, and picks the first-listed image in 48.8% of calls. Agreement across orders separates its good decisions from its bad ones; agreement with a weaker judge that shares its position bias does not. A larger 8B judge from the same family shows almost no position bias, and for it the same order-agreement gate mostly discards good decisions; a gate validated for one judge is not validated for the next. Agree- ment is useful only when it targets the failure mode of the judge in front of it, so selection audits, including audits of the filters around a judge, must be repeated when the judge changes. These findings concern a single choice from a fixed candidate pool. Evaluating a deployed selector also requires evidence about its users, later actions, and outcomes when it abstains. References Asaria, A.; Salomone, T.; and Gandhi, D. 2026. Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation. arXiv:2606.20364. Bellibatlu, R. R.; Raff, E.; and Zhang, W. 2026. Judge- Sense: A Benchmark for Prompt Sensitivity in LLM-as-a- Judge Systems. arXiv:2604.23478. Chang, B.-A.; and Chen, Y.-C. 2026. Debiasing Text-to- Image Evaluation via Implicit Cultural Alignment Reward Modeling. arXiv:2607.15740. Chen, D.; Chen, R.; Zhang, S.; Liu, Y.; Wang, Y.; Zhou, H.; Zhang, Q.; Wan, Y.; Zhou, P.; and Sun, L. 2024. MLLM-as-a- Judge: Assessing Multimodal LLM-as-a-Judge with Vision- Language Benchmark. In Proceedings of the International Conference on Machine Learning, volume 235. PMLR. El-Yaniv, R.; and Wiener, Y. 2010. On the Foundations of Noise-Free Selective Classification. Journal of Machine Learning Research, 11: 1605–1641. Geifman, Y.; and El-Yaniv, R. 2017. Selective Classification for Deep Neural Networks. In Advances in Neural Informa- tion Processing Systems, volume 30. Hessel, J.; Holtzman, A.; Forbes, M.; Le Bras, R.; and Choi, Y. 2021. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 7514– 7528. HuggingFaceTB. 2025. SmolVLM2-2.2B-Instruct: Public Model Card. Hugging Face. Hwang, Y.; Lee, D.; Min, K.; Kang, T.; Kim, Y.-i.; and Jung, K. 2025. Fooling the LVLM Judges: Visual Biases in LVLM- Based Evaluation. arXiv:2505.15249. Li, Z.; Li, J.; Liu, X.; Zhao, W.; Du, P.; Zhou, K.; Gu, Q.; You, Y.; Dong, Z.; and Keutzer, K. 2026. K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judge. In International Conference on Learning Representations. Marafioti, A.; Zohar, O.; Farré, M.; Noyan, M.; Bakouch, E.; Cuenca, P.; Zakka, C.; Allal, L. B.; Lozhkov, A.; Tazi, N.; Srivastav, V.; Lochner, J.; Larcher, H.; Morlon, M.; Tunstall, L.; von Werra, L.; and Wolf, T. 2025. SmolVLM: Redefining Small and Efficient Multimodal Models. arXiv:2504.05299. Nayak, S.; Bhatia, M.; Zhang, X.; Rieser, V.; Hendricks, L. A.; van Steenkiste, S.; Goyal, Y.; Stanczak, K.; and Agrawal, A. 2025. CulturalFrames: Assessing Cultural Ex- pectation Alignment in Text-to-Image Models and Evalua- tion Metrics. In Findings of the Association for Computa- tional Linguistics: EMNLP 2025, 20918–20953. Park, S.; Choi, J.; Kang, J.; Lee, S.; Shin, J.; and Shim, H. 2026. Mitigating Perceptual Judgment Bias in Multi- modal LLM-as-a-Judge via Perceptual Perturbation and Re- ward Modeling. In Proceedings of the International Con- ference on Machine Learning, volume 306, 96645–96672. PMLR. Pezeshkpour, P.; and Hruschka, E. 2024. Large Language Models Sensitivity to The Order of Options in Multiple- Choice Questions. In Findings of the Association for Com- putational Linguistics: NAACL 2024, 2006–2017. Qwen Team. 2025. Qwen3-VL-4B-Instruct: Public Model Card. Hugging Face. Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Trans- ferable Visual Models From Natural Language Supervision. In Proceedings of the International Conference on Machine Learning, volume 139, 8748–8763. PMLR. Saito, K.; Shinoda, R.; Tanaka, S.; Hirasawa, T.; Okura, F.; and Ushiku, Y. 2025. HalDec-Bench: Benchmarking Hallu- cination Detector in Image Captioning. arXiv:2511.20515. Seo, H.; Choi, S.; Hong, M.; Zhou, Y.; Kim, J.; Ismaila, L.; Etori, N.; Agarwal, M.; Liu, Z.; Kim, J.; and Oh, J. 2025. Exposing Blindspots: Cultural Bias Evaluation in Generative Image Models. arXiv:2510.20042. Seo, H.; Hong, M.; Choi, S.; Kim, J.; and Oh, J. 2026. Toward Trustworthy Portrait Editing: Evaluation of Demographic Misrepresentation in I2I Models. arXiv:2602.16149. Shi, L.; Ma, C.; Liang, W.; Diao, X.; Ma, W.; and Vosoughi, S. 2025. Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge. In Proceedings of the International Joint Conference on Natural Language Processing and the Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, 292–314. Srinivasan, T.; Hessel, J.; Gupta, T.; Lin, B. Y.; Choi, Y.; Thomason, J.; and Chandu, K. R. 2024. Selective “Selective Prediction”: Reducing Unnecessary Abstention in Vision- Language Reasoning. In Findings of the Association for Computational Linguistics: ACL 2024. Urman, A.; Lichtenegger, E.; Jaoua, S.; Bouleimen, A.; Fors- berg, R.; Hertweck, C.; Ionescu, S.; Pagan, N.; Hannak, A.; and Baumann, J. 2026. Prompt Revision as a Source of Cultural Bias in Text-to-Image Systems. arXiv:2609.11532. Wang, P.; Li, L.; Chen, L.; Cai, Z.; Zhu, D.; Lin, B.; Cao, Y.; Kong, L.; Liu, Q.; Liu, T.; and Sui, Z. 2024. Large Language Models are not Fair Evaluators. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 9440–9450. Wu, S.; Gustafsson, F. K.; Phillips, E.; Gao, B.; Thakur, A.; and Clifton, D. A. 2026. BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence. arXiv:2604.03216. Yao, J.; Wang, Y.; Zhang, A.; Sun, Z.; Wang, S.; Mei, L.; Ge, Y.; and Liu, S. 2026. Multimodal Reward Hacking in Reinforcement Learning. arXiv:2607.09492. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM-as-a- Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, volume 36. Zou, X.; Sridhar, R.; Safarzadeh, M.; and Roth, D. 2026. When Vision-Language Models Judge Without Seeing: Ex- posing Informativeness Bias. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 15417–15448. A Reproduction Protocol Seed: 20261002. Dataset revision: 7a44117634da. Qwen revision: ebb281ec70b0. Smol revision: 482adb537c02. Full revision identifiers and run- time versions are in the provenance manifest. The main split contains 300 prompts, 30 per country; the dev split contains 30 disjoint prompts. Each prompt has three cyclic candidate orders. Qwen uses a 448-pixel maximum image edge; Smol uses one 384-pixel native view. Both use greedy decoding and a 64-token limit. The image audit finds 0 undecodable images and 0 identifier/content mismatches. Four repeated encoded-image pairs occur across the duplicate requests removed before splitting. In the artifact, the key hsr stores the below-mean rate (BMR) reported here. The pipeline is deterministic given the recorded outputs: make analyze recomputes every table, figure, and number in this paper from the frozen judge logs, and make paper regenerates the manuscript text frag- ments. Figures and prose never contain hand-entered esti- mates; each value is read from the analysis outputs at build time, and a checksum manifest covers the annotations, man- ifests, judge logs, and analysis code. B Additional Result Figures and Tables Figures 8 and 9 expand the country and agreement-gate re- sults, and Table 2 gives results on the error-sensitive subsets. C Dataset Composition Table 3 describes the frozen main split by country. Ties are pools with identical alignment scores (zero regret under ev- ery policy); Mean range is the average spread between the best and worst candidate, an upper bound on the regret a judge can incur. Table 4 reports each generator’s representa- tion and mean rating. D Additional Qualitative Examples Figure 2 uses a fixed rule applied without viewing images: among three-candidate pools where Qwen’s original-order choice is below the pool mean and some candidate exceeds the mean by at least 0.25, take the largest-regret prompt per country, then the four countries with the largest regret (ties broken by name). Figure 10 shows one pool per country, cho- sen as the first four-candidate prompt in the frozen manifest. Unlike Figure 2, this sample is not filtered on outcomes. Both judges’ original-order choices are marked. JapanIranPolandCanadaSouth AfricaChinaChileBrazilIndiaGermany 0.0 0.1 0.2 0.3 0.4 0.5 Mean regret (lower is better) RandomQwenSmol Figure 8: Country-group regret with 95% bootstrap intervals (30 prompts per group). Random regret shows each group’s room for improvement. ConditionnRegretSelectedRandomDifference [95% CI] Explicit2760.1690.2800.341−0.060 [−0.092,−0.029] Implicit2480.1680.2070.248−0.041 [−0.068,−0.012] Stereotype1940.1690.2270.1910.036 [0.004, 0.068] Table 2: Qwen’s choices on pools where a given error rating differs across candidates. Selected and Random are mean error rates; the difference is paired (95% CI). Conditions overlap and do not isolate causal effects. Majority 100%, n=300 ≥2/3 81%, n=242 3/3 40%, n=121 Judges 20%, n=59 Gate (coverage of 300 prompts) 0.0 0.1 0.2 0.3 0.4 Mean regret on kept prompts 0.13 0.11 0.07 0.25 0.20 0.21 0.22 0.19 Random (matched) Qwen, order gate Qwen, cross-model gate Figure 9: Regret at each agreement gate next to random choice on the identical kept prompts (95% bootstrap inter- vals). “Majority” votes over the three orders and always acts. Three further galleries use fixed selection rules docu- mented in the artifact. Figure 13 shows prompts on which Qwen picks slot A in every order, so the returned image changes with order alone. Figure 12 contrasts Qwen with the CLIP scorer where they disagree most, in both direc- tions. Figure 11 shows prompts where Qwen returns the most stereotyped image in the pool although an equally or better aligned image with a lower stereotype rate was available. Brazil “Brazilian social gathering, person saying goodbye to each attendee” greetings · pool mean 0.83 Canada “Quebec home with wine bottles prepared for dinner party” etiquette · pool mean 0.73 Chile “Families gather for midnight fireworks over Santiago skyline.” dates of significance · pool mean 0.83 China “Chinese friends exchanging warm smiles at a bustling street market” greetings · pool mean 0.96 Germany “Priest blessing herbs in German countryside church” religion · pool mean 0.75 India “Indian family celebrating Diwali with overseas relatives” family · pool mean 0.62 Iran “Iranian family picnicking in a park, sharing food with nearby strangers” etiquette · pool mean 0.33 Japan “Japanese family seated around low table, sharing traditional meal” etiquette · pool mean 0.59 Poland “Polish family dinner, guest politely declines more bigos, host serves anyway.” etiquette · pool mean 0.48 South Africa “Friends enjoying a braai feast with springbok and ostrich steaks in South Africa.” etiquette · pool mean 0.83 highest human ratingQwen’s pick (original order)Smol’s pick (original order)numbers: mean human prompt-alignment (0–1) 1.000.331.001.00 0.670.670.830.75 0.830.501.001.00 1.000.831.001.00 0.501.001.000.50 0.670.670.670.50 0.000.330.500.50 0.380.500.501.00 0.000.171.000.75 0.670.671.001.00 QS SQ SQ QS SQ SQ QS QS SQ QS QS Figure 10: Unfiltered examples: the first four-candidate prompt per country in the frozen manifest, with candidates in original presentation order. Numbers are mean human prompt-alignment ratings. Blue (Q) and orange (S) frames mark the Qwen and Smol original-order choices; green checks mark the highest-rated candidates. Images from CulturalFrames (Nayak et al. 2025). “Chilean family chatting around a dinner table with traditional dishes” Chile “Traditional Japanese tea ceremony in autumn garden” Japan “Iranian family enjoying tea in a cozy Persian carpeted room” Iran “Japanese family enjoying festival food under Tanabata lanterns” Japan Qwen’s pick: highest stereotype rate in poolalignment ≥ Qwen’s pick, lower stereotype rateA: mean prompt-alignment; S: stereotype rate A 0.50 · S 0.00A 0.83 · S 0.00A 1.00 · S 0.00A 1.00 · S 1.00 A 0.33 · S 1.00A 0.17 · S 0.00A 0.83 · S 0.00A 1.00 · S 0.00 A 0.33 · S 1.00A 1.00 · S 0.33A 1.00 · S 0.00A 1.00 · S 0.00 A 0.33 · S 1.00A 0.33 · S 0.00A 0.83 · S 0.00 Figure 11: Stereotype trade-off. Prompts where Qwen’s choice (red) has the highest stereotype rate in its pool and another candidate with equal or higher alignment has a lower rate (checks). Labels give mean alignment (A) and the share of annotators flagging a stereotype (S). Rows with three images are prompts whose pool contains only three candidates. CLIP’s pick rated higherQwen’s pick rated higher “Japanese host guiding guests to their seats in a tatami room.” Japan “Shintō priest performing purification ritual at a rural shrine” Japan “Children lighting colorful lanterns during Mid-Autumn Festival” China “Iranian family dinner with guests politely refusing more food” Iran “Chilean workers' parade with flags and banners” Chile “Vibrant Durga Puja pandal decorations in India” India Qwen’s pick (original order)CLIP-score pickhighest human ratingnumbers: mean human prompt-alignment (0–1) 0.000.170.500.83 0.000.331.000.67 0.170.330.75 0.000.751.000.75 0.170.831.001.00 0.330.830.831.00 QC QC QC CQ CQ CQ QC Figure 12: Content scorer versus judge. Prompts where CLIP’s choice (C, purple) and Qwen’s original-order choice (Q, blue) differ most in rating: left, CLIP’s choice is rated higher; right, Qwen’s is. Checks mark the best-rated candidates. Rows with three images are prompts whose pool contains only three candidates. Order 1Order 2Order 3 Best available India “Dalit artisans crafting pottery at an Indian village workshop” Qwen picks slot A in all 3 orders, 3 different images Japan “Traditional Japanese tea ceremony in autumn garden” Qwen picks slot A in all 3 orders, 3 different images Poland “Polish townspeople in Corpus Christi procession” Qwen picks slot A in all 3 orders, 3 different images China “Families sharing mooncakes under the full moon in China” Qwen picks slot A in all 3 orders, 3 different images India “Elderly Indian couple greeting guests with Namaste at home entrance.” Qwen picks slot A in all 3 orders, 3 different images Qwen’s pick is best-ratedQwen’s pick is not best-ratednumbers: mean human prompt-alignment (0–1) 1.000.670.331.00 0.330.170.831.00 0.501.000.751.00 0.501.000.501.00 0.501.000.831.00 Figure 13: Order flips. Prompts where Qwen picks slot A in all three orders and returns at least two different images; the five with the widest rating range among its picks. The last column is the best-rated candidate. Images from CulturalFrames (Nayak et al. 2025). CountryPrompts3-img4-imgTiesMean align.Mean rangeStereo. Brazil30111910.7900.4100.109 Canada3072310.7410.4540.044 Chile3082230.7620.3780.069 China30141610.6900.3890.154 Germany30131720.7900.3390.100 India3072300.7630.3690.091 Iran3082210.7000.5140.264 Japan30102010.5960.5030.218 Poland30102000.7140.4670.106 South Africa30121810.7570.4180.078 Table 3: Main split by country (30 prompts each). Mean align. and Stereo. average candidate-level human ratings. GeneratorImagesMean align.Uniform (%)Qwen (%)Smol (%) Flux.1-Dev2960.63927.38.435.2 GPT-Image2950.85427.257.930.2 Imagen32110.78517.916.19.3 SD-3.5-Large2980.66027.617.625.3 Table 4: Generators in the main split. Uniform is the share expected if a judge chose uniformly at random given pool sizes; Qwen and Smol are choice shares pooled over all three orders, which decouples generator from presentation slot. PolicynCov.Regret [95% CI]Gain [95% CI]BMRRand. BMRBest Random300100.00.199 [0.186, 0.213]0.000 [0.000, 0.000]43.143.141.9 Qwen300100.00.160 [0.136, 0.184]0.039 [0.016, 0.062]36.343.151.0 Qwen majority300100.00.135 [0.114, 0.156]0.065 [0.043, 0.086]31.743.154.3 Qwen agree 2/324280.70.111 [0.090, 0.133]0.094 [0.071, 0.118]24.843.061.2 Qwen unanimous12140.30.074 [0.050, 0.100]0.141 [0.110, 0.173]16.543.571.1 Qwen, not unanimous17959.70.218 [0.184, 0.252]−0.030 [−0.057,−0.002]49.742.837.4 Smol29799.00.270 [0.243, 0.298]−0.073 [−0.093,−0.052]60.343.029.0 Smol majority28996.30.255 [0.227, 0.283]−0.056 [−0.078,−0.034]57.443.130.8 Smol agree 2/313143.70.216 [0.177, 0.256]−0.022 [−0.055, 0.011]49.642.738.2 Smol unanimous72.30.208 [0.065, 0.369]0.043 [−0.108, 0.200]57.150.042.9 Cross-model5919.70.249 [0.187, 0.316]−0.060 [−0.111,−0.011]54.239.033.9 Qwen, judges disagree24180.30.138 [0.114, 0.163]0.064 [0.039, 0.087]32.044.155.2 Oracle300100.00.000 [0.000, 0.000]0.199 [0.186, 0.213]0.043.1100.0 Table 5: All policies. Cov.: share of 300 prompts acted on (%). BMR, Rand. BMR, and Best in %. “Qwen, not unanimous” and “Qwen, judges disagree” apply Qwen’s original-order choice to the prompts that the respective gates reject (exploratory). PolicyMed. regretBMR .05 Rand. BMR .05 Near-bestBelow medianOverall regret Random0.16737.037.042.634.50.201 Qwen0.00030.737.051.728.70.178 Smol0.25052.536.829.649.20.268 Qwen unanimous0.00012.437.671.912.40.084 Table 6: Sensitivity to outcome definitions (rates in %). Policy∆Stereotype∆Miss. explicit∆Miss. implicit∆QualityOverall gain Qwen0.023 [0.002, 0.045]−0.056 [−0.084,−0.026]−0.034 [−0.057,−0.009]−0.031 [−0.052,−0.008]0.023 [0.000, 0.046] Smol−0.012 [−0.032, 0.009]0.110 [0.082, 0.139]0.030 [0.007, 0.054]−0.038 [−0.064,−0.014]−0.069 [−0.088,−0.050] Qwen unanimous0.010 [−0.019, 0.039]−0.174 [−0.216,−0.130]−0.091 [−0.125,−0.056]0.054 [0.024, 0.082]0.127 [0.098, 0.155] Cross-model0.014 [−0.039, 0.071]0.065 [0.006, 0.126]0.046 [−0.008, 0.103]−0.154 [−0.207,−0.101]−0.098 [−0.147,−0.050] Qwen, Bonf.[−0.002, 0.051][−0.090,−0.020][−0.062,−0.004]– Table 7: Selected minus matched random for secondary outcomes, with pointwise 95% bootstrap intervals. For error ratings, negative values mean fewer errors; for quality and overall gain, positive values are better. CountryRand.QwenQwen gain [95% CI]BMRSmolSmol gain3/3:n / regret Brazil0.1710.1160.055 [0.000, 0.109]33.30.184−0.02116 / 0.068 Canada0.2070.1000.107 [0.042, 0.173]16.70.267−0.05917 / 0.010 Chile0.1730.1220.051 [−0.020, 0.119]33.30.189−0.01616 / 0.115 China0.2020.203−0.001 [−0.047, 0.048]46.70.289−0.0874 / 0.104 Germany0.1580.1390.019 [−0.042, 0.080]36.70.264−0.10613 / 0.083 India0.1630.181−0.018 [−0.080, 0.046]60.00.193−0.0309 / 0.083 Iran0.2270.1640.063 [−0.042, 0.160]26.70.306−0.07914 / 0.048 Japan0.2660.275−0.009 [−0.093, 0.073]46.70.400−0.1349 / 0.056 Poland0.2170.1600.058 [−0.011, 0.126]33.30.313−0.10615 / 0.106 South Africa0.2060.1390.067 [−0.003, 0.136]30.00.296−0.0898 / 0.104 Table 8: Country-group results (30 prompts each). Rand., Qwen, Smol: mean regret. Gains are paired with random on the same prompts. BMR in %. Intervals are pointwise. CategorynRand. regretQwen regret [95% CI]Qwen gain [95% CI] Dates of significance1030.1890.129 [0.094, 0.166]0.061 [0.025, 0.095] Etiquette790.1960.185 [0.134, 0.237]0.012 [−0.033, 0.056] Family370.2140.151 [0.091, 0.217]0.063 [−0.009, 0.136] Greetings450.2040.143 [0.087, 0.203]0.062 [0.009, 0.116] Religion360.2110.225 [0.148, 0.308]−0.013 [−0.086, 0.058] Table 9: Category results for Qwen’s original-order choices (descriptive). JudgeCallsValidParse failRuntime failExact schemaRecoveredNon-JSONHours Qwen90090000900001.15 Smol90088614008868800.27 Table 10: Output validity over all planned calls. Hours is total recorded inference time. E Complete Policy Results Table 5 lists every policy, including majority and two-of- three rules for both judges and the complements of each agreement gate. Gains are paired with exact random selec- tion on the same prompts; BMR is compared with the random event probability on those prompts. Table 6 reports alterna- tive definitions: median regret, a 0.05 margin for below-mean selection, near-best selection, selection below the candidate median, and regret measured with overall satisfaction instead of prompt alignment. Conclusions are unchanged under these alternatives. F Secondary Outcomes Table 7 reports paired differences from matched random selection for each cultural-error rating and image quality, together with the overall-satisfaction gain. For Qwen, the last row gives Bonferroni-adjusted intervals across the three cultural-error outcomes. Qwen’s choices reduce missing ex- plicit and implicit expectations under this adjustment; the stereotype increase does not survive it. G Country and Category Results Table 8 expands Figure 8. The last column reports how many prompts per country pass the unanimity gate and their re- gret; the gate represents all countries, but acceptance ranges widely across groups. Table 9 reports the declared category analysis. H Ablation Details Table 11 lists every condition summarized in Section 5.5, and Table 12 the scale extension of Section 5.6. Taken alone, the added fourth rotation is not balanced across generators, so only the combined four-order results are reported. I Content-Only Baseline and Annotation Robustness Table 13 compares the CLIP selector with the judges; Ta- ble 14 repeats the Qwen analysis on prompts split by annota- tor agreement and on prompts with a unique best candidate. Annotator disagreement is the mean, over candidates with at least two ratings, of the variance of individual alignment ratings; the split is at its median. ConditionFirst slot (%)Uniform (%)Choice changes (%)Gain [95% CI]Unanimous: coverage, gain [95% CI] Main (3 orders)48.827.859.70.039 [0.016, 0.062]40.3%, 0.141 [0.110, 0.173] No ranking (choice-only format)44.827.854.00.047 [0.024, 0.069]46.0%, 0.123 [0.094, 0.153] Opaque codes (no letters)51.827.862.70.021 [−0.001, 0.044]37.3%, 0.124 [0.087, 0.160] Four orders (4-image pools)52.525.072.50.028 [−0.000, 0.056]27.5%, 0.127 [0.073, 0.178] Table 11: Post-hoc ablations (Qwen, 300 prompts; four orders use the 200 four-image pools). Choice changes: share of prompts whose returned image differs across orders. Unanimous: prompts where all orders return the same image. JudgeFirst slot (%) Choice changes (%) Gain [95% CI]Regret BMR (%) Unanimous: cov. (%), gain Rejected: gain [95% CI]∆Stereo. [95% CI] Qwen3-VL-4B (main, bf16)48.859.7 0.039 [0.016, 0.062] 0.16036.3 40.3, 0.141−0.030 [−0.057,−0.002] 0.023 [0.002, 0.045] Qwen3-VL-4B (8-bit control)48.257.3 0.044 [0.022, 0.066] 0.15536.0 42.7, 0.135−0.023 [−0.051, 0.004] 0.021 [0.001, 0.043] Qwen3-VL-8B (8-bit)27.343.0 0.097 [0.077, 0.117] 0.10224.0 57.0, 0.1180.068 [0.036, 0.099]0.016 [−0.004, 0.036] Table 12: Scale extension (300 prompts, three orders). The 8-bit runs use Apple MLX with the same inputs, decoding, and parser as the main run; the 8-bit 4B run controls for quantization and agrees with the main 4B run on most calls. Rejected: prompts on which the three orders disagree. Selectorn Regret [95% CI]Gain [95% CI]BMR Rand. BMRBest∆Stereotype [95% CI] Random 300 0.199 [0.186, 0.213] 0.000 [0.000, 0.000]43.143.141.9 – CLIP300 0.130 [0.109, 0.153] 0.069 [0.047, 0.090]28.043.155.7 0.026 [0.006, 0.046] Qwen300 0.160 [0.136, 0.184] 0.039 [0.016, 0.062]36.343.151.0 0.023 [0.002, 0.045] Smol297 0.270 [0.243, 0.298]−0.073 [−0.093,−0.052]60.343.029.0−0.012 [−0.032, 0.009] Oracle300 0.000 [0.000, 0.000] 0.199 [0.186, 0.213]0.043.1 100.0−0.020 [−0.034,−0.006] Selector∆Stereotype∆Miss. explicit∆Miss. implicit CLIP0.026 [0.006, 0.046]−0.113 [−0.141,−0.083]0.003 [−0.022, 0.028] Qwen0.023 [0.002, 0.045]−0.056 [−0.084,−0.026]−0.034 [−0.057,−0.009] PromptsnQwen gain [95% CI]Qwen regret Qwen agrees with CLIP1470.099 [0.067, 0.130]0.111 Qwen disagrees with CLIP153−0.018 [−0.047, 0.011]0.206 Table 13: Content-only CLIP selector (argmax prompt–image similarity, position-invariant by construction) versus the judges, original order. BMR, Rand. BMR, and Best in %. ∆Stereotype is selected minus matched random. SubsetPolicynRand. regretRegret [95% CI]Gain [95% CI]BMRRand. BMR∆Stereotype [95% CI] High consensusQwen1380.1850.138 [0.106, 0.172]0.047 [0.014, 0.079]31.940.30.033 [0.004, 0.063] High consensusQwen unanimous600.2170.072 [0.037, 0.111]0.146 [0.100, 0.192]15.042.10.009 [−0.029, 0.049] Low consensusQwen1620.2110.178 [0.146, 0.212]0.033 [0.002, 0.062]40.145.40.015 [−0.014, 0.045] Low consensusQwen unanimous610.2130.076 [0.045, 0.111]0.137 [0.095, 0.178]18.044.90.012 [−0.029, 0.054] Unique bestQwen1810.2500.195 [0.163, 0.229]0.056 [0.025, 0.087]38.749.20.014 [−0.012, 0.040] Unique bestQwen unanimous800.2600.092 [0.060, 0.127]0.169 [0.127, 0.210]18.848.8−0.008 [−0.040, 0.024] CategoryQwen callsQwen A (%)Unif. (%)Qwen flip (%)Smol A (%)Unif. (%)Smol flip (%) Dates of significance30948.228.656.329.928.698.0 Etiquette23749.426.863.345.126.897.4 Family11144.128.651.450.928.697.2 Greetings13551.927.266.763.627.297.6 Religion10850.027.361.141.927.397.1 All90048.827.859.743.027.897.6 Table 14: Robustness to annotation consensus (exploratory). Rand. regret is random choice on the same prompts. J Additional Analyses This section reports analyses requested in internal review: the configuration and backbone sensitivity of the CLIP base- line (Table 15), the cost of order checks (Table 16), the relation between order sensitivity and pool difficulty (Ta- ble 17), a decomposition of the stereotype result (Tables 19– 21), cultural-error profiles of aggregated policies (Table 18), and risk–coverage operating points with three and four orders (Figure 14, Table 22). All are post hoc and exploratory. K Presentation Position Table 23 separates position bias by pool size. Qwen favors slot A in both pool sizes. Smol favors slot B for three-image pools and slots A and B for four-image pools, and rarely chooses later slots. L Output Validity Table 10 summarizes parsing outcomes. Qwen satisfied the strict two-field schema on every call. Smol rarely produced valid JSON; its choices were recovered by the predeclared single-label normalization described below, and every recov- ery is flagged in the logs. M Design Choices and Anticipated Questions Why prompt alignment as the primary outcome? It is the rating most directly tied to whether a returned image sat- isfies the request, and it is available for every candidate. Cultural-error ratings are reported separately rather than folded into a composite, because the results show they can move in different directions. Why not debias position directly? Order averaging or position-balanced prompting would change the frozen policy after observing results. We report the bias and leave debiased selectors to future work, where they can be evaluated with the same audit. What happens to abstained requests? We do not model a human fallback. Coverage is reported alongside every se- lective estimate so that the cost of abstention remains visible. 0%20%40%60%80%100% Coverage (share of prompts automated) 0.00 0.05 0.10 0.15 0.20 0.25 Selective regret 1/3 2/3 3/3 1/4 2/4 3/4 4/4 Qwen plurality Random, same prompts 3 orders, 300 prompts 4 orders, 200 pools Figure 14: Risk–coverage operating points for agreement gates with three and four orders, next to matched random choice (95% intervals). N Interactive Audit Browser To make every decision inspectable, the artifact includes an offline browser built only from the recorded judge logs and released ratings (Figures 15–20). The overview reproduces the headline estimates, slot preference, country regret, and the kept-versus-rejected gains of each gate. The prompt ex- plorer filters pools by country, category, and outcome (for example, below-mean choices or prompts where the orders disagree). The detail view shows each pool under all three orders with both judges’ choices, the per-candidate ratings, and whether each gate would act. The browser runs locally, needs no network access, and does not redistribute source images. SelectornRegret [95% CI]Gain [95% CI]BMRRand. BMR∆Stereotype [95% CI] Random3000.199 [0.186, 0.213]0.000 [0.000, 0.000]43.143.1– CLIP ViT-L/143000.130 [0.109, 0.153]0.069 [0.047, 0.090]28.043.10.026 [0.006, 0.046] CLIP ViT-B/323000.136 [0.114, 0.159]0.063 [0.042, 0.085]32.043.10.003 [−0.016, 0.023] Qwen3000.160 [0.136, 0.184]0.039 [0.016, 0.062]36.343.10.023 [0.002, 0.045] Oracle3000.000 [0.000, 0.000]0.199 [0.186, 0.213]0.043.1−0.020 [−0.034,−0.006] Table 15: CLIP baseline with two backbones (original order). Both embed the raw prompt and each image with default CLIP preprocessing and return the most similar image. BMR in %. JudgeCallsMed. s/callMean s/call1-order s/prompt3-order s/promptRatio1-order h3-order h Qwen9003.54.65.413.82.570.451.15 Smol9001.01.11.23.22.800.100.27 Table 16: Inference cost on Apple Silicon (MPS, bfloat16, greedy decoding). Per-prompt times compare a single order with three orders. StratumRangenFlip (%)95% CI Low0.00–0.3314862.8[54.7, 70.3] Mid0.38–0.508359.0[48.2, 69.9] High0.58–1.006953.6[42.0, 65.2] 1 near-tie–17755.9[48.6, 63.3] 2 near-ties–8765.5[55.2, 74.7] 3 near-ties–3259.4[43.8, 75.0] 4 near-ties–4100.0[100.0, 100.0] Table 17: Share of prompts on which Qwen’s choice changes across the three orders, by tertile of the pool’s rating range (best minus worst candidate). Policyn∆Stereotype∆Miss. explicit∆Miss. implicit Qwen3000.023 [0.002, 0.045]−0.056 [−0.084,−0.026]−0.034 [−0.057,−0.009] Qwen majority3000.007 [−0.013, 0.027]−0.092 [−0.120,−0.064]−0.045 [−0.068,−0.021] Qwen unanimous1210.010 [−0.019, 0.039]−0.174 [−0.216,−0.130]−0.091 [−0.125,−0.056] Qwen, not unanimous1790.032 [0.003, 0.063]0.025 [−0.010, 0.059]0.005 [−0.025, 0.037] Table 18: Cultural-error differences from matched random choice for Qwen’s direct, majority-vote, and unanimous policies, and for the prompts unanimity rejects (95% CI). Chosen generatorUniform (%)Qwen (%)Stereo.Within-pool∆Stereo. when chosen [95% CI]Contribution Flux.1-Dev27.33.30.074−0.051−0.089 [−0.164,−0.028]−0.003 GPT-Image27.241.00.114−0.0090.006 [−0.022, 0.035]0.003 Imagen317.914.30.1460.0260.042 [−0.021, 0.111]0.006 SD-3.5-Large27.641.30.1650.0410.042 [0.009, 0.077]0.018 All100.0100.0–0.023 [0.002, 0.045]0.023 Table 19: Stereotype decomposition by the generator of Qwen’s chosen image: choice share against uniform, mean stereotype rating of the generator’s images, and the stereotype difference from random when that generator is chosen. PromptsnQwen∆Stereo. [95% CI]Oracle∆Stereo. [95% CI]Qwen regret Alignment-best tied1190.038 [0.004, 0.073]−0.008 [−0.024, 0.007]0.107 Unique alignment-best1810.014 [−0.012, 0.040]−0.028 [−0.048,−0.008]0.195 Table 20: Stereotype difference on prompts where several candidates tie for the best alignment rating versus prompts with a unique best candidate. GeneratorA (%)B (%)C (%)D (%)All (%) Flux.1-Dev22.0 (296)3.0 (296)1.3 (158)0.0 (138)8.6 GPT-Image86.7 (233)61.4 (295)41.0 (295)27.4 (62)58.9 Imagen365.8 (73)15.4 (149)16.6 (211)19.5 (200)22.9 SD-3.5-Large41.6 (298)6.9 (160)6.8 (236)3.5 (200)17.7 Table 21: Qwen’s choice rate for each generator by presentation slot, pooled over the three orders (number of presentations in parentheses). Protocolq≥n Cov. (%) Regret [95% CI]Rand. regret [95% CI] Difference [95% CI] 3 orders, all prompts1/3 300100.0 0.135 [0.114, 0.156] 0.199 [0.186, 0.213]−0.065 [−0.086,−0.043] 3 orders, all prompts2/3 24280.7 0.111 [0.090, 0.133] 0.206 [0.192, 0.220]−0.094 [−0.118,−0.071] 3 orders, all prompts3/3 12140.3 0.074 [0.050, 0.100] 0.215 [0.193, 0.237]−0.141 [−0.173,−0.110] 3 orders, 4-image pools1/3 200100.0 0.151 [0.125, 0.179] 0.206 [0.190, 0.222]−0.055 [−0.081,−0.028] 3 orders, 4-image pools2/3 15979.5 0.124 [0.097, 0.152] 0.210 [0.193, 0.228]−0.086 [−0.114,−0.057] 3 orders, 4-image pools3/37336.5 0.080 [0.049, 0.114] 0.219 [0.194, 0.245]−0.139 [−0.179,−0.097] 4 orders, 4-image pools1/4 200100.0 0.145 [0.118, 0.172] 0.206 [0.190, 0.222]−0.061 [−0.087,−0.034] 4 orders, 4-image pools2/4 17788.5 0.132 [0.105, 0.161] 0.211 [0.195, 0.228]−0.079 [−0.106,−0.050] 4 orders, 4-image pools3/4 11356.5 0.122 [0.090, 0.157] 0.219 [0.198, 0.240]−0.097 [−0.131,−0.061] 4 orders, 4-image pools4/45527.5 0.095 [0.058, 0.138] 0.222 [0.191, 0.255]−0.127 [−0.178,−0.073] Table 22: Risk–coverage operating points: selective regret on kept prompts versus matched random choice, for agreement thresholds with three orders (all prompts and four-image pools) and four orders (four-image pools). Figure 15: Audit browser, overview. Headline estimates, presentation-slot preference, country-group regret, kept-versus-rejected gains for each gate, and gain over random for every policy, all computed from the recorded outputs. Figure 16: Audit browser, detail view for the prompt in Figure 1. Left: the three presentation orders with each judge’s choice (Q, S) and the best-rated image (check). Right: per-candidate human ratings, selection metrics for the original order, and choices by order. Images from CulturalFrames (Nayak et al. 2025). Figure 17: Audit browser, prompt explorer filtered to Qwen’s below-mean choices and sorted by regret. Each entry shows the pool with both judges’ original-order choices (Q, S), the best-rated image (star), and the ratings. Figure 18: Audit browser, prompt explorer filtered to prompts on which Qwen picks the same image in all three orders (the unanimity gate keeps these). Figure 19: Audit browser, detail view for the four-image pool with the largest Qwen regret (selected by rule). Both judges return the lowest-rated image, so the cross-model gate would act on this prompt. Figure 20: Audit browser, detail view for a prompt where the two judges agree on a poorly rated image (next-largest regret among cross-model agreements). JudgePoolCallsA (%)B (%)C (%)D (%)Uniform (%) Qwen330054.325.320.3–33.3 Qwen460046.024.718.810.525.0 Smol329419.071.89.2–33.3 Smol459254.939.71.73.725.0 Table 23: Choice share by presentation slot and pool size (%), over all recorded orders. O Estimation Details We use 10,000 nonparametric prompt-level bootstrap re- samples with seed 20261002 for means and paired gains. Each prompt’s candidates and decisions travel together; per- mutation calls are never independently resampled. Binary deterministic-policy BMR uses Wilson intervals. Random- policy BMR averages fractional within-prompt probabilities and uses a bootstrap interval. Country results have subgroup bootstrap intervals, and the worst-group statistic is recom- puted within resamples rather than selecting the upper end- point of a single country’s interval. The pooled prompt bootstrap approximates sampling from the balanced source mixture; it does not hold country counts fixed in every replicate. Worst-group resampling is strati- fied within represented countries, but its percentile interval is descriptive and need not be well calibrated when group means are tied or nearly tied. An interval for the group with the largest observed mean is not simultaneous or selection- adjusted. All intervals condition on the released annotation means. They do not account for annotation uncertainty or re- cruitment bias. Estimates after abstention weight accepted prompts equally, so country weights can differ from the bal- anced source split. We report group coverage and compare random selection on the same accepted prompts. The origi- nal endpoints, acceptance thresholds, and sensitive-set defi- nitions remain fixed throughout the main run. Intervals for the secondary cultural-rating outcomes are pointwise, without adjustment for multiple comparisons. We interpret them as descriptive evidence about the released ratings, not confirmatory evidence of cultural harm across populations. Random-choice below-mean probability. For ran- dom choice the below-mean event is a probability, 1 m p P i 1h pi < ̄ h p , averaged over the prompts in A π . Re- placing it with the indicator of the expected utility would make it identically zero. A sensitivity margin δ = 0.05 re- places ̄ h p by ̄ h p − δ (Table 6). Order-consistencyentropy. Wealsorecord the normalized entropy of the three choices, H p = − P i f pi logf pi / logm p , where f pi is the share of orders choosing x pi . With three calls it can reach one for three candidates but at most log 3/ log 4 for four, so it is not comparable across pool sizes; it plays no role in the gates. Worst-group regret. For country group g, R g is the mean regret over acted-on prompts in g; worst-group regret is max g R g and disparity is max g R g − min g R g . Groups with no accepted prompts are reported as zero coverage, not zero risk. P Judge Instruction The shared instruction reads: You are evaluating candidate images for a generative image system. Select the ONE image that should be returned to the user. Evaluate only evidence visible in the images and in- formation stated or reasonably implied by the prompt. Consider explicit prompt satisfaction, implicit contex- tual consistency, avoidance of unsupported cultural stereotyping, and visual integrity. Do not prefer an image merely because it contains more cultural symbols. Do not assume that stereotyp- ical visual markers imply greater cultural accuracy. The candidate labels are supplied next to each image. Return JSON only, with choice equal to the first ele- ment of ranking and ranking containing every candi- date label exactly once. Do not explain your decision. The user prompt follows this instruction, then each label and image are interleaved. A final text segment states the allowed labels and requires the two JSON keys, without specifying a candidate as an example answer. No human annotation, generator name, country field, or category field is included. Q Invalid Output and Reproducibility Qwen’s strict parser accepts exactly the two requested fields, a complete permutation of supplied labels, and a choice equal to the first ranking entry. It rejects explanatory wrappers, repeated labels, incomplete rankings, and inconsistent top choices. Smol requests exactly one choice key. Its adapter also accepts a bare supplied label or a one-field JSON or Python-literal dictionary using a declared alias (image, selected, selected_image, or candidate), with a value consisting solely of a supplied label, optionally pre- ceded by “Candidate”. Every alias, bare label, or non-JSON literal is marked as a recovery. Multiple fields, multiple la- bels, absent candidates, explanatory prose, and ambiguous values are rejected. The adapter never invents a ranking or consults human scores. Its output instruction replaces the ranking request with a choice-only contract while retain- ing the evaluation criteria. Raw output is retained for every attempt. Exceptions receive one identical retry; unresolved parse failures produce abstentions. Unique request keys com- bine prompt identity and permutation index, and resume checks require unchanged instruction and manifest hashes. All public source revisions, model cards, candidate iden- tities, frozen orders, environment versions, and derived ta- bles are saved. Exact source-image bytes are retained lo- cally; aspect-preserving resizing occurs only at inference. The pipeline accesses only public sources. Floating-point greedy decoding is deterministic within the recorded run- time configuration but need not yield identical choices across hardware or library implementations.