Paper deep dive
SABRE: Scalable and Automated Benchmarking of VLMs under Stress
Zixuan Lan, Luzhe Sun, Matthew R. Walter, Jiawei Zhou
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind, making weaknesses hard to identify. Building stress tests is costly: samples must satisfy controlled conditions, remain answerable, and challenge current models. We present SABRE, a scalable, automated pipeline that converts a Test Primer (a Markdown Task Design with Data Schema) into structured specifications, generated or edited images, and question-answer pairs. Automated filtering removes candidates solved by a Filtering VLM, while human review verifies candidate validity and supports annotation correction and localized image repair. We instantiate SABRE-Prior to test whether VLMs follow visual evidence instead of relying on world priors -- learned expectations about familiar objects and scenes. Its 600 images and 1,000 questions span Context (unexpected entities in familiar scenes), Texture (counterfactual materials), Attribute (noncanonical component counts), and Language Elicitation (answers suggested by language but unsupported by the image). Across six VLMs, macro-average accuracy ranges from 17.8% to 31.3% (22.6% mean). A real-image Attribute control is comparably difficult for the Filtering VLM. SABRE-Counting and SABRE-Spatial pilots show that the workflow supports other stress-test settings. These results establish SABRE as a reusable framework for constructing and refreshing VLM stress tests rather than a single fixed benchmark.
Tags
Links
- Source: https://arxiv.org/abs/2608.07435v1
- Canonical: https://arxiv.org/abs/2608.07435v1
Trouble viewing inline? Open PDF directly →
Full Text
84,376 characters extracted from source content.
Expand or collapse full text
SABRE: Scalable and Automated Benchmarking of VLMs under Stress Zixuan Lan ∗,† University of Chicago zixuanlan@uchicago.edu Luzhe Sun ∗ Toyota Technological Institute at Chicago luzhesun@ttic.edu Matthew R. Walter Toyota Technological Institute at Chicago mwalter@ttic.edu Jiawei Zhou Stony Brook University jiawei.zhou.1@stonybrook.edu Abstract Vision-language models (VLMs) are improv- ing rapidly, but benchmark development lags behind, making weaknesses hard to identify. Building stress tests is costly: samples must satisfy controlled conditions, remain answer- able, and challenge current models. We present SABRE, a scalable, automated pipeline that converts a Test Primer(a Markdown Task De- sign with Data Schema) into structured spec- ifications, generated or edited images, and question–answer pairs. Automated filtering re- moves candidates solved by a Filtering VLM, while human review verifies candidate validity and supports annotation correction and local- ized image repair. We instantiate SABRE-Prior to test whether VLMs follow visual evidence instead of relying on world priors—learned ex- pectations about familiar objects and scenes. Its 600 images and 1,000 questions span Context (unexpected entities in familiar scenes), Tex- ture (counterfactual materials), Attribute (non- canonical component counts), and Language Elicitation (answers suggested by language but unsupported by the image). Across six VLMs, macro-average accuracy ranges from 17.8% to 31.3% (22.6% mean). A real-image Attribute control is comparably difficult for the Filter- ing VLM. SABRE-Counting and SABRE- Spatial pilots show that the workflow supports other stress-test settings. These results estab- lish SABRE as a reusable framework for con- structing and refreshing VLM stress tests rather than a single fixed benchmark. Project web- site and resources will be available athttps: //zesearch.github.io/vlm-SABRE/. 1 Introduction Vision-language models (VLMs) are advancing rapidly, while reliable benchmarks remain slow to design, construct, and validate(Fu et al., 2024; Yang et al., 2025). As fixed benchmarks saturate, * Equal contribution. † Work done as a research intern at Stony Brook University. they reveal less about the weaknesses that remain and provide less guidance for further model devel- opment. This paper asks a direct question: Can we construct targeted and reliable VLM stress tests fast enough to keep pace with model development? Large-scale visual benchmarks have traditionally been built by collecting real images and annotating them with human labor. ImageNet (Deng et al., 2009) and MS COCO (Lin et al., 2015) are land- mark examples of this approach. This approach is especially costly for stress tests, which evaluates a target capability under deliberately demanding, distracting, or counterintuitive conditions while keeping the task valid and answerable. Build- ing stress tests from real-image collections is slow and labor-intensive, benchmark designers must devise enough rare and counterintuitive exam- ples to probe the target failure mode systematically which makes it extremely hard to scale up. To re- duce annotation costs, several approaches automati- cally derive questions or labels from existing image collections. For example, POPE constructs object- existence questions from images and object anno- tations, while AutoConverter automatically turns existing visual questions into challenging multiple- choice questions (Li et al., 2023; Zhang et al., 2025). These approaches show that much of the an- notation process can be automated. However, they still inherit the images from source datasets where the visual evidence in these images may not be suf- ficient for complicated stress test setting. Image generation alone does not produce a valid bench- mark: generators may omit requested objects, alter unrelated regions, or produce incorrect counts(Hu et al., 2023; Huang et al., 2025; Ghosh et al., 2023), causing questions and reference answers derived from the benchmark specification to disagree with the actual image. Benchmark construction there- fore requires both specification-faithful generation and explicit verification. To address these challenges, we introduce a mod- arXiv:2608.07435v1 [cs.CV] 7 Aug 2026 UserTask Design Describe the task you want to test in natural language. Data Schema (a) (b) (c) ⋯ Base Image Description Area of Interest Image Edition Plan LLM Nano BananaFlux Image Generation and Edition Attribute TextureSpatial Language Context Counting Various Category Test Image Question & Answer Builder Q1 A1 Q2 A2 Q3 A3 Q4 A4 Automated Filtering (Reject Sampling) Q1 A1 Q2 A2 Q3 A3 Q4 A4 Edit Filtering VLM Discard Convenient Human Evaluation Platform Scalable Customized VLMs Stress Test SABRE Scalable and Automated Benchmarking of VLMs under Stress Sample Specifications Test Primer ×N Figure 1: Overview of the SABRE benchmark-construction pipeline. ular pipeline in which a benchmark designer pro- vides a Test Primer which consists of a natural- language Markdown Task Design, a Data Schema that defines task-specific fields and validation rules, and question format, such as open generation or multiple choice. The pipeline converts this spec- ification into structured sample specifications. It then generates the required images, applies task- oriented edits when needed, and builds the corre- sponding question–answer pairs. Each candidate is evaluated by a filtering VLM through a model-in- the-loop Automated filtering. Candidates solved by the filtering model are discarded, while those expos- ing a potential failure are retained. Because auto- matic filtering establishes difficulty but not validity, every retained candidate must undergo human ver- ification.For tasks that require a specified image edit, reviewers compare the base imageI b with the edited imageI e to verify that the requested change is present and unrelated content is preserved. Re- viewers can revise questions, reference answers, and region annotations, or repair localized image defects without regenerating the entire image. The same interface also accepts uploaded real images, which pass through the same screening and cura- tion stages. We first instantiate the pipeline as SABRE-Prior, which tests whether VLMs follow visual evidence when it conflicts with world priors, namely expecta- tions learned from training data about how familiar objects, scenes, and situations normally appear(Liu et al., 2025; Lee et al., 2025; Luo et al., 2025; Vo et al., 2026; Guan et al., 2024). Such conflicts are underrepresented in conventional image collections because they are intentionally unusual and often require precise image edits. Nevertheless, these atypical scenes, edited con- tent, and noncanonical object configurations can test a basic requirement of visual grounding: a model should answer from the image supplied by the user rather than from what it expects to see. Us- ing the pipeline, we construct SABRE-Prior across four complementary subsets: unexpected objects in familiar scenes (Context), answers suggested by question wording but unsupported by the im- age (Language Elicitation), counterfactual object materials (Texture), and noncanonical component counts (Attribute). To examine the pipeline in different stress-test settings, we also apply it to Counting and Spatial reasoning using separate Test Primers. These two small-scale pilots are not intended as exhaustive evaluations on model capa- bility; instead, they examine whether the pipeline can construct new types of stress tests without task- specific redesign. We evaluate SABRE-Prior on six recent fron- tier VLMs. Across its four world-prior subsets, macro accuracy ranges from 17.8% to 31.3%, with an average of 22.6% across models. The distinct performance profiles across subsets reveal com- plementary failures in following visual evidence and rejecting unsupported prior expectations. To- gether, the SABRE-Prior results and the two pilot instantiations support the central contribution of this work: SABRE is not a single fixed benchmark, but a reusable framework for constructing, verify- ing, and refreshing targeted VLM stress tests as models evolve. 2 Related Work VLM Evaluation and Stress Testing. VLM evaluation has expanded from task-specific VQA to broad suites covering perception, reasoning, instruction following, and hallucination, such as MME, MMBench, and M-Vet, alongside tar- geted stress tests such as POPE and MMVP (Fu et al., 2025; Liu et al., 2024; Yu et al., 2024; Li et al., 2023; Tong et al., 2024). These benchmarks show that strong aggregate performance can coex- ist with systematic failures on carefully selected visual cases. However, most are fixed evaluation sets focused on what to measure, whereas our work studies how targeted stress tests can be constructed, screened, and maintained as models and evaluation needs evolve. Automated and Generative Benchmark Con- struction. Prior work reduces benchmark con- struction costs by automatically generating ques- tions, answers, and distractors from existing image collections. For example, POPE derives object- existence questions from image annotations, while AutoConverter transforms existing VQA questions into challenging multiple-choice evaluations (Li et al., 2023; Zhang et al., 2025). Although these methods automate substantial annotation work, they cannot control the visual evidence contained in the source images. Image generation, editing, and procedural rendering address this limitation by di- rectly constructing evaluation images. ImageNet-D and JourneyBench use generated images to intro- duce controlled or unusual visual conditions, while Vision-Language Bootstrapping, Auto-Comp, and InfiniBench use dynamic image editing, synthetic image pairs, and customizable 3D scenes (Zhang et al., 2024; Wang et al., 2024; Yang et al., 2025; Sbrolli et al., 2026; Wang et al., 2026). How- ever, generative construction makes sample valid- ity dependent on whether each image realizes its intended specification. Our pipeline therefore in- tegrates structured sample specifications, image generation and targeted editing, pressure screen- ing, human verification, annotation revision, and localized repair. It also supports generated images and uploaded real images within the same curation workflow. World Priors and Visual Evidence Conflicts.A growing body of work studies whether VLMs fol- low visual evidence when it conflicts with lan- guage or world knowledge. The PhD benchmark includes a counter-common-sense subset, PhD-ccs, that uses AI-generated images violating everyday expectations to evaluate visual hallucination (Liu et al., 2025). VLind-Bench measures language pri- ors using DALL-E 3-generated counterfactual im- ages together with auxiliary tests of commonsense knowledge, visual perception, and commonsense bias, thereby reducing alternative explanations for a model’s failure (Lee et al., 2025). ViLP constructs out-of-distribution image–question–answer triplets in which one condition can be answered from tex- tual priors while the others require evidence from generated images (Luo et al., 2025). VLMBias makes controlled counterfactual modifications to familiar subjects, such as well-known logos, ani- mals, flags, and game pieces, to test whether mem- orized canonical knowledge overrides visual count- ing and identification (Vo et al., 2026). Hallusion- Bench is broader than a world-prior benchmark: it uses human-collected and human-edited control pairs to diagnose both language hallucination and visual illusion, although some edited cases explic- itly place visual content in conflict with common knowledge (Guan et al., 2024). These benchmarks directly motivate our world-prior case study, and we include them as comparison benchmarks in our experiments. Our primary goal, however, is not to introduce another isolated test of prior reliance, but to use this demanding setting to instantiate and evaluate a general pipeline for scalable stress-test construction. 3 Method 3.1 Overview and Problem Formulation Our goal is to automatically create a VLM stress- test benchmark based on the pipeline. The pipeline takes aTest Primer, denoted byS, which consists of a natural-language MarkdownTask Design, aData Schemathat defines task-specific fields and validation rules, and question format, such as open generation or multiple choice. Given S, the pipeline automatically constructs a set of Candidate Samples before human verification: C(S) =x i N i=1 . Each candidatex i = (I i , q i , y i )consists of an im- age I i , a question q i , and its reference answer y i . As illustrated in Figure 1, an LLM first converts the test primerSintoSample Specifications. Each sample specification is a JSON file containing the information required by the subsequent steps, such as image generation and editing prompts. Let z i denote thei-th sample specification. An im- age generation model, denoted byG, generates the required images. When a design requires an image modification, an image editing model, de- noted byE, edits the generated image. After can- didate set construction, the candidate samples en- terAutomated Filtering. A VLM, denoted by Mand called theFiltering VLM, serves as the Automated Filteringand evaluates each candi- date sample. Candidate samples answered correctly Context Zoom InEdit Dentist Mirror Teaspoon → Is the dentist using a teaspoon? Zoom InEdit USB Drive Lighter → Is there a black lighter visible in this digital forensics hardware-imaging lab? Folding Feeler Nail Clipper → Zoom InEdit Is there a nail clipper visible in this engine repair workshop? Texture Zoom InEdit Wood Mallet Fabric Mallet → Does the mallet appear to have a fabric surface? Zoom InEdit Plastic Fabric → Does the traffic cone appear to have a fabric surface? Metal wrench Form wrench → Zoom InEdit Is there a foam wrench on the workbench in the auto repair shop? Attribute How many tentacles are visible on the octopus? World Prior: 8 Test Edition: 12 Language Elicitation Background: It’s early in the morning at a quiet office desk, and someone has just come back from the break area after setting down their bag and opening a laptop for the day. A mug has been placed beside the notebook and pen, ready for the first few minutes of work. Question: What drink is most likely inside the mug? Choose: A. Coffee B. Tea C. Water D. Unknown How many circular lamps are visible on the traffic light? World Prior: 3 Test Edition: 5 How many tines does this fork have? World Prior: 4 Test Edition: 6 Figure 2: Samples of SABRE-Prior in 4 different subsets. byMare filtered out, while those answered incor- rectly are retained and sent to our Convenient Hu- man Evaluation Platform, for human verification and image repair. Candidate Samples that pass hu- man verification becomeSamplesand are included in the final benchmark. 3.2 Modular Task Recipes A test primerSdefines one stress-test topic. It contains a natural-language Markdown task design, an data schema, and a selected question format. The Markdown file states what to test and how to construct the images. It can specify the scene, the visual content to add, remove, or modify, and the content that must remain unchanged. The question format specifies how the VLM should answer, such as open generation or multiple choice. The data schema controls the structure of the JSON output. It defines the data types (e.g., text or lists) and other constraints. The pipeline uses the default data schema when the task recipe does not provide a custom one. A design LLM readsSand produces sample specifications in JSON. Each sample specification contains the information needed to construct one candidate sample, including an image generation or editing prompt, a question, and its reference answer. A new stress-test topic only requires a new test primer. The pipeline reuses the same image genera- tion, pressure screening, human verification, image repair, and export workflow for all topics. This separation provides extensibility without requiring a new benchmark-construction pipeline for each topic. 3.3 Automatic Candidate Sample Construction Each sample specificationz i contains an image prompt, a questionq i , and a reference answer y i . For a single-image task, the image generation modelGgenerates an imageI i from the prompt p i : I i = G(p i ). Some stress-test topics test whether a VLM re- sponds to a specific visual change, such as remov- ing an object or changing an object attribute. These topics require image editing. The edit changes the target visual evidence while keeping the rest of the scene unchanged. This isolates the target change and tests whether the VLM updates its an- swer based on that change. For an editing task,z i contains a base-generation promptp b i and an editing promptp e i . The pipeline usesGto generate the base imageI b i and the image editing model E to produce the edited image I e i : I b i = G(p b i ),I e i = E(I b i , p e i ). The editing prompt states what to change and what to preserve. A real image can replaceI b i , which skips the image generation step. The pipeline pairs each applicable image with one question and its reference answer. Each sam- ple specification produces exactly one candidate sample: x = (I, q, y). After human verification, each accepted sample forms or joins a case. A case contains one sample for one-to-one tasks and multiple samples when several image–question pairs derive from the same underlying image construction or visual interven- tion. The pipeline stores the prompts, model identi- fiers, and generation records as metadata. 3.4 Model-in-the-Loop Pressure Screening Automated filtering removes candidate samples that the Filtering VLMManswers correctly and retains those that it answers incorrectly. For each candidate samplex i = (I i , q i , y i ),Mproduces a response r i = M(I i , q i ). The evaluator comparesr i with the reference an- swery i using the scoring rule for the selected ques- tion format: e i = Eval(r i , y i ),e i ∈0, 1, wheree i = 1indicates a correct response ande i = 0 indicates an incorrect response. The automated filtering retains only the candi- date samples that M answers incorrectly: C pressure (S; M) =x i ∈C(S)| e i = 0. The filtering VLM measures candidate sample’s difficulty, not candidate sample’s validity. An incor- rect response fromMmay reveal a genuine model failure, but it may also result from an unsuccessful image generation or edit, an ambiguous question, or an incorrect reference answer. Every retained candidate sample must therefore undergo human verification before it can become a sample. 3.5 Human Verification and Localized Image Repair Candidate samples retained by the automated filter- ing enter human verification platform. Reviewers inspect the imageI i , questionq i , reference answer y i , sample specificationz i . A candidate sample passes verification only when the required visual evidence is present, the requested edit is correctly applied when needed, the question is unambigu- ous, and the reference answer matches the image. Reviewers can accept or reject a candidate sample, revise its question or reference answer, or repair a local image defect. For image repair, the reviewer selects a bounding boxb i and provides a repair instructionu i . The platform expands the selected region to include its surrounding visual context and crops the resulting patchP i from the image. It marks the target region inside the patch and sends the marked patch, rather than the full image, to Gemini 3.1 Flash Image Preview withu i . Restricting the model input to a local patch focuses the edit on the selected defect and prevents unintended changes to the rest of the image. The repaired patch e P i cannot be directly pasted into the image because a hard boundary may intro- duce visible color and texture discontinuities. We therefore construct a soft maskA i by expanding the target region and applying Gaussian blur to its boundary. The platform blends the repaired and # of yellow parking cones: 24 # of salmon nigiri: 21 # of whole mushrooms: 55 # of red cap test tube: 76# of yellow bike helmet: 51# of eggs: 56 Coordinate system: the object slot in the front- left-bottom grid cell is coordinate (1,1,1), not a geometric corner point. The x axis increases from left to right, the y axis increases from front to back, and the z axis increases from bottom to top. Coordinates are positive integers. What color and shape is the object at coordinate (3,1,2)? Answer only as two words: color shape. GT Answer: Blue Cube x y z -Counting S A B R E -Spatial S A B R E Figure 3: SABRE-counting and SABRE-spatial original patches as P ⋆ i = A i ⊙ e P i + (1− A i )⊙ P i . It then placesP ⋆ i back into the original image. The center of the mask preserves the repaired content, while the smooth boundary gradually transitions to the original pixels. This soft blending reduces rect- angular seams and integrates the repaired region more naturally into the surrounding scene. The reviewer compares the repaired image with the original and may accept it, repair it again, or re- ject the candidate sample. Our questionnaire study shows that the localized repair tool produces more successful and visually natural repairs than direct whole-image Gemini editing and the other evalu- ated methods. Participants consistently preferred the results produced by our tool. The full compar- ison and questionnaire results are provided in the appendix. Letx ⋆ i denote the candidate sample after human verification and any reviewer revisions or image repairs. The reviewer assigns a validity decision v i ∈ 0, 1, wherev i = 1indicates acceptance. The final benchmark is D(S; M) =x ⋆ i | x i ∈C pressure (S; M), v i = 1 . Each item inD(S; M)originates from a pressure- selected candidate sample and passes human verifi- cation. We call each retained item a sample. The curation platform also supports real-image authoring. Users can upload a real image, provide or revise its question and reference answer, and apply localized image repair when needed. RANKMODELCONTEXTTEXTUREATTRIBUTELANGUAGEMACRO AVG. 1 Claude 4.6 1040175831.3 2 Kimi-k2.6 752171723.3 3 Qwen 3.5 346142923.0 4 Gemini 3.5 052261122.3 5 GPT-5.4 128202318.0 6 Grok-4.3 428162317.8 Figure 4: Accuracy (%) of six frontier VLMs on SABRE-Prior. Bars show point estimates, and whiskers indicate the corresponding confidence intervals. Context and Texture report strict All4 accuracy: a case is correct only when all four Base–Edited yes/no probes are answered correctly. Attribute uses normalized exact-match counting accuracy, while Language Elicitation uses four-option multiple-choice accuracy. Macro Average is computed across the four subset accuracies. 4 Benchmark Instantiation We first use our pipeline to construct SABRE-Prior, a benchmark that tests whether VLMs follow vi- sual evidence or learned expectations about what is usually true. We refer to these learned expectations as world priors. SABRE-Prior Stress-Test Subsets.The bench- mark contains four complementary subsets. Con- text places unexpected objects in familiar scenes, such as replacing a microscope in a laboratory with a toaster. Language Elicitation uses question wording to suggest an answer that is unsupported by the image, such as asking what is inside a closed box. Texture assigns familiar objects counterfac- tual materials, such as giving a ceramic mug a furry surface. Attribute modifies canonical component counts while preserving object identity, such as showing a chair with five legs. Context and Texture use paired base–edit images with four complementary yes/no probes. Language Elicitation uses single-image, four-option questions whose correct answer is unknown, while Attribute uses open-ended counting questions over edited images. SABRE-Prior Construction, Scale, and Evalu- ation. All four subsets use the shared workflow in Section 3. For each subset, GPT-5.4 reads its task primerSand produces sample specifications. FLUX.2 [flex], FLUX.2 [pro] (Black Forest Labs, 2025), and Gemini 3.1 Flash Image (Nano Banana 2) (Raisinghani, 2026) generate or edit the images specified by these specifications. Gemini 3.5 Flash serves as the filtering VLMM. Candidate sam- ples retained by the automated filtering undergo hu- man verification and localized repair when needed. Each subset contains 100 cases. The final bench- mark contains 400 cases, 600 images, and 1,000 samples, with one question per sample. Context and Texture are evaluated using paired Base and Edited images. Context asks whether the source and target objects are visible in each image, with expected answers yes/no for Base and no/yes for Edited. Texture uses the same four- probe structure for the expected and counterfactual materials, again producing yes/no for Base and no/yes for Edited. A case is counted as correct only when all four yes/no samples are correct. Attribute presents a single edited image with an open-ended component-counting question; the extracted nu- meric response must exactly match the annotated visible count after normalizing digits and number words. Language Elicitation uses a single-image, four-option question whose wording suggests an answer that cannot be verified visually; the cor- rect option is unknown, with its position balanced across the dataset. We report each subset’s primary accuracy and their macro-average as the overall score. 4.1 Additional Pipeline Instantiations To evaluate the extensibility of our pipeline, we use the same workflow to construct two additional 20-example pilots: Counting and Spatial. Counting presents dense scenes containing dozens of target objects and visually similar distractors, and asks the model to report the number of visible target objects. Responses are scored by normalized exact count. Spatial presents dense multi-layer 3D lat- Source Image Edited Image Source Image Edited Image Source Image Edited Image One Handle Two Handles Four StringsSix StringsFour LegsTen Legs Rubber boot Cardboard boot Fabric CapLeather Cap Wood boardFabric board Aw lCrayfishWrenchGeckoHandleCrab Figure 5: Examples of generated images from SABRE Figure 6: Additional examples of generated images from SABRE. The first row shows Spatial QA instances, where the question-answer format follows the same structure as illustrated in Figure 3. The second row presents Counting instances, including examples that require counting pink slippers, pens, oranges, and mugs from left to right. The third row provides additional Attribute instances. tices and includes multiple question formats, such as identifying the color and shape at a specified coordinate or counting objects that satisfy spatial and visual constraints, for example red cubes on the top layer. Responses are scored by normalized exact attribute matching or exact count, depending on the question type. 5 Experiments 5.1 Experimental Setting We evaluate six frontier VLMs on SABRE-Prior: Gemini 3.5 Flash (Google DeepMind, 2026), GPT- 5.4 (OpenAI, 2026), Claude 4.6 Sonnet (Anthropic, 2026), Kimi-k2.6 (Moonshot AI, 2026), Qwen 3.5 27B (Qwen Team, 2026), and Grok-4.3 (xAI, 2026). These models represent the current state of the art, and our stress tests are designed to iden- tify their blindspots for future improvement. All models are evaluated with zero-shot prompts us- ing greedy decoding (i.e., temperature= 0) when supported. We estimate 95% confidence intervals using a percentile bootstrap with 20,000 resam- ples, resampling cases within each subset. For each resample, we recompute every SABRE-Prior subset accuracy and their macro-average, defined BenchmarkImagesConstruction MethodQuestion TypesGemini 3.5 Flash Acc. (%) PhD-CCS750DALL-E 3Yes/No82.3 VLind-Bench842DALL-E 3Yes/No90.0 ViLP900DALL-E 3 / FLUXOpen identification70.3 HallusionBench181Manual editingYes/No81.6 VLMBias1,392Semi-automated generationCounting and Yes/No60.3 SABRE-Prior600Automated generationAll above and MCQ22.3 Table 1: Same-model comparison using each benchmark’s primary evaluation protocol as the unweighted mean of the subset accuracies. Detailed model identifiers, access dates, prompts, and response-parsing rules are provided in the Ap- pendix. Baselines We additionally evaluate VCD (Leng et al., 2024) and SoM (Yang et al., 2023) on the open-source Qwen 3.5 27B model to examine whether visual-enhancement methods improve per- formance. We also construct a real-image vari- ant of Attribute subset to test whether model fail- ures persist when the edited source images are real rather than generated. We also show more exam- ples shown in 5 and 6. 5.2 Main Results Figure 4 reports the primary subset scores of the six evaluated VLMs. Macro-average accuracy ranges from 17.8% to 31.3%, with a mean of 22.6% across models. Because SABRE-Prior uses pres- sure screening and evaluates different behaviors with different question formats, the macro-average summarizes performance on this benchmark rather than providing a general ranking of model capa- bility. Table 1 provides complementary evidence. Gemini 3.5 Flash achieves 60.3%–90.0% accuracy on five existing benchmarks but only 22.3% on SABRE-Prior. Its low score on SABRE-Prior partly reflects its role as the Filtering VLM. How- ever, the other five models, which were not used during pressure screening, also score below 32%. These results show that the constructed cases ex- pose failures across frontier VLMs rather than only the Filtering VLM. Context obtains the lowest scores under the strict criterion: no model exceeds 10%, and the mean is 4.2%. This reflects the difficulty of consistently recognizing both the original and counterfactual objects across the Base–Edit intervention. Tex- ture scores are higher, ranging from 28% to 52%, suggesting that visible material evidence is more accessible than context-incongruent object identity, although canonical material associations still exert substantial pressure. Attribute ranges from 14% to 26%, combining fine-grained structural percep- tion and exact counting with the need to override canonical component counts. Language Elicitation shows the widest variation, from 11% to 58%. This subset should not be interpreted as a pure measure of visual recognition: the queried information is intentionally unavailable in the image. Instead, it tests whether a model recognizes insufficient visual evidence and rejects a plausible answer induced by the question wording. The large variation there- fore reveals differences in visual grounding, prior reliance, and willingness to abstain. Models perform very differently across subsets. Claude 4.6 achieves the highest macro-average, largely because of its strong Language Elicitation score. Gemini 3.5 Flash and Kimi-k2.6 perform best on Texture, while Gemini 3.5 Flash leads on Attribute but answers no Context cases correctly. The model ranking therefore changes across sub- sets, and no model performs best on all of them. These results show that SABRE-Prior exposes dis- tinct model weaknesses that a single aggregate score would hide. 5.3 Comparison with Existing Benchmarks We further compare SABRE-Prior with five related benchmarks using Gemini 3.5 Flash. As shown in Table 1, the model achieves between 60.33% and 90.02% on PhD-CCS, VLind-Bench, ViLP, HallusionBench, and VLMBias, but only 22.30% on SABRE-Prior. In particular, its accuracy ex- ceeds 80% on three of the existing benchmarks, suggesting that many previously constructed world- prior and hallucination cases are handled reliably by recent frontier models. In contrast, the cases produced by our pipeline continue to expose sub- stantial failures. The benchmarks use different question formats and scoring rules, so their accuracy values are not directly equivalent. However, the results show that MethodCtxTexAttLangAvg. Qwen 3.5 27B346142923.0 + VCD134162719.5 + SoM624122516.8 Table 2: Effect of visual-enhancement methods on Qwen 3.5 27B. Results use the same four subset metrics as Figure 4; Avg. denotes their macro-average. several existing benchmarks provide limited head- room for a current frontier VLM. The low Gemini 3.5 Flash score on SABRE-Prior partly reflects its role as the Filtering VLM, but it also demon- strates that our pipeline can construct new cases that target weaknesses not exposed by increasingly saturated benchmarks. Unlike a fixed test set, our pipeline can generate new stress-test candidates on demand using updated task recipes and Filter- ing VLMs. This allows the evaluation to adapt as models improve. The pipeline supports flexi- ble question formats, including yes/no, multiple- choice, and open-ended counting questions. It can therefore produce scalable stress tests across differ- ent task formats rather than remaining tied to one predefined benchmark. 5.4 Effect of Visual-Enhancement Methods A useful stress-test benchmark should not only expose model failures but also measure whether proposed methods fix them. We use SABRE- Prior to evaluate two visual-enhancement meth- ods on Qwen 3.5 27B. Visual Contrastive Decod- ing (VCD) contrasts predictions from original and perturbed visual inputs, while Set-of-Mark (SoM) adds region markers to improve visual grounding. We evaluate both methods using the same questions and subset metrics as the standard Qwen 3.5 27B setting. As shown in Table 2, neither method consistently improves performance. VCD and SoM improve selected subsets but reduce accuracy on others, re- sulting in lower macro-averages than the standard model. These results indicate that the failures ex- posed by SABRE-Prior cannot be fixed simply by highlighting visual regions or changing the decod- ing signal. This experiment also demonstrates the diagnos- tic value of SABRE-Prior. It shows where a mitiga- tion method helps, where it introduces regressions, and which model weaknesses remain unresolved. The benchmark can therefore support the develop- ment and evaluation of methods designed to ad- Image SourceCorrect / TotalAccuracy Generated (Attribute Modify)26 / 10026% Real-Image Control6 / 2030% Table 3: Real-image control. Exact-match counting accuracy of Gemini 3.5 Flash on generated and real- image Attribute samples. MODELCOUNTINGSPATIAL Claude 4.6 1/200/20 Kimi-k2.6 0/200/20 Qwen 3.5 1/200/20 Gemini 3.5 0/200/20 GPT-5.4 1/200/20 Grok-4.3 1/200/20 Figure 7: Results on two additional stress-test instan- tiations. Each cell represents one sample, and colored cells indicate correct model responses. dress specific VLM failures. 5.5 Real-Image Control We test whether the low model accuracy on SABRE-Prior is caused by generated-image ar- tifacts. We construct 20 control cases from real images using the same editing and evaluation proce- dure as the Attribute. As shown in Table 3, Gemini 3.5 Flash achieves 30% accuracy on the real-image cases, compared with 26% on the generated-image cases. The similarly low accuracies suggest that the benchmark difficulty is not primarily caused by generated-image artifacts. They also support the reliability of the generated stress tests, which ex- pose model failures at a level comparable to tests constructed from real photographs. The pipeline supports both real and generated source images, while generated images provide greater scalability and more precise control over the visual content being tested. 5.6 Additional Stress-Test Instantiations We use two different test primers, to construct addi- tional stress tests for counting and spatial reasoning. Each stress test contains 20 samples and uses the same generation, pressure screening, human veri- fication, and repair workflow. Figure 7 shows that both stress tests are challenging for all six VLMs. Each model answers at most one of the 20 Counting samples correctly, and none answers any Spatial sample correctly. These results contrast with the strong performance of frontier VLMs on many es- tablished benchmarks, which can become saturated as models improve. They also demonstrate the extensibility of our pipeline: different task specifi- cations can be used to construct challenging stress tests for new visual capabilities without changing the shared workflow. 6 Conclusion and Limitations We introduced SABRE, a scalable and modular pipeline for constructing VLM stress tests from natural-language task specifications. The pipeline converts each task specification into structured case specifications, constructs the required images and question–answer pairs, retains challenging candi- dates through failure-only pressure screening, and supports human verification and localized repair. Using this pipeline, we constructed SABRE-Prior, covering four forms of conflict between visual evi- dence and world priors. Six recent VLMs achieve only 17.8%–31.3% macro accuracy and exhibit distinct failure profiles. Representative Inference- time mitigation methods provide no consistent im- provement, suggesting that the benchmark exposes persistent weaknesses in using visual evidence to override prior expectations rather than errors that can be removed by generic inference-time interven- tions. We also apply the same pipeline to Count- ing and Spatial reasoning through separate task specifications, demonstrating its use across differ- ent stress-test settings. A limitation shared by any fixed benchmark suite is that its coverage reflects a finite set of task specifications at the time of re- lease. The modular design of SABRE mitigates this limitation by allowing new task specifications and construction modules to be added as VLM ca- pabilities evolve. References Anthropic. 2026.Introducing Claude Sonnet 4.6.https://w.anthropic.com/news/ claude-sonnet-4-6. Accessed: 2026-07-20. Black Forest Labs. 2025. FLUX.2: Frontier visual intelligence.https://bfl.ai/blog/flux-2. Ac- cessed: 2026-07-20. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. 2025. Mme: A compre- hensive evaluation benchmark for multimodal large language models. Preprint, arXiv:2306.13394. Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, Caifeng Shan, and Ran He. 2024. Mme-survey: A comprehensive survey on evaluation of multimodal llms. Preprint, arXiv:2411.15296. Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt. 2023.Geneval: An object-focused framework for evaluating text-to-image alignment. Preprint, arXiv:2310.11513. Google DeepMind. 2026.Gemini 3.5 flash: Model card.https://deepmind.google/models/ model-cards/gemini-3-5-flash/ .Accessed: 2026-07-20. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2024. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. Preprint, arXiv:2310.14566. Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. 2023. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In 2023 IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 20349–20360. Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 2025.T2i- compbench++: An enhanced and comprehensive benchmark for compositional text-to-image gener- ation. Preprint, arXiv:2307.06350. Kang-il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. 2025. VLind-bench: Measuring language pri- ors in large vision-language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4129–4144, Albuquerque, New Mexico. Association for Computational Linguistics. Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision- language models through visual contrastive decod- ing. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13872– 13882. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucina- tion in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305, Sin- gapore. Association for Computational Linguistics. Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dol- lár. 2015. Microsoft coco: Common objects in con- text. Preprint, arXiv:1405.0312. Jiazhen Liu, Yuhan Fu, Ruobing Xie, Runquan Xie, Xingwu Sun, Fengzong Lian, Zhanhui Kang, and Xirong Li. 2025.Phd: A chatgpt-prompted vi- sual hallucination evaluation dataset.Preprint, arXiv:2403.11116. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024. Mmbench: Is your multi-modal model an all-around player? Preprint, arXiv:2307.06281. Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, and Honglak Lee. 2025. Probing visual language priors in VLMs. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 41120–41156. PMLR. Moonshot AI. 2026. Kimi K2.6: Advancing open- source coding.https://w.kimi.com/blog/ kimi-k2-6. Accessed: 2026-07-20. OpenAI. 2026.Introducing GPT-5.4.https:// openai.com/index/introducing-gpt-5-4/. Ac- cessed: 2026-07-20. Qwen Team. 2026. Qwen3.5: Towards native multi- modal agents.https://qwen.ai/blog?id=qwen3. 5. Accessed: 2026-07-20. Naina Raisinghani. 2026.Nano banana 2: Com- bining pro capabilities with lightning-fast speed. https://blog.google/innovation-and-ai/ technology/ai/nano-banana-2/ .Accessed: 2026-07-20. Cristian Sbrolli, Matteo Matteucci, and Toshihiko Yamasaki. 2026.Auto-comp:An automated pipeline for scalable compositional probing of contrastive vision-language models.Preprint, arXiv:2602.02043. Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 9568–9578. An Vo, Khai-Nguyen Nguyen, Mohammad Reza Tae- siri, Vy Tuong Dang, Anh Totti Nguyen, and Daey- oung Kim. 2026. Vision language models are biased. Preprint, arXiv:2505.23941. Haoming Wang, Qiyao Xue, and Wei Gao. 2026. In- finibench: Infinite benchmarking for visual spatial reasoning with customizable scene complexity. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 21594–21605. Zhecan Wang, Junzhang Liu, Chia-Wei Tang, Hani Alo- mari, Anushka Sivakumar, Rui Sun, Wenhao Li, Md. Atabuzzaman, Hammad Ayyubi, Haoxuan You, Alvi Ishmam, Kai-Wei Chang, Shih-Fu Chang, and Chris Thomas. 2024. Journeybench: A challenging one- stop vision-language understanding benchmark of generated images. In Advances in Neural Informa- tion Processing Systems, volume 37, pages 63110– 63123. Curran Associates, Inc. xAI. 2026.Grok 4.3.https://docs.x.ai/ developers/models/grok-4.3. Accessed: 2026- 07-20. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chun- yuan Li, and Jianfeng Gao. 2023.Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. Preprint, arXiv:2310.11441. Yue Yang, Shuibo Zhang, Kaipeng Zhang, Yi Bin, Yu Wang, Ping Luo, and Wenqi Shao. 2025. Dy- namic multimodal evaluation with flexible complex- ity by vision-language bootstrapping. In Interna- tional Conference on Learning Representations, vol- ume 2025, pages 21399–21420. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. M-vet: Evaluating large multimodal models for integrated capabilities. In Proceedings of the 41st International Conference on Machine Learn- ing, volume 235 of Proceedings of Machine Learning Research, pages 57730–57754. PMLR. Chenshuang Zhang, Fei Pan, Junmo Kim, In So Kweon, and Chengzhi Mao. 2024. Imagenet-d: Benchmark- ing neural network robustness on diffusion synthetic object. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21752–21762. Yuhui Zhang, Yuchang Su, Yiming Liu, Xiaohan Wang, James Burgess, Elaine Sui, Chenyu Wang, Josiah Aklilu, Alejandro Lozano, Anjiang Wei, Ludwig Schmidt, and Serena Yeung-Levy. 2025. Automated generation of challenging multiple-choice questions for vision language model evaluation. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 29580– 29590. A Detailed Experimental Setup Evaluated Models. Table 4 lists the exact model identifiers and inference interfaces used in our experiments. All models were accessed on July 20, 2026. API models were evaluated using the versions exposed under these identifiers on that date. Each model received one image–question sample at a time without in-context examples. Inference Prompts. We use zero-shot, format-specific prompts and do not request explanations or chain-of-thought reasoning. Context and Texture use the following prompt: Question: question Answer with only one word: yes or no. Attribute and Counting use: Question: question Answer with only the count or shortest possible count phrase. Do not explain. Language Elicitation presents the question followed by four options labeled A–D and uses: Question: question and options Answer with only the option letter. Do not explain. Spatial uses the same short-answer instruction as Counting. Depending on the question, the expected response is either a count or a short combination of attributes such as a color and shape. For the local Qwen evaluation, these instructions are wrapped in the model’s standard chat template with a format-only system message requesting no reasoning or additional text. Decoding and Image Input.The maximum output length is restricted to 8–16 tokens for API models, depending on the response format. Qwen uses at most 32 newly generated tokens. Images are supplied at their original stored resolution to API models. GPT-5.4 uses the high image-detail setting. For local Qwen inference, images are converted to RGB and processed using the checkpoint’s AutoProcessor. Qwen is evaluated with do_sample=false, bfloat16 precision, batch size 1, and automatic device placement. The four subsets are distributed across four CUDA devices, with the subsets evaluated independently. Response Parsing.Yes/no responses are lowercased and mapped to yes or no when the response begins with the corresponding word. If necessary, we accept the answer when exactly one of the two words occurs in the response; ambiguous or unparseable outputs are marked incorrect. Multiple-choice responses are parsed as the first standalone letter from A to D. For open-count questions, responses are lowercased, punctuation is removed, and number words from zero to twenty are mapped to digits. We then extract the first valid numeric count. Equivalent forms such as “seven” and “7” are therefore treated identically, while explanations or other text do not receive partial credit. Spatial attribute responses are normalized for capitalization, spacing, and punctuation before exact matching. An output that cannot be normalized to the required response format is counted as incorrect. Scoring.Attribute, Language Elicitation, Counting, and Spatial are scored independently at the sample level. Context and Texture contain four samples for each Base–Edited case: two questions are asked about the Base image and two about the Edited image. A case is correct only if all four responses are correct, which we refer to as strict All4 accuracy. Language Elicitation is evaluated using four-option multiple-choice accuracy, with the position of the unknown answer balanced across the dataset. Attribute and Counting use normalized exact-count accuracy. Spatial uses normalized exact count or exact attribute matching according to the question type. The overall SABRE-Prior score is the unweighted macro-average of the Context, Texture, Attribute, and Language Elicitation subset accuracies. Confidence Intervals.We compute 95% confidence intervals using a percentile bootstrap with 20,000 resamples and random seed 42. Resampling is performed over cases within each subset. Thus, the four samples belonging to a Context or Texture case remain grouped and are resampled together. For every bootstrap replicate, we recompute the four subset accuracies and their unweighted macro-average. The reported interval endpoints are the 2.5th and 97.5th percentiles of the resulting bootstrap distributions. Reported NameModel IdentifierInterfaceInference Configuration Gemini 3.5 Flashgemini-3.5-flashGoogle Gen AI APITemperature 0; thinking budget 0 GPT-5.4gpt-5.4 OpenAI Responses APITemperature 0; reasoning effort none; image detail high Claude 4.6claude-sonnet-4-6OpenRouterTemperature 0 Kimi-k2.6kimi-k2.6Official Moonshot APIThinking disabled; provider-supported deterministic configuration Qwen 3.5 27BQwen3.5-27BLocal Hugging Face Transformers Greedy decoding; thinking disabled; bfloat16 Grok-4.3grok-4.3OpenRouterTemperature 0 Table 4: Models and inference configurations used in our experiments. Greedy decoding is used whenever supported by the corresponding inference interface. FieldTypePurpose concept_idStringUnique identifier used to reject duplicate specifications and connect subse- quent construction records. pressure_test_typeStringName of the stress-test topic, such as context_prior. objectiveStringCapability or failure mode that the specification is intended to test. generation_modeEnumEither paired_edit or single_image. base_promptStringPrompt used to construct the Base image for a paired intervention. edit_promptString Local transformation applied to the Base image to produce the Edited image. image_promptStringPrompt used for tasks that require only one generated image. task_attributesListTask-specific structured values, such as source entity, target entity, material, count, or spatial relation. questionsListQuestion specifications containing the probe identifier, image role, question type, prompt, reference answer, evaluation rule, and optional multiple-choice options. review_targetsListInstructions identifying the visual evidence and image region that a human reviewer should verify. generation_notesListAdditional constraints used during image generation, editing, and review. Table 5: Principal fields in the default Sample Specification schema. B Test Primers and Sample Specifications A benchmark designer begins by providing a Test Primer, which consists of a natural-language Task Design written in Markdown and a Data Schema that defines the required output fields and validation rules. The Task Design describes the capability or failure mode to test, the visual construction requirements, and the desired question format. The schema converts these requirements into a machine-verifiable contract. A designer may use the default schema provided by SABRE or replace it with a task-specific Pydantic schema when additional fields or validation rules are required. In our implementation, GPT-5.4 receives the Task Design together with the schema and generates one structured Sample Specification at a time. The output is accepted only if it satisfies the schema; invalid outputs and duplicate concept identifiers are returned to the design model with validation feedback for regeneration. Valid specifications are stored as JSONL records and passed to the image-construction and question-building modules. Default Data Schema. Table 5 summarizes the principal fields in the default Sample Specification schema. The schema supports paired image editing and single-image generation, as well as yes/no, multiple-choice, and open-generation questions. The schema also enforces cross-field constraints. A paired_edit specification must contain both a Base prompt and an edit prompt, and its questions may refer only to the Base or Edited image. A single_image specification must instead contain an image prompt, and all questions must refer to that image. Yes/no questions require yes_no_exact evaluation and a reference answer of either yes or no; multiple-choice answers must identify one of the provided options; and open-generation questions must specify an exact-match, counting, or containment-based evaluation rule. Sample Specification Context / paired_edit "concept_id": "ctx01_repack_tape_bagel", "pressure_test_type": "context_prior", "generation_mode": "paired_edit", "objective": "Test whether a VLM follows visible evidence when an expected object is replaced by an unexpected object.", "base_prompt": "Generate a documentary photograph of a busy warehouse repacking floor. Include exactly one tan roll of masking tape on the lower shelf. No bagel or additional tape roll is visible. ...", "edit_prompt": "Replace only the masking tape with one plain bagel at the same location. Preserve all other objects, people, lighting, camera properties, and scene content. ...", "task_attributes": [ "name": "source_entity", "value": "roll of masking tape", "name": "target_entity", "value": "plain bagel", "name": "review_location", "value": "lower shelf of the central repacking bench" ], "questions": [ "probe_id": "base_source", "image_role": "base", "question_type": "yes_no", "prompt": "Is a roll of masking tape visible?", "answer": "yes", "eval_type": "yes_no_exact", "probe_id": "base_target", "image_role": "base", "question_type": "yes_no", "prompt": "Is a plain bagel visible?", "answer": "no", "eval_type": "yes_no_exact", "probe_id": "edited_source", "image_role": "edited", "question_type": "yes_no", "prompt": "Is a roll of masking tape visible?", "answer": "no", "eval_type": "yes_no_exact", "probe_id": "edited_target", "image_role": "edited", "question_type": "yes_no", "prompt": "Is a plain bagel visible?", "answer": "yes", "eval_type": "yes_no_exact" ], "review_targets": [ "label": "source object", "image_role": "base", "description": "Verify one tape roll at the specified location.", "label": "target object", "image_role": "edited", "description": "Verify that one bagel replaces the tape roll." ] Figure 8: Abbreviated Context Sample Specification. The record connects visual construction, question definition, and human verification within a schema-validated specification. Long prompts are shortened for presentation. Context Task Design. We use the Context subset as a running example. Its Task Design instructs the design model to construct a natural Base scene containing one contextually expected source entity and no target entity. The Edited image must replace only the source entity with a visually compatible but contextually unexpected target while preserving the rest of the scene. The Task Design also defines four complementary yes/no probes: 1. Source entity visible in Base: yes. 2. Target entity visible in Base: no. 3. Source entity visible in Edited: no. 4. Target entity visible in Edited: yes. The Markdown further requires both entities to remain human-recognizable while occupying a minor region of a visually dense scene. It prohibits duplicate source or target instances, readable labels, large isolated objects, obvious editing artifacts, and changes outside the specified region. Example Context Sample Specification. The following abbreviated record is an actual schema- compatible specification generated for the Context task. Long image prompts are shortened only for presentation 8. Question Instantiation. The questions are specified before image construction; the Question Builder does not use another model to generate them. Once the required images are available, it associates each question with the appropriate image through its image_role, assigns a unique identifier, and exports the canonical records used for filtering, human review, and final evaluation. For Context, the four probes share the same case identifier and Base–Edited assets and are evaluated jointly using All4. Other subsets use the same question-instantiation interface with their task-specific InstantiationImage ModeTask-SpecificConstruc- tion QuestionFor- mat Primary Validity Requirement ContextPairedBase– Edited Replace one contextually ex- pected source entity with a visually compatible but con- textually unexpected target. Fouryes/no probes The source appears only in Base, the target appears only in Edited, and all non-target content is preserved. Texture PairedBase– Edited Change the visible surface of a familiar entity from its canonical material to a coun- terfactual material while pre- serving its identity and ge- ometry. Fouryes/no probes Canonical and counterfactual surface cues are visually distinguishable, and the intervention does not change the entity or surrounding scene. Attribute Single final im- age Construct a familiar entity with a noncanonical number of visible components while preserving its category iden- tity. Open-ended count Every counted component is visible and individually distinguishable, and the annotated count matches the image exactly. Language Elici- tation Single image Construct a scene in which the question suggests a plau- sible answer although the queried evidence is hidden, occluded, unreadable, or outside the frame. Four-option MCQ The requested information cannot be determined from the image, making unknown the uniquely supported answer. Counting Independently scored images Construct dense scenes con- taining many instances of a precisely defined target cat- egory together with visually similar distractors. Open-ended count Every target instance has a visible boundary, distractors are distinguishable, and the reference count is exact. Spatial Independently scored images Construct dense multi-layer 3D lattices with controlled object colors, shapes, co- ordinates, and spatial rela- tions. Open-ended attribute or count Each question has one visually verifiable answer under an explicitly defined coordinate system or spatial constraint. Table 6: Task-specific instantiations of the shared SABRE pipeline. The main SABRE-Prior benchmark contains Context, Texture, Attribute, and Language Elicitation. Counting and Spatial are reported separately as additional pipeline instantiations. question formats and scoring rules. Complete Test Primers and schemas will be released with the benchmark-construction code. C Task-Specific Instantiations We instantiate the pipeline on the four SABRE-Prior subsets and two additional pilots. Each instantiation uses a different Test Primer, visual construction requirement, question format, and validity criterion, while retaining the same construction, filtering, verification, repair, and export workflow. Table 6 summarizes these settings. Context and Texture use paired Base–Edited cases, whereas Attribute and Language Elicitation use a single final image. Counting and Spatial are evaluated at the individual-image level and do not use the All4 case-level metric. Context. A Context specification defines a dense, coherent Base scene containing exactly one source entity that is natural for the scene and no instance of the target entity. The source is placed in a precisely described review location and remains identifiable without becoming the focal object. The target is selected to have a similar apparent size, orientation, color, or silhouette so that it can occupy the same location after editing while remaining contextually unexpected. The edit instruction replaces only the source with the target and preserves the camera, people, lighting, composition, surrounding objects, and local occlusion pattern. The specification contains four question records: base_source, base_target, edited_source, and edited_target, with reference answers yes, no, no, and yes, respectively. A candidate is visually valid only when both entities are human-recognizable in their intended images, neither entity appears in the wrong image, and no unrelated scene content changes in a way that affects the questions. Texture.A Texture specification begins with a familiar entity displaying its expected surface, such as a plastic watering can or a wooden rolling pin. The counterfactual material is chosen to conflict with the entity’s canonical appearance while providing recognizable visual evidence, such as visible fabric fibers, foam structure, glass transparency, or a paper-like surface. The intervention changes the visible material of the target entity but preserves its category, geometry, scale, pose, and location. The four probes ask whether the entity has the canonical or counterfactual surface in the Base and Edited images. Their answer pattern is again yes/no for Base and no/yes for Edited. Human validation checks that the surface change covers the intended visible region, that the material cues are sufficiently clear for direct visual judgment, and that the edit does not deform the entity or alter neighboring objects. Attribute.Attribute specifications target familiar objects whose components normally follow a canonical count, including animal legs, chair legs, shirt buttons, instrument holes, and repeated symbols on playing cards. The image is constructed or edited to contain a noncanonical number of visible components while retaining the object’s recognizable identity. The intervention must produce complete components rather than duplicated fragments, merged structures, shadows, or ambiguous partial instances. Each exported image is associated with one open-ended question asking for the visible component count. The reference answer is derived from the verified image rather than assumed from the construction prompt. During validation, the reviewer confirms the object identity, checks each visible component, and ensures that occlusion or malformed geometry does not make the count ambiguous. Language Elicitation.Language Elicitation specifications construct scenes in which contextual wording makes one answer appear plausible although the necessary evidence is not available in the image. Examples include asking about the contents of a closed opaque container, text on a face-down object, information hidden by glare or blur, or content outside the image boundary. The image prompt must ensure that the queried information is genuinely unavailable and that no incidental visual cue reveals the intended distractor. Each question contains three plausible content answers and one unknown option. The plausible alternatives are selected to reflect answers that may be suggested by the scene or question wording, rather than visually supported answers. The position of unknown is shuffled and balanced across the subset. A candidate is valid only when none of the three content options can be confirmed from the image and unknown is therefore the uniquely supported response. Counting.Counting specifications define a target category using visible properties such as object type, color, pattern, or location. The scene contains dozens of target instances distributed across natural clusters, together with distractors sharing selected properties with the target. For example, a scene may contain yellow lemons among other yellow or round objects, or four-hole blue buttons among buttons and beads with similar colors and shapes. Targets are not arranged in regular rows and may touch or overlap slightly, but each instance must retain a visible boundary. The construction process may produce Base and Edited images with different target counts. The edit adds or removes a specified number of targets while preserving the scene and distractors. Unlike Context and Texture, each image is exported as an independent open-count sample. Human verification determines the final visible count and rejects images containing merged, duplicated, heavily occluded, or otherwise uncountable target instances. Spatial.Spatial specifications construct dense multi-layer 3D lattices containing repeated colored shapes at controlled cells. A fixed coordinate convention defines the front-left-bottom cell as the origin cell, with the horizontal axis increasing from left to right, depth increasing from front to back, and height increasing from bottom to top. The same convention is stated explicitly in every coordinate question to avoid dependence on an unstated viewpoint. The task supports coordinate lookup questions, such as identifying the color and shape at a specified cell, as well as constrained counting questions, such as counting red cubes on the top layer. Specifications may also move a target object between cells while preserving the lattice, camera, and all other objects. As with Counting, Base and Edited images are evaluated independently. A candidate is valid only when Context Repair Review Current 1/1 · Reviewed 1/1 gecko_wrench_workbench__v01 · standalone · gecko_wrench_workbench__v01 generated_without_screeningSaved: unsure Exactly one small silver adjustable wrench.→Exactly one small realistic gecko. Review location Front-right area of the midground workbench cluster, diagonally positioned beside and partly overlapped by a grease-stained rag. Local cluster A crowded mixed-tool cluster containing pliers, hex keys, tire lever, socket, chain, spokes, washers, bolts, rag, and unbranded bicycle components; exactly one adjustable wrench is present in Base and no additional adjustable wrench appears anywhere. Edited visual cues — Human verifiability Verify that exactly one recognizable small silver adjustable wrench is visible in the front-right midground tool cluster and that no other adjustable wrench appears anywhere in the scene.; Verify that exactly one recognizable small gecko occupies the wrench location, with toe-padded feet, head, mottled body, and tapering tail visible, and that no adjustable wrench remains anywhere.; Verify that the rag, surrounding tools, people, bicycles, camera angle, lighting, focus, composition, and all non-designated scene content are unchanged. Base: mark the source entity Drag a box or click for a default box Edited: mark the inserted entity or repair region Drag a box or click for a default box Showing original edited image. Decision KeepNeeds recheckEdited replacement failedBase invalidScene changedUnsure Notes Optional: ambiguity, inaccurate box, or rejection reason Questions and answers Editable benchmark content. The model prediction and result are the historical filter-gate record. ProbeQuestionAnswerFilter predictionFilter result probe_01 Is there an adjustable wrench in the Base image? yes Incorrect probe_02 Is there a gecko in the Base image? no Incorrect probe_03 Is there an adjustable wrench in the Edited image? no Incorrect probe_04 Is there a gecko in the Edited image? yes Incorrect Figure 9: Overview of the SABRE annonation platform. the lattice orientation is visually recoverable, the queried cells or layers remain visible, the target objects are distinguishable from distractors, and the reference answer can be verified without relying on the generation prompt. D Human Verification and Curation Platform D.1 Annotation Platform Figure 9 shows the platform used to verify candidates retained by automated pressure filtering. For paired-image cases, the Base and Edited images are displayed side by side together with the intended source-to- target transformation and a natural-language description of the review location. This location cue helps reviewers find the relevant evidence in visually dense scenes. Reviewers mark the source entity in the Base image and the inserted entity or edited region in the Edited image using bounding boxes. The normalized box coordinates are stored as curation metadata, allowing the regions to be highlighted during later inspection and reused for localized repair. These annotations are not provided to the evaluated VLMs. Reviewers inspect whether the Base image is valid, whether the intended transformation is correctly realized, and whether unrelated scene content is preserved. The interface also presents the four pressure-screening probes, including their reference answers, filtering-model predictions, and correctness. A valid case is marked as Keep; otherwise, it can be marked as requiring reinspection, having a failed edit, having an invalid Base image, containing unintended scene changes, or being uncertain. Reviewers may also record notes explaining ambiguous cases or rejection decisions. The platform additionally provides localized image repair and real-image authoring tools. Reviewers can select a defective region in an Edited image, generate a repaired candidate, compare it with the current image, and either accept the repair or restore the original version. The real-image authoring interface supports importing photographs, specifying source and target entities, constructing edited candidates, defining question–answer pairs, and exporting the resulting annotations in the same structured format. D.2 Localized Image Repair Fine-grained full-image editing is frequently unreliable, especially when the requested change affects only a small object in a visually complex scene. A generation model may ignore the requested change, leave visible remnants of the original entity, or modify unrelated objects and scene content. Repeatedly SubsetMetricGemini 3.5 Flash GPT-5.4 Claude 4.6 Kimi-k2.6 Qwen 3.5 27B Grok-4.3 Context Prior All4 (P)↑0110734 Edit2↑0213976 Q1 Base+↑979972859397 Q2 Base-↑826794848562 Q3 Edit-↑033317107 Q4 Edit+↑868252586888 Texture All4 (P)↑522840524628 Edit2↑522841544729 Q1 Base+↑9910098989999 Q2 Base-↑1009910099100100 Q3 Edit-↑813054696338 Q4 Edit+↑588265676565 AttributeCount Acc. (P)↑262017171416 Language Elicitation MCQ Acc. (P)↑112358172923 OverallMacro Avg.↑22.318.031.323.323.017.8 Table 7: Detailed results on SABRE-Prior. Rows marked with (P) are the primary subset metrics used to compute the macro-average. For Context Prior and Texture, All4 requires all four Base–Edited probes to be correct, while Edit2 requires both Edited-image probes to be correct. Q1 and Q2 respectively test the expected and counterfactual evidence in the Base image; Q3 and Q4 test the same evidence after editing, with expected answers yes/no/no/yes. Attribute reports normalized exact-match counting accuracy, and Language Elicitation reports four-option multiple- choice accuracy. All values are percentages. Bold values indicate the highest score in each row. regenerating the complete image can therefore destroy otherwise valid visual evidence and weaken the controlled relationship between the Base and Edited images. To address this problem, our curation platform provides localized patch repair, as illustrated in Figure 10. When a reviewer identifies a defective region, they draw a bounding box around the failed edit or local artifact and select a repair operation. The system expands the selected region slightly to retain its surrounding visual context, crops the resulting patch, and sends only this marked patch to the image generation model together with a task-specific repair instruction. The repaired patch is resized when necessary and blended back into the original image using a soft mask. This procedure preserves the image outside the local neighborhood and reduces unintended changes to unrelated scene content. The platform supports four repair operations: removing an unwanted entity, swapping one entity for another, cleaning local visual artifacts, and restoring the background. Figure 10 presents the two entity-level operations. For Swap Entity, the reviewer specifies both the source and target entities; for Remove Entity, the selected object is removed and the exposed region is completed using its local context. Reviewers may further refine the repair instruction to specify expected appearance, scale, lighting, or content preservation constraints. A generated repair is treated as a candidate rather than being accepted automatically. The platform displays the original Edited image and repair preview side by side, marks the case as requiring reinspection, and allows the reviewer to accept the repair or return to the previous image. When a repair is accepted, the previous Edited image is retained as a backup and the repair operation is recorded. The repaired case must then pass human verification before it is included in the benchmark. D.3 Blind User Study of Localized Repair Quality We conduct a randomized, method-anonymous user study to evaluate the perceptual quality of images produced by the localized repair tool. The survey interface are shown in Figures 11. The study contains 20 image-repair cases. Each question first presents an original image with a red bounding box indicating the entity to be removed, followed by four candidate repair results. Participants select the result that most naturally and seamlessly realizes the requested edit. The instructions emphasize that the marked entity should be completely removed while unrelated image content should remain unchanged. We compare four methods: Gemini prompt-only full-image editing, OpenCV inpainting, Gemini patch Repair MethodVotes Preference (%) Gemini full-image prompt164.0 OpenCV inpainting00.0 Gemini patch + hard paste102.5 Ours: patch + soft blend37493.5 Table 8: Method preferences from 20 participants over 20 image-repair cases, yielding 400 selections in total. generation with hard pasting, and our Gemini patch generation with soft-mask blending. The hard-paste baseline and our method use the same generated patch, isolating the effect of the compositing strategy. Candidate results are displayed only as Method 1–Method 4, without revealing their underlying methods. Their display positions are independently shuffled for every question, and the system records both the randomized order and the selected method. Each question requires one selection and cannot be skipped. Forty survey sessions were initiated. We analyze the 20 sessions that completed all 20 required comparisons, yielding 400 valid selections. We use entity removal throughout the study to maintain a consistent and unambiguous evaluation objective. The study therefore directly evaluates the removal setting and the shared patch-generation and compositing components of the localized repair mechanism. Entity swapping is demonstrated qualitatively in Figure 10 but is not included in this user study. As shown in Table 8, our method is selected in 374 of the 400 comparisons, corresponding to an overall preference rate of 93.5%. Because every participant completes the same 20 questions, this is also the mean participant-level preference rate. The standard deviation across participants is 8.1 percentage points, with individual preference rates ranging from 70% to 100%. Participants therefore show a clear preference for localized patch generation with soft-mask blending over full-image editing, OpenCV inpainting, and hard patch compositing on the tested entity-removal cases. D.4 Illustrative Pipeline Cost Exact historical costs cannot be recovered because early task exploration and benchmark construction shared the same API accounts. We therefore report an illustrative marginal-cost estimate for the stabilized Context recipe using the Gemini-based image pipeline. For every 100 initial candidates, we estimate that approximately 50 pass pressure filtering and enter human review, of which 30 are accepted directly, 10 are accepted after localized repair, and 10 are rejected. This yields approximately 40 retained cases. E Additional Results Table 7 localizes the failures within the paired-image subsets. Base-image accuracy is generally high, whereas performance drops after editing. Indicating that models may recognize the inserted target while still reporting the replaced source. These results motivate the strict All4 metric over individual-probe accuracy. StageModel100 Initial CandidatesEstimated Cost Cost per 100 Re- tained Sample specifica- tion GPT-5.4100 structured outputs$2.0–$5.0$5.0–$12.5 Base-image gener- ation Gemini 3.1 Flash Image 100 image generations$6.7$16.8 Targetedimage editing Gemini 3.1 Flash Image 100 image edits$6.7$16.8 Pressure filteringGemini 3.5 Flash 400 image–question evaluations$1.1–$2.2$2.8–$5.5 Localized repair Gemini 3.1 Flash Image approximately 10 repair calls$0.7–$1.4$1.8–$3.5 Total API cost–$17.2–$22.0$43.0–$55.0 Humanverifica- tion –50 reviewed cases25–50 min1.0–2.1 h Repair and rein- spection –approximately 10 cases10–20 min0.4–0.8 h Total active hu- man time –35–70 min1.5–2.9 h Table 9: Illustrative marginal cost of the stabilized Context recipe. Image costs assume 1K outputs from Gemini 3.1 Flash Image. The estimate includes sample-specification generation, Base-image generation, targeted editing, pressure filtering, localized repair, and active human review. It excludes one-time task exploration, prompt development, framework engineering, and final benchmark evaluation. Costs and retention rates vary with the task specification and required visual intervention. Context Repair Review Current 1/1 · Reviewed 1/1 gecko_wrench_workbench__v01 · standalone · gecko_wrench_workbench__v01 generated_without_screeningSaved: needs_recheck Exactly one small silver adjustable wrench.→Exactly one small realistic gecko. Review location Front-right area of the midground workbench cluster, diagonally positioned beside and partly overlapped by a grease-stained rag. Local cluster A crowded mixed-tool cluster containing pliers, hex keys, tire lever, socket, chain, spokes, washers, bolts, rag, and unbranded bicycle components; exactly one adjustable wrench is present in Base and no additional adjustable wrench appears anywhere. Edited visual cues — Human verifiability Verify that exactly one recognizable small silver adjustable wrench is visible in the front-right midground tool cluster and that no other adjustable wrench appears anywhere in the scene.; Verify that exactly one recognizable small gecko occupies the wrench location, with toe-padded feet, head, mottled body, and tapering tail visible, and that no adjustable wrench remains anywhere.; Verify that the rag, surrounding tools, people, bicycles, camera angle, lighting, focus, composition, and all non-designated scene content are unchanged. Upload image for free repair Choose FileNo file chosen Use current benchmark image Current edited image Draw the repair box here Repair input: current edited image. Repair preview Appears after running patch repair Preview: repairs/gecko_wrench_workbench__v01__20260805T181554__gemini_patch__repaired.jpg Repair edited image Draw a box on the Edited image around the residual entity, replacement target, or local artifact. Repair only that marked region, then re-review the repaired image before keeping it. Repair status: needs_recheck Current repaired image: repairs/gecko_wrench_workbench__v01__20260805T181554__gemini_patch__repaired.jpg Repair type Swap entity Engine Gemini local patch repair Gemini model gemini-3.1-flash-image-preview Source entity realistic gecko Target entity small Repair instruction Replace the realistic gecko inside the marked region with small. The new small must be clearly visible and naturally integrated with the scene. Match the surrounding lighting, perspective, scale, shadows, and occlusion. Remove all visible traces of the realistic gecko. Keep everything outside the marked region unchanged. Swap the realistic gecko into a small ball dont make others changed Questions and answers Editable benchmark content. The model prediction and result are the historical filter-gate record. ProbeQuestionAnswerFilter predictionFilter result probe_01 Is there an adjustable wrench in the Base image? yes Incorrect probe_02 Is there a gecko in the Base image? no Incorrect probe_03 Is there an adjustable wrench in the Edited image? no Incorrect Context Repair Review Current 1/1 · Reviewed 1/1 gecko_wrench_workbench__v01 · standalone · gecko_wrench_workbench__v01 generated_without_screeningSaved: needs_recheck Exactly one small silver adjustable wrench.→Exactly one small realistic gecko. Review location Front-right area of the midground workbench cluster, diagonally positioned beside and partly overlapped by a grease-stained rag. Local cluster A crowded mixed-tool cluster containing pliers, hex keys, tire lever, socket, chain, spokes, washers, bolts, rag, and unbranded bicycle components; exactly one adjustable wrench is present in Base and no additional adjustable wrench appears anywhere. Edited visual cues — Human verifiability Verify that exactly one recognizable small silver adjustable wrench is visible in the front-right midground tool cluster and that no other adjustable wrench appears anywhere in the scene.; Verify that exactly one recognizable small gecko occupies the wrench location, with toe-padded feet, head, mottled body, and tapering tail visible, and that no adjustable wrench remains anywhere.; Verify that the rag, surrounding tools, people, bicycles, camera angle, lighting, focus, composition, and all non-designated scene content are unchanged. Upload image for free repair Choose FileNo file chosen Use current benchmark image Current edited image Draw the repair box here Repair input: current edited image. Repair preview Appears after running patch repair Preview: repairs/gecko_wrench_workbench__v01__20260805T181506__gemini_patch__repaired.jpg Repair edited image Draw a box on the Edited image around the residual entity, replacement target, or local artifact. Repair only that marked region, then re-review the repaired image before keeping it. Repair status: needs_recheck Current repaired image: repairs/gecko_wrench_workbench__v01__20260805T181506__gemini_patch__repaired.jpg Repair type Remove entity Engine Gemini local patch repair Gemini model gemini-3.1-flash-image-preview Entity to remove realistic gecko Repair instruction Remove the realistic gecko inside the marked region. Do not leave any visible part, silhouette, shadow, label, or texture of the realistic gecko. Remove the realistic gecko only, dont make others changed Questions and answers Editable benchmark content. The model prediction and result are the historical filter-gate record. ProbeQuestionAnswerFilter predictionFilter result probe_01 Is there an adjustable wrench in the Base image? yes Incorrect probe_02 Is there a gecko in the Base image? no Incorrect probe_03 Is there an adjustable wrench in the Edited image? no Incorrect Figure 10: Localized image repair in the curation platform. Top: Swap Entity replaces the bagel inside the selected region with a ball. Bottom: Remove Entity removes the bagel and reconstructs the surrounding tabletop. The current Edited image is shown on the left and the repair preview on the right. Reviewers can accept the repaired image or revert to the original Edited image. Image Editing Quality Evaluation We are comparing four methods for removing objects from images. Please help us evaluate which result looks best. Best viewed on a desktop or laptop computer HOW EACH QUESTION WORKS → M 1M 2 M 3M 4 1 Original image has a red box marking the entity to remove 2 4 anonymous methods each attempt the removal 3 Click the result that looks most natural and seamless ✦ Only the entity inside the red box should be removed — nothing else in the image should change ✦ Select the result that looks most natural and seamless ✦ 20 questions total — take as long as you need ✦ If a result fails to remove the entity, or accidentally removes/changes something else — that is the method's failure, not your mistake. Please judge honestly based on what you see. No right or wrong answers — your honest impression is what matters. Start Evaluation Original Figure 11: Instructions for the method-anonymous repair-quality study. Participants are asked to select the most natural result that removes only the marked entity while preserving all unrelated image content.