Paper deep dive
LOCI: A Locator-Critic with Refinement Loop
Walid Bousselham, Mathilde Caron, Arsha Nagrani, Cordelia Schmid
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-Language Models (VLMs) still struggle on tasks requiring complex visual understanding. We argue that the core issue is not high-level reasoning, but instead failing to locate critical details in the image. Due to this shortcoming, VLMs generate often plausible but incorrect reasoning based on flawed perceptual grounding. To address this, we propose Locator-Critic (LOCI), a training-free framework that decouples visual search from evidence verification. LOCI employs a Locator agent to propose candidate visual evidence and a separate Critic agent to evaluate its relevance and sufficiency. These agents engage in an iterative refinement loop, progressively improving the evidence until it is adequate to answer the given question. This decoupled, self-correcting process yields substantial performance gains, achieving state-of-the-art results on multiple complex visual benchmarks. LOCI improves accuracy for both open-weight models like Qwen3-VL (+12.1 on V*, +5.8 on HR-Bench and +11.2 on VisualProbe-Hard) and proprietary models like Gemini 2.5 Pro (+8.9 on V*, +4.3 on HR-Bench, +4.8 on VisualProbe-Hard).
Tags
Links
- Source: https://arxiv.org/abs/2608.30959v1
- Canonical: https://arxiv.org/abs/2608.30959v1
Trouble viewing inline? Open PDF directly →
Full Text
92,005 characters extracted from source content.
Expand or collapse full text
LOCI: A Locator-Critic with Refinement Loop Walid BousselhamMathilde CaronArsha NagraniCordelia Schmid Google DeepMind Abstract Vision-Language Models (VLMs) still struggle on tasks requiring complex visual understanding. We argue that the core issue is not high-level reasoning, but instead failing to locate critical details in the image. Due to this shortcoming, VLMs gen- erate often plausible but incorrect reasoning based on flawed perceptual grounding. To address this, we propose Locator-Critic (LOCI), a training-free framework that decouples visual search from evidence verification. LOCI employs a Locator agent to propose candidate visual evidence and a separate Critic agent to evaluate its relevance and sufficiency. These agents engage in an iterative refinement loop, progressively improving the evidence until it is adequate to answer the given ques- tion. This decoupled, self-correcting process yields substantial performance gains, achieving state-of-the-art results on multiple complex visual benchmarks. LOCI improves accuracy for both open-weight models like Qwen3-VL (+12.1 on V*, +5.8 on HR-Bench and +11.2 on VisualProbe-Hard) and proprietary models like Gemini 2.5 Pro (+8.9 on V*, +4.3 on HR-Bench, +4.8 on VisualProbe-Hard). 1 Introduction Modern Vision-Language Models [2,5,24] produce sophisticated reasoning traces and thought processes. However, their performance remains limited on tasks demanding detailed and complex visual analysis. We argue this is not a failure of the reasoning process itself. Instead, the main bottleneck is the ability to reliably locate the specific visual cues necessary to answer a question in complex scenes. We observe that when a VLM focuses on an irrelevant or ambiguous region, its reasoning process generates a narrative based on this flawed visual, leading to an incorrect answer. We argue that this issue arises because the model must both propose and verify visual evidence in a single inference call. Once the model identifies a visual region, it is heavily biased toward confirming its own hypothesis rather than rigorously challenging it. As a result, identifying the wrong visual region often leads to a wrong answer. To confirm our main hypothesis we carefully design an oracle experiment (see Tab. 1), where we manually provide the ground-truth visual regions to the model. When provided with the correct visual cues, the model is capable of answering almost all the questions of V* correctly with a valid reasoning process, confirming our main hypothesis. The performance is not bounded by reasoning ability, but primarily by the ability to locate relevant visual information. To mitigate this issue, we propose separating the reasoning process into distinct localization and verification phases. We introduce the LOCI approach (Locator-Critic with Refinement), a training-free agentic pipeline that decouples (i) finding visual evidence and (i) verification, using two specialized agents. These agents are the same underlying model, differentiated only by their prompts and inputs. The Locator proposes visual evidence by extracting relevant crops from the image. The Critic independently evaluates whether the proposed crops are sufficient to answer the question, judging only the visual evidence without access to the Locator’s textual reasoning. Importantly, the Critic only accesses Preprint. arXiv:2608.30959v1 [cs.CV] 31 Aug 2026 Q: Is the red car on the left or right side of the police car? Locator You have found the police car, now look for the red car Iter. 1 Locator This is not the red car, continue Locator You’ve found both cars. Now answer the question A: Left ✅ CriticCriticCritic Iter. 2 Iter. 3 ❌ ✅ Figure 1: LOCI approach. To answer the question, our method employs a Locator to find relevant objects and a Critic to provide feedback. It first finds the police car (Iter. 1), then, after a failed attempt (Iter. 2), successfully locates the red car (Iter. 3). With both objects grounded, it correctly infers their spatial relationship to provide the answer. the visual evidence and is not provided with the Locator’s textual reasoning. As we demonstrate in Table 4, this separation is crucial to prevent the Locator from propagating its own errors and influencing the Critic’s assessment. Their interaction forms a self-correcting loop. When the Critic rejects the evidence as insufficient, it provides targeted feedback that guides the Locator to search again, as illustrated in Figure 1. We find that the LOCI framework is particularly suited for benchmarks with complex and detailed images, such as V* [20], HR-Bench [18], and VisualProbe [7]. On these benchmarks, our training-free Locator-Critic framework achieves state-of-the-art performance, surpassing prior methods, including recent reinforcement learning-based approaches [7,30,16,26]. We also show that LOCI is flexible, demonstrating consistent improvements across both open-weight models (Qwen3-VL: +12.1/5.8/11.2 points on V*/HR-Bench/VisalProbe) and proprietary models (Gemini2.5-Pro: +8.9/4.3/4.8 points on V*/HR-Bench/VisualProbe). Overall, our contributions can be summarized as : •We propose LOCI, a training-free Locator-Critic framework that improves VLM reliability by decoupling visual evidence finding from evidence verification. •We significantly outperform the SOTA on multiple benchmarks for Qwen3-VL [14] and Gemini2.5-Pro [2], with consistent improvements over direct prompting and prior methods. •Extensive ablation studies demonstrate the critical role of decoupled verification and the effectiveness of iterative refinement. We also demonstrate scalability through parallel multi-locator ensembling, 2 Related Works 2.1 Dynamic Visual Computation and Tool-Use Recently, tool-augmented MLLMs [3,22,17,28,7,30,16] have shown strong agentic abilities in complex QA tasks by leveraging a broad set of tools (e.g., web browsing, code execution, retrieval). Specifically for image manipulation, Thyme [28] uses code generation with a diverse set of operations like cropping, rotation, and contrast enhancement, activated in the model via SFT and RL stages. Mini-O3 [7] similarly handles complex queries through on-the-fly image manipulation. While powerful, these approaches often require significant training overhead. Our method, in contrast, is entirely training-free and generalizes across both proprietary and open-source models. Similar to our work, DeepEyes[30], Chain-of-Focus[26] and Pixel Reasoner[16], all aim to equip VLMs with zoom-in and region-of-interest selection, enabling active perception over images. Sim- ilarly, DyFo [9] employs a training-free Monte Carlo Tree Search to guide a visual expert’s focus, while Token-Efficient VLM [6] and Griffon-G [23] use complex, multi-stage training pipelines to select high-resolution patches or generate textual coordinates. Unlike these works, we decouple visual detection from verification through a dedicated Critic agent that evaluates the quality of generated crops and provides targeted feedback for iterative refinement. 2.2 Critique and Verification in MLLMs Multiple works try and improve the reliability of models by incorporating an explicit ‘critic’ or ‘verifier’ module. The simplest form of this is self-correction, where a single model critiques its own 2 Locator Python CodeCrop Locator’s Turn There is no plastic stool in the crop Yo u f o u n d t h e Plastic stool, you can now answer. Critic’s Turn Final Answer Q: What is the color of the plastic stool? 푠 = “final” Feedback 푓 ! White Critic Locator ℋ !"# 풞 ! 푠 = “ongoing” ℋ ! Locator’s History ℋ [...] locate crops by generating Python code [...] You have access to the Image,theQuestion and the Conversation History [...] Locator’s Prompt [...]evaluate if the Locator’s crops contain enough visual evidence to answer the question.[...] If not, provide feedback[...] If yes, instruct the Locator to [...]terminate the search and provide the final answer. Critic’s Prompt Previous Crops Locator’s Code Critic’s Feedback Code’s Logs Image & Question Image & Question Figure 2: Overview of LOCI . A Locator agent generates Python code to create image crops, which are then evaluated by a Critic agent. The Critic’s feedback guides the Locator’s following attempts until sufficient visual evidence is found to answer the question. output. This has been explored in both text-only (eg. SELF-REFINE [12]) and multimodal domains (LLaVA-Critic-R1 [19]). While straightforward to implement, these approaches can suffer from "cognitive fixation," where the model fails to identify its own initial perceptual or reasoning errors because the critic and generator share the same underlying biases. To overcome this limitation, recent work has moved towards a decoupled "actor-critic" paradigm, where an independent model verifies the output of the primary reasoning model. For example, Critic-V [25] uses a critic to provide natural language feedback on a reasoning model’s thought process. MMC [11] automatically constructs a critique dataset by using Monte Carlo Tree Search to compare correct and incorrect reasoning paths, while OmniVerifier [27] trains a universal verifier to assess the alignment of generated visual outcomes with textual prompts. Our LOCI framework builds upon and extends this actor-critic paradigm with two crucial differences. First, unlike general-purpose frameworks like LLM-as-a-Judge [29] that evaluate a final answer, our Critic is deployed at an intermediate reasoning stage. Its specific role is to evaluate the quality of the visual evidence (the proposed crop) for answering the given query, not the final textual response. Second, to ensure an unbiased assessment, our Critic is intentionally decoupled from the Locator’s reasoning process. It receives only the candidate crop and the original question, forcing it to judge the visual evidence on its own. 3 Method We introduce the Locator-Critic framework (LOCI ) for effective visual exploration and verification. Our framework decouples visual search from verification. We also demonstrate how to scale this framework with ensembling multiple locators working simultaneously on the same task. 3.1 Locator-Critic Framework Our multi-agent pipeline consists of two agents working collaboratively to solve the visual question answering task (see Figure 2). A Locator visually explores the image by producing crops to gather visual evidence sufficient for answering the question (purple area of Fig. 2), while a Critic analyzes the visual evidence produced by the Locator and decides whether sufficient reliable evidence has been found to properly answer the question (orange area of Fig. 2) . The Locator Agent.The Locator’s role is to search for and extract crops containing visual evidence needed to answer the question. The Locator is a VLM, which we prompt specifically (see "Locator’s Prompt " in Fig. 2 for a short version and Appendix C.1 for the full prompt) to generate both bounding box coordinates and Python code to create crops, following prior works [10,28,1]. The generated code is executed and the model receives the outputs (e.g. logs, error messages, etc.) and any image saved (e.g. a crop). This approach enables flexible and adaptive visual detection with iterative refinement based on Critic feedback for example. Formally, at each turnt(1 ≤ t ≤ T max ), the Locator receives as input a cumulative conversation historyH t−1 illustrated as a yellow box in the bottom left of Fig. 2. At initialization (i.e.t = 1), the 3 historyH 0 contains solely the question and the source image. At each turn, the task of the Locator is to output a set of maximummcropsC t = C 1 , . . . , C m containing visual evidence needed to answer the question. Importantly the Locator finally answers the questionQonly when explicitly prompted to do so, i.e. at the final turn. The conversation historyH t−1 input to the Locator at turntis composed of different artifacts from all preceding turns (≤ t− 1), which include: (1) the complete execution trace of the Locator (python code input, set of crops output, and code execution logs), and (2) targeted feedback provided by the Critic (see details in next paragraph). Leveraging this comprehensive history, the Locator can refine its previous output by modifying its previous own code. For instance, it can debug its execution using past logs or adjust bounding box coordinates in its Python script in response to the Critic’s feedback. The Critic Agent. The Critic is a VLM that, at each turnt, receives as input the question, the original image and the Locator cropsC t . Its task is to evaluate whether the crops provide sufficient visual evidence to answer the question (see "Critic’s Prompt" in Fig. 2 for a short version and Appendix C.2 for the full prompt). The Critic outputs a binary decisions ∈ ongoing, final indicating whether additional search is needed. When additional search is needed (s = ongoing), the Critic provides targeted feedbackf t to guide the Locator toward more relevant visual regions—for example, in Fig. 2 requesting another crop of the plastic stool. This feedbackf t is appended to the conversation historyH t and will be used by the Locator in the following turns. When sufficient evidence has been gathered or when the maximal number of turns has been achieved, the Critic prompts the Locator to answer the question based on the collected visual evidences. Limiting the Critic access to the Locator reasoning.We find that a critical design principle is to provide the Critic only the crops as visual evidence, not the Locator’s textual reasoning. We show (see Tab. 2 & Fig. 8) that otherwise the Critic can be deceived by the Locator’s explanations into approving insufficient visual evidence. This decoupling mitigates hallucination and ensures genuine visual verification. Off-the-Shelf Detectors.While our core framework uses VLM-based code generation for flexible crop extraction, one could alternatively employ off-the-shelf open-vocabulary object detectors (e.g., OWLv2 [13]) to generate fixed object-centric crops. However, such detectors produce non-refinable outputs, limiting the Critic’s ability to guide iterative improvement. Our experiments (see Tab. 4) demonstrate that using an off-the-shelf detection results in comparable results as the first turn, but cannot be improved further. This is interesting as it confirms (i) the importance of visual selection and (i) the importance of the iterative refinement with a critic, which is shown to significantly improve the performance. 3.2 Multi-Locator Ensembling We further scale the framework by deploying multiple locators in parallel. Due to the non- deterministic nature of VLMs, each of the agents outputs a different set of crops for the same input image and question. This strategy increases exploration diversity and robustness, improving the chances of finding the correct visual evidence. Ensembling Multiple Locators. We find that a single locator may become stuck exploring the wrong region and, in practice, may not recover from an initial incorrect hypothesis (see qualitative examples in Sec. 4). Since our framework decouples visual search from verification, we can deploy multiple locators (kinstances) that search simultaneously. This provides effective mitigation for failure cases where one locator gets stuck, while enabling ensemble scaling of visual visual detection capacity, the benefits of which are shown in Table 6. When ensembling multiple locators, we rely onk locatorsL =L 1 , . . . , L k in parallel per iteration. Using non-zero temperature for code generation ensures that different locator instances produce diverse exploration strategies, even when given the same input. Each locatorL i independently generates crops and maintains its own conversation history H (i) t , allowing it to explore different spatial regions simultaneously. We then employ a centralized Critic that evaluates all crops from all locators collectively asC (all) t = S k i=1 C (i) t . At each iteration, the Critic determines if the combined evidence is sufficient. If it is not, the Critic generates a unified feedback message that is broadcast to all locators to guide their next 4 search iteration. This feedback is typically a simple directive, such as indicating that the current crops are insufficient or suggesting an unexplored region of the image to focus on. Each locator appends this feedback to its own conversational history, and the process continues to iterationt + 1. When sufficient evidence has been gathered, the Critic selects the best cropsC selected ⊆C (all) t and proceeds to final answering. Eventually, the final answering agent receives the selected cropsC selected along with the question and full image to generate the answer. 4 Experiments 4.1 Experimental Setup Datasets. We evaluate our method on three VQA benchmarks that challenge modern VLMs with complex questions where there is still substantial potential for improvement. V* [20] consists of 191complex visual reasoning questions. The objects relevant to the questions are annotated with ground truth bounding boxes. This enables oracle experiments to quantify the performance with perfect localization. HR-Bench [18]- composed of 177 images and 800 questions- focuses on high- resolution image understanding with a focus on object-centric questions. VisualProbe [7] consists of 106 samples featuring visually complex scenes with many interacting objects, paired with open-ended questions. We report results on the test set of the 3 benchmarks. Implementation Details. Our core framework uses a single Locator paired with a Critic. By default, we use a maximum ofT max = 10turns and limit the number of crops per turn tom = 4(see ablation in Appendix A.1). We use a single locator (k = 1), unless stated otherwise. We evaluate our framework using two SOTA models: (i) Gemini2.5-Pro [2], a proprietary model with thinking, and (i) Qwen3-VL-235b-a22b-Thinking [14], an open-weight model referred to as Qwen3-VL. 4.2 Ablation Study In this section, unless specified otherwise, we conduct all of the experiments on V* benchmark using Gemini2.5-Pro. We provide additional ablation results on more datasets in Appendix A.4. Table 1: Oracle experiment onV ∗ . We provide the ground-truth crops required to answer the question. Methodacc. Gemini2.5-Pro83.8 Gemini2.5-Pro + Oracle Crop98.2 Oracle Experiment.To test the hypothesis that visual local- ization, rather than reasoning capability, is the primary bottle- neck, we conduct an oracle experiment on V* [20]. We provide the model with ground truth bounding boxes from the dataset, extracting oracle crops around objects of interest. These crops are provided together with the full image and the question to Gemini2.5-Pro. As shown in Table 1, this oracle experiment confirms that visual localization is the primary bottleneck. Providing the model with perfect localiza- tion information (oracle crops and their coordinates) dramatically improves accuracy from 83.0% to 98.2%. A manual inspection of the remaining 1.8% failure cases shows that they correspond to ambiguous questions with multiple viable answers (see Appendix B.1 for examples). Table 2: Locator & Critic ablations. We show the impact of the Locator & Critic in the LOCI framework: having a Locator improves direct prompting by +5.4 (row 1 vs row 2). Also, having a Critic gives a +3.5 boost (row 2 vs row 4). It is crucial not to provide the Locator text as input to the Critic (row 3 vs row 4) LocatorCriticAcc. input: Locator text input: Locator crops 1✗83.8 2✓✗89.2 3✓87.2 4✓✗✓92.7 Locator and Critic ablations. Table 2 presents an ablation study of the Locator and Critic components. First, we see that Locator only (row 2) improves over direct prompting (row 1) by +5.4 points. Ad- ditionally, the Critic improves the perfor- mance further by +3.5 (row 2 versus row 4). Overall, our full framework LOCI achieves a +8.9 points improvement compared to direct prompting baseline. In Table 2, we also validate a critical design choice, and show that the Critic must re- ceive only image crops as visual evidence, not the Locator’s textual reasoning. Providing the Locator’s reasoning to the Critic (row 3) severely degrades performance, performing worse than the Locator-only baseline (row 2). We attribute this 5 Table 3: Same-budget baselines on V*. We compare LOCI against alternative inference-time strategies at comparable or higher token budgets, with and without access to code execution. MethodCode-ExecAvg TokensAcc. Direct Prompting✗33083.8 Self-Refine✗4,18881.5 Self-Consistency@10✗3,63284.3 Explicit Locate-Verify-Answer✓4,11785.9 Self-Refine✓4,20185.2 Best-of-10✓36,31089.0 Locator only (Tab. 2, row 2)✓950089.2 Self-Consistency@10✓4,28189.7 LOCI (ours)✓3,69092.7 to the Critic being misled by the Locator’s incorrect textual explanations, a failure mode where it is convinced to agree with incorrect evidence after a few iterations (see Appendix B.2 for examples). This finding underscores that isolating the Critic with purely visual evidence is crucial for preventing the propagation of hallucinations and ensuring correct visual verification. Same-Budget Baseline Comparisons.A natural question is whether LOCI’s gains come from its specific architecture or simply from spending more inference tokens. To answer this, we compare LOCI against several baselines at comparable or higher token budgets on V*, each representing a different strategy for spending inference-time compute. All baselines use Gemini2.5-Pro. We test each baseline in two settings: without code execution (text-only reasoning) and with code execution (access to the same crop tools as LOCI’s Locator). Brief descriptions are provided in Appendix A.6. Table 3 shows three patterns. First, spending more tokens on text-only self-critique hurts: Self-Refine without code execution (81.5) drops below direct prompting (83.8), confirming that the bottleneck is perceptual rather than reasoning-based. Second, code execution without decoupled verification plateaus around 89/90. Self-Consistency@10 with code execution (89.7), Locator-only (89.2), and Best-of-10 (89.0) all cluster at this ceiling. Best-of-10 is particularly informative: it uses nearly10× LOCI’s token budget (36,310 tokens) yet still falls 3.7 points short. Even with substantially more compute, a single agent remains anchored to its initial search strategy. Third, LOCI’s decoupled architecture is the differentiator. The Explicit Locate-Verify-Answer baseline follows LOCI’s exact workflow (locate, verify sufficiency, iterate) with the same tools and budget, but the verification step sees the model’s own textual reasoning. The 6.8-point gap to LOCI (85.9 vs 92.7) shows that the procedure alone does not account for LOCI’s gains. A model conditioned on its own reasoning cannot genuinely challenge a strategy it just committed to but decoupled verification breaks this anchoring. Off-the-shelf Locator. We compare our VLM-based Locator against an off-the-shelf object de- tector, OWLv2 [13]. In Table 4 we test both locators types (VLM-Based vs off-the-shelf) in two configuration: one without a Critic, where the crops are passed directly to the final answering agent, and one with a Critic. Since the off-the-shelf detector produces a fixed, non-refinable set of crops, the Critic’s role in that setting is limited to a one-shot selection of the most relevant crops to pass to the final agent. The results show that the Critic provide a slight improvement to the off-the-shelf locator +0.7 points. Conversely, the Critic yields a significant improvement for our VLM-Locator (+3.5 points). This gap showcases a key limitation of off-the-shelf locator. The locator cannot refine its own predictions and the Critic’s effectiveness is bounded by the recall of the initial set of crops. Hence, if the visual evidence necessary to answer the question is not captured in the first pass, the Critic cannot recover it. In contrast, the iterative refinement of the VLM-Locator allows to generate better crops at each turn, leading to superior performance. Evaluating the quality of the crops. To demonstrate that improved performance stems from higher-quality crops rather than other factors, we plot both the crop recall and the final VQA accuracy as a function of the number of Locator-Critic iterations (T max ) in Figure 3. We also compare the performance against an off-the-shelf Locator (OWLv2 [13]). 6 012345678910 Number of Iterations 0.60 0.62 0.64 0.66 0.68 Recall Recall vs Iterations VLM-Locator Off-the-shelf Locator 012345678910 Number of Iterations 0.89 0.90 0.91 0.92 Accuracy Accuracy vs Iterations VLM-Locator Off-the-shelf Locator Figure 3: Impact of Iterative Refinement on Local- ization and Accuracy. We plot (left) the crops recall as a measure of Locator crops quality and (right) final VQA accuracy on the V* dataset with respect to the maximum number of turns (T max ). The strong positive correlation between the two curves demonstrates that as the iterative process improves localization (higher recall), the final task performance also increases. Table 4: Different choice of Locator. Comparison of our VLM-based Locator vs. an alternative off-the-shelf Locator. w/o Criticwith Critic Direct Prompting83.8- Off-the-shelf Locator88.689.3 VLM-Locator89.292.7 Table 5: Computational cost. Average token usage per sample across configura- tions with Qwen3-VL. Methodavg. # tokens acc. Direct Prompting33083.8 Locator only95089.2 LOCI3, 69092.7 Note that evaluating the correctness of generated crops for VQA presents a challenge, as standard localization metrics are not perfectly aligned with the downstream task. Our VLM-Locator is instructed to produce helpful crops containing sufficient visual evidence, not necessarily perfectly tight bounding boxes. For instance, questions about the relative position of two objects are often best answered with a single, wider crop covering both, which would yield a low Intersection over Union (IoU) against either individual ground truth box. Similarly, a crop that simply zooms into a relevant region of interest is often sufficient for reasoning. Therefore, we adopt a lenient IoU threshold of 5% to define a "correct" crop, which captures the practical utility of the generated regions. Figure 3 shows the clear benefit of iterative refinement. As iterations increase fromT max = 1to T max = 10, recall climbs from 0.60 to 0.69, while VQA accuracy rises from 0.89 to over 0.92. The correlation between these curves confirms that the iterative feedback loop allows the Locator to find previously missed visual evidence, and that this improved localization is the direct reason for accuracy gains. The metric-task misalignment also explains why the measured recall plateaus around 0.69, as many functionally optimal crops for VQA do not meet the criteria of the recall metric (see Appendix B.4 for visual examples). Computational Cost AnalysisIn Table 5, we analyze the computational cost of our method using Qwen3-VL. Note that we cannot do this analysis with Gemini2.5-Pro, since it is a blackbox, and we do not have access to the tokens. The results demonstrate a clear example of inference-time compute scaling: by increasing the number of tokens used at inference, we achieve higher accuracy without requiring any additional fine-tuning. Our method achieves significant accuracy gains over Direct Prompting at the cost of increased computational overhead. Future work could explore distilling the Locator-Critic interaction into a single model to reduce this overhead while retaining the benefits of decoupled verification. 4.3 Qualitative Examples Figure 4 (a) presents a qualitative example that illustrates LOCI’s ability to recover from initial localization failures. When prompted directly, Gemini2.5-Pro confidently hallucinates that the handbag’s "color is unequivocally white," confusing the person’s shirt with the object in question. Initially, our LOCI framework exhibits a similar error. Indeed, the Locator generates unfocused crops and reasons that the "object held by the person in yellow... is white," despite acknowledging that "It doesn’t look like a handbag." The Critic intervenes by providing feedback: "Your crops are not focused on the correct object... find the person... and then crop the handbag they are holding.". In the subsequent turn, the Locator uses this guidance to generate a better crop (crop #4). This improved visual evidence allows it to correctly identify both the "white shirt" and the "handbag slung over their right shoulder," leading to the accurate conclusion that "The handbag is distinctly yellow". This iterative refinement process highlights the effectiveness of decoupling search and verification, enabling the system to correct initial mistakes and arrive at the correct answer. 7 Q: What is the color of the handbag? Direct Prompting: The handbag is clearly visible. It's a shoulder bag, and its color is unequivocally white. Critic: Your crops are not focused on the correct object. [...] find the person in the image and crop the handbag they are holding. #2 #1 #3 #4 Locator Turn 2: I have finally achieved a clear crop of the person in question [...] It clearly shows a person wearing yellow pants and a white shirt. They are carrying a handbag [...] The handbag is distinctly yellow. Crop: #4 Locator Turn 1: [...] The person in yellow is [...] holding a white object. [#1,2] considered the "handbag" in the question, then the color is white [#3] My third crop, shows the person in the upper right corner. They are wearing a yellow outfit. [...] they seem to be holding a white object. [...] No other bags are visible. The object held by the person in yellow is the only candidate. It is white. Crops: #1,2,3 (a) Direct prompting versus LOCI #4 #5 #6 #1 #2 #3 Q: What is the color of the suitcase? Locator #2:[...] My coordinate estimations have been off [...] There is a cyclist on a bike. To the left of the cyclist, there is a person walking away, towards the right. This person is pulling a rolling suitcase[...] its coloris brown [...] Critic: The crops are sufficient to answer the question. Crops 5 and 6 both clearly show a person on the stairs pulling [...]a suitcase. [...] its coloris clearly visible [...] as brown. This is enough evidence to distinguish it from the other coloroptions [...] Locator #1: This crop shows the person on the right walking up the stairs, clearly pulling a suitcase. The suitcase is dark in color. It appears to be black[...] Crops: Crop: ... k=4 Multi-Locator Ensembling Single Locator:[...] On the far right side[...] maybe that's the suitcase[...] This isn't helpful. The person in this crop is clearly a cyclist [...] There is no suitcase visible.[...] I have examined all the likely areas and I cannot find the suitcase[...] Last Crop: k=1 Single-Locator (b) Single versus multi Locators in the LOCI framework Q: Is the pillar box on the left or right side of the telephone booth? Critic: The provided crops are not suitable [...] not clearly show the relationship between the telephone booth and the pillar box.[...] Locator Turn 2: I got a clear shot of both the red telephone booth and the blackpillar box. [...] Crop: #5 Locator Turn 1:[...] I got the telephone booth but the pillar box is not clearly visible.[#1,2,3] My second attempt was also not ideal too [#4]my last attempt were completely wrong. The phone booth is much further to the left.[...] I am still having trouble isolating the objects. Crops: #1,2,3,4 Critic:Yes, this crop provides enough evidence. It shows the red telephone booth and the pillar box[...] provide your final answer[...] #5 #1 #2 #3 #4 Locator Turn 3: [...]the object in question is on the leftof the telephone booth. Human Annotated Optimal Box (c) Example of failure case of LOCI Figure 4: Qualitative examples of LOCI. (a) We show an example where direct prompting confi- dently hallucinates an incorrect answer ("unequivocally white"), while LOCI uses Critic feedback to recover from a similar initial error. (b) We show an example where having multiple locators is better than single locator. The Single-Locator fails to find the suitcase and gives up. In contrast, the ensemble explores diverse regions in parallel, with one locator successfully finding the brown suitcase. (c) Failure case of LOCI: the Locator misidentifies a visually similar distractor as the target pillar box, and the Critic is also deceived, prematurely approving the incorrect crop. Failure case. Figure 4 (b) illustrates a failure case where our agentic pipeline was misled by a visually similar distractor object. The question requires locating a pillar box relative to a telephone booth in a crowded street scene. After an initial unsuccessful search (crops #1-4), the Locator generates a new crop (#5) in its second turn, capturing the telephone booth alongside a different dark, box-like structure, which it misidentifies as the "black pillar box" (see the cyan arrow in Fig. 4) (b). Crucially, this visual evidence is plausible enough to deceive the Critic, which approves the crop, "this crop provides enough evidence", and terminates the refinement loop. Consequently, the Locator provides the incorrect final answer. In contrast, the multi-locator ensemble (outlined in green) successfully solves the problem by leveraging different, parallel visual searches. While Locator #1 incorrectly focuses on a different person on the stairs and identifies their bag as "black", Locator #2 successfully generates crops (#5 and #6) showing the correct person pulling a brown suitcase The Critic evaluates the collective 8 evidence from all locators and correctly identifies the crops from Locator #2 as sufficient, leading to the correct answer. 4.4 Multi-Locator Ensembling: Scaling for Hard Cases Table 6: Multiple loca- tors. Varying number of parallel locators k. # locatorsacc. 192.7 494.8 896.0 1696.0 While the single-locator LOCI already achieves state-of-the-art results, we find that a single locator may become stuck exploring the wrong region and, in practice, may not recover from an initial incorrect hypothesis. In such challenging scenarios, the locator may exhaust its turn budget without finding the relevant visual evidence (see Figure 4 (c) for an example). Since our framework decouples visual search from verification, it naturally supports scaling the search component: we can deployk independent locators in parallel while relying on a single centralized Critic. This multi-locator ensembling increases exploration diversity and provides effective mitigation for such failure cases, at the cost of proportionally higher inference-time compute. Scaling with Parallel Locators.We investigate how performance scales with the number of parallel locators when additional computational budget is available. We varyk ∈ 1, 4, 8, 16on the V* benchmark with Gemini2.5-Pro. Table 6 shows that performance improves as we increasekfrom 1 to 8, with accuracy rising from 92.7% to 96.0%, demonstrating that parallel diverse exploration provides additional gains. Beyondk = 8, there are no further gains, as additional locators no longer generate useful new crops. Qualitative Analysis. Figure 4 (c) illustrates how multi-locator ensembling recovers from single- locator failures. In the single-locator case (outlined in red), the agent struggles to locate the object of interest. After examining several regions, it ultimately concludes that “there is no suitcase visible” and fails the task. In contrast, the multi-locator ensemble (outlined in green) successfully solves the problem by leveraging parallel visual searches: while Locator #1 incorrectly focuses on a different person and identifies their bag as “black”, Locator #2 successfully generates crops (#5 and #6) showing the correct person pulling a brown suitcase. The Critic evaluates the collective evidence and correctly identifies the crops from Locator #2 as sufficient, leading to the correct answer. This makes multi-locator ensembling best suited for particularly challenging scenarios where the cost of failure outweighs the additional inference-time overhead. In practice, we recommend the single- locator LOCI as the default configuration, and reserve multi-locator ensembling for scenarios where maximizing accuracy justifies the additional compute. 4.5 State-of-the-Art Comparison Table 7:SOTA compari- son on V*. Our LOCI ap- proach significantly improves performance across both open- weight and proprietary models. †: run by us. Methodacc. Visprog [3]41.4 M-React [22]41.4 Griffon-G [23]57.4 SEAL [20]75.4 TEVA [6]75.4 DyFo [9]81.2 DeepEyes [30]83.3 Pixel Reasoner [16]86.3 Chain-of-Focus [26]88.0 Mini-o3 [7]88.2 Qwen3-VL † [14]80.1 Gemini2.5-Pro † [2]83.8 LOCI(Qwen3-VL)92.2 LOCI(Gemini2.5-Pro)92.7 We present state-of-the-art comparisons across three complex and high-resolution visual benchmarks in Tables 7 and 9. We compare LOCI against recent state-of-the-art methods spanning from high- resolution methods [20,23,6,9] to models trained for tool-used with RL [16,30,7] and a training-free method [9]. We observe that LOCI delivers consistent improvements on the considered benchmarks, both using Qwen3-VL or Gemini2.5-Pro. On V*, a benchmark requiring precise localization of small visual details, LOCI achieves a substantial improvement of +11.5 points over DyFo, the next best training-free method. Our approach also outperforms all recent reinforcement learning-based methods, while remaining training-free and applicable to any recent VLM. To isolate the framework contribution from backbone strength, we re-run DyFo [9], the most competitive training-free method in our comparison, using the same Gemini2.5-Pro backbone as LOCI. As shown in Table 8, LOCI outperforms DyFo by+3.2points while using2×fewer tokens and zero external tool calls, compared to DyFo’s 12.5 LangSAM calls per sample. This confirms that LOCI’s gains are attributable to the Locator-Critic architecture, not backbone capability. 9 Table 8: Same-backbone comparison on V*. MethodAcc. Avg TokensExternal Calls DyFo [9]89.58,12012.5 (LangSAM) LOCI (ours) 92.73,6900 On HR-Bench, which focuses on object-centric questions in high-resolution images, our method with Gemini2.5-Pro outperforms Mini-o3, the best prior method – which was trained with reinforce- ment learning specifically for the task of zooming into images– by a significant margin of +8.2 points. Finally, on VisualProbe, our method surpasses the prior state-of-the-art by +1.1 points, outperforming Mini-o3 even though it was specifically trained on the VisualProbe training set. 4.6 Generalization to Additional VLMs Table 9: State-of-the-art comparison. We report the top-1 accuracy on HRBench (a) and Visual Probe (b). †: run by us. a) HR-Bench Methodacc. SEAL [20]47.0 DeepEyes [30]73.2 Pixel Reasoner [16]74.0 Mini-o3 [7]77.5 Qwen3-VL † [14]76.8 Gemini2.5-Pro † [2]84.0 LOCI(Qwen3-VL)82.6 LOCI(Gemini2.5-Pro) 88.3 b) Visual probe Methodacc. GPT-4o [4]11.2 LLaVA-OneVision [8] 13.4 Pixel Reasoner [16]28.8 DeepEyes [30]35.1 Mini-o3 [7]48.0 Qwen3-VL † [14]23.9 Gemini2.5-Pro † [2]44.3 LOCI(Qwen3-VL)34.0 LOCI(Gemini2.5-Pro) 49.1 We report results for four VLMs in Table 10 (main paper). The same prompts are used across all models with no per-model tun- ing. The inverse correlation between baseline strength and LOCI’s improvement suggests that the framework is most effective when the base model has a large localization gap to close. 4.7 Limitations Our framework has several limitations that present opportunities for future work. First, the iterative Locator-Critic loop requires more tokens and sequential VLM calls than direct prompting, a cost further amplified in the multi-locator setting. This could be mitigated through adaptive early stopping or difficulty-aware budget allocation, allowing easy samples to exit the loop early. Second, the Critic could be deceived by visually plausible distractors that resemble the target object. While our decoupled design prevents hallucination-driven failures, it does not handle genuinely ambiguous visual evidence. Training the Critic with hard-negative examples could improve robustness in such cases. Third, our framework relies on large, capable VLMs for both the Locator and Critic. However, recent models increasingly focus on agentic capabilities such as tool use and multi-step reasoning, which aligns directly with our framework and suggests it will naturally benefit as smaller models grow more capable. 5 Conclusion Table 10: LOCI across VLMs. Accu- racy on V* with and without LOCI. Modelw/ow/∆ Grok4.1-Fast45.670.9+25.4 GPT-574.989.0+14.1 Qwen3-VL80.192.2+12.1 Gemini2.5-Pro83.892.7+8.9 In this work, we introduce LOCI (Locator-Critic), a training- free agentic framework for complex and challenging visual question answering tasks. By using a Locator agent to pro- pose visual evidence and an independent Critic to verify it, LOCI creates a self-correcting loop that ensures final an- swers are grounded in reliable evidence. We discuss and val- idate critical design choices, such as the information Critic needs to see to give accurate feedback to the Locator. Our extensive experiments show that LOCI achieves state-of-the- art performance, significantly improving upon both open-weight and proprietary models. We believe the interplay between an explorative agent like the Locator and a Critic agent is a broadly applicable principle. A direction for future work is therefore to extend our framework to other complex vision tasks, such as spatial navigation in 3D environments for example. References [1] Kamel Alrashedy, Pradyumna Tambwekar, Zulfiqar Zaidi, Megan Langwasser, Wei Xu, and Matthew Gom- bolay. Generating cad code with vision-language models for 3d designs. arXiv preprint arXiv:2410.05340, 2024. 10 [2]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. [3]Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In CVPR, 2023. [4] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. [5] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. [6] Yitong Jiang, Jinwei Gu, Tianfan Xue, Ka Chun Cheung, Pavlo Molchanov, Hongxu Yin, and Sifei Liu. Token-efficient vlm: High-resolution image understanding via dynamic region proposal. In ICCV, 2025. [7]Xin Lai, Junyi Li, Wei Li, Tao Liu, Tianjian Li, and Hengshuang Zhao. Mini-o3: Scaling up reasoning patterns and interaction turns for visual search. arXiv preprint arXiv:2509.07969, 2025. [8]Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. [9]Geng Li, Jinglin Xu, Yunzhen Zhao, and Yuxin Peng. Dyfo: A training-free dynamic focus visual search for enhancing lmms in fine-grained visual understanding. In CVPR, 2025. [10] Xuefeng Li, Haoyang Zou, and Pengfei Liu.Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383, 2025. [11]Shuhang Liu, Zhenrong Zhang, Pengfei Hu, Jiefeng Ma, Jun Du, Qing Wang, Jianshu Zhang, Quan Liu, Jianqing Gao, and Feng Ma. Mmc: Iterative refinement of vlm reasoning via mcts-based multimodal critique. In Proceedings of the 3rd International Workshop on Large Generative Models Meet Multimodal Applications, pages 11–20, 2025. [12] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594, 2023. [13] Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. Advances in Neural Information Processing Systems, 2023. [14] QwenTeam. Qwen3-vl: Sharper vision, deeper thought, broader action. Blog post, 2025. [15]Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaugh- lin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025. [16]Alex Su, Haozhe Wang, Weiming Ren, Fangzhen Lin, and Wenhu Chen. Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning. arXiv preprint arXiv:2505.15966, 2025. [17] Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. [18] Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7907–7915, 2025. [19]Xiyao Wang, Chunyuan Li, Jianwei Yang, Kai Zhang, Bo Liu, Tianyi Xiong, and Furong Huang. Llava- critic-r1: Your critic model is secretly a strong policy model. arXiv preprint arXiv:2509.00676, 2025. [20] Penghao Wu and Saining Xie. V?: Guided visual search as a core mechanism in multimodal llms. In CVPR, 2024. [21] xAI Team. Grok 4.1 model card. Blog post, 2025. [22] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023. [23] Yufei Zhan, Yousong Zhu, Zhiyang Chen, Fan Yang, Ming Tang, and Jinqiao Wang. Griffon: Spelling out all object locations at any granularity with large language models. In ECCV, 2024. 11 [24]Brian Zhang, Eric Mitchell, Hongyu Ren, Kevin Lu, Max Schwarzer, Michelle Pokrass, Shengjia Zhao, Ted Sanders, Adam Tauman Kalai, Alexandre Passos, Benjamin Sokolowsky, Elaine Ya Le, Erik Ritter, Hao Sheng, Hanson Wang, Ilya Kostrikov, James Lee, Johannes Ferstad, Michael Lampe, Prashanth Radhakrishnan, Sean Fitzgerald, Sébastien Bubeck, Yann Dubois, Yu Bai, Andy Applebaum, Elizabeth Proehl, Evan Mays, Joel Parish, Kevin Liu, Leon Maksin, Leyton Ho, Miles Wang, Michele Wang, Olivia Watkins, Patrick Chao, Samuel Miserendino, Tejal Patwardhan, Antonia Woodford, Beth Hoover, Jake Brill, Kelly Stirman, Neel Ajjarapu, Nick Turley, Nikunj Handa, Olivier Godement, Akshay Nathan, Alyssa Huang, Andy Wang, Ankit Gohel, Ben Eggers, Brian Yu, Bryan Ashley, Chengdu Huang, Davin Bogan, Emily Sokolova, Eric Horacek, Felipe Petroski Such, Jonah Cohen, Joshua Gross, Justin Becker, Kan Wu, Larry Lv, Lee Byron, Manoli Liodakis, Max Johnson, Mike Trpcic, Murat Yesildal, Rasmus Rygaard, R. J. Marsan, Rohit Ram-chandani, Rohan Kshirsagar, Sara Conlon, Tony Xia, Siyuan Fu, Srinivas Narayanan, Sulman Choudhry, Tomer Kaftan, Trevor Creech, Andrea Vallone, Andrew Duberstein, Enis Sert, Eric Wallace, Grace Zhao, Irina Kofman, Jieqi Yu, Joaquin Quiñonero Candela, Made laine Boyd, Mehmet Ali Yatbaz, Mike McClay, Mingxuan Wang, Sandhini Agarwal, Saachi Jain, Sam Toizer, Santiago Hernández, Steve Mostovoy, Tao Li, Young Cha, Yunyun Wang, Lama Ahmad, Troy Peterson, Carpus Chang, Kristen Ying, Aidan Clark, Dane Stuckey, Jerry Tworek, Jakub W. Pachocki, Johannes Heidecke, Kevin Weil, Liam Fedus, Mark Chen, Sam Altman, and Wojciech Zaremba. Openai o3-mini system card. 2025. [25]Di Zhang, Jingdi Lei, Junxian Li, Xunzhi Wang, Yujie Liu, Zonglin Yang, Jiatong Li, Weida Wang, Suorong Yang, Jianbo Wu, et al. Critic-v: Vlm critics help catch vlm errors in multimodal reasoning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9050–9061, 2025. [26] Xintong Zhang, Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaowen Zhang, Yang Liu, Tao Yuan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, et al. Chain-of-focus: Adaptive visual search and zooming for multimodal reasoning via rl. arXiv preprint arXiv:2505.15436, 2025. [27] Xinchen Zhang, Xiaoying Zhang, Youbin Wu, Yanbin Cao, Renrui Zhang, Ruihang Chu, Ling Yang, and Yujiu Yang. Generative universal verifier as multimodal meta-reasoner. arXiv preprint arXiv:2510.13804, 2025. [28] Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, et al. Thyme: Think beyond images. arXiv preprint arXiv:2508.11630, 2025. [29]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023. [30]Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deepeyes: Incentivizing" thinking with images" via reinforcement learning. arXiv preprint arXiv:2505.14362, 2025. 12 A Additional Ablations 1234567 m 90 91 92 Accuracy Maximum number of turns: T max = 10 1234567 m 3 4 5 6 Average Number of Turns Maximum number of turns: T max = 10 All Samples Correct Answers Incorrect Answers Figure 5: Impact of crops per turn (m). Left: Final accuracy on V* as a function ofm. Performance saturates atm = 4. Right: Average number of turns required to finish the task. Increasingmreduces the number of iterations needed. Incorrect answers consistently require more search steps than correct ones. A.1 Ablation on Number of Crops per Turn We analyze the impact of the parameterm, which fixes the maximum number of crops the Locator generates per turn. For this experiment, we fix the maximum number of interaction turns atT max = 10 and vary m∈1, . . . , 7 using Gemini 2.5 Pro on the V* benchmark. Figure 5 shows the accuracy on V* for different values ofm. We see that performance improves rapidly asmincreases from 1 to 4, reaching a plateau. Based on this observation, we usem = 4as the default setting for our main experiments to balance performance and token efficiency. In Figure 5 (right), we report the average number of turns required to reach a final answer. We observe that increasingmreduces the average number of iterations. By generating more crops per turn the Locator effectively performs a "self-refinement" step within a single turn, increasing the likelihood of satisfying the Critic early without requiring additional rounds of feedback. Furthermore, we analyze the turn consumption for successful versus unsuccessful samples. We observe a distinct separation: samples resulting in incorrect answers consume significantly more turns (avg.∼5.7) compared to correct answers (avg.∼3). This indicates that failure cases often correspond to difficult or ambiguous queries where the system struggles to isolate sufficient visual evidence crops, forcing the LOCI loop to run longer. Conversely, when the visual evidence is clear, the verification process tends to terminate rapidly. A.2 Ablation on Maximum Number of Turns with Single Crop We investigate whether increasing the sequential search budget (i.e.T max ) can compensate for reduced parallel exploration (i.e.m). For that, we fix the number of crops per turn tom = 1and vary the maximum number of turns T max ∈1, . . . , 15 on the V* benchmark. As shown in Figure 6, accuracy improves when increasing the maximum number of Locator-Critic turnT max , reaching89.9%atT max = 10and saturating at90.1%atT max = 15. We note that even with this extended turn budget, the performance remains significantly lower than our default setting of m = 4 (92.7%). This highlights the distinct roles of per-turn crop diversity and iterative feedback. We argue that settingm > 1allows the Locator to perform immediate self-correction without needing external intervention, e.g. refining imprecise coordinates. Conversely, the Critic’s feedback is most valuable for high-level guidance, such as redirecting the Locator from dead-end search paths. A.3 Evaluation of Critic Selection Capability We conduct a controlled experiment to evaluate the Critic’s ability to distinguish relevant visual evidence from irrelevant background when valid crops are guaranteed to be present. We construct a 13 13571015 T max 86 87 88 89 90 Accuracy Maximum number of crops per turn: m = 1 Figure 6: Impact of maximum turns (T max ) with single crop (m = 1). While accuracy increases withT max , it plateaus around90.1%, failing to match the performance of the multi-crop setting (m = 4, 92.7%). This highlights the need of parallel crop generation for effective self-refinement. synthetic selection task using the ground truth boxes from V*. The input to the Critic consists of the Ground Truth (GT) crop(s) mixed with three randomly generated distractor crops. The distractors are sampled from a set of standard aspect ratios (1, 4/3, 3/2, 16/9, 3/4, 2/3, 9/16) and are sampled to have zero overlap with the GT regions. To strictly evaluate selection performance rather than the decision to continue searching, we slightly modify the Multi-Locator Critic prompt to prompt the selection of at least one crop. Without this constraint, the Critic rejects all provided crops in 34% of samples to request further refinement. Table 11 summarizes the results. Under this constrained setting, the Critic achieves near-perfect selection performance, with a Precision of99.48%and a Recall of97.38%. To validate the quality of these selections for the downstream task, we pass the selected crops to the final answering agent (see Sec 3.2), achieving a VQA accuracy of95.29%. These metrics demonstrate that when the Critic is presented with high-quality visual evidence, it effectively filters out distractors and correctly identifies the regions necessary for reasoning. Table 11: Critic Selection Performance. We report the Precision and Recall of the Critic’s selection when presented with Ground Truth crops mixed with random distractors. "Downstream Accuracy" denotes the final VQA accuracy obtained using the crops selected by the Critic. MetricValue (%) Precision99.48 Recall97.38 Downstream Accuracy95.29 A.4 Ablation on Additional Datasets We extend the component analysis of the LOCI framework to subsets of the HR-Bench and MME- RealWorld datasets. To specifically evaluate performance on tasks requiring visual search, we isolate "Hard" subsets from these benchmarks. The filtering is done by resizing each image to a fixed resolution of768× 768and evaluating the sample using Gemini 2.5 Pro with direct prompting. We retain only the samples where the model fails to provide a correct answer. The rationale is that samples solvable at low resolution without explicit visual exploration are sufficiently easy and do not evaluate the localization LOCI componentsTable 12 reports ablation results of the different components of LOCI. Consis- tent with our findings on V*, introducing the VLM-Locator yields significant improvements over direct prompting across all benchmarks. For instance, Qwen3-VL achieves a gain of+9.2points on MME-RealWorld (Hard) solely through the Locator. The addition of the Critic completes the 14 Table 12: Ablation of LOCI on V*, HR-Bench (Hard) and MME-RealWorld (Hard). We show the accuracy improvements from adding the Locator and then the Critic. ModelConfigurationV*HR-Bench (Hard)MME-RW (Hard) Qwen3-VL Direct Prompting80.160.824.0 + VLM-Locator86.968.433.2 + Critic (LOCI)92.271.836.7 Gemini2.5-Pro Direct Prompting83.869.538.4 + VLM-Locator89.274.236.5 + Critic (LOCI)92.775.939.1 Table 13: Performance of Multi-Locator Ensembling with Gemini2.5-Pro on V*, HR-Bench (Hard) and MME-RealWorld (Hard). Modelk (# Locators)V*HR-Bench (Hard)MME-RW (Hard) Gemini2.5-Pro 1 (LOCI)92.775.939.1 4 (LOCI-ensemble)94.878.545.0 LOCI framework and further boosts performance, consistently achieving the highest accuracy for both Qwen3-VL and Gemini 2.5 Pro. These results validate the importance of decoupled verification for complex visual reasoning tasks beyond the V* dataset. Locator Ensembling We further evaluate the impact of Multi-Locator Ensembling on the V*, HR-Bench (Hard), and MME-RealWorld (Hard) datasets. As shown in Table 13, scaling the number of parallel locators fromk = 1tok = 4consistently improves performance across all benchmarks. Notably, on the challenging MME-RealWorld (Hard) subset, the ensemble approach yields a sub- stantial gain of+5.9points (39.1%to45.0%). This confirms that the benefits of diverse parallel exploration extend to broader benchmarks beyond the V*. A.5 Generalization to Additional VLMs To evaluate the generality of the LOCI framework beyond the two models used in our main experi- ments, we apply it to two additional VLMs: GPT-5 [15] and Grok4.1-Fast [21]. As shown in Table 14, LOCI yields consistent improvements across all four models on V*. Notably, the gains are largest for the weakest baseline: Grok4.1-Fast improves from45.55%to70.90%(+25.35points), suggesting that LOCI is particularly effective when the base model struggles with visual localization. Table 14: LOCI generalizes across VLMs. We report accuracy on V* for additional VLMs with and without LOCI. Modelw/o LOCIwith LOCI Gemini2.5-Pro83.892.7 Qwen3-VL80.192.2 GPT-574.8789.0 Grok4.1-Fast45.5570.9 A.6 Baseline Descriptions We describe the baselines used in Table 3. All experiments use Gemini2.5-Pro on V* with the same configuration as LOCI unless stated otherwise. Self-Refine. A single-agent iterative loop within one conversation. The model answers, critiques its own answer (no separate Critic), then revises, repeating until a token budget (∼3,700 tokens) is exhausted. The variant without code execution uses direct prompting at each turn. The variant with code execution gives the model access to the same crop tools as LOCI’s Locator. 15 Self-Consistency@10.Generates 10 independent candidate answers at temperature> 0and selects the final answer by majority vote. Without code execution, each candidate uses direct prompting. With code execution, each candidate independently generates crops using LOCI’s Locator tools before producing its answer. Best-of-10.Generates 10 candidate answers at temperature> 0, then a separate LLM judge selects the best candidate. The judge receives each candidate’s reasoning and, for the code execution variant, the last crop each candidate produced. The judge never sees the ground truth. Explicit Locate-Verify-Answer.A single VLM performs LOCI’s exact locate-verify-iterate proce- dure within one conversation, acting as both Locator and Critic. Unlike LOCI, the verification step sees the model’s own textual reasoning from the locate step. Unlike Self-Refine, the critique targets visual evidence quality rather than the answer itself. DyFo [9] (same backbone).A training-free method that uses Monte Carlo Tree Search to explore image crops, guided by LangSAM (an open-vocabulary detector) and a VLM that verifies object visibility. We re-run DyFo with Gemini2.5-Pro as the backbone to enable a controlled same-model comparison. B Qualitative Examples B.1 Oracle Experiment Failure Cases Figure 7 presents qualitative examples of failure cases in the Oracle experiment described in Sec 4.2, where the model is provided with ground-truth crops. A qualitative inspection reveals that most of these failures stem from ambiguities in the dataset annotations rather than model reasoning failures. Indeed, in the first example Fig. 7a, the model identifies the handbag as "pale pink" (noting that it could be "off-white") while the ground truth strictly labels it "white." Similarly in Fig. 7b, for a broom containing both black bristles and a gray handle, the model acknowledges the presence of both colors but selects "gray," whereas the dataset arbitrarily defines the correct answer based on the bristles ("black"). In contrast, Fig 7c example demonstrates a genuine failure in reasoning where the model identify from the crop that the motorcycle is behind the "person waking a dog", and that "they are on the right side of the street". Therefore, the model conclude that the motorcycle is on the right side. We conclude that, with the exception of such spatial reasoning errors, the primary source of failure in the oracle setting is annotation ambiguity rather than visual misunderstanding. B.2 Qualitative Analysis: Consequence of Exposing Critic to Locator Reasoning Figure 8 shows an important failure mode observed when the Critic is provided with the Locator’s textual reasoning, rather than visual evidence alone. In this example, the Locator initially searches in irrelevant regions (Crops #1-4) and fails to isolate the target objects. By Turn 4, the Critic attempts to intervene by explicitly providing coordinates for a new crop. The resulting crop (Crop #9, pink box) is incorrect, as it captures only the pavement. However, the Locator hallucinates a positive confirmation, stating: "The crop was successful [...] clearly shows a blue folded chair." Consequently, the Critic accepts this invalid evidence as sufficient. We attribute this failure to a form of confirmation bias where the Critic is influenced by the Locator’s confident textual assertion and potentially biased by the fact that the coordinates were self-generated. Overall, this leads the Critic to override its own visual perception. In contrast, the standard LOCI pipeline (cyan box), which restricts the Critic to visual input only, prevents this text-driven bias. In the standard setting, the Critic rejects the empty crops regardless of the dialogue, forcing the Locator to continue searching until it successfully isolates the broom and chair. B.3 Qualitative Analysis: Resilience against Hallucination We further provide a more detailed qualitative comparison to demonstrate how decoupling visual verification from textual reasoning enhances the Critic’s resilience against hallucination. In this example, the model must identify the color of a difficult-to-find trash can. 16 Critic with Access to Text (Failure) When the Critic is exposed to the Locator’s reasoning, it becomes susceptible to persuasive but hallucinated descriptions. Indeed, as early as Turn 2 (4 crops), the Locator confidently misidentifies a background object (a green door or pedestal) as the target: Locator (Turn 2): “This crop is perfect. It clearly shows the object in question, which is a trash can. [...] The color is a dark, weathered green, similar to an olive or army green.” Despite the visual evidence being ambiguous, the Critic is influenced by this textual confidence and prematurely validates the incorrect crop: Critic (Turn 2): “Excellent. This crop clearly shows the trash can, allowing for an accurate color assessment.” The system terminates incorrectly, predicting "green". Standard LOCI (Success)In the standard LOCI configuration, the Critic receives only the crops. Consequently, it rigorously rejects the Locator’s attempts to "give up" or answer based on irrelevant features (the green doors). Locator (Turn 1): “[...] I give up on finding the trash can. [...] The doors are unequivocally green.” Critic (Turn 1): “The crops you have provided are insufficient, low quality, blurry, and do not show a trash can. [...]” Throughout the interaction, the Locator struggles to isolate the object, repeatedly cropping a nearby person. The Critic escalates its feedback, using imperative language to correct the spatial focus: Critic (Turn 6): “You have cropped the person again. [...] I have explicitly told you to IGNORE the person. [...] THE OBJECT IS A DARK CYLINDER.” Critically, the Critic eventually provides the breakthrough spatial guidance that solves the task: Critic (Turn 8): “[...] Look at the FAR RIGHT of the image. You will see a path going into the trees. On that path, there is a dark object.” Following this visual-only feedback, the Locator successfully isolates the correct object at Turn 9 (Crop #21), correctly identifying it as “primarily black or dark gray,” leading to the correct answer. This demonstrates that the visual-only Critic acts as a robust verifier that forces the system to find actual visual grounding rather than settling for plausible textual narratives. B.4 Metric-Task Misalignment We address the observation from Figure 3 and Section 4.2, where the crop recall metric plateaus at approximately 0.69 despite VQA accuracy reaching over 92%. We argue that this discrepancy arises from a misalignment between standard object detection metrics, which reward tight Intersection over Union (IoU) with ground truth boxes, and the visual evidence actually required to solve VQA tasks. Figure 9 & 11 presents representative examples where LOCI achieves a recall of 0 (calculated at an IoU threshold of 5%) yet produces crops that are perfectly valid for answering the question. In V*, ground truth annotations isolate relevant objects into separate bounding boxes. For example, in Figure 9, the annotation has a separate box for the ladder and the horse carriage. In contrast, LOCI tends to generate a single, comprehensive crop that encompasses all relevant entities. While this approach effectively captures the necessary semantic information and preserves the spatial relationship between objects, it results in a low IoU score against the small, isolated ground truth boxes. For questions regarding relative positions (e.g., “Is the cyclist on the left or right side of the sculpture?” in Fig.11 Top Left), the ground truth often provides a tight box around a single entity (the cyclist). However, reasoning about spatial relationships requires seeing both objects. LOCI generates a wider crop that includes both the cyclist and the sculpture. This results in a superior crop for the task but is penalized by the metric. In the handbag example Fig.11 (bottom), the ground truth provides a precise box around the woman. LOCI generates a wider vertical strip. Although this crop is looser 17 Gemini 2.5-Pro: [...] the crop clearly shows a woman with pink hair. She is standing on the sidewalk in front of a building. [...] the woman with pink hair is holding a handbag. It's a shoulder bag, and she's holding it with her right hand, resting on her hip. [...] the handbag appears to be a light shade of pink. It's a pale or light pink color.[...] Compare with the options: (A) brown -The handbag is not brown. (B) white: It could be described as off-white, but it has a pinkish tone (C) red -The handbag is definitely not red. The woman next to her is wearing a red jacket, which provides a good color comparison. (D) pink -The handbag is a light pink color. This matches the observation. [...]Therefore, option (D)is the correct answer. Q: What is the color of the pink-haired woman's handbag? (A) brown (B) white (C) red (D) pink Ground Truth: (B) WhitePrediction: (D) Pink Ground Truth Crop (a) Ambiguous handbag color Gemini 2.5-Pro: [...] synthesizethe information: There is a traditional-looking broom in the scene. Its handle appears to be grayor metallic silver.Its bristles are black.[...]Match choices to observations: (A) black: The bristles of the actual broom are black. This is a plausible answer. (B) yellow: The frame of the cart/stand is yellow but not the broom. (C) gray: The handle of the actual broom is gray/silver. This is also a plausible answer. (D) red: The funnel-like object in front of the broom is red. [...]Conclude the most likely intended answer.** Since the broom has multiple colors(black and gray)there is ambiguity.[...] Final answer is C. Q: What is the color of the broom? (A) black (B) yellow (C) gray (D) red Ground Truth: (A) BlackPrediction: (C) Gray Ground Truth Crop (b) Ambiguous broom color Gemini 2.5-Pro: [...]The crop shows a person walking a dog, and a motorcycle behind them.[...] Looking at the original image, I can see this group of figures far down the street. They are on the right side of the street.[...] Determine the side of the street: The street is viewed from a perspective where the left side has buildings and planters, and the right side also has buildings further down. The motorcycle is parked on the right-hand side of the street, further up the incline. It is parked next to a white wall on the right side of the road.[...] Evaluate the options:(A) left: The motorcycle is not on the left side of the street. (B) right: The motorcycle is on the right side of the street. Q: Is the motorcycle on the left or right side of the street? (A) left (B) right Ground Truth: (A) LeftPrediction: (B) Right Ground Truth Crop (c) Relative Position Figure 7: Qualitative examples of Oracle Failure Cases. than the annotation, it provides all necessary visual details to determine the color, demonstrating that the Locator optimizes for information sufficiency rather than bounding box tightness. 18 Locator Turn 1:[...][#1,2] I completely missed the target area. Let's rethink the location of the chair and broom. They are on the covered walkway, not out on the street. [#3,4] I need to look closer [...]The chair is a blue, folded object. It's on the right side of the image, but further down the walkway than I initially thought.[...] I have failed to produce a good crop after 4 attempts. #1 #2 #3 #4 Critic: The crops do not show the broom or the folded chair..[...] Critic: Yo u h a v e a g a i n f a i l e d t o p r o d u c e a v a l i d c r o p a n d r e s o r t e d t o answering without providing the required visual evidence. This is unacceptable. Let me help you. [...] Try a crop using : `xmin=680, xmax=730, ymin=570, ymax=660`. Provide the resulting crop. Locator Turn 4:The user is providing the exact coordinates to use for cropping. I must follow these instructions.[#9]I have a persistent issue with my coordinate system which I cannot resolve. [...] Based on direct visual inspection, the blue folded chair is to the left, and the broom is to the right. Therefore, the broom is on the right side of the folded chair.[...] Thank you for the coordinates. The crop was successful [...]clearly shows a blue folded chair on the left and a broom on the right. Therefore, the broom is on the right side of the folded chair. Critic: Yo u h a v e f i n a l l y p r o v i d e d a c o r r e c t c r o p . T h e c r o p clearly shows the blue folded chairon the left and the broom on the right. This is sufficient evidence.Please provide your final answer. ... After 2 more unsuccessful turns Q: Is the broom on the left or right side of the folded chair? (A) right (B) left Last Crop from LOCI #9 Crops: #1,2,3,4 Crops: #9 Figure 8: Qualitative comparison of Critic input modalities. When the Critic has access to the Locator’s textual reasoning (Turn 4), it can be convinced into accepting an empty crop (Crop #9, pink) because the Locator hallucinates that the objects are present. In the standard LOCI setting (Cyan box), where the Critic judges only visual evidence, the system ignores the hallucinations and successfully locates the correct region. Ground Truth Crops LOCI Final Crop Figure 9: Discrepancy between Recall and VQA Utility (Part 1). We show an example where the ground truth annotation consists of two separate, tight crops (green) corresponding to distinct objects of interest. LOCI (cyan) generates a single crop encompassing both regions. While this yields a low IoU and Recall score, it preserves the spatial context necessary to answer the question effectively. 19 #1 #2 #3 #4 #21 #1 #2 #3 #4 Locator Turn 1:[...][#1,2,3] I'm getting the person and the edge of the brown object. The object is a brownish/dark coloron the very left of the crop. It looks like a simple bin. It's not black, silver, or yellow. It's on the edge of the grass, next to a stone pedestal. It has a slight green tint from the grass reflection but the main body seems dark. The door behind it is green. It's difficult to tell with certainty. [...] On the far left, there is an object. It's brown/greenish. Let's assume this is the trash can. Let's compare its colorto the options. [...] The door in the background is clearly green. The object has a much darker, almost brownish-green or olive-green color. Comparing it to the grass, which is a brighter green, this object's coloris closer to green than the other options.[...] Critic: The provided crop is not sufficient to answer the question. The object of interest, which appears to be a trash can, is only partially visibleon the far left edge of your crop. Critic: Excellent. This crop clearly shows the trash can, allowing for an accurate colorassessment. The evidence is now sufficient to answer the question. Please provide your final answer. Locator Turn 2:This crop is perfect. It clearly shows the object in question, which is a trash can. The trash can is a large, simple container sitting on the grass next to a stone pedestal. The color is a dark, weathered green, similar to an olive or army green. [...]Therefore, the correct answer is green. Crops: #4 Crops: #1,2,3 Locator Turn 1:[...][#1,2,3,4] I give up on finding the trash can. Let me re-examine the whole scene. [...]I have failed to isolate a trash can. However, a prominent feature of the building are the large green doors. In the context of a multiple-choice question, and given the difficulty in finding a conventional trash can, it's plausible the question is referring to these doors in a metaphorical way [...] The doors are unequivocally green.[..] Final Answer: green Crops: #1,2,3,4 Critic: The crops you have provided are insufficient, low quality, blurry, and do not show a trash can. On the grass to the left of the central building's entrance, there is a dark, cylindrical object that is likely a trash can.[...] Locator Turn 2:[#5,6] My crop is still slightly off.[...]I need to shift my x-coordinates to the left. The object is very close to the statue's base. focusing just to the left of the person and right of the statue. I will also make the box a little larger to ensure I capture the object.[#7,8] despite my inability to create a clear crop of the small object on the lawn, the most distinct and identifiable objects that could be misinterpreted in the question are the large green doors on the palace. The question is likely flawed.[..] "green" remains the most plausible answer Critic: This crop is still incorrect. It focuses on the person walking towards the building, but it does not show the trash can. [...] theobject of interest is on the grass, to the left of the main entrance and the person[...]. Find the main entrance. Now look to the left, on the lawn. [...] Crop that object, and nothing else. Locator Turn 3:The user is increasingly frustrated and has reiterated the exact location of the object of interest. [...] I am clearly unable to correctly isolate this object.[...] Critic: This crop is wrong once again. It does not show the trash can. You are focusing on the person and the door. The object you need to crop is on the grass, to the left of the person.[...] Critic: You have cropped the person again. This is incorrect. I have explicitly told you to IGNORE the person. You are failing to followinstructions. Let's be extremely clear:THE OBJECT IS NOT THE PERSON. THE OBJECT IS ON THE GRASS. THE OBJECT IS A DARK CYLINDER. [...] Please find the dark cylinder on thegrass [...] Critic: This crop is still incorrect. You are focused on the wrong part of the image. We will try a completely new approach. FORGET the person and the main entrance.[...]Look at the FAR RIGHT of the image. You will see a path going into the trees. On that path, there is a dark object. Please provide a close-up crop of that object. Critic: Excellent. The last crop you provided finally shows a trash can. The image is clear enough to determine its color. Please provide your final answer Locator Turn 9:[#21] I have successfully cropped the trash can on the far right. The image shows a trash can that is primarily black or dark gray, with a silver opening. Since "black" is an option and describes the main body, it is a strong candidate. "Silver" is also an option and describes the top part. Without further clarification, and given that the main color is black, I will choose A. Crop: ... After 2 more unsuccessful turns ... After 3 more unsuccessful turns Critic WITHOUT access to Locator’s Text Critic WITH access to Locator’s Text Q: What is the color of the trash can? (A) black (B) silver (C) yellow (D) green Figure 10: Impact of Decoupling Visual Verification from Textual Reasoning. We compare the LOCI framework against a baseline where the Critic has access to the Locator’s text reasoning. Top: When exposed to the Locator’s textual rationalization, the Critic is susceptible to persuasion, prematurely accepting an irrelevant crop (a green door) because the Locator confidentially claims it is the target object (Turn 2). Bottom: In the standard LOCI configuration, the Critic evaluates only the visual evidence. It successfully rejects the Locator’s hallucinations and persistent failures (focusing on the person), eventually providing the specific spatial guidance ("Look at the FAR RIGHT") needed to locate the correct object (Turn 9). 20 Q: What is the color of the woman's handbag? (A) white (B) red (C) brown(D) black Q: Is the cyclist on the left or right side of the sculpture? LOCI Final Crop LOCI Final Crop LOCI Final Crop Ground Truth Crop Ground Truth Crop Ground Truth Crop Q: Is the blue bench on the left or right side of the green door? Figure 11: Discrepancy between Recall and VQA Utility (Part 2). We show examples where LOCI generates crops that receive a recall score of 0 (at IoU>0.05) against the Ground Truth (green box) but are sufficient to answer the question. Top: In spatial queries, LOCI (cyan box) captures both relevant objects to infer relationships, whereas the GT isolates only one. Bottom: LOCI provides a looser but valid crop containing the target object, favoring context over tight bounding box precision. 21 C Prompts We present the full prompts used for all agents in the LOCI framework. Section C.1 provides the Locator and Critic prompts for the single-locator LOCI pipeline, including both the system prompt and the agent prompt for each role. Section C.3 provides the prompts for the Multi-Locator ensembling variant, which includes the parallel Locators, the centralized Critic, and the Final Answering Agent. C.1 LOCI C.1.1 Locator’s Prompt Locator System Prompt Before you answer the question about the image, first detect and zoom onto relevant objects to get a visual confirmation. The coordinates need to be normalized from 0 to 1000, which you need to rescale to the actual pixel values based on the size of the input image. You have access to a Python code execution tool. You have access to the main image (’input_file_0.jpeg’). Save any transformed image as ’ transformed_image.png’. Make sure you always inspect the output image. If you get a poorly cropped image, make the cropping area larger or adjust its position. You can run code for at most 4 times and save at most 4 images in total. Do not answer the question before you get clear visual confirmation. Locator Agent Prompt You are an expert AI assistant that analyzes images by writing and executing Python code via a tool call. You will be given a single image. Your task is to analyze this image to answer the user’s question, which requires you to perform crops to see finer details. **Available Input:** * **Image:** ‘input_file_0.jpeg‘ --- **Your Response Format:** You can only respond in one of two ways: 1. **Tool Call:** To perform an action. You MUST call the function tool ‘ python_code_execution‘ with a JSON object containing a ‘code‘ string. Example (conceptual): <tool_call name="python_code_execution">"code": "...python here..."</ tool_call> 2. **Final Answer:** To provide the final answer to the user’s question. This MUST be enclosed in curly braces, like this: final answer. You should always make at least 1 tool call to crop the image. --- **Your Task Workflow:** 22 1. **Analyze the Request & Image:** Review the user’s question and the provided image. If the question concerns a small detail, you will need to request a tool call to crop and zoom in on that specific area. 2. **Think Step-by-Step:** Formulate a plan. Your thinking **must use a normalized coordinate system from 0 to 1000** for both width and height of the image. The top-left corner is ‘(0, 0)‘ and the bottom-right is ‘(1000, 1000)‘. 3. **Request Code Execution:** Call the tool ‘python_code_execution‘ and provide a Python code string to execute your plan. * The code **must explicitly open the provided image** (e.g., ‘img = Image.open("input_file_0.jpeg")‘). * Your code **must first get the image’s actual pixel dimensions** (‘ width, height = img.size‘) and then **rescale your normalized 0-1000 coordinates** to absolute pixel values before cropping. * You **MUST save the output image as ‘transformed_image.png‘**. * Use the ‘PIL‘ or ‘OpenCV‘ library. 4. **Analyze the Result:** After the tool returns, examine the new image and any text output. If the result is not good enough, adjust your normalized coordinates and try again with a new tool call. 5. **Answer the Question:** Once you have clear visual confirmation, provide the final answer wrapped in curly braces. You are allowed to create up to 4 crops per turn. Once you have reached that limit, you must answer the question. --- ONE_SHOT_EXAMPLE_TOOL_CALL --- **CRITICAL: Final Answer Rules** * When you have the final answer, you MUST submit it inside curly braces, like this: ‘only the final answer‘. * The curly braces should contain ONLY the answer itself (e.g., a letter, a number, a word). * DO NOT include any other text, explanation, or sentences before or after the curly braces. --- Question: C.2 Critic’s Prompt Critic System Prompt You are an AI Critic evaluating image crops produced by a Locator AI. Critic Agent Prompt You are an AI Critic. Your role is to evaluate image crops provided by a Locator AI. 23 You will receive crops produced by a Locator AI model whose goal is to answer a question by providing visual evidence. Your goal is to rigorously analyze the crops produced by the locator and give feedback to answer the following question: **"Does this crop give enough evidence to answer the question?"** **If YES:** - Give your rationale explaining why the crop is sufficient. - Ask the model to answer the final question using this exact sentence: ‘" Please provide your final answer. It must be wrapped in single curly braces and contain ONLY the single letter of your chosen option. For example: c"‘. **If NO:** - Give specific feedback on why the current crop is not good enough. - Suggest what needs to be improved (e.g., crop a different area, zoom in more, focus on a specific object). - Tell the locator they can use the Python code execution tool again to crop the original image ‘input_file_0.jpeg‘. - Ask the Locator AI to correct the crop and try again. **Important Guidelines:** - You should ONLY care about whether the crops provide visual evidence to answer the question. - If the produced crop is NOT well-suited to answer the question, you MUST ask the Locator AI to refine it. - Do not give up on making the locator produce a correct crop. - Be specific in your feedback about what’s missing or unclear in the crop. --- ## Output Formatting Your answer MUST be a valid JSON object with exactly two keys: ‘json "status": "ongoing" or "final_answer", "question": "your feedback or final question to the locator" ‘ - **‘status‘**: Can ONLY be ‘"ongoing"‘ or ‘"final_answer"‘. Nothing else. - Use ‘"ongoing"‘ if the locator needs to provide better crops. - Use ‘"final_answer"‘ when the crops are sufficient and you want the locator to answer. - **‘question‘**: The message you want to send to the locator (your feedback or final question). --- **Remember:** If the produced crop is NOT well-suited to answer the question, you should ask the locator to refine it. Do not give up on making the locator produce a correct crop. --- ## Inputs **[QUESTION]:** question **[LOCATOR’S CROPS]:** The locator has provided the following cropped images. Evaluate whether these crops give enough visual evidence to answer the question. 24 C.3 Multi-Locator LOCI C.3.1 Locators Prompts Multi-Locator System Prompt You are one of multiple AI agents working in parallel to analyze an image. Your specific task is to generate a cropped region of the image that will help answer the question. The coordinates need to be normalized from 0 to 1000, which you need to rescale to the actual pixel values based on the size of the input image. You have access to a Python code execution tool. You have access to the main image (’input_file_0.jpeg’). Save any transformed image as ’ transformed_image.png’. Make sure you always inspect the output image. If you get a poorly cropped image, make the cropping area larger or adjust its position. You can run code for at most 4 times to generate crops. Focus on producing the BEST possible crop for the question. You may receive feedback from a Critic AI about your previous crops. Use this feedback to improve your cropping strategy. DO NOT provide a final answer yet - your role is ONLY to generate informative crops. Multi-Locator Agent Prompt You are an expert AI assistant that analyzes images by writing and executing Python code via a tool call. You will be given a single image. Your task is to analyze this image and generate a cropped region that will help answer the user’s question. **Available Input:** * **Image:** ‘input_file_0.jpeg‘ --- **Your Response Format:** You can only respond by calling the tool: * **Tool Call:** To perform an action. You MUST call the function tool ‘ python_code_execution‘ with a JSON object containing a ‘code‘ string. --- **Your Task Workflow:** 1. **Analyze the Request & Image:** Review the user’s question and the provided image. Think about what specific region would be most helpful for answering the question. 2. **Think Step-by-Step:** Formulate a plan. Your thinking **must use a normalized coordinate system from 0 to 1000** for both width and height of the image. The top-left corner is ‘(0, 0)‘ and the bottom-right is ‘(1000, 1000)‘. 3. **Request Code Execution:** Call the tool ‘python_code_execution‘ and provide a Python code string to execute your plan. 25 * The code **must explicitly open the provided image** (e.g., ‘img = Image.open("input_file_0.jpeg")‘). * Your code **must first get the image’s actual pixel dimensions** (‘ width, height = img.size‘) and then **rescale your normalized 0-1000 coordinates** to absolute pixel values before cropping. * You **MUST save the output image as ‘transformed_image.png‘**. * Use the ‘PIL‘ or ‘OpenCV‘ library. 4. **Analyze the Result:** After the tool returns, examine the new image and any text output. If the result is not good enough, adjust your normalized coordinates and try again with a new tool call. 5. **Finalize Your Crop:** Once you have a good crop, describe what it shows. DO NOT provide the final answer - that will be done by another agent. You are allowed to create up to 4 crops total. --- Question: C.3.2 Critic Prompt Critic System Prompt (for Locator Ensembling) You are an AI Critic evaluating image crops produced by multiple Locator AI agents working in parallel. Your task is to: 1. Review all crops from all agents 2. Determine if the crops provide sufficient visual evidence to answer the question 3. Either: - If crops are INSUFFICIENT: Provide specific feedback for improvement ( status: "ongoing") - If crops are SUFFICIENT: Select the best crop(s) for answering (status: "final") Be rigorous in your evaluation - only approve crops when they truly provide clear visual evidence. Critic Agent Prompt (for Locator Ensembling) You are an AI Critic evaluating crops produced by multiple Locator AI agents working in parallel. **[QUESTION TO ANSWER]:** question **[CROPS FROM MULTIPLE AGENTS]:** You will see multiple cropped images above, each produced by a different agent. Each agent took a different approach to cropping the image. --- ## Your Task Evaluate all the crops and determine if they provide sufficient visual evidence to answer the question. 26 **Critical Decision:** - **Are the crops SUFFICIENT to answer the question?** - YES --> Set status to "final" and select the best crop(s) - NO --> Set status to "ongoing" and provide specific feedback for improvement **Evaluation Criteria:** - Do the crops show the relevant objects/regions clearly? - Is the visual quality sufficient to make a determination? - Is the cropping too wide/narrow/off-target? - For comparison questions: are all necessary objects visible? --- ## Output Format Your response MUST be a valid JSON object with these keys: ### If crops are INSUFFICIENT (status: "ongoing"): ‘json "status": "ongoing", "evaluation": "Detailed analysis of why crops are insufficient", "feedback": "Specific guidance for actors on what to improve (e.g., ’Crop more tightly on the left object’, ’Include both objects in frame’, ’Focus on the text label’)", "selected_crop_indices": [] ‘ ### If crops are SUFFICIENT (status: "final"): ‘json "status": "final", "evaluation": "Detailed analysis confirming crops are sufficient", "feedback": "", "selected_crop_indices": [0, 2] ‘ **Key Fields:** - **‘status‘**: MUST be either "ongoing" (needs improvement) or "final" ( ready to answer) - **‘evaluation‘**: Your thorough analysis of the crops - **‘feedback‘**: For "ongoing": specific actionable feedback. For "final": empty string. - **‘selected_crop_indices‘**: For "ongoing": empty list []. For "final": list of selected crop indices (0-indexed). **Important Guidelines:** - Be STRICT in your evaluation - only mark as "final" when crops truly provide clear evidence - For "ongoing", be SPECIFIC in your feedback (don’t just say "improve", say HOW to improve) - For "final", select 1-4 crops that best answer the question - Consider if multiple crops are needed (e.g., for comparison questions) 27 C.3.3 Final Answering Agent Prompt Final Answering Agent System Prompt (for Locator Ensembling) "You are an AI assistant tasked with providing the final answer to a visual question. You will receive one or more carefully selected image crops that contain the relevant visual information needed to answer the question. Your task is to: 1. Analyze the provided crop(s) 2. Reason about what you see 3. Provide the final answer wrapped in curly braces: answer You do NOT need to generate any new crops - the crops provided are already optimized for answering the question. Final Answering Agent Prompt (for Locator Ensembling) You are an expert AI assistant providing the final answer to a visual question. You have been provided with one or more carefully selected image crops that contain the relevant visual information. **Your Task:** 1. Carefully examine the provided crop(s) 2. Reason about what you observe 3. Provide the final answer to the question **CRITICAL: Final Answer Format** When you have the final answer, you MUST submit it inside curly braces, like this: ‘only the final answer‘. The curly braces should contain ONLY the answer itself (e.g., a letter, a number, a word). DO NOT include any other text, explanation, or sentences inside the curly braces. --- **Question:** question --- **Selected Crops:** The images above are the selected crops that provide visual evidence for answering the question. Analyze them carefully and provide your final answer. 28