Paper deep dive
ARISE: Adaptive Agentic Reasoning with Image-grounded Self-Evaluation for Interpretable IBD Assessment
Pronoma Banerjee, Anuva Shah, Jason Wu, Md. Masudur Rahman, Sanjay Mohanty, Satya Kurada, Juan P. Wachs
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Inflammatory bowel disease (IBD) requires frequent imaging-based assessment, yet interpretation of modalities such as wireless capsule endoscopy (WCE) and intestinal ultrasound remains heavily dependent on specialist expertise. Vision-Language Models (VLMs) demonstrate significant potential in multimodal medical image analysis, but their clinical adoption is hindered by their insufficient domain-specific reasoning, susceptibility to hallucination, scarcity of high quality training data in fine-grained diagnostics and limited interpretability. We introduce ARISE (Adaptive Agentic Reasoning with Image-grounded Self-Evaluation), an autonomous planning framework that models few-shot medical image understanding as a sequential agentic workflow. ARISE structures agent execution into a transparent 5-stage workflow: hypothesis generation, image-grounded evidence summarization, evidence-conditioned refinement, symbolic verification, and final diagnosis. We apply ARISE to IBD assessment across two independent patient cohorts: wireless capsule endoscopy (WCE) images for Crohn's disease and B-mode ultrasound data for ulcerative colitis. ARISE consistently improves diagnostic performance over baseline VLMs while exposing where reasoning succeeds or fails, providing a more interpretable basis for clinical decision support and realistic deployment.
Tags
Links
- Source: https://arxiv.org/abs/2610.04777v1
- Canonical: https://arxiv.org/abs/2610.04777v1
Trouble viewing inline? Open PDF directly →
Full Text
49,218 characters extracted from source content.
Expand or collapse full text
ARISE: Adaptive Agentic Reasoning with Image-grounded Self-Evaluation for Interpretable IBD Assessment Pronoma Banerjee Anuva Shah Jason Wu Md. Masudur Rahman Sanjay Mohanty Satya Kurada Juan P. Wachs Abstract Inflammatory bowel disease (IBD) requires frequent imaging-based assessment, yet interpretation of modalities such as wireless capsule endoscopy (WCE) and intestinal ultrasound remains heavily dependent on specialist expertise. Vision-Language Models (VLMs) demonstrate significant potential in multimodal medical image analysis, but their clinical adoption is hindered by their insufficient domain-specific reasoning, susceptibility to hallucination, scarcity of good quality training data in fine-grained diagnostics and limited interpretability. We introduce ARISE (Adaptive Agentic Reasoning with Image-grounded Self-Evaluation), an autonomous planning framework that models few-shot medical image understanding as a sequential agentic workflow. ARISE structures agent execution into a transparent 5-stage workflow: hypothesis generation, image-grounded evidence summarization, evidence-conditioned refinement, symbolic verification, and final diagnosis. We apply ARISE to IBD assessment across two independent patient cohorts: wireless capsule endoscopy (WCE) images for Crohn’s disease and B-Mode ultrasound data for Ulcerative Colitis. ARISE consistently improves diagnostic performance over baseline VLMs while exposing where reasoning succeeds or fails, providing a more interpretable basis for clinical decision support and realistic deployment. 1 Purdue University, West Lafayette, IN 47907, USA 2 Indiana University School of Medicine, Indianapolis, IN 46202, USA jpwachs@purdue.edu Introduction Inflammatory bowel disease (IBD), including Crohn’s disease (CD) and ulcerative colitis (UC), is a chronic relapsing condition that requires repeated assessment of disease activity to prevent progressive bowel damage and other complications (Ng and others 2017; Turner and others 2021; Torres et al. 2017). Two imaging modalities are central to this monitoring workflow. Wireless capsule endoscopy (WCE) (Iddan et al. 2000) provides direct visualization of small-bowel mucosal abnormalities in Crohn’s disease, while intestinal ultrasound (IUS) (Maaser and others 2020) offers a radiation-free means of assessing bowel-wall features such as colon wall thickening and loss of stratification, both established markers of activity in ulcerative colitis (Bots et al. 2018). Despite their clinical value, both modalities share a critical bottleneck: accurate interpretation depends heavily on scarce, subspecialty-trained experts (Vallée and others 2020; Leenings et al. 2025). In resource-limited and non-tertiary settings, this dependence delays diagnosis and monitoring precisely where timely intervention matters most (Kaplan and Windsor 2021). Vision-language models (VLMs) offer a promising way to support medical image interpretation without task-specific training (Radford and others 2021; Alayrac and others 2022; Liu et al. 2023; Achiam and others 2023). Yet their translation into IBD assessment is hindered by four well-documented obstacles (Hager and others 2024). First, general-purpose VLMs exhibit a domain-reasoning gap: fine-grained IBD diagnosis requires integrating subtle, co-occurring visual cues over disease progression rather than recognizing salient objects, and these cues are underrepresented in general training distributions (Du et al. 2025; Zhang et al. 2025). Second, VLMs remain prone to hallucination and to internally contradictory rationales, an unacceptable failure mode in a clinical setting (Ji and others 2023; Li et al. 2025; Madaan and others 2023). Third, high-quality annotated IBD imaging data are scarce, ruling out conventional fine-tuning in most practical deployments (Shakeri and others 2024; Vallée and others 2020). Finally, even accurate VLMs act as black boxes, providing no way to distinguish sound visual reasoning from chance or spurious shortcut (Byun et al. 2025; Byun et al. 2026); inference-time methods such as test-time prompt tuning and retrieval-augmented in-context learning avoid retraining, but do not necessarily produce reasoning that can be inspected or verified(Shu and others 2022; Ram and others 2023; Zhan et al. 2026; Xia et al. 2025). Neurosymbolic methods that translate natural-language problems into symbolic form for solver-backed verification improve logical faithfulness, but they have been developed for text-only reasoning rather than image-grounded medical diagnosis, where hypotheses must be tied to observable visual findings (Pan et al. 2023; Olausson and others 2023; Garcez and Lamb 2023). Moreover, most medical-agent evaluations emphasize final-task accuracy, making it difficult to identify where failures arise within a multi-stage reasoning process (Liu and others 2024; Fu et al. 2025; Hager and others 2024). As recent workflow-aware benchmarks have argued, final-output metrics alone cannot reveal where or why an agent fails (Schmidgall et al. 2024; Jiang and others 2025). In short, no existing few-shot adaptation approach produces a reasoning state that is simultaneously grounded in the image evidence, verified for internal consistency, and open to expert inspection before deployment. We introduce ARISE (Adaptive Agentic Reasoning with Image-grounded Self-Evaluation), an autonomous framework that adapts a frozen VLM to few-shot IBD assessment by constructing, grounding, verifying, and refining an explicit set of diagnostic hypotheses, and that evaluates both its intermediate reasoning stages and its final diagnosis. ARISE follows five stages: (S1) generation of diagnostic hypotheses from task and clinical knowledge, (S2) image-grounded summarization of findings from a small support set, (S3) evidence-conditioned refinement of those hypotheses, (S4) symbolic verification and repair using first-order logic and an SMT solver, and (S5) final diagnosis of unseen images. This design makes transparency and correctness mutually reinforcing, and constrains hallucination without fine-tuning. Each stage produces an inspectable artifact and an associated score, allowing ARISE to evaluate both the quality of its own reasoning process and the accuracy of the resulting diagnosis evaluated in two settings: Wireless Capsule Endoscopy (WCE) for Crohn’s disease and B-mode ultrasound for ulcerative colitis. Our contributions are fourfold: First, we introduce a few-shot adaptation strategy in which domain knowledge is represented as image-grounded, iteratively refined hypotheses rather than learned parameters. Second, we extend solver-backed neurosymbolic reasoning to image-grounded medical diagnosis by verifying and repairing VLM-generated hypotheses before prediction. Third, we introduce stage-level evaluation that separates agentic process quality from task performance, revealing where reasoning failures occur rather than reducing evaluation to a final label. Finally, we demonstrate the framework across independent WCE (Vallée and others 2020) and B-mode ultrasound (Leenings et al. 2025) cohorts and multiple VLM families, showing that structured, verifiable reasoning can improve diagnostic performance across modalities and model backbones. Methodology Figure 1: An overview of the ARISE workflow for the Crohn’s disease WCE Pathological vs Non-Pathological image classification. The workflow starts with initial hypotheses generation and a parallel summarization of visual cues from select examples. The initial hypotheses are first updated using the summary of visual cues, followed by a logical consistency check via an SMT solver. The final hypotheses are used for the final diagnosis stage. We describe ARISE (Adaptive Agentic Reasoning with Image-grounded Self-Evaluation), an autonomous agent that adapts a frozen vision-language model (VLM) to few-shot inflammatory bowel disease (IBD) assessment by executing a transparent, five-stage reasoning workflow. Unlike a single opaque prompt, ARISE carries an explicit workflow state across stages, invokes an external symbolic tool, and emits an inspectable artifact at every stage that receives its own score. No VLM parameters are updated: adaptation is stored entirely as an auditable hypothesis state. This eliminates the dependency on massive datasets, bypassing the conventional fine-tuning bottleneck that typically hinders medical AI deployment. We summarize a run as an agentic (process) score over the five workflow stages plus a task (outcome) score, and combine them into a single overall score. Problem Formulation Let =(xi,yi)i=1ND=\(x_i,y_i)\_i=1^N be an IBD imaging dataset, where xix_i is an input image (a wireless capsule endoscopy (WCE) frame for the Crohn’s cohort or a B-mode ultrasound frame for the ulcerative colitis (UC) cohort) and yi∈y_i is a diagnostic label. We operate in the few-shot regime: a small labeled support set =(xj,yj)j=1mS=\(x_j,y_j)\_j=1^m is available for adaptation, and a disjoint query set Q is held out. Each task carries a task description T (modality, label space Y, acquisition context, requested output) and a clinical cue description C enumerating expert knowledge about the visual findings associated with each label. The ARISE agent is the tuple (Mθ,π,V,Γ)(M_θ,π,V, ): MθM_θ is the frozen VLM; π is a deterministic controller that sequences the stages and triggers repair loops; V is a symbolic satisfiability-modulo-theories (SMT) verifier (Z3); and Γ is a prompt-based parser that maps each natural-language hypothesis to a typed first-order-logic (FOL) premise. The agent maintains a state σ(t)=(H(t),Φ(t),ℰ(t),(t)),σ^(t)= (H^(t),\ ^(t),\ E^(t),\ z^(t) ), (1) comprising the hypothesis set H(t)H^(t), its premises Φ(t)=Γ(H(t)) ^(t)= (H^(t)), an evidence memory ℰ(t)E^(t), and the vector of stage scores (t)=(s1,…,s5)z^(t)=(s_1,…,s_5). The exact scores are described in details in Appendix A. The goal is to construct a verified hypothesis state H∗H that yields an accurate query prediction y^=ftask(x,H∗,ℰ∗,Mθ) y=f_task(x,H ,E ,M_θ) while producing an inspectable, individually-scored reasoning trace. Stage S1: Hypothesis Generation ARISE composes a planning prompt from task and clinical knowledge, p0=Plan(T,C,)p_0=Plan(T,C,Y), and the VLM generates concise, checkable hypotheses H(0)=h1,…,hK=Mθ(p0,τ,ptop),H^(0)=\h_1,…,h_K\=M_θ(p_0;\ τ,p_top), (2) where the sampling temperature τ and nucleus sampling parameter ptopp_top can be varied to encourage hypothesis diversity. For example, in the WCE six-lesion task, p0p_0 may ask: “Given Crohn’s disease capsule endoscopy images and the label set AU,E,O,S,U3−10,U>10\AU,E,O,S,U3-10,U>10\, generate concise visual hypotheses linking observable mucosal findings to each label. Consider mucosal breaks, white ulcer base, red halo, diffuse redness, enlarged villi, and luminal narrowing. Avoid claims that cannot be checked from the image.” Stage S2: Image-Grounded Evidence Summarization ARISE initializes an empty evidence memory ℰ(0)=∅E^(0)= and picks one example image at a time, sequentially. At step t it selects (xt,yt)(x_t,y_t) and asks the VLM to report only the observable evidence in xtx_t supporting yty_t under the current hypotheses: et=Mθ(Observe(xt,yt,p0)).e_t=M_θ (Observe(x_t,y_t,p_0) ). (3) The output is structured: ete_t tags each cited finding against a per-cohort controlled vocabulary V (for endoscopy, this will be mucosal break, white ulcer base, red halo, diffuse erythema, enlarged villi, luminal narrowing, etc.). This makes scoring a set comparison rather than text parsing, which is simpler to evaluate. The generated visual summary is appended as ℰ(t)=ℰ(t−1)∪etE^(t)=E^(t-1)∪\e_t\. The final evidence set is denoted as ℰ∗E^*. The grounding score s2s_2 measures agreement between model-identified findings and expert-annotated findings, emphasizing visual fidelity over unsupported inference. Stage S3: Evidence-Conditioned Refinement Given the accumulated evidence, the agent proposes a refined set H(ℰ∗)=Mθ(Refine(p0,H(0),ℰ∗))H^(E^*)=M_θ(Refine(p_0,H^(0),E^*)). The refinement score s3s_3 measures whether the refined hypothesis set improves downstream prediction performance relative to the initial hypotheses on a held-out validation subset. Figure 2: Example of stages S2 and S3 on the Crohn’s disease Ulceration Diagnosis task with GPT-5.4 Stages S2 and S3 are jointly described in Figure 2. Stage S4: Symbolic Verification The generated hypotheses H(ℰ∗)H^(E^*) are converted from natural language to first order logic statements Φ using an LLM parser Γ , Φ=Γ(H(ℰ∗)) = (H^(E^*)). For example, the natural language statement ”A mucosal break with a white base indicates aphthoid ulceration” is translated to the form (MucosalBreak(x)∧WhiteBase(x))→AU(MucosalBreak(x) (x)) . These are then passed into a Satisfiability Modulo Theories Solver. V(Φ)=1,if Φ is satisfiable,0,otherwise.V( )= cases1,&if is satisfiable,\\ 0,&otherwise. cases (4) If V(Φ)=0V( )=0, V returns the minimal conflicting set b and the LLM repairs the hypotheses, H∗=Mθ(Repair(p0,H(ℰ∗),b))H^*=M_θ(Repair(p_0,H^(E^*),b)), giving the final hypotheses set H∗H^* to be used for diagnosis. The verification score s4s_4 measures the fidelity of the natural-language-to-logic translation, evaluated through bidirectional entailment between the original and reconstructed hypotheses. This is done using a natural language inference model, details are provided in the Appendix A. Stage S5: Final Diagnosis With the verified state H∗H , the model performs a downstream classification task, classifying an input x x with a corresponding label y y. A few examples of pictures and labels are added for in-context learning. For each classification, we also output the hypothesis hh it uses for the classification. The stage score s5s_5 measures whether the prediction belongs to the valid label space and whether the model grounds its prediction in the verified hypothesis set. The task score for the entire agentic system is measured as the average of the classification accuracy, F1-score, precision, and recall on Q. Workflow-Aware Scoring: Agentic ++ Task The five workflow stages are aggregated into a single agentic (process) score, and the outcome is the task score: Agentic =∑k=15wksk,∑k=15wk=1, = _k=1^5w_k\,s_k, _k=1^5w_k=1, (5) Overall =12Agentic+12Task. = 12\, Agentic+ 12\, Task. (6) Weights emphasize the trust-critical stages, with grounding (s2s_2) assigned the highest weight and symbolic verification and refinement the second highest; our default is =(w1,…,w5)=(0.20,0.30,0.20,0.20,0.10)w=(w_1,…,w_5)=(0.20,0.30,0.20,0.20,0.10). Experiments Datasets and Tasks ARISE is evaluated across two IBD imaging domains: Crohn’s disease (CD) using wireless capsule endoscopy (WCE) and ulcerative colitis (UC) using B-mode intestinal ultrasound. Detailed dataset composition, support/few-shot examples/test splits, and task definitions are provided in Appendix B. For Crohn’s disease, we use the CrohnIPI dataset (Vallée and others 2020), a wireless capsule endoscopy (WCE) collection of 3,498 RGB images independently reviewed by multiple clinical experts. We evaluate three tasks of increasing diagnostic granularity: (1) Pathological vs non-pathological image classification, (2) ulceration diagnosis (ulceration vs. no ulceration), and (3) six-lesion classification for pathological images, with labels being aphthoid ulceration, stenosis, edema, erythema, ulceration between 33 and 1010 m, and ulceration >10>10 m. For Ulcerative Colitis, ultrasound data is taken from the C-TRUS dataset (Leenings et al. 2025), containing a total of 800 ultrasound images collected from 13 patients. We evaluate colon-wall stratification status, classifying whether bowel-wall layering is preserved or lost, a clinically relevant marker of intestinal inflammation. Models and Prompting We evaluate ARISE using instruction-following VLMs from three model families: GPT, Gemini, and Qwen. We test GPT-5.4 (Afridi et al. 2026), GPT-4o-mini (Achiam and others 2023), Gemini-2.5 (Comanici and others 2025), and Qwen2-VL (Wang et al. 2024), with Qwen2-VL serving as an open-weight VLM baseline. All models are evaluated on identical query splits. Automatic hypothesis generation and logical premise construction use a reasoning-oriented language model, while symbolic contradiction detection is performed using the Z3 SMT solver (De Moura and Bjørner 2008). We perform 2 major experiments. 1. We compare the diagnostic accuracy, F1 score, precision, and recall of the base VLM and the VLM with the final generated hypotheses (task score after stage 5). 2. We then compare the agentic scores for every stage across models, and the overall task score for every model, on the 6-lesion classification task. This would tell us which stages are strongest and weakest, and which models tend to outperform the others in both workflow and task-level performance. Crucially, these evaluations jointly establish the clinical value of verifiable reasoning over standard prompting, while opening the black box to reveal exactly how process-level reasoning quality drives final diagnostic accuracy. Results Diagnostic Performance: Base VLM vs. ARISE Table 1 compares base-model accuracy with the same VLMs operating within the ARISE workflow across the three Crohn’s disease WCE tasks. ARISE improves accuracy for every model, with the largest accuracy gain observed as the task complexity increases (6-lesion classification being the hardest) and for the weaker baseline models. Qwen2-VL and Gemini-2.5 close much of the gap to the stronger GPT baselines once wrapped in the ARISE workflow, suggesting that the framework’s gains come from structuring and verifying reasoning rather than from any one backbone’s raw visual capability. Detailed task-level results for this dataset, including accuracy, F1 score, precision, and recall for all model–task combinations, are provided in the Appendix. For the ulcerative-colitis cohort, Table 2 reports the corresponding accuracy, F1, precision, and recall for the B-mode ultrasound colon-wall-stratification task (preserved vs. lost). The ARISE framework results in a consistent improvement over all baseline models even when diagnosing from noisy B-mode ultrasound images, showing that the performance improvement generalizes across different tasks and modalities. Table 1: Base and ARISE classification accuracy across the three Crohn’s disease WCE tasks. Task 1: Pathological vs non-pathological image classification, Task 2: Ulceration diagnosis, Task 3: 6-lesion classification Model Task 1 Task 2 Task 3 GPT-5.4 (Base) 0.69 0.66 0.32 GPT-5.4 (ARISE) 0.88 0.83 0.79 GPT-4o-mini (Base) 0.61 0.61 0.29 GPT-4o-mini (ARISE) 0.88 0.81 0.74 Gemini-2.5 (Base) 0.66 0.33 0.37 Gemini-2.5 (ARISE) 0.85 0.71 0.58 Qwen2-VL (Base) 0.42 0.66 0.22 Qwen2-VL (ARISE) 0.83 0.74 0.64 Table 2: Colon wall stratification status classification Model Accuracy F1 Precision Recall GPT-5.4 (Base) 0.60 0.66 0.69 0.63 GPT-5.4 (ARISE) 0.87 0.78 0.83 0.74 GPT-4o-mini (Base) 0.59 0.58 0.58 0.47 GPT-4o-mini (ARISE) 0.77 0.75 0.76 0.74 Gemini-2.5 (Base) 0.26 0.37 0.63 0.22 Gemini-2.5 (ARISE) 0.71 0.78 0.76 0.72 Qwen2-VL (Base) 0.66 0.64 0.63 0.64 Qwen2-VL (ARISE) 0.74 0.74 0.75 0.76 Stage-Level Reasoning Quality Figure 3 reports the five individual stage scores (s1 through s5) for all four models on the six-lesion classification task, together with the mean across models at each stage. Hypothesis generation (S1) and symbolic verification (S4) are consistently strong across models, while evidence-guided refinement (S3) is the most challenging stage and exhibits the greatest model variation. The contrast between the two kinds of stages points to a broader pattern: agents are consistently better at rule-based steps than at visually-grounded refinement, where they must revisit the image evidence itself and use it to revise their own hypotheses rather than follow a fixed procedure. The strongest models also perform better during refinement and final diagnosis, indicating that visually grounded adaptation is a key factor separating stronger from weaker agents. Figure 3: Stage-wise agentic scores across models on the Crohn’s disease 6-lesion classification task for the different workflow stages: hypothesis generation (S1), visual grounding (S2), refinement (S3), symbolic verification (S4), and final diagnosis (S5). Rule-based stages are consistently strong, while evidence-guided refinement shows the greatest difficulty and model variation. Agentic and Task-level scores across models We examine how each model’s intermediate reasoning quality (the agentic score) relates to its final classification performance (the task score) on the six-lesion classification task. Figure 4 reports the overall, agentic, and task scores for all four models, ranked by overall score. Across models, stronger agentic performance generally corresponds to stronger diagnostic performance, but the relationship is not one-to-one. The rule-based stages, such as hypothesis generation and symbolic verification, are consistently strong across all models, whereas the largest differences emerge during evidence-guided refinement and final diagnosis, where the model must use visual evidence to revise and apply its hypotheses. Hence weaker models can maintain relatively high agentic scores by successfully completing structured stages such as hypothesis generation and verification, while still struggling to translate that reasoning into the correct final classification. The resulting process–outcome gap highlights the value of ARISE’s stage-level evaluation, which reveals where reasoning quality stops carrying through to diagnostic performance. Figure 4: Overall, agentic, and task scores for the 4 evaluated agents on the Crohn’s disease 6-lesion classification task. Qualitative Visual Grounding To illustrate the mechanism behind these quantitative gains, Figure 5 presents a representative B-mode ultrasound case from the colon-wall-stratification task with ground-truth label ”lost.” Queried directly, GPT-5.4 hedges between competing visual interpretations and concludes an incorrect diagnosis. Under the ARISE workflow, the same underlying model grounds its answer in concrete, vocabulary-anchored observations (”the bowel wall appears diffusely hypoechoic and homogeneous, with no separable alternating layers or consistent banding along the segment”) and correctly predicts ”lost.” This case is typical of the pattern underlying the quantitative improvements described earlier: ARISE’s evidence-summarization (S2) and symbolic-verification (S4) stages guide the model away from ambiguous, hedged visual claims and toward grounded and verifiable ones, which is where the framework’s accuracy gains originate. Figure 5: Qualitative example of improvement of visual grounding with ARISE. The base VLM produces ambiguous visual cues, ultimately leading to the wrong classification. ARISE improves visual grounding and ultimately yields the correct classification. Path to Deployment Figure 6: Illustration of an AI-enabled decision-support interface for explainable IBD assessment using intestinal ultrasound, designed to make model outputs accessible and interpretable to nurses. ARISE is designed as a clinician-in-the-loop decision-support system rather than an autonomous diagnostic tool, emphasizing interpretability, error detection, and compatibility with existing IBD imaging workflows. The current framework has been evaluated across two complementary IBD imaging settings: 3,498 wireless capsule endoscopy (WCE) images from the publicly available CrohnIPI dataset (Vallée and others 2020) and B-mode intestinal ultrasound data obtained from the C-TRUS dataset (Leenings et al. 2025) and annotated by an expert at the Indiana University (IU) School of Medicine. Clinical experts informed the visual cues, decision boundaries, and interpretation guidelines used by ARISE. Unlike black-box deep-learning based models, ARISE exposes the visual findings, diagnostic hypotheses, symbolic verification results, refinement steps, and final recommendation, allowing clinicians to inspect or override reasoning before acting on the output. The next deployment stage is retrospective clinical validation using anonymized IU School of Medicine data from approximately 40 adults with Crohn’s disease, collected by a gastrointestinal ultrasound expert certified by the International Bowel Ultrasound Group (IBUS), an international organization supporting standardized training, research, and clinical adoption of intestinal ultrasound for IBD care. Expert annotations of bowel-wall stratification and clinical assessments will serve as reference standards for evaluating ARISE predictions. Stage-level errors will be analyzed across disease severity, patient variation, image quality, and acquisition hardware. For patients with anonymized, clinically indicated endoscopy data, ARISE will also estimate the Simple Endoscopic Score for Crohn’s Disease (SES-CD) (Daperno et al. 2004), from AI-identified ulcer size, ulcerated surface, and luminal narrowing across the terminal ileum and four colonic segments. This will be compared to the SES-CD score assessments by experts. Following retrospective validation, ARISE will be incorporated into a software-based clinical interface capable of processing endoscopy and B-mode ultrasound data streams. In an initial shadow-mode study, ARISE outputs will be compared with specialist interpretation without influencing patient care. A subsequent clinician-in-the-loop pilot will allow specialists to accept, question, or override intermediate reasoning and final recommendations, providing direct evidence of where the system’s explanations are clinically useful or insufficient. Finally, nurses and generalist clinicians will participate in a small case study evaluating whether ARISE’s visual-cue explanations improve non-expert interpretation and diagnostic accuracy relative to unaided assessment (Figure 6). Successful prospective evaluation would support testing ARISE across multiple hospitals and integrating it with existing clinical imaging and reporting systems. This staged progression, from retrospective validation, to silent real-time testing, to clinician-assisted evaluation and multi-site deployment, provides a practical path toward safe, interpretable use of ARISE in IBD assessment. Conclusion We presented ARISE, a five-stage agentic framework that adapts a frozen vision-language model to few-shot IBD assessment by generating, grounding, verifying, and refining diagnostic hypotheses without any fine-tuning. Across two independent cohorts (WCE for Crohn’s disease and B-mode ultrasound for ulcerative colitis) and three VLM families, ARISE improved accuracy, F1, precision, and recall over baselines on every model–task combination, with the largest gains on the hardest task and weakest baseline model, while its stage-level scoring exposed cases where strong intermediate reasoning did not fully carry through to final accuracy, highlighting a gap a single outcome metric would hide. Beyond raw accuracy, ARISE’s key contribution is auditability: every reasoning stage is stored as an inspectable artifact rather than collapsed into an opaque prediction, directly addressing the domain-reasoning gap, hallucination risk, data scarcity, and black-box behavior limiting clinical VLM adoption. Future work extends this toward improving the model with the agentic scores and deployment in real-world clinical settings. References Achiam et al. (2023) J. Achiam et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: Introduction, Models and Prompting. Afridi et al. (2026) H. Afridi, H. Ullah, S. D. Khan, and M. Ullah From gpt-3 to gpt-5: mapping their capabilities, scope, limitations, and consequences. arXiv preprint arXiv:2604.10332. Cited by: Models and Prompting. Alayrac et al. (2022) J. Alayrac et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35, p. 23716–23736. Cited by: Introduction. Bots et al. (2018) S. Bots, K. Nylund, M. Löwenberg, K. Gecse, O. H. Gilja, and G. D’Haens Ultrasound for assessing disease activity in ibd patients: a systematic review of activity scores. Journal of Crohn’s and Colitis 12 (8), p. 920–929. Cited by: Introduction. Byun et al. (2025) J. Y. Byun, Y. Park, N. Azizan, and R. Chellappa Test-time-scaling for zero-shot diagnosis with visual-language reasoning. arXiv preprint arXiv:2506.11166. Cited by: Introduction. Byun et al. (2026) J. Y. Byun, Y. Park, J. Corbeil, and A. B. Abacha Overconfidence and calibration in medical vqa: empirical findings and hallucination-aware mitigation. arXiv preprint arXiv:2604.02543. Cited by: Introduction. Comanici et al. (2025) G. Comanici et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv. External Links: 2507.06261, Link Cited by: Models and Prompting. Daperno et al. (2004) M. Daperno, G. D’Haens, G. Van Assche, F. Baert, P. Bulois, V. Maunoury, R. Sostegni, R. Rocca, A. Pera, A. Gevers, J. Mary, J. Colombel, and P. Rutgeerts Development and validation of a new, simplified endoscopic activity score for crohn’s disease: the ses-cd. Gastrointestinal Endoscopy 60 (4), p. 505–512. External Links: ISSN 0016-5107, Document, Link Cited by: Path to Deployment. De Moura and Bjørner (2008) L. De Moura and N. Bjørner Z3: an efficient smt solver. In International conference on Tools and Algorithms for the Construction and Analysis of Systems, p. 337–340. Cited by: Models and Prompting. Du et al. (2025) Y. Du, N. Yu, and S. Wang Medical knowledge intervention prompt tuning for medical image classification. IEEE Transactions on Medical Imaging 44 (12), p. 4945–4959. External Links: Document Cited by: Introduction. Fu et al. (2025) C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, Y. Wu, R. Ji, C. Shan, and R. He MME: a comprehensive evaluation benchmark for multimodal large language models. In Advances in Neural Information Processing Systems, Vol. 38. Note: Datasets and Benchmarks Track Cited by: Introduction. Garcez and Lamb (2023) A. d. Garcez and L. C. Lamb Neurosymbolic ai: the 3rd wave. Artificial Intelligence Review 56 (11), p. 12387–12406. Cited by: Introduction. Hager et al. (2024) P. Hager et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature medicine 30 (9), p. 2613–2622. Cited by: Introduction, Introduction. He et al. (2021) P. He, X. Liu, J. Gao, and W. Chen DeBERTa: decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations (ICLR), Cited by: Appendix A. Iddan et al. (2000) G. Iddan, G. Meron, A. Glukhovsky, and P. Swain Wireless capsule endoscopy. Nature 405 (6785), p. 417. Cited by: Introduction. Ji et al. (2023) Z. Ji et al. Survey of hallucination in natural language generation. ACM computing surveys 55 (12), p. 1–38. Cited by: Introduction. Jiang et al. (2025) Y. Jiang et al. MedAgentBench: a virtual ehr environment to benchmark medical llm agents. NEJM AI 2 (9), p. AIdbp2500144. Cited by: Introduction. Kaplan and Windsor (2021) G. G. Kaplan and J. W. Windsor The four epidemiological stages in the global evolution of inflammatory bowel disease. Nature Reviews Gastroenterology & Hepatology 18 (1), p. 56–66. Cited by: Introduction. Leenings et al. (2025) R. Leenings, M. Konowski, N. R. Winter, J. Ernsting, L. Fisch, C. Barkhau, U. Dannlowski, A. Lügering, X. Jiang, and T. Hahn C-trus: a novel dataset and initial benchmark for colon wall segmentation in transabdominal ultrasound. In Simplifying Medical Ultrasound, p. 101–111. Cited by: Ulcerative colitis (B-Mode Ultrasound), Introduction, Introduction, Datasets and Tasks, Path to Deployment. Li et al. (2025) L. Li, J. Qu, L. Song, Y. Zhou, Y. Qin, T. Yang, and Y. Zhao Treble counterfactual vlms: a causal approach to hallucination. In Findings of the Association for Computational Linguistics: EMNLP 2025, p. 18423–18434. External Links: Document Cited by: Introduction. Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning. Advances in neural information processing systems 36, p. 34892–34916. Cited by: Introduction. Liu et al. (2024) Y. Liu et al. Mmbench: is your multi-modal model an all-around player?. In European conference on computer vision, p. 216–233. Cited by: Introduction. Maaser et al. (2020) C. Maaser et al. Intestinal ultrasound for monitoring therapeutic response in patients with ulcerative colitis: results from the trust&uc study. Gut 69 (9), p. 1629–1636. Cited by: Introduction. Madaan et al. (2023) A. Madaan et al. Self-refine: iterative refinement with self-feedback. Advances in neural information processing systems 36, p. 46534–46594. Cited by: Introduction. Ng et al. (2017) S. C. Ng et al. Worldwide incidence and prevalence of inflammatory bowel disease in the 21st century: a systematic review of population-based studies. The Lancet 390 (10114), p. 2769–2778. Cited by: Introduction. Olausson et al. (2023) T. Olausson et al. LINC: a neurosymbolic approach for logical reasoning by combining language models with first-order logic provers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 5153–5176. Cited by: Introduction. Pan et al. (2023) L. Pan, A. Albalak, X. Wang, and W. Wang Logic-lm: empowering large language models with symbolic solvers for faithful logical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 3806–3824. Cited by: Introduction. Radford et al. (2021) A. Radford et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: Introduction. Ram et al. (2023) O. Ram et al. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics 11, p. 1316–1331. Cited by: Introduction. Schmidgall et al. (2024) S. Schmidgall, R. Ziaei, C. Harris, E. Reis, J. Jopling, and M. Moor AgentClinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments. arXiv preprint arXiv:2405.07960. Cited by: Introduction. Shakeri et al. (2024) F. Shakeri et al. Few-shot adaptation of medical vision-language models. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 553–563. Cited by: Introduction. Shu et al. (2022) M. Shu et al. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems 35, p. 14274–14289. Cited by: Introduction. Torres et al. (2017) J. Torres, S. Mehandru, J. Colombel, and L. Peyrin-Biroulet Crohn’s disease. The Lancet 389 (10080), p. 1741–1755. Cited by: Introduction. Turner et al. (2021) D. Turner et al. STRIDE-i: an update on the selecting therapeutic targets in inflammatory bowel disease (stride) initiative of the international organization for the study of ibd (ioibd). Gastroenterology 160 (5), p. 1570–1583. Cited by: Introduction. Vallée et al. (2020) R. Vallée et al. CrohnIPI: an endoscopic image database for the evaluation of automatic crohn’s disease lesions recognition algorithms. In Medical Imaging 2020: Biomedical Applications in Molecular, Structural, and Functional Imaging, Vol. 11317, p. 440–446. Cited by: Crohn’s disease (Wireless Capsule Endoscopy), Introduction, Introduction, Introduction, Datasets and Tasks, Path to Deployment. Wang et al. (2024) P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. External Links: 2409.12191, Link Cited by: Models and Prompting. Xia et al. (2025) P. Xia, K. Zhu, H. Li, T. Wang, W. Shi, S. Wang, L. Zhang, J. Zou, and H. Yao MMed-rag: versatile multimodal rag system for medical vision language models. In International Conference on Learning Representations, Cited by: Introduction. Zhan et al. (2026) Z. Zhan, S. Zhou, X. Zhou, Y. Xiao, J. Wang, J. Deng, H. Zhu, Y. Hou, Y. Song, M. Lin, and R. Zhang Retrieval-augmented in-context learning for multimodal large language models in disease classification. Journal of Biomedical Informatics 178, p. 105017. External Links: Document Cited by: Introduction. Zhang et al. (2025) J. Zhang, G. Wang, M. K. Kalra, and P. Yan Disease-informed adaptation of vision-language models. IEEE Transactions on Medical Imaging 44 (10), p. 3984–3996. External Links: Document Cited by: Introduction. Appendix A A. Stage Scoring Definitions and Equations Notation. Let Y denote the task label space, H(0)H^(0) the initial hypothesis set, ℰ∗E^* the accumulated image-grounded evidence, H∗H^* the final verified hypothesis set, V the controlled clinical vocabulary, Γ the natural-language-to-first-order-logic parser, and Φ=Γ(H) = (H) the corresponding logical premises. y y denotes the predicted label and h h the hypothesis cited in support of that prediction. Score s1s_1. S1 is scored by a three-item rubric averaged to [0,1][0,1]: s1 s_1 =12([H(0)≠∅]⏟non-empty+∑y∈[∃h∈H(0),x→hy]||⏟per-label coverage) = 12 ( 1\! [H^(0)≠ ]_non-empty+ _y 1 [∃ h∈ H^(0),x→^hy\ ]|Y|_per-label coverage ) (A1) where the first term indicates that the hypotheses set generated has at least 1 hypothesis, and the 2nd term indicates that there is at least one hypothesis per label. The best agentic score (s1=1s_1=1) is obtained when both conditions hold. Score s2s_2. Let ℱ(et)F(e_t) be the findings cited by the agent and ℱ⋆(xt)F (x_t) the label-relevant findings marked present by expert annotation of the (few) support images. We report precision and recall separately and combine them, weighting fidelity above completeness since a fabricated finding is the clinical dealbreaker: Prec =|ℱ(et)∩ℱ⋆(xt)||ℱ(et)|,Rec=|ℱ(et)∩ℱ⋆(xt)||ℱ⋆(xt)|, = |F(e_t) (x_t)||F(e_t)|, = |F(e_t) (x_t)||F (x_t)|, (A2) Obs =|c∈et:c∈||et|, = |\c∈ e_t:\,c \||e_t|, (A3) s2 s_2 =0.5Prec+0.4Rec+0.1Obs =0.5Prec+0.4Rec+0.1Obs (A4) where ObsObs (observability) is the fraction of claims referencing a concrete vocabulary finding rather than an abstract inference. Score s3s_3. Refinement is scored by the improvement in prediction accuracy of downstream task before and after the refinement step, when compared against the ground truth label, for a smaller validation set of 50 observations. s3=AccH(ℰ∗)−AccH(0) s_3=Acc_H^(E^*)-Acc_H^(0) (A5) Score s4s_4. We score the translation fidelity of the LLM parser using a Natural Language Inference (NLI) model (DeBERTa-v3-large) (He et al. 2021). First, we compute the natural language hypothesis back from the first order logic: H∗~=Γ−1(Φ) H^*= ^-1( ) (A6) We then test whether H∗H^* and H∗ H^* mutually entail one another (each statement implies the other). The NLI gives us the entailment probability (probability of one statement implying the other) and we get the score as the min of 2 probabilities. s4=min[Pe(H∗→H~∗),Pe(H~∗→H∗)] s_4= [P_e\! (H^*→ H^* ),P_e\! ( H^*→ H^* ) ] (A7) This ensures that the 2 statements are equivalent and the first order logic statement was correct. S5: Final diagnosis. The process score checks whether the prediction lies in the permitted label space and whether the model cites a hypothesis from the verified set as support: s5=12[(y^∈)+(h^∈H∗)].s_5= 12 [1( y )+1( h∈ H^*) ]. (A8) Task score. Diagnostic outcome quality is measured separately from process quality as the mean of accuracy, F1 score, precision, and recall on the held-out query set: TASK=14[Acc+F1+Prec+Rec].TASK= 14 [Acc+F1+Prec+Rec ]. (A9) Workflow-level aggregation. The five stage scores are combined using the weights reported in the main manuscript, with the greatest weight assigned to image grounding: AGENTIC =∑k=15wksk, = _k=1^5w_ks_k, (A10) OVERALL =12AGENTIC+12TASK. = 12AGENTIC+ 12TASK. (A11) where (w1,…,w5)=(0.20,0.30,0.20,0.20,0.10)(w_1,…,w_5)=(0.20,0.30,0.20,0.20,0.10) B. Datasets and experimental protocol Crohn’s disease (Wireless Capsule Endoscopy) The CrohnIPI dataset (Vallée and others 2020) contains 3,498 expert-reviewed WCE RGB images. The manuscript reports 60.55% normal and 38.85% pathological images. Pathological images include erythema, edema, aphthoid ulceration, ulceration between 3 and 10m, ulceration greater than 10m, and stenosis; some images contain co-occurring labels. Ulcerative colitis (B-Mode Ultrasound) The C-TRUS dataset (Leenings et al. 2025) contains 800 B-mode ultrasound images from 13 patients. The study filters 275 good-quality images for the colon-wall stratification experiment. These raw datapoints were annotated for colon-wall stratification classification, and segmentation of the colon wall, by an expert at IU School of Medicine. The datasets, tasks and number of support and in-context examples and number of test samples are mentioned in Table A1 Table A1: Dataset and task splits used in the experiments. AU: aphthoid ulceration; S: stenosis; O: edema; E: erythema; U3-10: ulceration between 3 and 10 m; U>>10: ulceration >10>10 m. Task Modality Output labels Support ICL Test Normal vs. pathological classification WCE Normal; pathological 7 18 1434 Six-lesion classification WCE AU; S; O; E; U3–10; U>>10 7 12 535 Ulceration diagnosis WCE Ulceration; no ulceration 7 12 535 Colon-wall stratification status B-mode ultrasound Preserved; lost 4 10 211 Experimental Protocol 1. Base VLM evaluation: Each backbone is first evaluated in a query-only setting on the held-out test split. 2. ARISE evaluation: The same backbone is evaluated on the same query split after the five-stage ARISE workflow constructs the verified hypothesis state H∗H^*. 3. Task-level comparison: Accuracy, F1 score, precision, and recall are reported for Base and ARISE on all four clinical tasks. 4. Stage-level analysis: Scores S1–S5 are compared across the four VLMs on the Crohn’s disease six-lesion classification task. S3 is computed on a separate 50-example validation subset. 5. Process vs. outcome analysis: The weighted AGENTIC score is compared with the TASK score to identify cases in which a well-formed reasoning process does not fully translate to correct final classification. Appendix B C. Results for Crohn’s disease Tables A2, A3 and A4 below report the full task-level metrics for each backbone in the query-only Base setting and within ARISE. Table A2: Pathological vs Non-Pathological WCE image Model Accuracy F1 Precision Recall GPT-5.4 (Base) 0.69 0.65 0.59 0.72 GPT-5.4 (ARISE) 0.88 0.86 0.85 0.89 GPT-4o-mini (Base) ) 0.61 0.73 0.60 0.77 GPT-4o-mini (ARISE) 0.88 0.88 0.89 0.88 Gemini-2.5 (Base) 0.66 0.59 0.77 0.62 Gemini-2.5 (ARISE) 0.85 0.78 0.82 0.76 Qwen2-VL (Base) 0.42 0.29 0.21 0.50 Qwen2-VL (ARISE) 0.83 0.83 0.88 0.83 Table A3: WCE - Diagnosing ulceration Model Accuracy F1 Precision Recall GPT-5.4 (Base) 0.66 0.76 0.79 0.73 GPT-5.4 (ARISE) 0.83 0.78 0.83 0.74 GPT-4o-mini (Base) 0.61 0.67 0.86 0.59 GPT-4o-mini (ARISE) 0.81 0.84 0.88 0.81 Gemini-2.5 (Base) 0.33 0.18 0.78 0.10 Gemini-2.5 (ARISE) 0.71 0.78 0.86 0.72 Qwen2-VL (Base) 0.66 0.77 0.76 0.79 Qwen2-VL (ARISE) 0.74 0.84 0.75 0.96 Table A4: 6 lesion classification from Crohn’s disease WCE Model Accuracy F1 Precision Recall GPT-5.4 (Base) 0.32 0.30 0.35 0.32 GPT-5.4 (ARISE) 0.79 0.76 0.81 0.75 GPT-4o-mini (Base) 0.29 0.34 0.32 0.26 GPT-4o-mini(ARISE) 0.74 0.75 0.77 0.74 Gemini-2.5 (Base) 0.37 0.35 0.47 0.37 Gemini-2.5(ARISE) 0.58 0.53 0.58 0.63 Qwen2-VL (Base) 0.22 0.11 0.07 0.22 Qwen2-VL (ARISE) 0.64 0.58 0.58 0.64 D. Prompt Templates and Implementation Prompt A: Normal vs. Pathological WCE Classification The binary prompt assigns each WCE frame to exactly one of two labels, N (normal/non-pathological) or P (pathological). The prompt defines pathological findings to include aphthoid ulceration, ulceration, erythema, edema, stenosis, focal mucosal breaks, depressed lesions, white/yellow ulcer bases, red halos, abnormal redness, swelling, or luminal narrowing. The decision rules instruct the model to choose N only when the mucosa appears continuous and no focal abnormality is visible, and to choose P whenever a definite abnormality is present. In the baseline setting, no refined hypotheses or in-context examples are provided. In the ARISE setting, the refined hypothesis list is inserted under a Supporting rules field and the model is asked to return the predicted class together with the ID of the supporting hypothesis when available. Prompt A template: You are a medical assistant specialized in wireless capsule endoscopy image classification. Classify each WCE image into exactly one binary label: N = normal-appearing / non-pathological frame. P = pathological / abnormal frame. Supporting rules: [refined hypotheses, ARISE setting only] Decision rules: choose N only for normal, continuous mucosa without focal abnormality; choose P if any visible abnormality is present; do not output a lesion subtype. Prompt B: Six-Lesion WCE Classification Prompt B is applied only to frames already considered pathological and assigns exactly one lesion label: aphthoid ulceration (AU), erythema (E), edema (O), stenosis (S), ulceration 3–10 m (U3-10), or ulceration >>10 m (U>10). The prompt includes a short description of each class and an ulcer-first decision hierarchy: visible mucosal break, depression, erosion, crater, or white/yellow ulcer base directs the model toward an ulcer subtype; intact-surface redness favors erythema; swollen/enlarged villi with a continuous surface favors edema; and dominant luminal narrowing favors stenosis. The baseline Prompt B uses a fixed set of short class-discrimination rules. In the ARISE setting, these fixed rules are replaced by the refined hypothesis state loaded from disk. Prompt B template: You are a medical assistant specialized in wireless capsule endoscopy image classification. This image is already considered pathological. Do not choose N. Classify the pathological subtype into exactly one of: AU, E, O, S, U3-10, U>10. Supporting rules: [fixed short rules for baseline; refined hypotheses for ARISE] Decision hierarchy: ulcerative surface loss → ulcer subtype; intact-surface redness → E; swollen/enlarged villi with continuous surface → O; dominant luminal narrowing → S. Support and In-Context Examples A set of support examples are selected for each task for Stage S2. Seprately, ARISE runs use manually selected labeled image examples as multimodal in-context demonstrations. The demonstrations contain the example image and its class label; the explanatory text stored with each example is used during the final diagnostic evaluation. For Prompt A, GPT and Gemini use four ICL examples, while Qwen2-VL-2B uses two. For Prompt B, the GPT notebook permits up to 12 examples, Gemini uses four, and Qwen2-VL-2B uses two. Qwen uses fewer examples to reduce GPU memory consumption on a Colab T4.