Paper deep dive
Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery
Jingbo Yue, Bruce Coburn, Jinge Ma, Jui-Feng Chi, Fengqing Zhu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/9/2026, 8:21:38 AM
Summary
The paper proposes a framework for improving image-based nutrition estimation by using multimodal large language models (MLLMs) to verify food identity and portion estimation geometry. The system performs an initial inventory, audits semantic and geometric evidence, and triggers a single targeted recovery pass for missing or unsupported items. This approach improves mass and energy accuracy compared to baselines without requiring task-specific fine-tuning.
Entities (10)
Relation Signals (7)
Multimodal Large Language Models → usedfor → Food-Item Verification
confidence 95% · We propose a framework that uses multimodal large language models (MLLMs) to inventory visible foods and separately verify food identity
Proposed Framework → improves → Energy Accuracy
confidence 94% · and energy accuracy relative to an adapted retrieval baseline
Proposed Framework → improves → Mass Accuracy
confidence 94% · item-level grounding improves mass accuracy across all tested settings
SAM 3 → usedfor → Food Segmentation
confidence 92% · For each normalized item t_i, SAM 3 receives a deterministic short phrase... each proposal is independently verified
Qwen3.5-35B-A3B → evaluatedon → ACETADA
confidence 90% · and 790 ACETADA (ACE) images... We evaluate... Qwen3.5-35B-A3B (Q35B)
gemma-4-26B-A4B → evaluatedon → Nutrition5k
confidence 90% · The main experiments use 1,658 Nutrition5k (N5k) images... We evaluate... Gemma 4 26B A4B (G26B)
DietAI24 → uses → FNDDS
confidence 88% · DietAI24 grounds meal descriptions to the Food and Nutrient Database for Dietary Studies (FNDDS)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Single-image nutrition estimation can fail silently when visible foods are missed. Even when a food is correctly identified, its proposed region may not support portion estimation. We propose a framework that uses multimodal large language models (MLLMs) to inventory visible foods and separately verify food identity and whether each proposed 2D region supports portion estimation. One whole-image review uses these verification results to identify unresolved gaps and omitted foods, triggering at most one targeted recovery pass. Recovered regions are re-verified without access to the recovery prompt, then reconciled into a final item set for nutrition estimation. The framework requires no task-specific fine-tuning. Matched evaluation on common valid-output samples shows that item-level grounding improves mass accuracy across all tested settings and energy accuracy relative to an adapted retrieval baseline, with item-identity precision and recall also improving, while post-recovery visual coverage is assessed separately at inference time without ground-truth annotations.
Tags
Links
- Source: https://arxiv.org/abs/2610.11144v1
- Canonical: https://arxiv.org/abs/2610.11144v1
Trouble viewing inline? Open PDF directly →
Full Text
28,615 characters extracted from source content.
Expand or collapse full text
IMPROVING IMAGE-BASED NUTRITION ESTIMATION THROUGH MULTIMODAL FOOD-ITEM VERIFICATION AND RECOVERY Jingbo Yue Bruce Coburn Jinge Ma Jui-Feng Chi Fengqing Zhu †thanks: Corresponding author: Fengqing Zhu (zhu0@purdue.edu). Abstract Single-image nutrition estimation can fail silently when visible foods are missed. Even when a food is correctly identified, its proposed region may not support portion estimation. We propose a framework that uses multimodal large language models (MLLMs) to inventory visible foods and separately verify food identity and whether each proposed 2D region supports portion estimation. One whole-image review uses these verification results to identify unresolved gaps and omitted foods, triggering at most one targeted recovery pass. Recovered regions are re-verified without access to the recovery prompt, then reconciled into a final item set for nutrition estimation. The framework requires no task-specific fine-tuning. Matched evaluation on common valid-output samples shows that item-level grounding improves mass accuracy across all tested settings and energy accuracy relative to an adapted retrieval baseline, with item-identity precision and recall also improving, while post-recovery visual coverage is assessed separately at inference time without ground-truth annotations. Index Terms: image-based nutrition estimation, multimodal large language models, food segmentation, visual grounding, portion estimation †address: Elmore Family School of Electrical and Computer Engineering, Purdue University West Lafayette, IN, USA yue53, coburn6, ma859, chi96, zhu0@purdue.edu 1 Introduction Meal images rarely present foods as isolated, well-separated objects. Touching foods, small side items, liquids with weak visual cues for portion estimation, and mixed dishes with components that are not independently portionable make single-view dietary assessment particularly challenging. Multimodal large language models (MLLMs) offer a natural interface for jointly interpreting food appearance and contextual cues. However, a single holistic pass can miss fine or contextually difficult visual evidence; guided visual search has shown benefits in visually crowded scenes [17]. In dietary assessment, missed visible foods can remain unaccounted for in downstream portion and nutrition estimation. Recent systems improve nutrition estimation by structuring intermediate reasoning or grounding meal descriptions to nutritional databases [8, 20]. However, for single-view 2D dietary images, downstream nutrition estimation requires two different types of evidence for each visible food: reliable semantic identity for database grounding and usable image-space evidence for portion reasoning. These requirements are not equivalent: a region may support food identity while being too partial for portion estimation, whereas a well-localized region may still have ambiguous food identity. Our framework explicitly audits both before downstream nutrition estimation. Our central design principle is that each segmentation proposal used to localize an inventory item is fallible evidence, not item truth. Each proposal is independently verified along these two evidence axes, and two explicit support checks identify initial food inventory items lacking either type of evidence. One whole-image review, conditioned on the verified evidence state, then addresses these gaps while also identifying visible foods omitted from the initial food inventory. Unsupported initial food hypotheses can be suppressed or left unresolved, while visually supported gaps receive at most one targeted recovery attempt. Recovery comprises one round of targeted re-segmentation and independent re-verification without recovery phrases or target provenance. The resulting evidence is reconciled deterministically into the final item set used for downstream food-code and portion estimation. Our contributions are twofold. First, we introduce a single-round item-grounded verification and recovery framework that audits semantic and geometric evidence before downstream estimation, without task-specific fine-tuning. Second, we establish a multi-level protocol spanning pre-reconciliation evidence, post-reconciliation identity quality, and matched downstream nutrition accuracy to test whether grounding improves nutrition estimates, with post-recovery visual coverage assessed separately without ground-truth labels. Figure 1: One-round item-grounded recovery. Verified initial and recovery evidence are re-audited and reconciled into F. Coverage assessment and nutrition estimation follow. The example shows fruit-salad geometry repair and meat-and-rice evidence recovery. 2 Related Work Dietary MLLMs and nutrition grounding. Recent dietary MLLM systems increasingly structure intermediate reasoning rather than directly estimating nutrition from an image. A nutritionist-inspired two-step approach separates food analysis from numerical estimation [8], while DietAI24 grounds meal descriptions to the Food and Nutrient Database for Dietary Studies (FNDDS) through retrieval before portion and nutrient inference [20]. DietDelta further performs food-item-level reasoning from paired before-and-after images to estimate consumed amounts [15]. Unlike these systems, we explicitly verify and, when necessary, recover the visual food-item representation before downstream grounding and portion inference. Active visual search and verification. Guided visual search can recover evidence missed by holistic perception [17], while DeepEyes, VLM-R3, and Woodpecker further explore active or post-hoc visual verification [22, 7, 21]. Dietary estimation differs because recovered evidence must support both food identity for database grounding and geometry for portion reasoning. We therefore use one whole-image re-inspection driven by explicit evidence gaps and hide recovery provenance before re-verification. Food segmentation and portion evidence. FoodSeg103 provides large-scale food segmentation supervision [18], while earlier dietary systems used recognition feedback to refine segmentation hypotheses [24, 23, 16]. We instead use open-vocabulary MLLM/SAM evidence to recover missing foods and reconcile food identity with geometry for portion estimation. Accurate food isolation is central to volume-oriented pipelines such as VolTex [1], while monocular approaches use 3D scaling or point-cloud reconstruction [14, 9]. Accordingly, segmentation serves as evidence for item-level verification and recovery within the downstream nutrition pipeline. 3 Methodology 3.1 Item proposals and evidence audit Fig. 1 shows our framework. Given a meal image I, an MLLM produces a structured initial food inventory. Each nutrition item must be semantically meaningful, independently localizable, and independently portionable; inseparable mixed foods remain atomic. Deterministic normalization preserves uncertainty and portion cues, assigns stable IDs, and flags parent/component overlap risks. Proposal generation and verification: For each normalized item tit_i, SAM 3 [3] receives a deterministic short phrase at confidence threshold 0.500.50; thereafter, scores serve only as provenance. Each proposal is shown to an MLLM verifier as a gray-background masked target with a 15%15\%-padded context crop. The verifier independently assesses food identity, inventory assignment, segmentation quality, target scope, and portion geometry; the SAM prompt does not determine assignment. Geometry denotes 2D image-space region evidence, not explicit depth or volume reconstruction. Item-level evidence audit: Proposal-level verification results are summarized into two item-level indicators: Ps(i) P_s(i) =[verified semantic support exists], =1[verified semantic support exists], (1) Pg(i) P_g(i) =[grouped geometry is provisionally ready], =1[grouped geometry is provisionally ready], (2) where [⋅]1[·] is the indicator function. Ps(i)=1P_s(i)=1 if at least one successfully verified proposal is assigned to item i; this indicates semantic support, not final semantic readiness. To determine Pg(i)P_g(i), we retain successfully verified, in-scope proposals with correct or partial segmentation and a resolved geometry type. At least one must have correct segmentation and portion-usable geometry. Any other proposal with mask A must overlap at least one such supporting mask B with IoU≥0.80≥ 0.80 or IoM=|A∩B|/min(|A|,|B|)≥0.90=|A∩ B|/ (|A|,|B|)≥ 0.90. For each A, let UAU_A be the union of supporting masks B satisfying either overlap threshold; we additionally require |A∩UA|/|A|≥0.90|A∩ U_A|/|A|≥ 0.90. The known-item gap set is Q=i:¬Ps(i)∨¬Pg(i). Q=\i: P_s(i) P_g(i)\. (3) Since Q is restricted to initial inventory items, entirely omitted foods require nomination by the whole-image residual review. 3.2 Residual review, recovery and reconciliation Every meal receives one whole-image residual review conditioned on the verified evidence state. It receives the unmodified image, initial food inventory, Q, and accepted-proposal summaries, but no nutrition-database information, ground truth, or proposal-source identity. For each known gap it may recover, suppress unsupported, or abstain unresolved, and it may nominate omitted visible foods. Residual review retains the original image: subtracting fallible proposal masks could irreversibly remove neighboring or overlapping visual evidence. Recovery lowers the SAM threshold from 0.500.50 to 0.100.10; verification and reconciliation criteria remain unchanged. Within one recovery round, each target is segmented once and re-verified without recovery provenance. After source-blind re-verification, verified initial and recovery proposals are combined and re-audited. Deterministic reconciliation merges evidence sharing the same identity, suppresses parent/component redundancy, and leaves ambiguous geometry unresolved. Mask count does not determine serving count. The resulting representation is F=fii=1K,fi=idi,si,Gi,risem,rigeo,Πi, F=\f_i\_i=1^K, f_i=\id_i,s_i,G_i,r_i sem,r_i geo, _i\, (4) where idiid_i, sis_i, GiG_i, and Πi _i denote ID, semantics, union geometry, and provenance; risemr_i sem, rigeor_i geo indicate semantic and geometric readiness. After F is frozen, a ground-truth-free whole-image MLLM assigns meal-level Coverage: C1 (visually complete), C2 (mostly covered), or C3 (needs manual review); AU denotes unavailable assessment. C1 does not imply full semantic and geometric readiness for every item. Coverage is read-only, uses no numerical threshold, and cannot modify F or trigger further recovery. C1+C2 forms the broader downstream visual cohort, while C1 denotes the visually complete subset used for the primary matched nutrition comparison. 3.3 Food-code grounding and nutrition A deterministic adapter maps each eligible Final-F item to a per-item query while preserving idiid_i and provenance Πi _i. Each item is independently grounded to FNDDS using the hybrid retrieval and constrained food-code selection in Sec. 4.1. The selected code is immutable; legal portions are obtained by exact-code lookup with no semantic retrieval or cross-code fallback and presented with item-local geometry and preserved portion cues. The portion model selects a legal portion and a positive multiplier or abstains. Abstention is terminal: no default weight is imputed, and missing grams for a required Final-F item mark the meal incomplete. Let Feff⊆F_ eff F contain items with a valid code, legal portion, and positive multiplier. For i∈Feffi∈ F_ eff, g^i=w(pi)mi g_i=w(p_i)m_i, y^i,q=g^ivci,q/100 y_i,q= g_iv_c_i,q/100, and Y^meal,q=∑i∈Feffy^i,q Y_meal,q= _i∈ F_ eff y_i,q, where cic_i, pip_i, w(pi)w(p_i), and mim_i denote the code, portion, portion weight, and multiplier; vci,qv_c_i,q is nutrient q per 100 g. Meal mass is ∑i∈Feffg^i _i∈ F_ eff g_i. 4 Experimental Setup 4.1 Datasets, models, and implementation The main experiments use 1,658 Nutrition5k (N5k) images obtained with the public DietAI24 preprocessing code [12, 20], retaining the full preprocessed pool rather than the 1,000-image evaluation subset reported in DietAI24, and 790 ACETADA (ACE) images [4], totaling 2,448 meals. Nutrition5k supplies fine-grained ingredient/component metadata, not a one-to-one visible-food item ontology [12]; a recent manual validation also documented omissions in its original labels [11]. Because our inventory unit is an independently localizable and portionable visible food item, we denote its recovery results N5k∗ and use the fixed 12,491 component references only as a normalization denominator, not as item-level ground truth. ACE uses 4,589 structured food-item references directly. Thus the ρ metrics are reference-normalized evidence-state rates instead of identity recall. The “plate only” sentinel is excluded from item and numerical evaluation, consistent with prior Nutrition5k curation [2]. We evaluate two representative open-weight MLLMs from distinct families, Gemma 4 26B A4B (G26B) [6] and Qwen3.5-35B-A3B (Q35B) [10], under the same FNDDS 2021–2023 grounding pipeline [13]. Dataset-agnostic prompts, verification contracts, and SAM thresholds are frozen before formal evaluation and shared across datasets and models, without tuning on evaluation annotations. Visual grounding receives no reference labels or item counts, FNDDS codes, or nutrient values. For food-code grounding, BGE-based dense retrieval (BAAI/bge-small-en-v1.5 [19]) and BM25 rankings are combined using weighted reciprocal-rank fusion [5]: RRF(c)=∑r∈ℛwr/(k+rankr(c))RRF(c)= _r w_r/(k+rank_r(c)), with k=60k=60 and equal dense/sparse family weights. Retrieval depth is 50 per branch. 4.2 Item-recovery evaluation Recovery is evaluated on two evidence states, independently for each model–dataset pair. Initial is the state after initial proposal verification and before residual review. Post-recovery combines the initial verified evidence with source-blind verified recovery evidence, before deterministic reconciliation. Existing initial inventory identities remain represented regardless of their support state; a novel residual target is added only when recovery yields a successfully verified novel-item assignment. The same semantic and grouped-geometry audits from Sec. 3.1 are then recomputed on the combined evidence. Let NrefN_ ref denote the fixed dataset normalization count, NrepN_ rep the number of represented item identities, NsN_s the number with Ps=1P_s=1, and NgN_g the number with Pg=1P_g=1. We report ρitem=NrepNref,ρsem=NsNref,ρgeo=NgNref. _ item= N_ repN_ ref, _ sem= N_sN_ ref, _ geo= N_gN_ ref. (5) For N5k∗, Nref=12,491N_ ref=12,491; for ACE, Nref=4,589N_ ref=4,589. These are reference-normalized representation and verifier-assessed support rates, not ground-truth identity or segmentation scores. Table 1: Pre-reconciliation recovery on N5k∗/ACE: reference-normalized rates (%); parentheses show changes from Initial (p). Model Evidence state ρitem _ item N5k∗/ACE ρsem _ sem N5k∗/ACE ρgeo _ geo N5k∗/ACE G26B Initial 35.38/93.01 27.08/77.86 26.50/76.14 Post-recovery (ours) 35.44/93.07 (+0.06/+0.07 p) 31.41/86.01 (+4.33/+8.15 p) 30.85/84.24 (+4.36/+8.11 p) Q35B Initial 35.80/90.98 27.70/80.67 27.52/80.58 Post-recovery (ours) 35.94/91.00 (+0.14/+0.02 p) 32.77/87.60 (+5.07/+6.93 p) 32.45/87.49 (+4.93/+6.91 p) Manual food-identity audit: Separately from pre-reconciliation recovery evaluation, we manually audit a fixed sample of 150 randomly selected, reference-screened ACETADA C1 meals per model, from G26B/Q35B C1 cohorts of 472/680 meals. Because N5k∗ component references do not map one-to-one to visible-food items (Sec. 4.1), identity precision and recall are reported only on ACE. The initial inventory and Final-F are evaluated on the same meals against the same reference foods under strict one-to-one semantic matching. We report micro-averaged precision and recall, Pmicro=∑nTPn∑n(TPn+FPn)P_ micro= _nTP_n _n(TP_n+FP_n) and Rmicro=∑nTPn∑n(TPn+FNn)R_ micro= _nTP_n _n(TP_n+FN_n). 4.3 Nutrition baselines and matched evaluation For controlled end-to-end comparison, we use Direct, a whole-image nutrition estimate from the same MLLM without item-level grounding, and an adapted DietAI24 reimplementation [20]. Following its meal-level retrieval design, adapted DietAI24 expands one holistic meal description into multiple queries and infers food codes from a shared FNDDS candidate pool; our method instead retrieves independently for each grounded Final-F item. All three methods use the same evaluated MLLM; adapted DietAI24 and ours also share embeddings and the FNDDS release. Together, the two baselines provide controlled whole-image and retrieval-grounded comparisons; methods with different input, training, or 3D-resource assumptions are discussed qualitatively. Because valid-output subsets differ across methods, the primary nutrition comparison for model m and dataset d uses the common C1 intersection S∩(m,d)=SDirect(m,d)∩SAdapted(m,d)∩SOurs,C1(m,d),S_∩^(m,d)=S_ Direct^(m,d)∩ S_ Adapted^(m,d)∩ S_ Ours,C1^(m,d), (6) where SOurs,C1(m,d)S_ Ours,C1^(m,d) contains complete-output C1 meals and the baseline sets contain valid outputs. Direct returns valid outputs for all meals, so adapted DietAI24 determines the intersection with our complete-output C1 set. C1+C2 provides broader-cohort context. We report meal-level mass and energy mean absolute error (MAE), MAEq=N−1∑j=1N|Y^j,q−Yj,q|MAE_q=N^-1 _j=1^N| Y_j,q-Y_j,q|, for q∈mass,energyq∈\mass,energy\. Fig. 2 uses Q35B/ACE C1 predictions without restriction to the baseline intersection. ℛfullR_ full contains meals with risem=rigeo=1r_i sem=r_i geo=1 for every i∈Feffi∈ F_ eff; ℛgapR_ gap contains the remainder. 5 Results and Discussion 5.1 Item-level recovery Table 1 compares the initial and post-recovery pre-reconciliation evidence states for G26B and Q35B. Across N5k∗ and ACE, recovery increases ρitem _ item by 0.020.02–0.140.14 percentage points (p), ρsem _ sem by 4.334.33–8.158.15 p, and ρgeo _ geo by 4.364.36–8.118.11 p. These gains mainly reflect repaired semantic and geometric support for represented items, with limited additions of verified omitted foods. Because N5k∗ uses component-level references, its absolute rates are not directly comparable with ACE; within-dataset changes are the relevant recovery signal. 5.2 Final-F identity quality Table 2 reports a manual audit of 150 ACETADA C1 meals per model using strict one-to-one semantic matching. Final-F raises micro P/R from 98.35/93.02%98.35/93.02\% to 98.62/94.34%98.62/94.34\% for G26B and from 90.54/87.19%90.54/87.19\% to 93.73/87.87%93.73/87.87\% for Q35B, improving both precision and recall. G26B begins near the precision ceiling (98.35%98.35\%). This audit evaluates the complete inventory-to-F transformation, whereas Table 1 measures pre-reconciliation evidence support instead of identity correctness. Table 2: Manual food-identity audit on 150 ACETADA C1 meals per model. Values are micro precision/recall (%); superscripts show changes from Initial (p). G26B Q35B Representation PmicroP_ micro RmicroR_ micro PmicroP_ micro RmicroR_ micro Initial inventory 98.35 93.02 90.54 87.19 Final-F (Ours) 98.62↑0.27 94.34↑1.32 93.73↑3.19 87.87↑0.68 5.3 Matched-sample nutrition accuracy Table 3 compares Direct, adapted DietAI24, and our method on matched C1 cohorts. Ours achieves the lowest mass MAE in all four model–dataset settings and lower energy MAE than adapted DietAI24 in all four, while the energy comparison with Direct is dataset-dependent. These are end-to-end results, not isolated recovery effects. C1 denotes visual completeness but does not guarantee nutrition accuracy. Across both datasets, the post-recovery C1 pools contain 1,351/2,448 meals (55.2%55.2\%) for G26B and 1,875/2,448 (76.6%76.6\%) for Q35B; the corresponding C1+C2 pools contain 2,262 (92.4%92.4\%) and 2,220 (90.7%90.7\%), respectively. The matched C1 intersections contain n∩=719n_∩=719 (G26B) / 904904 (Q35B) on N5k and 383383 (G26B) / 494494 (Q35B) on ACE. ACE C1+C2 meals route roughly twice as many effective Final-F items as N5k (5.15.1 vs. 2.52.5 per meal), consistent with the larger adapted DietAI24 gap on ACE, where a shared meal-level retrieval context must disambiguate more foods. Within Q35B/ACE C1 meals, Fig. 2 shows lower MAE for the fully semantic- and geometry-ready group ℛfullR_ full than for ℛgapR_ gap in both mass (143.8143.8 vs. 148.1148.1 g) and energy (164.4164.4 vs. 181.3181.3 kcal). Table 3: Meal-level MAE (mass g / energy kcal; lower is better). Direct, Adapted DietAI24, and Ours (C1) use the same matched C1 samples; Ours (C1+C2) reports our broader-cohort result. Model Direct Adapted DietAI24 Ours (C1) Ours (C1+C2) Nutrition5k (N5k) G26B 81.1/119.3 123.7/164.9 74.3/107.4 80.8/111.2 Q35B 114.3/138.8 150.8/189.1 74.2/96.9 77.1/97.1 ACETADA (ACE) G26B 217.8/168.1 368.7/250.6 140.5/177.1 159.4/180.4 Q35B 192.5/166.2 438.4/262.9 144.8/172.5 148.1/173.8 Figure 2: Q35B/ACETADA C1 mass and energy predictions without baseline-intersection filtering, grouped by ℛfullR_ full and ℛgapR_ gap. 5.4 Limitations and future work Visual completeness does not eliminate ambiguity in low-contrast adjacent foods or mixed dishes. Verification adds inference overhead. Motivated by the item-density and readiness patterns, future work will quantify this cost and explore adaptive recovery for unresolved items while preserving bounded inference for typical meals. 6 Conclusion We presented a food-item verification and recovery framework for image-based nutrition estimation without task-specific fine-tuning. Recovery primarily strengthens semantic and geometric support while changing represented-item rates by at most 0.140.14 percentage points. On matched C1 cohorts, our method achieves the lowest mass MAE across all four model–dataset settings and lower energy MAE than adapted DietAI24 in all four, while energy gains over Direct are dataset-dependent. These results reinforce the distinction between visual evidence quality and downstream nutrition accuracy. 7 Compliance with Ethical Standards This study uses existing Nutrition5k and ACETADA data [12, 4] for computational evaluation. No new human or animal experiments were conducted. References [1] A. AlMughrabi, U. Haroon, R. Marques, and P. Radeva (2025) VolTex: food volume estimation using text-guided segmentation and neural surface reconstruction. In Proc. IEEE/CVF CVPR Workshops, p. 450–457. Cited by: §2. [2] R. Bianco, M. Marinoni, S. Coluccia, G. Carioni, F. Fiori, P. Gnagnarella, V. Edefonti, and M. Parpinel (2024) Tailoring the nutritional composition of Italian foods to the US Nutrition5k dataset for food image recognition: challenges and a comparative analysis. Nutrients 16 (19), p. 3339. External Links: Document Cited by: §4.1. [3] N. Carion, L. Gustafson, Y. Hu, S. Debnath, R. Hu, D. Suris Coll-Vinent, et al. (2026) SAM 3: segment anything with concepts. In Proc. ICLR, Cited by: §3.1. [4] B. Coburn, J. He, M. E. Rollo, S. S. Dhaliwal, D. A. Kerr, and F. Zhu (2025) Evaluating large multimodal models for nutrition analysis: a new benchmark enriched with contextual metadata. In Proc. IEEE-EMBS BHI, External Links: Document Cited by: §4.1, §7. [5] G. V. Cormack, C. L. A. Clarke, and S. Buettcher (2009) Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. In Proc. ACM SIGIR, p. 758–759. External Links: Document Cited by: §4.1. [6] Gemma Team, Google DeepMind (2026) Gemma 4 model card. Note: Google AI for Developers Cited by: §4.1. [7] C. Jiang, Y. Heng, W. Ye, H. Xu, M. Yan, J. Zhang, F. Huang, and S. Zhang (2025) VLM-R3^3: region recognition, reasoning, and refinement for enhanced multimodal chain-of-thought. In Adv. Neural Inf. Process. Syst., Vol. 38. External Links: Document Cited by: §2. [8] P. Khlaisamniang, K. Kerdthaisong, S. Vorathammathorn, N. Yongsatianchot, H. Phimsiri, A. Chinkamol, T. Thitseesaeng, K. Veerakanjana, K. Kachai, P. Ittichaiwong, and T. Saengja (2025) Decomposing food images for better nutrition analysis: a nutritionist-inspired two-step multimodal LLM approach. In Proc. IEEE/CVF CVPR Workshops, p. 482–491. Cited by: §1, §2. [9] J. Ma, X. Zhang, G. Vinod, S. Raghavan, J. He, and F. Zhu (2025) MFP3D: monocular food portion estimation leveraging 3D point clouds. In Pattern Recognition. ICPR 2024 International Workshops and Challenges, p. 49–62. External Links: Document Cited by: §2. [10] Qwen Team (2026) Qwen3.5 model release. Note: Qwen3.5-35B-A3B Cited by: §4.1. [11] S. Sterling, L. T. Berube, A. J. Glenn, A. Shaukat, S. Barua, M. E. Grams, and A. Tsirigos (2026) Vision-language models for image-based dietary assessment: a benchmark of accuracy, cost, and prompt strategies across ten models. bioRxiv. Note: Preprint 2026.07.26.740845 External Links: Document Cited by: §4.1. [12] Q. Thames, A. Karpur, W. Norris, F. Xia, L. Panait, T. Weyand, and J. Sim (2021) Nutrition5k: towards automatic nutritional understanding of generic food. In Proc. IEEE/CVF CVPR, p. 8899–8907. Cited by: §4.1, §7. [13] U.S. Department of Agriculture, Agricultural Research Service (2024) Food and nutrient database for dietary studies (FNDDS) 2021–2023. Note: Food Surveys Research Group Cited by: §4.1. [14] G. Vinod, J. He, Z. Shao, and F. Zhu (2024) Food portion estimation via 3D object scaling. In Proc. IEEE/CVF CVPR Workshops, p. 3741–3749. External Links: Document Cited by: §2. [15] G. Vinod, S. Raghavan, B. Coburn, and F. Zhu (2026) DietDelta: a vision-language approach for dietary assessment via before-and-after images. In Proc. IEEE/CVF CVPR Workshops, p. 9591–9600. Cited by: §2. [16] Y. Wang, Y. He, C. J. Boushey, F. Zhu, and E. J. Delp (2018) Context based image analysis with application in dietary assessment and evaluation. Multimedia Tools and Applications 77 (15), p. 19769–19794. External Links: Document Cited by: §2. [17] P. Wu and S. Xie (2024) V*: guided visual search as a core mechanism in multimodal LLMs. In Proc. IEEE/CVF CVPR, p. 13084–13094. Cited by: §1, §2. [18] X. Wu, X. Fu, Y. Liu, E. Lim, S. C. H. Hoi, and Q. Sun (2021) A large-scale benchmark for food image segmentation. In Proc. ACM Multimedia, Cited by: §2. [19] S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, D. Lian, and J. Nie (2024) C-Pack: packed resources for general Chinese embeddings. In Proc. ACM SIGIR, p. 641–649. Cited by: §4.1. [20] R. Yan, H. Luo, J. Lu, D. Liu, H. Posluszny, M. P. Dhaliwal, J. MacLeod, Y. Qin, C. Yang, T. J. Hartman, and X. Hu (2025) DietAI24 as a framework for comprehensive nutrition estimation using multimodal large language models. Communications Medicine 5, p. 458. External Links: Document Cited by: §1, §2, §4.1, §4.3. [21] S. Yin, C. Fu, S. Zhao, T. Xu, H. Wang, D. Sui, Y. Shen, K. Li, X. Sun, and E. Chen (2024) Woodpecker: hallucination correction for multimodal large language models. Science China Information Sciences 67, p. 220105. External Links: Document Cited by: §2. [22] Z. Zheng, M. Yang, J. Hong, C. Zhao, G. Xu, L. Yang, C. Shen, and X. Yu (2026) DeepEyes: incentivizing “thinking with images” via reinforcement learning. In Proc. ICLR, Cited by: §2. [23] F. Zhu, M. Bosch, N. Khanna, C. J. Boushey, and E. J. Delp (2015) Multiple hypotheses image segmentation and classification with application to dietary assessment. IEEE Journal of Biomedical and Health Informatics 19 (1), p. 377–388. External Links: Document Cited by: §2. [24] F. Zhu, M. Bosch, T. R. Schap, N. Khanna, D. S. Ebert, C. J. Boushey, and E. J. Delp (2011) Segmentation assisted food classification for dietary assessment. In Proc. SPIE, Computational Imaging IX, Vol. 7873, p. 78730B. External Links: Document Cited by: §2.