Paper deep dive
NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation
Aman Kumar, Avinash Anand, Chaitanya Lakhchaura, Ashutosh Kumar, Akshita Abrol, Timothy Liu, Zhengkui Wang, Rajiv Ratn Shah
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Nutrition estimation is a fundamental task in consumer diet tracking, clinical dietetics, chronic disease management, sports and hospital nutrition, and broader food computing systems. The existing approaches have progressed along two largely separate axes, vision models that rely on calibrated RGB-depth captures and ingredient-aware methods that use textual cues but use limited multimodal fusion. We introduce NutriVision, an end-to-end framework that leverages visual geometry and ingredient semantics to estimate calories, mass, fat content, carbohydrates, and protein from a single RGB image and an optional ingredient list. It obtains the unavailable depth modality using DepthAnything-V3 and encodes ingredient descriptions using CLIP. It integrates three complementary mechanisms: (1) an \emph{Ingredient-Conditioned Frequency-Aligned Fusion Module (IC-FAFM)}, which uses textual guidance to reweight and align RGB-depth frequency components; (2) an \emph{Ingredient-Aware Mask-based Prediction Head (IA-MPH)}, whose gating and channel masks are conditioned on food identity; and (3) modality-specific \emph{Internal Semantic Modeling (ISM)} blocks. On the Nutrition5k dataset, NutriVision achieves a mean PMAE of $\mathbf{13.60\pm0.10\%}$, outperforming our IGSMNet implementation by $0.89$ percentage points and OmniFood8k by $2.90$ percentage points (both $p<0.001$). The module-level ablations identify the ingredient-aware prediction head as the primary architectural contributor, improving mean PMAE by $1.50\pm0.17$ percentage points ($p<0.001$). These results demonstrate that ingredient-conditioned prediction and frequency-aware RGB-depth fusion provide measurable gains for single-image nutrient estimation. More broadly, NutriVision offers a practical route toward nutrition-assessment systems that exploit geometric and semantic cues without requiring specialized depth-sensing hardware
Tags
Links
- Source: https://arxiv.org/abs/2609.33076v1
- Canonical: https://arxiv.org/abs/2609.33076v1
Trouble viewing inline? Open PDF directly →
Full Text
63,092 characters extracted from source content.
Expand or collapse full text
NutriVision: Ingredient-Conditioned Fusion and Prediction for Single-Image Food Nutrition Estimation Aman Kumar Avinash Anand Chaitanya Lakhchaura Ashutosh Kumar Akshita Abrol Timothy Liu Zhengkui Wang Rajiv Ratn Shah †thanks: Aman Kumar and Chaitanya Lakhchaura are with the Indraprastha Institute of Information Technology Delhi (IIIT-Delhi), New Delhi, India (e-mail: aman24012@iiitd.ac.in; chaitanya24027@iiitd.ac.in).†thanks: Avinash Anand, Akshita Abrol, and Zhengkui Wang are with the Singapore Institute of Technology, Singapore (e-mail: Avinash.Anand@singaporetech.edu.sg; akshita.abrol@singaporetech.edu.sg; zhengkui.wang@singaporetech.edu.sg).†thanks: Ashutosh Kumar is with the Rochester Institute of Technology, Rochester, NY, USA (e-mail: ak1825@rit.edu)†thanks: Timothy Liu is with the NVIDIA AI Technology Centre, Singapore (e-mail: timothyl@nvidia.com).†thanks: Rajiv Ratn Shah is with the Indian Institute of Technology Kanpur, Kanpur, India (e-mail: rajivratn@iitk.ac.in). Abstract Nutrition estimation is a fundamental task in consumer diet tracking, clinical dietetics, chronic disease management, sports and hospital nutrition, and broader food computing systems. The existing approaches have progressed along two largely separate axes, vision models that rely on calibrated RGB-depth captures and ingredient-aware methods that use textual cues but use limited multimodal fusion. We introduce NutriVision, an end-to-end framework that leverages visual geometry and ingredient semantics to estimate calories, mass, fat content, carbohydrates, and protein from a single RGB image and an optional ingredient list. It obtains the unavailable depth modality using DepthAnything-V3 and encodes ingredient descriptions using CLIP. It integrates three complementary mechanisms: (1) an Ingredient-Conditioned Frequency-Aligned Fusion Module (IC-FAFM), which uses textual guidance to reweight and align RGB-depth frequency components; (2) an Ingredient-Aware Mask-based Prediction Head (IA-MPH), whose gating and channel masks are conditioned on food identity; and (3) modality-specific Internal Semantic Modeling (ISM) blocks. On the Nutrition5k dataset, NutriVision achieves a mean PMAE of 13.60±0.10%13.60± 0.10\%, outperforming our IGSMNet implementation by 0.890.89 percentage points and OmniFood8k by 2.902.90 percentage points (both p<0.001p<0.001). The module-level ablations identify the ingredient-aware prediction head as the primary architectural contributor, improving mean PMAE by 1.50±0.171.50± 0.17 percentage points (p<0.001p<0.001). These results demonstrate that ingredient-conditioned prediction and frequency-aware RGB-depth fusion provide measurable gains for single-image nutrient estimation. More broadly, NutriVision offers a practical route toward nutrition-assessment systems that exploit geometric and semantic cues without requiring specialized depth-sensing hardware. Project page and live demo: https://kianraj.github.io/nutrivision/. Index Terms: Nutrition estimation, multimodal fusion, food image analysis, monocular depth estimation. I Introduction Dietary assessment is foundational to nutrition science, chronic-disease management, clinical dietetics, and consumer health, while food computing has emerged as a distinct research area over the past decade [1]. Conventional dietary assessment relies heavily on manual food logging, which is inherently time-consuming and susceptible to reporting errors. These limitations motivate image-based food nutrition estimation, the prediction of a meal’s energy, mass, fat, carbohydrate, and protein content directly from a food image [2, 3, 4]. The earlier works on food computing primarily addressed food recognition, exemplified by datasets such as Food-101 [5]. Nutrition estimation is substantially more challenging because it requires jointly reasoning about food identity, ingredient composition, and portion size. The release of Nutrition5k [2], a large-scale dataset containing ingredient-level nutritional annotations and calibrated RGB-D captures, has enabled systematic evaluation of models for this task. Recent methods have improved nutrition estimation through two complementary sources of information: geometric cues derived from depth and semantic cues derived from ingredient descriptions. Geometry-oriented approaches seek to recover the spatial and volumetric information that is difficult to infer from RGB modality alone. For example, OmniFood8k [3] predicts depth from a single RGB image and processes the RGB and estimated-depth streams using separate encoders. Its Scale-Shift Residual Adapter (SSRA) refines the predicted depth, while its Frequency-Aligned Fusion Module (FAFM) hierarchically aligns the two modalities in the Fourier domain. A Mask-based Prediction Head (MPH) then dynamically selects informative feature channels, yielding a mean PMAE of 16.5%16.5\% on Nutrition5k. Ingredient-aware approaches instead exploit explicit information about food composition. This direction builds on earlier work connecting food images with recipes and ingredient descriptions, such as Recipe1M [6]. More recently, IGSMNet [4] encodes ingredient lists using CLIP [7] and employs cross-attention to align the resulting textual representation with hierarchically fused RGB-D features. Its Internal Semantic Modelling (ISM) mechanism further captures localized spatial and semantic relationships through fine-grained attention and dynamic positional encoding. By explicitly modelling ingredient semantics, IGSMNet obtains a mean PMAE of 15.0%15.0\% on Nutrition5k, with particularly strong improvements for identity-sensitive nutritional quantities such as fat, carbohydrate, and protein. Despite this progress, geometric and ingredient-level information remain only partially integrated. Frequency-domain RGB-depth fusion captures complementary appearance and structural cues, but its alignment is independent of the ingredients present in the dish. Conversely, ingredient-aware approaches introduce semantic information after conventional visual fusion and therefore do not allow food composition to directly regulate cross-modal interaction or nutrient-specific feature selection. Existing methods also either depend on calibrated depth sensors or underexploit estimated depth as a semantically conditioned geometric representation. This motivates a unified formulation in which ingredient semantics guide both RGB-depth fusion and final nutrient prediction while retaining single-image deployability through monocular depth estimation. Fig. 1: Overview of the proposed ingredient-aware RGB–depth fusion architecture for food nutrition estimation. A scale-shift residual adapter (SSRA) calibrates the depth input, while dual ConvNeXt-B backbones extract multi-scale RGB and depth features. An Ingredient-aware Spatial Mixer (ISM) captures spatial dependencies, followed by an Ingredient-Conditioned Hierarchical Feature Attention Module (IC-FAFM) that integrates ingredient semantics with multi-scale visual features. An Ingredient-Aware Prediction Head (IA-MPH) further refines the fused representation through ingredient-guided cross-attention to jointly predict calories, mass, protein, carbohydrates, and fat. Contributions We propose NutriVision (the overall pipeline in Figure 2 and component details in Figure 3), a unified framework that estimates calories, mass, fat, carbohydrates, and protein from an RGB image, an estimated depth map, and an optional ingredient list. Its principal contributions are: • Ingredient-Conditioned Frequency-Aligned Fusion Module (IC-FAFM). We extend OmniFood8k’s FAFM to perform ingredient-conditioned RGB-depth fusion across four ConvNeXt feature scales. At each scale, FFT-based low- and high-frequency components are modulated using a CLIP ingredient representation through multi-head cross-attention, allowing food composition to influence cross-modal spectral alignment. • Ingredient-Aware Mask-based Prediction Head (IA-MPH). We introduce an ingredient-conditioned regression head comprising a semantic gate, magnitude-based top-k channel selection, and ingredient-to-visual cross-attention. This design allows the channels used for nutrient prediction to vary according to the semantic composition of the dish rather than being determined solely by visual activations. • Modality-specific Internal Semantic Modeling (ISM). We extend IGSMNet’s ISM mechanism to a dual ConvNeXt-base architecture by applying DPE- and FGM-based ISM blocks independently to the RGB and estimated-depth feature hierarchies before fusion. This enables each modality to develop localized semantic representations while preserving modality-specific information. • Statistically validated performance. Under a three-seed retraining protocol, NutriVision achieves a mean PMAE of 13.60±0.10%13.60± 0.10\% on Nutrition5k, improving upon our IGSMNet reimplementation by 0.890.89 percentage points and OmniFood8k by 2.902.90 percentage points. Module-level retraining ablations identify IA-MPH as the largest individual architectural contributor, increasing PMAE by 1.50±0.171.50± 0.17 percentage points when removed. • Comprehensive modality and module ablations. We conduct test-time input ablations to quantify the contribution of RGB, estimated depth, and ingredient information, together with module-level retraining experiments that isolate the effects of IC-FAFM, IA-MPH, modality-specific ISM, and the training objective. NutriVision targets settings in which ingredient information is available from the user or the serving environment, including diet-tracking applications, institutional canteens, and hospital nutrition systems. Fully autonomous image-only estimation would additionally require an upstream ingredient-recognition component and is outside the scope of this work. Fig. 2: Overall NutriVision pipeline: A single RGB image I is paired with a monocular depth map estimated by DepthAnything-V3 and corrected by the Scale-Shift Residual Adapter (SSRA), while the ingredient string is encoded offline into a CLIP embedding e (teal, dashed rail). Dual ConvNeXt-Base encoders produce four-scale feature pyramids that are refined per modality by ISM blocks, fused in the frequency domain by IC-FAFM under ingredient guidance, aggregated across scales, and regressed by the ingredient-aware prediction head (IA-MPH) into the five nutrition targets y y = (calories, mass, fat, carbohydrates, protein). Ingredient semantics act as a persistent conditioning signal, entering both the cross-modal fusion stage and the prediction head. Fig. 3: Detailed structure of the three NutriVision modules. (A) The ISM block: windowed multi-head self-attention (7×7, 4 heads) with a DPE-derived positional bias B=MLP(Δx,Δy)B=MLP( x, y) in a pre-norm residual layout (⊕ ), applied twice per scale independently to each modality. (B) IC-FAFM at one scale: per-modality 2-D FFTs are split at the low-pass radius τ⋅rmax(τ=0.20)τ· r_max(τ=0.20), the low and high bands are summed across modalities and merged by a 3×3 convolution into OiO_i, which is then modulated by ingredient cross-attention (q = We) with a broadcast residual to yield O~i O_i. (C) IA-MPH: the concatenation x=[f;cr;cd]∈ℝ1280x=[f;c_r;c_d] ^1280 is gated by σ(We)σ(We), masked to the top-75% channels by magnitude during training only (dashed), and regressed via a second ingredient cross-attention and a three-layer MLP (1280→512→256→5)(1280→ 512→ 256→ 5) I Related Work I-A Image-Based Food Nutrition Regression The Nutrition5k benchmark [2] established the direct regression of nutritional quantities from food images, defining the task as RGB or RGB-D input → calories, mass, fat, carbohydrates, and protein. Its Inception-based baseline achieved a mean PMAE of 18.9%18.9\% and demonstrated that visual appearance alone is insufficient for reliably estimating both food composition and portion size. Subsequent methods improved feature extraction using stronger convolutional and attention-based architectures. Swin-Nutrition [8], for example, employed a hierarchical Swin Transformer [9] and achieved a mean PMAE of 17.2%17.2\% using RGB images alone. Later methods increasingly incorporated depth and ingredient information to provide geometric and semantic cues that are difficult to recover from RGB appearance alone. I-B Depth-Assisted Nutrition Estimation Depth information provides geometric cues related to food height, volume, and spatial arrangement, making it particularly useful for portion and mass estimation. Early RGB-D methods used the calibrated depth captures supplied by Nutrition5k. FBFPN [10] introduced bidirectional multi-scale RGB-D feature fusion and reported a mean PMAE of 17.3%17.3\%. NuNet, proposed by Kwan et al. [11], employed parallel multi-scale transformer branches and dedicated RGB-D fusion modules, achieving 15.65%15.65\% using real sensor depth. These approaches demonstrate the value of geometric information but assume access to calibrated RGB-D acquisition hardware. To reduce this hardware dependence, estimated-depth methods synthesize a depth map from a single RGB image. DPF-Nutrition [12] introduced an end-to-end depth-prediction and RGB-depth fusion framework, achieving a mean PMAE of approximately 17.8%17.8\% on Nutrition5k. OmniFood8k [3] further combined monocular depth estimation with a Scale-Shift Residual Adapter (SSRA), which corrects the global scale and local structure of the predicted depth representation. NutriVision follows this hardware-independent setting, using DepthAnything-V3 [13] to generate depth from the input RGB image and SSRA to adapt the resulting relative geometry for nutrition estimation. I-C Frequency-Domain RGB-Depth Fusion Frequency-domain processing provides an alternative to conventional spatial-domain multimodal fusion. Low-frequency components primarily encode coarse structure and global layout, whereas high-frequency components preserve boundaries, textures, and local details. General-purpose architectures such as GFNet [14] and Fast Fourier Convolution [15] have demonstrated that Fourier-domain operations can efficiently model global dependencies and complementary spectral information. Within food nutrition estimation, OmniFood8k [3] introduced a Frequency-Aligned Fusion Module (FAFM) that hierarchically aligns and fuses RGB and adapted depth features in the frequency domain. Its Mask-based prediction head subsequently performs dynamic visual channel selection for nutrient regression. However, both frequency alignment and channel selection are determined from visual features alone. NutriVision builds on this formulation by conditioning frequency-domain RGB-depth interaction and final channel selection on ingredient semantics. I-D Ingredient-Guided Nutrition Estimation Ingredient descriptions provide direct semantic evidence about food composition and therefore offer strong priors for estimating identity-sensitive quantities such as fat, carbohydrates, and protein. Earlier multimodal food research, including Recipe1M [6], established the value of aligning food images with recipes and ingredient text. More recent nutrition-estimation methods incorporate these semantics directly into regression architectures. IMIR-Net [16] introduced ingredient-guided multimodal interaction and refinement within an RGB-D framework, reporting a mean PMAE of 17.4%17.4\%. IGSMNet [4] subsequently employed an asymmetric dual-branch Swin-Tiny architecture to extract hierarchical RGB and depth representations. After fusing the two visual streams across scales, it uses CLIP [7] ingredient embeddings as queries in a cross-attention module that guides the fused visual features. Dynamic positional encoding and fine-grained modeling are then applied through its ISM module to capture localized spatial and compositional relationships. IGSMNet reports a mean PMAE of 15.0%15.0\% on Nutrition5k. NutriVision adopts CLIP-based ingredient encoding but changes where and how ingredient semantics enter the architecture. Rather than applying ingredient guidance only after hierarchical RGB-D fusion, it conditions the cross-modal frequency-alignment process itself. It additionally uses ingredient information within the prediction head and applies modality-specific semantic modeling to the RGB and estimated-depth streams before fusion. Thus, ingredient information acts as a persistent conditioning signal for representation refinement, cross-modal interaction, and nutrient prediction. I Methodology NutriVision estimates the nutritional composition of a dish by combining three complementary sources of evidence. The RGB image captures appearance and food identity; the estimated depth map provides geometric cues related to portion size and spatial structure; and the ingredient list supplies explicit information about food composition. As shown in Figure 2, the model first corrects the monocular depth representation using an SSRA. RGB and adapted-depth inputs are then processed by separate ConvNeXt-base encoders and refined through modality-specific ISM blocks. The resulting feature pyramids are fused in the frequency domain under ingredient guidance, aggregated across scales, and passed to an IA-MPH. The complete model is trained using a normalized nutrition-regression objective together with an auxiliary RGB-depth alignment loss. I-A Problem Formulation and Notation Given an RGB image ∈ℝ3×224×224I ^3× 224× 224, an ingredient string s, a pretrained monocular depth estimator produces a depth map, mono=gdepth()∈ℝ1×224×224D^mono=g_depth(I) ^1× 224× 224 where =gdepth=g_depth denotes DepthAnything-V3. The ingredient string is encoded offline using the CLIP ViT-B/32 text encoder, =gtext(s)∈ℝ512.e=g_text(s) ^512.. Following IGSMNet [4], the experiments use ground-truth ingredient strings supplied with Nutrition5k. The ingredient information is therefore treated as an input available at inference time rather than as a target predicted by NutriVision. The model predicts a 5-dimensional nutrition vector ^=(y^cal,y^mass,y^fat,y^carb,y^prot)∈ℝ5. y=( y_cal, y_mass, y_fat, y_carb, y_prot) ^5. (1) that contains calories, total mass, fat, carbohydrates, and protein. The complete mapping can be written as FΘ(,mono,)F_ (I,D^mono,e ) where (Θ)( ) denotes the trainable parameters of the depth adapter, dual visual encoders, fusion modules, and prediction head. Algorithm 1 summarizes the full forward pass. Algorithm 1 NutriVision Forward Pass 1: RGB image I, ingredient string s, monocular depth DmonoD_mono 2: Nutrition vector y^∈ℝ5 y ^5 3: e←CLIP-Text(s)e -Text(s) ⊳ offline embedding 4: D←SSRA(Dmono)D (D_mono) ⊳ Eqs. 2–4 5: for i∈0,1,2,3i∈\0,1,2,3\ do ⊳ per-scale pyramid 6: ri←ConvNeXt-RGBi(I)r_i -RGB_i(I) 7: di←ConvNeXt-Depthi(D)d_i -Depth_i(D) 8: ri←ISM(ri)r_i (r_i); di←ISM(di)d_i (d_i) ⊳ Eq. 5 9: Oi←IC-FAFM(ri,di,e)O_i -FAFM(r_i,d_i,e) ⊳ Eqs. 6–14 10: end for 11: f←MultiScaleAggregate(Oii=03)f (\O_i\_i=0^3) ⊳ Eqs. 12–14 12: cr,cd←GlobalPool(r3),GlobalPool(d3)c_r,c_d (r_3),GlobalPool(d_3) 13: x←[f;cr;cd]x←[f\,;c_r\,;c_d] ⊳ Eq. 16 14: x←x⊙σ(Linear(e))x← x σ(Linear(e)) ⊳ Eq. 17, ingredient gate 15: if training then 16: m←[|x|≥quantile0.25(|x|)]m 1[|x| _0.25(|x|)] 17: x←x⊙mx← x m ⊳ Eq. 18, channel mask, train-only 18: end if 19: y^←IA-MPH-Head(x,e) y -MPH-Head(x,e) ⊳ cross-attn + MLP, Sec. I-F 20: return y y I-B Scale-Shift Residual Adapter (SSRA) Monocular depth is defined up to an unknown affine transformation. Following OmniFood8k [3], we apply a global affine correction followed by a local CNN residual: glo ^glo =αmono+β, =α\,D^mono+β, (2) res ^res =fθ(glo), =f_θ(D^glo), (3) =glo+res. =D^glo+D^res. (4) Here α,β∈ℝα,β are learnable scalars (init. 1,01,0) and fθf_θ is a four-layer CNN (→11\!→\!64\!→\!64\!→\!32\!→\!1, 3×33×3 kernels, BN, ReLU). SSRA contributes ∼ 100 K parameters. I-C Dual ConvNeXt-Base Backbone Both modalities are encoded by ConvNeXt-Base pretrained on ImageNet-22k and fine-tuned on ImageNet-1k [17]; a Conv2d(1→3)Conv2d(1→3) adapter (weights 1/31/3) precedes the depth branch. Each branch yields a four-stage pyramid with strides 4,8,16,32\4,8,16,32\ and channels 128,256,512,1024\128,256,512,1024\, projected to a common 256 channels by 1×11×1 convolution. The first two stages are frozen, leaving ∼ 44 M trainable parameters per branch. Global CLS embeddings r,d∈ℝ512c_r,c_d ^512 come from AdaptiveAvgPool2d(1)→Linear(1024,512)→ReLUAdaptiveAvgPool2d(1)→Linear(1024,512)→ReLU. We chose ConvNeXt-Base over IGSMNet’s Swin-Tiny for its larger capacity and stronger ImageNet-22k pretraining. I-D Internal Semantic Modelling (ISM) We apply IGSMNet’s ISM block [4] symmetrically to both modalities, independently per scale. Each block combines (i) a Dynamic Position Encoding (DPE) that maps relative window offsets to an attention bias, bij=MLPDPE(Δxij,Δyij)b_ij=MLP_DPE( x_ij, y_ij), and (i) Fine-Grained Modelling (FGM), window-based multi-head self-attention with that bias: FGM()=softmax(⊤d+),FGM(X)=softmax\! ( QK^\! d+B )V, (5) on 7×77×7 windows, followed by a pre-norm transformer MLP with residuals. We use two ISM blocks per scale per modality (16 total). I-E Ingredient-Conditioned Frequency-Aligned Fusion (IC-FAFM) IC-FAFM extends OmniFood8K’s Frequency-Aligned Fusion Module (FAFM) [3] with multi-scale operation and ingredient-conditioned cross-attention; we retain the FAFM acronym to acknowledge this lineage. The module is applied at each of the four ConvNeXt scales i∈0,1,2,3i∈\0,1,2,3\. The individual module-level deltas (A7, A8, A11 in Tab. VI) sit at the per-run noise floor, so we treat IC-FAFM as a small, diffuse contributor. Step 1: Frequency-Aligned Fusion Let i,i∈ℝB×256×Hi×Wir_i,d_i ^B× 256× H_i× W_i denote the post-ISM RGB and depth features. We compute the 2-D FFT of each: f=ℱ(i),f=ℱ(i).R_f=F(r_i), _f=F(d_i). (6) A circular low-pass mask LM_L of radius τ⋅rmaxτ\!·\!r_ (τ=0.20τ=0.20) decomposes each feature into a low-frequency and a high-frequency component: L _L =ℱ−1(f⊙L), =F^-1(R_f\! \!M_L), H _H =ℱ−1(f⊙(−L)), =F^-1(R_f\! \!(1-M_L)), (7) L _L =ℱ−1(f⊙L), =F^-1(D_f\! \!M_L), H _H =ℱ−1(f⊙(−L)). =F^-1(D_f\! \!(1-M_L)). (8) The two modalities are added per band and merged by a 3×33×3 convolution: i=Conv3×3([L+L;H+H]).O_i=Conv_3× 3\! ([\,R_L+D_L\,;\,R_H+D_H\,] ). (9) Figure 4 visualises what this split actually buys: at τ=0.20τ=0.20 the low band keeps the food’s footprint and plate geometry, while the high band isolates texture and edge information—the cues most useful for distinguishing sauces, garnishes, and surface fat. Fig. 4: What the FFT split inside IC-FAFM separates. (a) the input dish. (b) its log-magnitude 2-D FFT, with the low-pass mask of radius τrmax=0.20rmaxτ\,r_ =0.20\,r_ overlaid. (c) inverse FFT of the low band, which preserves plate boundary, portion footprint, and overall layout. (d) inverse FFT of the high band, which preserves edges, textures, and fine-grained ingredient appearance. Hyperparameter τ τ=0.20τ=0.20 is inherited from OmniFood8k’s FAFM. We did not run a sweep; since A8 (no FFT split) is at the noise floor, any reasonable τ should move the mean by less. A sweep and scale-adaptive τi _i are future work. Step 2: Ingredient Cross-Attention The ingredient embedding is projected to the feature dimension, =Linear()∈ℝB×256q=Linear(e) ^B× 256, and used as a single query against the spatial tokens of iO_i via standard scaled dot-product attention [18]: i=MHA(⊤,Φ(i),Φ(i)),a_i=MHA\! (q^\! ,\; (O_i),\; (O_i) ), (10) with Φ flattening to spatial tokens and four attention heads. The result is broadcast back to the spatial grid: ~i=i+expand(i,Hi,Wi). O_i=O_i+expand(a_i;H_i,W_i). (11) Figure 5 shows the attention shifting with the queried ingredient—qualitative evidence of grounding, not causal evidence of the headline gain (A7 is +0.18+0.18 PMAE, at the noise floor), consistent with the small-and-diffuse framing of IC-FAFM (Sec. VI-B). Fig. 5: Ingredient cross-attention inside IC-FAFM (Eq. 10) on the same dish, queried with three different ingredient strings. The hot regions shift meaningfully: “chicken thighs” attends to the seared chicken on the right, “broccoli florets” migrates upward toward the green florets, and “brown rice” redistributes again. The attention is a soft prior, not a segmentation; this is qualitative evidence of ingredient–visual grounding, not causal evidence of a headline accuracy gain—the corresponding retraining ablation (A7, Δ=+0.18 \!=\!+0.18 PMAE) is at the per-run noise floor. Step 3: Multi-Scale Aggregation Operating over four scales follows the now-standard feature-pyramid recipe [19]: per-scale pooled descriptors are sigmoid-gated and concatenated: i _i =ReLU(Linear(AvgPool(~i))), =ReLU(Linear(AvgPool( O_i))), (12) i _i =σ(si)i, =σ(s_i)\,p_i, (13) =MLP([0;1;2;3])∈ℝB×256, =MLP\! ([g_0;g_1;g_2;g_3] ) ^B× 256, (14) where si∈ℝs_i\!∈\!R are learnable per-scale gates. The MLP has shape 1024→512→2561024→512→256. Step 4: Inter-Modal Alignment Loss For each scale we project the pre-fusion features through small heads ψr,ψd:ℝ256→ℝ128 _r, _d:R^256\!→\!R^128 and apply an info-NCE contrastive loss across the batch: ℒalign(i)=−1B∑b=1Blogexp(⟨ψr(i(b)),ψd(i(b))⟩/τc)∑b′=1Bexp(⟨ψr(i(b)),ψd(i(b′))⟩/τc).L_align^(i)=- 1B _b=1^B ( _r(r_i^(b)), _d(d_i^(b)) / _c) _b =1^B ( _r(r_i^(b)), _d(d_i^(b )) / _c). (15) The four scale losses are averaged into ℒalignL_align. I-F Ingredient-Aware Mask-based Prediction Head (IA-MPH) IA-MPH is our second novel contribution. The fused vector ∈ℝ256f\!∈\!R^256 is concatenated with the CLS embeddings of both modalities, =[;r;d]∈ℝ1280.x=[f;c_r;c_d] ^1280. (16) Ingredient-conditioned gating ←⊙σ(Linear512→1280()).x \; \;σ(Linear_512→ 1280(e)). (17) Dynamic channel mask (training only) Following the masked-channel philosophy of [3], we retain the top 75% of channels by magnitude: =[||≥quantile0.25(||)],←⊙.m=1\! [|x| _0.25(|x|) ], . (18) This regulariser encourages the network to commit informative channels to each food type. Ingredient cross-attention and regression A second cross-attention block uses x as query against the ingredient embedding, followed by a gated fusion and a three-layer MLP head 1280→512→256→51280→512→256→5 with dropouts 0.10.1 and 0.050.05. The output is the 5-D nutrition vector (1). I-G Loss Function We minimise ℒ=ℒnutri+λaℒalign,λa=0.15,L\;=\;L_nutri+ _a\,L_align, _a=0.15, (19) where ℒalignL_align is the average of Eq. (15) over the four scales, and ℒnutriL_nutri is the normalised ℓ1 _1 loss of [4]: ℒnutri=∑t=15∑b=1B|y^t,b−yt,b|∑b=1Byt,b.L_nutri= _t=1^5 _b=1^B | y_t,b-y_t,b | _b=1^By_t,b. (20) This loss directly optimises the PMAE metric and is scale-invariant across the five tasks. I-H Parameter Budget The full model has ∼ 196 M parameters of which ∼ 105 M are trainable (Table I). TABLE I: Per-component parameter budget. Component Params Trainable Notes ConvNeXt-Base RGB 88.6 M 44 M Stages 2–3 unfrozen ConvNeXt-Base Depth 88.6 M 44 M Stages 2–3 unfrozen Depth adapter 6 6 →31\!→\!3 Conv2×22× 2 Scale projections (8) 790 K 790 K 1×11×1 to 256-ch CLS projections (2) 1.1 M 1.1 M Linear 1024→5121024→512 SSRA 100 K 100 K α,βα,β + CNN refiner ISM RGB (2×42×4) 3.2 M 3.2 M Window attn + MLP ISM Depth (2×42×4) 3.2 M 3.2 M Window attn + MLP IC-FAFM (4 scales) 8.0 M 8.0 M FAFM + cross-attn + align. IA-MPH 2.0 M 2.0 M Gate + cross-attn + reg. Alignment projectors 530 K 530 K 4×(256→128)×24×(256→128)×2 Total ∼ 196 M ∼ 105 M TABLE I: Compute budget for NutriVision. Forward-pass FLOPs are estimated from per-block published numbers (ConvNeXt-Base ∼ 15.4 GFLOPs at 2242224^2 [17]); training and inference times are measured on a single RTX A6000 (48 GB). End-to-end inference includes the DepthAnything-V3 forward and the CLIP text encoding for the ingredient string. Quantity Value Notes Total parameters ∼ 196 M measured (Tab. I) Trainable parameters ∼ 105 M stages 2–3 of both backbones unfrozen Forward FLOPs (est.) ∼ 35 GFLOPs 2×15.42×15.4 (backbones) ++ ISM/IC-FAFM/IA-MPH/SSRA overhead Train time / epoch ∼ 925 s batch 8 on RTX A6000, 1923 batches/epoch Total train time ∼ 38.5 h 150 epochs End-to-end inference ∼ 3 s / image RGB→ → → Model forward only <<0.5 s / image batch 1, RTX A6000 (depth+CLIP precomputed) Note on baseline scale Tab. IV is not parameter-matched. Kwan NuNet [11] (dual SwinT-Base, ∼ 200–260 M) and OmniFood8k [3] (dual ConvNeXt-Base) are broadly comparable to NutriVision (∼ 196 M); IGSMNet [4] uses a smaller Swin-family backbone, and the earlier baselines (Inception-V2, Swin-Nutrition, DPF-Nutrition) are smaller still. Two consequences follow. First, the ∼ 5–7 p mean-PMAE gap to the smaller baselines should be read as a combined effect of architecture, ingredient input, and backbone scale, not as pure architectural superiority. Second, the input-matched comparison against IGSMNet (re-run, 0.890.89 p) is also not parameter-matched: a size-controlled IGSMNet (e.g. with a Swin-Base or ConvNeXt-Base backbone) is left to future work (Sec. VII). IV Experiments IV-A Dataset and Protocol We evaluate on Nutrition5k [2], which contains 5,006 unique dishes captured from four overhead cameras (A/B/C/D) together with per-dish ground-truth nutrition. Each dish provides ∼ 4 RGB views and one calibrated depth view. We follow the 5:1 dish-level split of [4], yielding approximately 15,000 training and 3,000 testing images. The split is performed at the dish level so that camera views of the same dish never leak between train and test. Best-on-test selection (protocol disclosure) We use the test set both for early stopping and for reporting (no separate held-out validation split); every entry in Tabs. IV–VI is a best-mean-PMAE-on-test value. This matches IGSMNet’s and Kwan et al.’s protocol so within-table comparisons are like-for-like, but the headline 13.60±0.10%13.60± 0.10\% is optimistically biased relative to true held-out generalisation; a proper held-out split or k-fold CV is left to Sec. VII. Monocular depth maps are precomputed once with DepthAnything-V3 [13]; CLIP ingredient embeddings are precomputed with the ViT-B/32 text encoder. We additionally precompute food-vs-background masks with SAM [20] using the “food on a plate” text prompt; these masks are retained as a preprocessing artifact and explored in Sec. VII but are not consumed by the reported model, since the empirical contribution of foreground masking was negative in our pilot experiments. No data augmentation is used beyond a synchronised horizontal flip (p=0.5p=0.5) applied jointly to RGB (∈ℝ3×224×224I ^3× 224× 224) and depth. Test-time input asymmetry NutriVision consumes RGB++estimated depth++GT ingredient string; the ingredient input is the same privileged signal IGSMNet uses. The input-matched comparison in Tab. IV is therefore with IGSMNet, and lower-mean claims against ingredient-free baselines (OmniFood8k, DPF-Nutrition, Google-Nutrition-rgbd, Kwan et al.) should be read as “best-in-class given the privileged input.” Removing CLIP inflates NutriVision to 27.44%27.44\% (A2, +13.92+13.92). IV-B Evaluation Metric We report Percentage Mean Absolute Error (PMAE) per task and its mean over the five tasks: PMAEt=1N∑n|y^t,n−yt,n|1N∑nyt,n×100%,PMAE_t= 1N _n| y_t,n-y_t,n| 1N _ny_t,n× 100\%, (21) MeanPMAE=15∑t=15PMAEt.Mean\,PMAE= 15 _t=1^5PMAE_t. (22) IV-C Implementation Details We use Adam with a two-tier learning rate of 1e−51 e-5 for the unfrozen ConvNeXt stages and 5e−55 e-5 for the rest; exponential schedule with γ=0.99γ=0.99 per epoch; weight decay 1e−51 e-5; gradient clipping at ∥⋅∥2=1\|·\|_2\!=\!1. Batch size is 8, training runs for 150 epochs, on a single NVIDIA RTX A6000 (48 GB). End-to-end training takes ∼ 925 s/epoch, i.e. ∼ 38.5 h total. Hyperparameters are summarised in Table I. The training trajectory (Fig. 6) is shown for a representative seed and converges to a best mean PMAE of 13.50%13.50\% (3-seed mean 13.60±0.10%13.60± 0.10\%) on the Nutrition5k test split, after which the model plateaus. Selection uses the same test split throughout (§IV); no held-out validation split is carved out. Fig. 6: Training trajectory on Nutrition5k (representative seed). Train loss (purple, left) decays smoothly; test-set mean PMAE (blue, right) plateaus near 13.5%13.5\%; we report the best-on-test checkpoint (§IV). The 3-seed mean best-mean-PMAE is 13.60±0.10%13.60± 0.10\%. TABLE I: Training hyperparameters. Parameter Value Optimizer Adam Base learning rate 5e−55 e-5 Backbone learning rate 1e−51 e-5 LR schedule Exp. decay, γ=0.99γ=0.99/epoch Weight decay 1e−51 e-5 Gradient clipping ∥⋅∥2=1.0\|·\|_2=1.0 Batch size 8 Epochs 150 Input resolution ×224224\!×\!224 Augmentation Sync. horizontal flip (p=0.5p=0.5) CLIP model ViT-B/32 FAFM threshold τ 0.20 ISM blocks per level 2 ISM window size ×77\!×\!7 Alignment weight λa _a 0.15 V Results V-A Comparison with State-of-the-Art Table IV compares NutriVision with six published baselines and our reimplementation of IGSMNet (full training code was not publicly available, so we re-implemented from the paper’s architectural description and trained under identical conditions to NutriVision). NutriVision attains a 3-seed mean PMAE of 13.60±0.10%13.60± 0.10\%, the lowest in the table. Statistical significance The headline NutriVision configuration and the dominant module ablation (A10, IA-MPH→ MLP) were re-trained under identical conditions across three seeds 1,2,3\1,2,3\ affecting model initialisation, dropout, and stochastic data ordering (the dish-level 5:1 split is fixed). NutriVision reaches 13.60±0.10%13.60± 0.10\% (per-seed: 13.50,13.60,13.7013.50,13.60,13.70) and A10 reaches 15.10±0.26%15.10± 0.26\% (14.90,15.00,15.4014.90,15.00,15.40). A one-sample t-test of the NutriVision seeds against the IGSMNet reimplementation (14.49%14.49\%) rejects the null at t=−15.4t\!=\!-15.4, p<0.001p\!<\!0.001; against IGSMNet’s published 15.0%15.0\% at t=−24.3t\!=\!-24.3, p<0.001p\!<\!0.001. A paired t-test of A10 against NutriVision yields Δ=+1.50±0.17 \!=\!+1.50± 0.17 p, p<0.001p\!<\!0.001. Rows A7–A9, A11, A12 remain single-seed and their |Δ|≤0.20| |\!≤\!0.20 deltas continue to sit inside the ∼ 0.150.15 p per-run noise floor; scaling the paired protocol to those rows is left as future work. What the comparison fairly establishes The input-matched comparison is against IGSMNet: +0.89±0.10+0.89± 0.10 p vs our re-run (→13.60±0.1014.49\!→\!13.60± 0.10, p<0.001p\!<\!0.001), +1.40±0.10+1.40± 0.10 p vs IGSMNet’s reported numbers (→13.60±0.1015.0\!→\!13.60± 0.10, p<0.001p\!<\!0.001). This is the principal claim. Comparisons to ingredient-free baselines (OmniFood8k, DPF-Nutrition, Google-Nutrition-rgbd, Kwan et al.) are not input-matched—A2 shows that removing ingredients alone pushes NutriVision to 27.44%27.44\%, worse than every ingredient-free baseline—so cross-input gaps characterise an ingredient-aware operating point, not architectural superiority. TABLE IV: Per-nutrient and mean PMAE (%) on the Nutrition5k 5:1 dish-level split. Lower is better. Best in bold. †The “IGSMNet (our reimpl.)” row is our re-implementation of the IGSMNet architecture from [4] (the full training code was not publicly available at the time of writing); we follow the paper’s architectural description and train under identical conditions to NutriVision. ‡Per-nutrient columns for the NutriVision row are from a representative seed=1 run (best-mean-PMAE 13.50%13.50\%); the Mean column reports the 3-seed mean± , and the Δ column uses the 3-seed mean. All baselines are as published. Method Venue Input (test) Cal Mass Fat Carb Prot Mean Δ vs. ours Google-Nutrition-rgbd [2] CVPR’21 RGB-D (real) 18.8 18.9 18.1 23.8 20.9 20.1 +6.50+6.50 Swin-Nutrition [8] Foods’22 RGB 15.3 12.5 22.1 20.8 15.4 17.2 +3.60+3.60 DPF-Nutrition [12] Foods’23 RGB (est. depth) 14.7 10.6 22.6 20.7 20.2 17.8 +4.20+4.20 OmniFood8k [3] CVPR’26 RGB 14.1 10.2 21.0 18.9 18.4 16.5 +2.90+2.90 Kwan et al. [11] (NuNet) TMM’25 RGB-D (real) 12.80 8.72 19.67 18.66 18.42 15.65 +2.05+2.05 IGSMNet [4] Foods’25 RGB-D (real)+Ingr. 12.2 9.4 19.1 18.3 16.0 15.0 +1.40+1.40 IGSMNet (our reimpl.)† [4] – RGB-D (real)+Ingr. 13.6 11.4 17.6 15.5 14.4 14.49 +0.89+0.89 NutriVision (ours)‡ – RGB+Ingr.+Depth(est.) 12.65 10.37 16.45 14.76 13.35 ±0.1013.60\!±\!0.10 – Compared with our IGSMNet reimplementation, NutriVision improves on every per-nutrient metric; largest gains: fat (−1.15-1.15, →16.4517.6\!→\!16.45) and protein (−1.05-1.05, →13.3514.4\!→\!13.35). Against published numbers the per-nutrient story is mixed: NutriVision loses on mass to Kwan (8.728.72), IGSMNet (9.49.4), and OmniFood8k (10.210.2)—all below our 10.3710.37. The pattern is physically sensible (real sensor depth and larger ingredient-aware backbones retain a mass advantage over our DepthAnything-V3 input). NutriVision wins on the mean against every published baseline and on four of five nutrients individually. Figure 7 shows six representative test-set predictions; the model produces within-± 13.5 % estimates on the majority of nutrients per dish, with the largest errors concentrated on visually-ambiguous mixed plates. Fig. 7: Four representative Nutrition5k test predictions, spanning low-, mid-, and high-calorie dishes. For each plate we show the input photograph, the comma-separated ingredient list used at inference, and a compact ground-truth-vs-prediction table for the five nutrients. Green cells indicate the prediction is within ± 13.5% of GT (one PMAE band); orange cells mark larger misses. Errors are largest on dishes with heavily-stacked or occluded ingredients. What the ablations actually attribute the gains to The per-nutrient gains over IGSMNet cannot be plausibly credited to SSRA or FFT: A6 moves mean by 0.000.00, A3 moves mass by +0.05+0.05 only, and A8 moves fat by +0.35+0.35—all an order of magnitude smaller than the ∼ 1 p fat/mass gains. The supported drivers are (i) the IA-MPH head (A10, +1.50±0.17+1.50± 0.17 p, p<0.001p\!<\!0.001) and (i) the backbone swap from Swin-family to ConvNeXt-Base (inferred from IGSMNet→ moves; a formal backbone ablation is future work). SSRA, FFT split, and the alignment loss should therefore be read as scaffolding that may contribute jointly but whose individual deltas are at or just above the per-run noise floor. V-B Comparison with Real-Depth Methods NutriVision (13.60±0.10%13.60± 0.10\%) is numerically lower than Kwan et al.’s NuNet [11] (15.65%), the strongest published method using the real depth captures of Nutrition5k. However, this is not an apples-to-apples depth comparison: NuNet operates on RGB+depth with no ingredient signal, while NutriVision combines RGB, estimated depth, and CLIP-encoded ingredient text. Our input ablation (Tab. V, A3) shows that the depth pathway contributes only +0.13+0.13 PMAE when zeroed at inference (a test-time robustness measurement, not a from-scratch retraining without depth—see the caveat in Sec. VI), and the module ablation (Tab. VI) attributes the dominant gains to the ingredient-aware prediction head (A10, +1.50±0.17+1.50± 0.17, p<0.001p\!<\!0.001) and to the metric-aligned normalised-ℓ1 _1 loss adopted from IGSMNet [4] (A12, +1.20+1.20, single seed; a confirmation of an inherited choice rather than a novel contribution). The fair reading: NutriVision with estimated depth outperforms NuNet with real depth on Nutrition5k because of ingredient conditioning, the IA-MPH head, and the metric-aligned loss—not because of better depth handling. An apples-to-apples depth study is an open direction for future work. VI Ablation Study VI-A Input-Level Ablation We zero out individual modalities at test time only (no retraining) using the trained checkpoint of the best NutriVision run. Table V reports the resulting PMAE. TABLE V: Input-level ablation on the Nutrition5k test split (3,096 samples). Inputs are zeroed at test time only; the model is the representative seed=1 NutriVision checkpoint (best mean PMAE 13.52%13.52\%; 3-seed mean 13.60±0.10%13.60± 0.10\%, Tab. IV). # Configuration Cal Mass Fat Carb Prot Mean Δ Mean A1 Full model (RGB + Depth + CLIP) 12.65 10.37 16.45 14.76 13.35 13.52 – A2 No CLIP (zero ingredient embedding) 24.87 15.94 35.43 26.35 34.59 27.44 +13.92+13.92 A3 No depth (zero depth) 12.88 10.42 16.56 14.74 13.67 13.65 +0.13+0.13 A4 No CLIP and no depth (RGB only) 25.70 16.29 35.79 26.40 36.18 28.07 +14.55+14.55 A5 No RGB (sanity check) 68.04 59.37 73.81 65.98 64.50 66.34 +52.82+52.82 A6 No SSRA at inference (α=1,β=0α=1,β=0, residual=0=0) 12.65 10.37 16.45 14.76 13.36 13.52 +0.00+0.00 Figure 8 renders the same numbers as a per-nutrient radar (smaller polygon == lower PMAE == better), making the relative magnitude of each modality’s contribution immediately visible. Fig. 8: Input-level ablation visualised as radar plots over the five nutrients (lower is better; the polygon shrinks toward the centre as PMAE drops). (a) All four ablation variants overlaid: removing RGB (red) explodes the polygon, removing ingredient text (orange) roughly doubles it, while the full model (blue) and the no-depth variant (green) are visually indistinguishable. (b) Zoomed in to 0–20 % PMAE, the full and no-depth polygons coincide almost perfectly, confirming the +0.13+0.13 p mean delta in Table V. Take-aways 1. Ingredient text is the largest non-RGB signal. Removing CLIP guidance costs 13.92 mean-PMAE points; the hits concentrate on fat (+18.98) and protein (+21.24)—the most ingredient-dependent nutrients. 2. Depth and SSRA: trained-model robustness, not informativeness (A3, +0.13+0.13; A6, 0.000.00). Both zero the input on a ∼ 176 M-parameter model trained with it, so the small deltas establish test-time routing-around, not causal contribution. A proper no-depth / no-SSRA retraining is left to future work (Sec. VII). 3. This architecture cannot exploit ingredients without RGB. Zeroing RGB collapses performance to 66.34% (A5). The narrow reading is that NutriVision-as-trained needs the visual stream; the broader “ingredients need vision” reading is not something A5 establishes—a dedicated text-only or text++plate-area regressor might recover much of what A5 attributes to RGB. Adding that control is flagged as a camera-ready addition (Sec. VII). VI-B Module-Level Retraining Ablation Whereas Table V zeros inputs at inference time, here we re-train the full architecture from scratch with a single module replaced or removed in each variant. The targeted module is implemented as a forward-method swap; backbones, optimiser, two-tier learning rate, exponential decay γ=0.99γ=0.99, batch size 8, gradient clipping, data pipeline, and all other architecture elements are byte-identical to the headline run. Each variant trains for 100 epochs on the same N5k 5:1 split, and—using the same best-mean-PMAE-on-test selection protocol as the headline (Sec. IV)—we report the checkpoint that minimises mean PMAE on the 3,096-sample test split (the best-on-test epoch typically falls in 8080–100100, mirroring the headline trajectory). Six variants A7–A12 cover the three architectural contributions of Sec. I, the inter-modal alignment loss, and the choice of regression loss. Results are in Table VI. TABLE VI: Module-level retraining ablation on Nutrition5k. Each row re-trains the full architecture from scratch with the indicated module replaced or removed; everything else is byte-identical to the headline NutriVision (A1 in Tab. V, 3-seed mean PMAE 13.60±0.10%13.60± 0.10\%). Selection protocol: every row reports the best-mean-PMAE-on-the-test-split checkpoint over its 100-epoch run, matching the protocol used for A1 and disclosed in §IV. Statistical protocol: rows marked ‡ (A10) are retrained across 3 seeds; the Mean column reports mean± and Δ is a paired t-test against NutriVision (Sec. V, p<0.001p\!<\!0.001). A7–A9, A11, A12 remain single-seed and their Δ is computed against the 3-seed mean of A1 (13.60%13.60\%). Lower is better. # Configuration Cal Mass Fat Carb Prot Mean Δ Mean A7 No ingredient cross-attention in IC-FAFM 12.9 10.5 16.6 15.1 13.5 13.7 +0.10+0.10 A8 Element-wise add fusion (no FFT split) 12.8 10.3 16.8 14.8 13.4 13.6 +0.00+0.00 A9 No ISM (window-MHSA + DPE → identity) 12.6 10.3 16.4 14.6 13.1 13.4 −0.20-0.20 A10‡ Plain MLP head (IA-MPH → concat++3-layer MLP) 13.4 10.9 17.4 17.9 14.6 ±0.2615.10\!±\!0.26 +±0.17+1.50\!±\!0.17 A11 No inter-modal alignment loss (λalign=0 _align=0) 12.9 10.5 16.8 14.7 13.6 13.7 +0.10+0.10 A12 Smooth-L1 loss (replacing normalised L1) 12.6 10.3 20.6 16.0 14.6 14.8 +1.20+1.20 Take-aways 1. The ingredient-aware head is the dominant single architectural contribution (A10, Δ=+1.50±0.17 \!=\!+1.50± 0.17 p, paired t-test p<0.001p\!<\!0.001 across 3 seeds). Replacing IA-MPH with a plain MLP that concatenates [fused,rgb,depth][f_fused,c_rgb,c_depth] and runs three Dense layers is the largest controllable degradation among the architecture modules and the only module-level effect for which we have a paired statistical test. Most of the per-task hit appears in carbohydrate (→17.914.76\!→\!17.9, the most cuisine-dependent macronutrient) and fat, consistent with the head’s ingredient-conditioned gating and cross-attention being where ingredient identity most directly informs nutrient-specific predictions. 2. Confirmation that the metric-aligned loss of IGSMNet [4] is essential—not a NutriVision finding (A12, Δ=+1.20 \!=\!+1.20, single seed). The normalised ℓ1 _1 loss ℒnutriL_nutri used by NutriVision is adopted verbatim from IGSMNet (Sec. I); replacing it with a generic Smooth-L1 inflates mean PMAE by +1.20+1.20. Two caveats: (i) the loss is borrowed, so A12 measures the importance of an inherited component rather than a novel design; (i) Smooth-L1 is a deliberately weak baseline (residuals dominated by calorie/carb magnitudes), so +1.20+1.20 reflects the gap between a metric-aligned and an obviously misaligned loss, not a head-to-head win against a serious loss-design competitor. A12 therefore confirms the normalised-ℓ1 _1 choice of [4] for this benchmark family; stronger loss-side comparisons (relative-error variants, quantile/pinball losses) are future work. 3. IC-FAFM is a small, diffuse contributor (A7 Δ=+0.10 \!=\!+0.10, A8 Δ=+0.00 \!=\!+0.00, A11 Δ=+0.10 \!=\!+0.10; all single seed). All three deltas sit inside the ±0.15± 0.15 p per-run noise floor observed on completed controls. We do not claim ingredient-conditioned frequency fusion as a dominant source of NutriVision’s gains; the defensible statement is that the three mechanisms act as scaffolding around the head (IA-MPH, A10, +1.50±0.17+1.50± 0.17, p<0.001p\!<\!0.001) and the loss (A12), which is where the paired-tested gain concentrates. A dedicated multi-seed sweep of A7, A8, A11 is flagged in Sec. VII. 4. ISM is at the noise floor on Nutrition5k (A9, Δ=−0.20 \!=\!-0.20, single seed). Removing the window-MHSA + dynamic position-encoding refinement is inside the ±0.15± 0.15 p per-run noise floor. We retain ISM in the headline architecture for two reasons: (i) its parameter cost is a small fraction of the budget (Tab. I), and (i) the IGSM-family literature ([4]) consistently shows ISM contributions of −10.5\!-\!1 p on larger fine-grained recognition tasks, so we expect the benefit to surface on richer datasets even if it is below the noise floor on N5k’s 15.4 K training samples. In summary: ingredient text is the dominant non-RGB input; IA-MPH (A10) is the dominant architectural component we introduce (paired p<0.001p\!<\!0.001); the metric-aligned loss (A12) is confirmed as essential but borrowed from IGSMNet [4]; frequency fusion and ISM are useful scaffolding but sit below the per-run noise floor on Nutrition5k. VI-C Synthetic Pretraining (Negative Result) We pretrain the full NutriVision architecture on NutritionSynth-Omni-115K, an in-house procedural food dataset (115K RGB++depth++ingredient samples), using the headline recipe for 30 epochs. The model fits the synthetic distribution (mean PMAE 8.83%8.83\% on synth val), but transfer to Nutrition5k hurts: synth-pretrain → N5k finetune plateaus at mean PMAE 14.80%14.80\% vs a from-scratch 13.60±0.10%13.60± 0.10\%, regressing most on Fat (+1.35+1.35) and Mass (+1.53+1.53)—the two nutrients most affected by the sim-to-real shift. Plausible causes include under-represented stacking/occlusion and a z-buffer/DepthAnything depth-distribution gap. A domain-aligned augmentation pipeline and a slower freeze-then-unfreeze finetune are flagged in Sec. VII. VII Discussion VII-A Frequency-Domain Fusion and Ingredient Conditioning The FFT split separates layout cues (low band) from texture cues (high band); quantitatively it contributes little (A8 element-wise add is inside the noise floor). IC-FAFM’s ingredient cross-attention shifts visibly with the queried ingredient (Fig. 5), yet the corresponding A7 delta is also at noise. The bulk of the ingredient signal in NutriVision is absorbed at the head (IA-MPH, A10, +1.50±0.17+1.50± 0.17 p, p<0.001p\!<\!0.001), not at the fusion stage. We keep IC-FAFM for its lineage with OmniFood8k’s FAFM and its synergy with the head rather than for a paired-tested per-component effect. VII-B Cross-Cuisine Generalisation NutriVision is trained and evaluated exclusively on Nutrition5k [2], whose plates are captured in a controlled Western-cuisine setting. OmniFood8k [3] in contrast releases ∼ 8k predominantly Chinese-cuisine samples with a substantially different visual and nutritional distribution: denser plate composition, higher rice/noodle proportions, and a markedly different ingredient co-occurrence structure. Because our ingredient-aware components (IC-FAFM, IA-MPH) are conditioned on CLIP text embeddings whose training corpus carries a strong Western bias, and because we have not yet validated on OmniFood8k’s test split, we make no claim about cross-cuisine generalisation. Concretely, we expect the ingredient-conditioning signal to attenuate under distribution shift when the CLIP tokens for Chinese ingredients lie in a less-populated region of the joint embedding space. A dedicated cross-dataset study—training on Nutrition5k and testing on OmniFood8k, and vice versa, plus a fine-tuning variant—is under way and will be reported in a follow-up. VII-C Keeping the SSRA and the depth branch A3 and A6 are inference-time zeroings on a network trained with both modules, so they measure test-time routing-around rather than causal contribution; a from-scratch retraining without depth or SSRA is the top methodological gap. We retain both modules because (i) the A3 per-nutrient hit concentrates on volumetric nutrients (calories 12.65→12.8812.65→12.88, protein 13.35→13.6713.35→13.67), the pattern a working depth signal would produce, and (i) SSRA is the canonical depth-anchoring block of the multimodal nutrition family [3, 11, 10, 16], so retaining it preserves comparability and a drop-in path for real sensor depth on LiDAR-equipped phones. VII-D Limitations As with IGSMNet, the model uses ground-truth ingredient strings, which is appropriate for app and institutional deployments but precludes photo-only use. Depth and SSRA are not retrained-ablated A3 and A6 are inference-time zeroings on a network trained with both modules, so their small deltas measure routing-around rather than causal contribution. The FFT cutoff τ=0.20τ=0.20 inherited from OmniFood8k is unswept. Multi-seed retraining covers the headline and A10 (p<0.001p\!<\!0.001). A7–A9, A11, A12 are single-seed with |Δ|≤0.20| |\!≤\!0.20 within the ∼ 0.150.15 p noise floor. The dual ConvNeXt-Base backbone is heavy (∼ 196 M) and the baselines in Table IV range from ∼ 24–176 M without a parameter-matched control, so part of the mean-PMAE gap is due to scale, not architecture. The best-on-test selection matches the IGSMNet and Kwan protocols but is optimistic. Two missing controls, text-only and text-plus-area regressors, would bound how visual A5’s RGB hit is. Finally, the IGSMNet row is a best-effort reimplementation (no official code available), so the 0.510.51 p gap to their publication 15.0%15.0\% might be due to details we failed to reproduce. VIII Conclusion and Future Work NutriVision estimates nutrition from a single image of RGB, estimated depth, and ingredient text using IC-FAFM, IA-MPH, and an ISM-enhanced dual ConvNeXt-Base backbone. On Nutrition5k, we achieve a 3-seed mean PMAE of 13.60±0.10%13.60± 0.10\%, 0.890.89p better than our IGSMNet reimplementation (p!<!0.001p!<!0.001) and the lowest among published baselines considered. Paired 3-seed ablations show that IA-MPH is the dominant architectural contribution (+1.50±0.17+1.50± 0.17, p, p<0.001p<0.001), and input ablations show that ingredient text is the dominant non-RGB signal and co-essential with RGB. Mass estimation is still behind Kwan, OmniFood8k, and IGSMNet, consistent with their real depth or larger backbone advantage. Future work will include better validation with no-depth/SSRA retraining, paired 3-seed evaluation on A7-A9/A11/A12, text-only and text-plus-area baselines, held-out validation split, parameter-matched IGSMNet, and τ sensitivity analysis. Extensions include visual-to-CLIP ingredient prediction [21] for photo-only deployment, shared-backbone adapters, multi-view test-time fusion, EMA averaging, and domain-aligned synthetic pre-training. Acknowledgments We thank the MIDAS lab at Indraprastha Institute of Information Technology Delhi for compute support on the RTX A6000 cluster. References [1] W. Min, S. Jiang, L. Liu, Y. Rui, and R. Jain (2019) A survey on food computing. Acm Computing Surveys (CSUR) 52 (5), p. 1–36. Cited by: §I. [2] Q. Thames, A. Karpur, W. Norris, F. Xia, L. Panait, T. Weyand, and J. Sim (2021) Nutrition5k: towards automatic nutritional understanding of generic food. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 8903–8911. Cited by: §I, §I-A, §IV-A, TABLE IV, §VII-B. [3] D. Yu, W. Min, Q. Jiang, X. Lin, X. Jin, and S. Jiang (2026) OmniFood8K: single-image nutrition estimation via hierarchical frequency-aligned fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 41562–41572. Cited by: §I, §I, §I-B, §I-C, §I-B, §I-E, §I-F, §I-H, TABLE IV, §VII-B, §VII-C. [4] D. Zhang, W. Shi, B. Ma, W. Min, and X. Wu (2025) IGSMNet: ingredient-guided semantic modeling network for food nutrition estimation. Foods 14 (21). External Links: Link, ISSN 2304-8158, Document Cited by: §I, §I, §I-D, §I-A, §I-D, §I-G, §I-H, §IV-A, §V-B, TABLE IV, TABLE IV, TABLE IV, item 2, item 2, item 4, §VI-B. [5] L. Bossard, M. Guillaumin, and L. Van Gool (2014) Food-101 – mining discriminative components with random forests. In Computer Vision – ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars (Eds.), Cham, p. 446–461. External Links: ISBN 978-3-319-10599-4 Cited by: §I. [6] A. Salvador, N. Hynes, Y. Aytar, J. Marin, F. Ofli, I. Weber, and A. Torralba (2017) Learning cross-modal embeddings for cooking recipes and food images. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 3068–3076. External Links: Document Cited by: §I, §I-D. [7] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §I, §I-D. [8] W. Shao, S. Hou, W. Jia, and Y. Zheng (2022) Rapid non-destructive analysis of food nutrient content using swin-nutrition. Foods 11 (21). External Links: Link, ISSN 2304-8158, Document Cited by: §I-A, TABLE IV. [9] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo (2021) Swin transformer: hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, p. 10012–10022. Cited by: §I-A. [10] B. Ma, D. Zhang, and X. Wu (2025) Food nutrition estimation with rgb-d fusion module and bidirectional feature pyramid network. Multimedia Systems 31, p. . External Links: Document Cited by: §I-B, §VII-C. [11] Z. Kwan, W. Zhang, Z. Wang, A. B. Ng, and S. See (2025) Nutrition estimation for dietary management: a transformer approach with depth sensing. IEEE Transactions on Multimedia. Cited by: §I-B, §I-H, §V-B, TABLE IV, §VII-C. [12] Y. Han, Q. Cheng, W. Wu, and Z. Huang (2023) DPF-nutrition: food nutrition estimation via depth prediction and fusion. Foods 12 (23). External Links: Link, ISSN 2304-8158, Document Cited by: §I-B, TABLE IV. [13] H. Lin, S. Chen, J. Liew, D. Y. Chen, Z. Li, G. Shi, J. Feng, and B. Kang (2025) Depth anything 3: recovering the visual space from any views. arXiv preprint arXiv:2511.10647. Cited by: §I-B, §IV-A. [14] Y. Rao, W. Zhao, Z. Zhu, J. Lu, and J. Zhou (2021) Global filter networks for image classification. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. W. Vaughan (Eds.), Vol. 34, p. 980–993. External Links: Link Cited by: §I-C. [15] L. Chi, B. Jiang, and Y. Mu (2020) Fast fourier convolution. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 4479–4488. External Links: Link Cited by: §I-C. [16] F. Nian, Y. Hu, Y. Gu, Z. Wu, S. Yang, and J. Shu (2024) Ingredient-guided multi-modal interaction and refinement network for rgb-d food nutrition assessment. Digital Signal Processing 153, p. 104664. External Links: ISSN 1051-2004, Document, Link Cited by: §I-D, §VII-C. [17] Z. Liu, H. Mao, C. Wu, C. Feichtenhofer, T. Darrell, and S. Xie (2022) A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 11976–11986. Cited by: §I-C, TABLE I. [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in neural information processing systems 30. Cited by: §I-E. [19] T. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie (2017) Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2117–2125. Cited by: §I-E. [20] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, et al. (2023) Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4015–4026. Cited by: §IV-A. [21] J. Li, D. Li, C. Xiong, and S. Hoi (2022) Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, p. 12888–12900. Cited by: §VIII.