Paper deep dive
OliveGemma: A 3 Billion Visual Language Model for Recognising the Mediterranean & European Diet
Dimitrios I. Zaridis, Traianos Tsiokris, Vasileios C. Pezoulas, Daphni Plati, Eugenia Mylona, Eleni Georga, Nikos Tsiknakis, Antonis Sakellarios, Dimitrios I. Fotiadis
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Image based dietary assessment offers a scalable alternative to self reported food diaries, yet fine-grained food recognition remains challenging due to high intra-class variability and visually similar dishes. This study presents OliveGemma, a vision language model for recognising and reasoning about Mediterranean and European cuisine. Built on the open-weight PaliGemma-2-3B architecture, OliveGemma is fine-tuned with LoRA on a unified corpus of 17,340 images from three European research project datasets (MedGR, ODIN, and VIPPSTAR), reconciled into a vocabulary of 216 composed dish categories and paired with 102,642 instruction style question-answer items covering dish recognition, likely and visible ingredients, class boundary discrimination, visual evidence and overall visual food understanding. Under a 3-fold cross-validation scheme, OliveGemma achieves a top-1 accuracy of 92.96% +/- 0.91%, exceeding the strongest CNN baseline (DenseNet-121) by 7.31% and outperforming zero-shot frontier models with exact instructions and bounded classes including Gemini Flash 3 and 3.5, GPT-5.4 Mini, and Claude Haiku 4.6 by 8%, 46%, and 64% respectively. Furthermore, OliveGemma demonstrates competitive performance on Top-3 and Top-5 accuracy, being second best across CNNs and frontier models, surpassed only by DenseNet-121. In addition, OliveGemma achieves 90.79% +/- 1.3% Exact-Set on the likely ingredients of the food categories. These results demonstrate that PEFT adaptation of a small VLM can surpass substantially larger proprietary models on specialised food recognition. The model is publicly available at this https URL and the experiments and results can be found at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.03428v1
- Canonical: https://arxiv.org/abs/2608.03428v1
Trouble viewing inline? Open PDF directly â
Full Text
67,230 characters extracted from source content.
Expand or collapse full text
OLIVEGEMMA: A 3 BILLION VISUALâLANGUAGE MODEL FOR RECOGNISING THE MEDITERRANEAN & EUROPEAN DIET Dimitrios I. Zaridis Unit of Medical Technology & Intelligent Information Systems University of Ioannina Ioannina, Greece dimzaridis@gmail.com Traianos Tsiokris* Unit of Medical Technology & Intelligent Information Systems University of Ioannina Ioannina, Greece akistsiokris@gmail.com Vasileios C. Pezoulas Unit of Medical Technology & Intelligent Information Systems University of Ioannina Ioannina, Greece bpezoulas@gmail.com Daphni Plati Unit of Medical Technology & Intelligent Information Systems University of Ioannina Ioannina, Greece daphni.plati@gmail.com Eugenia Mylona Unit of Medical Technology & Intelligent Information Systems University of Ioannina Ioannina, Greece & Department of Medical Physics, School of Medicine University of Patras Patras, Greece mylona.eugenia@gmail.com Eleni Georga Unit of Medical Technology & Intelligent Information Systems University of Ioannina Ioannina, Greece & egeorga@uoi.gr Nikos Tsiknakis Computational BioMedicine Laboratory Foundation for Research and Technology Hellas Heraklion, Greece tsiknakisn@ics.forth.gr Antonis Sakellarios Dept. of Mechanical and Aeronautics Engineering University of Patras Patras, Greece & asakellarios@upatras.gr Dimitrios I. Fotiadis Unit of Medical Technology & Intelligent Information Systems University of Ioannina Ioannina, Greece & Biomedical Research Institute FORTH Ioannina, Greece fotiadis@uoi.gr ABSTRACT Image-based dietary assessment offers a scalable alternative to error-prone self-reported food diaries, yet fine-grained food recognition remains challenging due to high intra-class variability, visually simi- lar dishes, long-tailed distributions, and the compositional nature of plated meals. This study presents OliveGemma, a domain-adapted visionâlanguage model for recognising and reasoning about Mediter- ranean and European cuisine. Built on the open-weight PaliGemma-2-3B architecture, OliveGemma is fine-tuned with Low-Rank Adaptation (LoRA) on a unified corpus of 17,340 images from three European research project datasets (MedGR, ODIN, and VIPPSTAR), reconciled into a vocabulary of 216 composed dish categories and paired with 102,642 instruction-style questionâanswer items cov- ering dish recognition, likely and visible ingredients, class-boundary discrimination, visual evidence and overall visual food understanding. Only 23.75M parameters (0.78% of the 3B backbone) are arXiv:2608.03428v1 [cs.CV] 4 Aug 2026 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet updated, yielding aâ 90MB adapter deployable on commodity hardware (CPU, 16 GB RAM), being open-source. Under a three-fold cross-validation scheme with an identical closed-vocabulary protocol, OliveGemma achieves a top-1 accuracy of92.96± 0.91%, exceeding the strongest CNN baseline (DenseNet-121) by7.31%and outperforming zero-shot frontier models with exact instructions and bounded classes (same 216 classes as OliveGemma and CNNs) including Gemini Flash 3 and 3.5, GPT-5.4 Mini, and Claude Haiku 4.6 byâ 18%, 46%, 64%, respectively. Furthermore, OliveGemma demonstrates competitive performance on Top-3 and Top-5 accuracy, being the second best across CNNs and Frontier models, only surpassed by DenseNet-121. In addition, OliveGemma achieves 90.79± 1.3%Exact-Set in Likely Ingredients of the food categories, detecting effectively the visually evident ingredients presented in the dish. These results demonstrate that parameter-efficient adapta- tion of a small open-weight VLM can surpass substantially larger proprietary systems on specialised food recognition while enabling privacy-preserving, reproducible deployment for dietary assessment. The model is publicly available athttps://huggingface.co/JamesZar/OliveGemma-3B, under the name JamesZar/OliveGemma-3B, while the experiments and the results shall be found from the following git repository https://github.com/tsiokris/OliveGemma. Keywords Large Language Models; Visual Language Models; Fine tuning; Food Recognition 1 Introduction Diet is widely recognised as one of the most important modifiable determinants of long-term health. Among healthy dietary patterns, the Mediterranean diet has consistently been associated with a reduced risk of cardiovascular disease, several chronic conditions, and improved longevity [1]. However, translating this evidence into clinical practice and everyday life requires accurate and practical methods for monitoring dietary intake. Traditional dietary assessment methods, such as self-reported food diaries and dietary recalls, are time consuming and sensitive to reporting and recall bias. Consequently, image based dietary assessment has emerged as a scalable alternative, enabling automatic analysis of meal photographs to identify consumed foods and estimate nutritional intake [2]. Automated food recognition forms the core component of such systems and has applications in clinical nutrition monitoring, consumer food-logging platforms and intelligent food service technologies [3]. Despite the progress in computer vision, food recognition remains a challenging task. Unlike many conventional object recognition problems, food images exhibit high intra-class variability, as the same dish may differ substantially in appearance depending on ingredients, preparation methods, plating style, or lighting conditions. On the other hand, visually similar dishes often belong to different categories, resulting in low inter-class variability. Food datasets also exhibit a pronounced long-tail distribution, where a small number of common dishes dominate while many categories contain relatively few examples. An additional challenge arises from the compositional nature of meals, as a single plate frequently contains multiple food components rather than a single homogeneous dish. These challenges become even more pronounced when combining heterogeneous datasets originating from different countries, languages, and annotation protocols, where identical dishes may appear under different names or different levels of semantic granularity. Early research addressed food recognition as a closed-vocabulary image classification problem, where convolutional neural networks (CNNs) are trained to assign each image to one predefined class. Benchmarks such as Food-101 [4] and later Food2K [5] established this paradigm and enabled substantial improvements in recognition accuracy. Nevertheless, conventional image classifiers remain inherently limited by their fixed label space, producing only a single class prediction without providing additional semantic understanding of the image. Recent advances in vision-language models (VLMs) have introduced a more flexible alternative by combining a pretrained visual encoder with a large language model capable of generating natural language responses. Architectures such as BLIP-2 [6], LLaVA [7], Qwen2.5-VL [8], and PaliGemma-2 [9] extend image recognition beyond fixed classification by supporting instruction following, open ended visual understanding and reasoning about image content. These capabilities are prominent for food analysis, where recognising a dish may also require distinguishing visually similar meals, identifying ingredients, or explaining the visual evidence supporting a prediction. Despite these advances, several limitations remain. General-purpose VLMs are not specifically trained for fine grained food recognition and often struggle to distinguish visually similar dishes, particularly when operating in specialised domains such as Mediterranean cuisine. In addition, the strongest commercial VLMs are accessible only through proprietary cloud based APIs, raising concerns regarding reproducibility, deployment cost, and privacy, particularly in clinical applications where patient images should remain within institutional infrastructure. Although full fine-tuning can adapt a foundation model to a specialised domain, updating billions of parameters is computationally demanding and increases the risk of catastrophic forgetting [10]. Parameter efficient finetuning methods, particularly Low-Rank Adaptation (LoRA), provide an attractive alternative by updating only a small fraction of the model parameters while 2 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet preserving the pretrained backbone [11,12]. However, the application of LoRA-based adaptation to composed, cross- dataset food recognition has received limited attention, and comparisons against both conventional CNN classifiers and contemporary frontier VLMs under identical evaluation conditions remain limited. Motivated by these limitations, this study presents OliveGemma, a domain adapted vision language model based on the open-weight PaliGemma-2-3B architecture. The model is finetuned using LoRA on a unified corpus of 17,340 food images collected from the MedGR, ODIN, and VIPPSTAR datasets and reconciled into a canonical vocabulary of 216 composed dish categories. By updating only 23.75 million trainable parameters (0.78% of the approximately three billion parameters of the original model), OliveGemma provides an efficient and reproducible solution for fine-grained food recognition while preserving the advantages of an open-weight deployment. Experimental evaluation demonstrates that the proposed approach achieves competitive recognition performance, outperforming several established CNN architectures and substantially exceeding the zero-shot performance of state of the art proprietary vision language models evaluated under an identical closed vocabulary protocol. 1.1 Related Work 1.1.1 Convolutional Neural Networks Food Image Recognition Automated food recognition has traditionally been formulated as a closed vocabulary image classification task and has advanced alongside the development of increasingly large datasets. Food-101 [4] established the standard benchmark for 101 Western dishes, while the UEC-Food datasets expanded the problem to Japanese cuisine [13]. Larger datasets, including ISIA Food-500 [14] and Food2K [5], further increased the label space to 500 and 2,000 food categories, respectively. Complementary resources such as Recipe1M [15] introduced paired recipe-image supervision, enabling multimodal learning for food understanding. Furthermore, especially for Mediterranean diet, there is the MedGR dataset, used in this analysis, with over 50K images depicting food plates of greek-italian cuisine [16]. Despite their scale and diversity, these datasets share a common assumption that each image is assigned to a single predefined class within a flat taxonomy. Consequently, they do not support composed meals containing multiple food components, nor do they address the reconciliation of heterogeneous food vocabularies originating from different countries. On the modeling side, CNNs have been the dominant approach for food image classification. Architectures such as ResNet [17], Inception (GoogLeNet) [18], DenseNet [19], and EfficientNet [20] have consistently demonstrated strong recognition performance, while more recent studies have adopted Vision Transformers as image classifiers [21]. Given the fine-grained nature of food recognition, several works have incorporated attention mechanisms and second-order or bilinear pooling techniques to capture subtle visual cues that distinguish highly similar dishes [22]. Although these methods achieve high accuracy within a fixed taxonomy, they remain inherently limited by the closed-vocabulary classification paradigm. Their predictions are restricted to predefined labels and they cannot explain or justify their decisions. 1.1.2 Vision Language Models and Instruction Tuning Recent advances in VLMs have shifted visual recognition from fixed-label classification toward open-ended language generation. This stack is built upon contrastively pretrained vision encoders, most notably CLIP [23] and its successor SigLIP [24], which align images and text within a shared embedding space. Generative VLMs combine these visual encoders with large language models to enable multimodal reasoning and natural language responses. BLIP-2 [6] bridges the visual and language components through a lightweight querying transformer, whereas instruction-tuned models such as LLaVA [7], the Qwen-VL family [25,8], InternVL [26], PaliGemma and PaliGemma-2 [27,9] extend this framework by following natural language instructions and reasoning over visual content. Among these architectures, PaliGemma-2 provides several characteristics that make it suitable for domain adaptation because it combines a SigLIP vision encoder with a Gemma-2 language decoder through a lightweight linear projection layer and is released as an open-weight model at a practical 3B parameter scale. Unlike chat-oriented assistants, PaliGemma-2 is designed as a transferable vision language foundation model and achieves strong performance across a wide range of downstream vision language tasks after task-specific finetuning [9,27]. These characteristics make it suitable backbone candidate for developing reproducible and locally deployable food recognition systems. Despite the rapid progress of VLMs, important challenges remain. General purpose models frequently underperform in specialised fine-grained domains when evaluated in the zero-shot setting, particularly when subtle visual differences separate food categories. Furthermore, many state-of-the-art commercial VLMs are accessible only through proprietary cloud APIs, limiting reproducibility while introducing deployment costs and privacy concerns. These limitations are especially relevant in clinical dietary assessment, where patient images may contain protected health information and therefore processing of those data may be prohibited by external services. 3 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet 1.1.3 Parameter-Efficient Fine-Tuning and Domain Adaptation Fully fine-tuning a multi billion parameter VLM is computationally expensive and risks catastrophic forgetting of the backboneâs general capabilities [10]. On the other hand parameter-efficient finetuning (PEFT) updates only a small set of added or selected parameters while the backbone stays frozen. The PEFT family includes bottleneck adapters [28], prefix and prompt-tuning [29,30], and Low-Rank Adaptation (LoRA) [11], grounded in the hypothesis that the weight update required to adapt a pretrained model has low intrinsic rank, together with its quantised variant QLoRA [12]. For multimodal foundation models, PEFT is frequently combined with staged adaptation strategies in which the language model is first aligned to the target task before selectively unfreezing components of the vision encoder. This approach has become a common strategy for transforming general purpose VLMs into domain-specific models. For instance, LLaVA-Med adapts LLaVA for biomedical image understanding [31], and GeoChat, which applies LoRA- based fine-tuning to remote sensing imagery while demonstrating that low-rank adaptation preserves the backbone modelâs general capabilities [32]. OliveGemma follows the same general pattern by adapting an open-weight vision-language model to the food domain using LoRA. However, to the best of our knowledge, no previous study has applied this strategy to composed label food recognition across multiple heterogeneous datasets, nor evaluated the resulting model against both conventionally trained CNN classifiers and zero-shot frontier VLMs using an identical dataset, label space, and evaluation protocol. 1.2 Contributions The main contributions of this study are summarized as follows. âą We propose an open-weight vision-language model, OliveGemma, based on PaliGemma-2-3B, adapted using LoRA for fine-grained composed food recognition. The model is trained on a unified dataset of 17,340 images compiled from the MedGR, ODIN, and VIPPSTAR datasets and reconciled into a canonical vocabulary of 216 food categories. âąWe perform a comparative evaluation against both conventional image classification models and existing vision-language models. The benchmark includes fine-tuned CNN architectures, OliveGemma, and zero-shot frontier VLMs from Google (Gemini), OpenAI (GPT), and Anthropic (Claude), all evaluated using the same dataset split and label space to isolate the effect of domain adaptation. âąWe extend food recognition beyond single-label classification through multimodal reasoning capabilities, including ingredient prediction and natural language generation of visual evidence and visible ingredients, supporting richer dietary assessment and explainable food analysis. âąOliveGemma can be deployed on local premises and can operate on CPU with 16GB of RAM, enabling privacy-preserving operalization. 2 Methodology 2.1 Overall Workflow Figure 1 illustrates the overall workflow followed in this study. Initially, the MedGR, ODIN, and VIPPSTAR datasets were collected, harmonized, and structured to support a unified experimental pipeline. A 3 fold cross validation scheme was then applied while preserving the original class imbalance and ensuring sufficient representation of all food categories across the three folds. For the CNN baseline experiments, four established image classification architectures pretrained on the Food-101 dataset were selected to leverage prior domain knowledge of food images. Each model was finetuned and evaluated independently on every fold. The proposed OliveGemma model was fine-tuned using question-answer (QA) pairs generated from the annotations already available in the employed datasets together with knowledge distilled from Anthropic Claude 4.6 Opus. Ap- proximately 61,071 QA pairs were generated for each fold. The training corpus was designed to capture multiple aspects of food understanding, including food recognition, visual reasoning between visually similar dishes, likely ingredients, visual evidence supporting the predicted class, food groups, cuisine identification, preparation steps, and uncertainty-aware responses. The latter encourages the model to explicitly acknowledge information that cannot be reliably inferred from a single image, reducing overconfident hallucinations during inference. To compare OliveGemma against frontier vision-language models, OpenAI GPT-5.4 Mini, Google Gemini Flash 3 and Gemini Flash 3.5, and Anthropic Claude Haiku 4.6 were evaluated. To ensure a consistent evaluation protocol, all models were prompted using the same system prompt, instructing them to act as food recognition experts. Furthermore, 4 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet Figure 1: The overall workflow of this study the complete canonical class vocabulary was provided, and each model was constrained to return exactly one food category from the predefined label set, thereby isolating the effect of model capability from differences in prompting or output formatting. 2.2 Dataset Description The dataset used in this study was constructed by merging three heterogeneous datasets, MedGR, ODIN, and VIPPSTAR each originating from a separate European research project. Combining them produced a single image collection D = (I i , s i , d i ) N i=1 ofN = 17,340images, wheres i denotes the source andd i the dish label. Across the raw material there were 250 distinct dish folders, but many of these described the same food under different languages or different levels of detail. For example, the Italian pollo and the English chicken refer to an identical dish yet appear as separate folders. To remove this redundancy, the source folders were reconciled into one canonical recognition vocabularyCwith|C| = 216composed dish labels, listed in appendix (Table 5). Every canonical label describes a complete plate and may itself comprise several components, so the recognition gold standardd i âCassigned to each image corresponds to this whole-plate canonical label. Every image was converted to RGB, minâmax normalised to [0, 255] and resized to the 448Ă 448 input expected by the SigLIP encoder. Each image is paired with one or more instruction-style questionâanswer items. The dish label itself was taken directly from the original dataset annotations, whereas the remaining contextual information about ingredients was distilled using Claude Opus 4.6 as a teacher model. Five attribute types relevant to recognition and fine-grained reasoning were retained for fine-tuning: the dish name (the food recognition task), the likely ingredients, the class boundary that separates a dish from a similar one, the visible ingredients, and the supporting visual evidence. This procedure yielded102,642QA pairs in total. Each training instance takes the form of a triplet(I,x,y)comprising the image, the promptxand the target answery, with the dish-name recognition task covering all216canonical classes ofC. 5 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet Recognition supervision was available only for the17,340images carrying a canonical dish-name answer, the remaining images contributing solely to the auxiliary attributes. These17,340images formed the recognition evaluation set used throughout the cross-model comparisons, and their assignment to folds was held fixed. 2.3 OliveGemma Fine Tuning Strategy The proposed model adapts the open-weight vision-language model PaliGemma-2-3B to the task of fine-grained food recognition and attribute reasoning. PaliGemma combines a SigLIP vision encoder with a Gemma-2 autoregressive language decoder through a linear projection layer that maps visual embeddings into the language modelâs token space. Given an input imageIof size448Ă 448and a textual promptx, the vision encoder produces1,024visual tokens that are prepended to the tokenized prompt. The decoder subsequently generates the target responsey = (y 1 ,...,y T )in an autoregressive manner. Training follows the standard supervised objective used by decoder-only language models. LetÎdenote the complete set of model parameters. The optimization objective is the next-token cross-entropy loss, computed only over the target answer tokens, while the image and prompt tokens are excluded from the loss calculation: L(Î) = â T X t=1 logp Î (y t |y <t , I, x). Here,p Î (y t | y <t ,I,x)denotes the probability assigned to the next token given the input image, textual prompt, and previously generated tokens. The sequencey <t represents all target tokens preceding positiont, andTdenotes the length of the target response. Rather than updating all approximately3.03Ă 10 9 model parameters, parameter-efficient fine-tuning is performed using Low-Rank Adaptation (LoRA). This approach freezes the pretrained backbone and learns only a small set of low-rank weight updates, reducing memory requirements while mitigating catastrophic forgetting. For every adapted projection matrix W 0 âR dĂk , the forward computation becomes h = W 0 z + âW z = W 0 z + α r BAz, B âR dĂr , AâR rĂk , r âȘ min(d,k), In this formulation,W 0 denotes the frozen pretrained weight matrix,zis the input activation, andhis the corresponding output activation. The trainable matricesAandBconstitute the LoRA adaptation, whose product has rank at mostr. Consequently, the number of trainable parameters is reduced from dk to r(d + k) for each adapted projection. The LoRA configuration follows the standard parameterization with rankr = 16, scaling factorα = 32, corresponding to an effective scaling ofα/r = 2, and a dropout rate of 0.05 applied to the low-rank branch. MatrixAis initialized from a zero-mean Gaussian distributionN (0,Ï 2 ), whileBis initialized to zero. Consequently,âW = 0at initialization, ensuring that optimization starts exactly from the pretrained PaliGemma-2 model. Throughout fine-tuning, only the LoRA parametersA,Bare updated, whereas the vision encoder, projection layer, and original language model weights remain frozen. 2.3.1 Training & Frozen Components LoRA modules are inserted into seven projection layers of every Gemma-2 decoder block which are the four self- attention projectionsW q ,W k ,W v ,W o and the three feed-forward (SwiGLU) projectionsW gate ,W up ,W down . Figure 2 illustrates the resulting architecture together with the decomposition of trainable and frozen components. The token embedding layer, multimodal projection layer, and language modeling head remain frozen throughout training. The SigLIP vision encoder follows a two-stage optimization schedule designed to first align the language decoder with the target output format before adapting the visual representations. During the firstS = 1,500optimization steps, the entire vision encoder is frozen, such thatâ Ξ v L⥠0. After this warm-up phase, the encoder is unfrozen and optimized jointly with the decoder using a learning rate reduced by a factor of ten, i.e., η v = Ïη with Ï = 0.1. This strategy results in only 23,752,704 trainable parameters, corresponding to approximately 0.78% of the full PaliGemma-2 backbone (3.03 billion parameters). The resulting LoRA adapter occupies approximately 90 MB of storage, enabling efficient training and deployment while preserving the pretrained model weights. |Ξ train | = 23,752,704 |z LoRA â 0.78% of|Î|, 6 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet Figure 2: Parameter-efficient fine-tuning of PaliGemma-2-3B. The food image is encoded by SigLIP (frozen for 1,500 steps, then fine-tuned at0.1Ăthe base learning rate), projected into the decoder token space (frozen projector and embeddings), and decoded by Gemma-2. Only rank-16 LoRA adapters on the attentionq,k,v,oand MLP gate, up, downprojections are trained, therefore the effective training parameter kept at23.75M (0.78%of the3B backbone). 2.4 CNN Models To establish conventional closed-vocabulary baselines, four representative CNN architectures were selected, namely, the ResNet-50 [17], the EfficientNet-B3 [20], the MobileNet-V3-Large [33], and the DenseNet-121 [19]. These models represent the principal families of image classification architectures while spanning different trade-offs between accuracy, computational cost, and parameter efficiency. ResNet-50 serves as the reference residual architecture, employing bottleneck residual blocks to facilitate stable optimization and producing a 2,048-dimensional pooled feature representation. EfficientNet-B3 represents the family of compound scaled networks, jointly scaling depth, width, and input resolution to achieve a favorable accuracy to computation trade off, offering 1536 embeddings. MobileNet- V3-Large combines inverted residual blocks, depthwise separable convolutions, squeeze-and-excitation modules, and hard-swish activations to provide an architecture optimized for mobile and edge devices, producing 1,280 image embeddings. DenseNet-121 employs densely connected convolutional blocks that promote feature reuse and efficient gradient propagation, generating a 1,024 embeddings that has proven effective for fine grained visual recognition tasks. To ensure that performance differences are attributable to the backbone architecture rather than the training procedure, all models follow an identical transfer learning protocol. Networks are initialized using the ImageNet-1K pretrained weights, while the convolutional feature extractor remains frozen throughout training. Only the final linear classification layer is optimized, mapping the pooled feature representation to the 216 food categories. Consequently, each model updates between approximately 255,000 and 510,000 trainable parameters. Input images are resized to224Ă 224pixels 7 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet and normalized using the standard ImageNet mean and standard deviation. Data augmentation consists of random resized cropping and random horizontal flipping during training. Optimization is performed using Adam with a learning rate of1Ă 10 â3 , a batch size of 32, and a training duration of 10 epochs. The checkpoint achieving the lowest validation loss is retained for evaluation. Following the evaluation protocol described in subsection 2.5, all CNN baselines are trained and evaluated using the same three-fold cross-validation splits employed for OliveGemma. Performance is reported using top-1, top-3, and top-5 accuracy on the held-out test fold, ensuring a consistent comparison across all evaluated models. 2.5 Evaluation The evaluation protocol was designed to ensure a consistent and fair comparison across all modelling approaches, including the CNN baselines, OliveGemma, and the frontier VLMs. A three-fold cross-validation scheme was adopted, where in each iteration one fold was used for training, one for validation, and the remaining fold for testing. The assignment of samples to each fold remained fixed throughout all experiments, ensuring that every model was evaluated on exactly the same training, validation, and test splits. To enable a fair comparison with frontier VLMs, the same evaluation protocol was applied to the Gemini, GPT, and Claude model families. All models received an identical system prompt instructing them to act as food recognition experts. Furthermore, the complete canonical class vocabulary was provided, and the models were constrained to predict only a food category belonging to this predefined label set. This converted the otherwise open vocabulary models into a closed-vocabulary classification setting directly comparable with the CNN baselines and OliveGemma. Model performance was evaluated using top-1, top-3, and top-5 accuracy. Given the relatively large number of food categories and the visual similarity among many dishes, these metrics provide a more informative assessment than top-1 accuracy alone by accounting for correct predictions appearing among the highest-ranked candidate classes. 3 Results 3.1 Comparison with CNN Baselines Table 1: Performance Evaluation across the 3-fold Cross-Validation Scheme between CNN models and OliveGemma ModelTop-1 (%)Top-3 (%)Top-5 (%) ResNet-5080.72 ± 0.8692.96 ± 1.1795.64 ± 1.16 EfficientNet-B377.08 ± 0.5691.26 ± 1.1694.77 ± 0.96 MobileNet-V3-Large83.64 ± 1.1295.01 ± 0.6897.05 ± 0.47 DenseNet-12185.65 ± 0.3895.82 ± 0.5697.87 ± 0.23 OliveGemma92.96 ± 0.9195.73 ± 0.3896.10 ± 0.44 The results in Table 1 reveal a clear ordering among the convolutional baselines. DenseNet-121 emerges as the strongest CNN at top-1 (85.65±0.38%), followed by MobileNet-V3-Large (83.64±1.12%), ResNet-50 (80.72±0.86%) and EfficientNet-B3 (77.08±0.56%). This ranking does not track raw model capacity or ImageNet performance, where EfficientNet-B3 and ResNet-50 typically lead, which is informative in itself. Because every backbone is frozen and only the linear head is trained, the comparison reflects the quality of each networkâs pretrained features for fine-grained food discrimination rather than its capacity to learn new representations. The advantage of DenseNet-121 is therefore consistent with its dense connectivity pattern, which promotes feature reuse across layers and tends to preserve the local texture and colour cues that separate visually adjacent dishes. The comparatively weak showing of EfficientNet-B3, despite its strong supervised ImageNet accuracy, suggests that compound-scaled features optimised for generic object categories transfer less readily to the texture-driven, compositional structure of plated food when the backbone cannot be adapted. A second observation concerns the gap between top-1 and the deeper ranks. All four CNNs recover sharply at top-3 and top-5, with DenseNet-121 reaching 95.82±0.56% and 97.87±0.23% respectively. The size of this jump, more than ten percentage points between top-1 and top-3 for every model, indicates that the correct label is usually present within the networkâs shortlist even when it is not promoted to first place. In other words, the frozen features carry enough signal to narrow each plate to a small candidate set, but the linear head alone cannot resolve the final fine-grained decision among visually similar classes. This pattern motivates the comparison that follows: where the CNN shortlists are well-formed yet the rank-1 decision is unreliable, the language conditioned predictions of OliveGemma are expected to convert that latent discriminative signal into a correct top-1 answer more often. Notably, OliveGemmaâs top-1 of 8 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet 92.96±0.9192.96±0.91% exceeds the best CNN by 7.31 points, while the CNN baselines retain a marginal edge at top-5, a trade-off examined in the Discussion. 3.2 Comparison with Gemini, OpenAI and Anthropic Proprietory Family Models OliveGemma was further evaluated against several proprietary frontier vision-language models from the Google (Gemini), OpenAI, and Anthropic model families. All proprietary models were evaluated in the zero-shot setting without any task-specific fine-tuning on the employed food datasets. To ensure a consistent comparison, every model was evaluated using the identical 3 fold cross-validation splits employed for OliveGemma and the CNN baselines. Furthermore, all models operated under the same closed-vocabulary protocol based on the canonical set of 216 food categories. The complete label vocabulary was incorporated into the system prompt and models were instructed to return a single line response whosecandidatesfield is a ranked top-5list in which every entry must be copied verbatim from the closed set, no invented, translated, split, or recombined labels, and composed multi-component plates (e.g. salmon, leek mashed potatoes) treated as one indivisible label. Decoding is deterministic (temperature0), the response budget is512tokens, images are sent at a maximum side of1,024pixels, and Gemini-3, OpenAI and Claude âthinkingâ is disabled (thinking_budget=0) so that no reasoning tokens leak into or truncate the structured output. Figure 3 presents the complete system prompt used throughout the evaluation. Performance was assessed using the same top-1, top-3, and top-5 exact-match metrics adopted for all experiments. Moreover, during inference each model receives this system instruction together with the food image and a recognition question (e.g. âWhat is the name of this dish?â), and the rankedcandidateslist is scored with the same top-1/top-3/top-5exact match used throughout. This protocol transformed otherwise open-vocabulary vision-language models into a directly comparable closed-vocabulary classification setting. Table 2 summarizes the average performance across the three cross-validation folds. OliveGemma achieved a mean top-1 accuracy of92.96± 0.91%, substantially outperforming all evaluated proprietary models. The strongest proprietary baseline, Gemini 3.5 Flash, achieved74.80± 0.45%top-1 accuracy, approximately 18 percentage points below OliveGemma. Similar improvements were observed over Gemini 3 Flash, while the performance gap increased considerably for OpenAI ChatGPT 5.4 Mini and Claude Haiku 4.6. The differences become smaller when Table 2: 3-fold dish-recognition comparison between zero-shot Gemini, OpenAI and Anthropic family models and OliveGemma, on the identical216-class closed vocabulary and test splits across folds. All models are scored with top-1/top-3/top-5 exact match. ModelTop-1 (%)Top-3 (%)Top-5 (%) Gemini 3 Flash74.17 ± 0.3589.76 ± 0.1692.76 ± 0.34 Gemini 3.5 Flash74.80 ± 0.4590.43 ± 0.0293.25 ± 0.19 OpenAI ChatGPT 5.4 Mini46.35 ± 1.9965.96 ± 2.2772.30 ± 2.02 Claude Haiku 4.628.61 ± 1.2642.29 ± 1.4247.59 ± 1.21 OliveGemma92.96 ± 0.9195.73 ± 0.3896.10 ± 0.44 considering top-3 and top-5 accuracy. Both Gemini models frequently included the correct food category among their highest-ranked predictions but failed to consistently assign it the highest confidence. This behavior is expected in fine-grained food recognition, where visually similar dishes often differ only by subtle ingredients, preparation methods, or presentation. In contrast, OliveGemma was specifically adapted to this domain through LoRA fine-tuning, enabling more reliable discrimination between closely related Mediterranean and European food categories. The comparatively lower performance of ChatGPT 5.4 Mini and Claude Haiku 4.6 further illustrates that general-purpose vision-language capabilities alone are insufficient for this task. Although these models demonstrate strong general visual understanding, they were not optimized for distinguishing highly similar food classes within a specialized closed vocabulary. These findings highlight the importance of domain adaptation and indicate that parameter-efficient fine-tuning can enable relatively small open-weight models to outperform substantially larger proprietary systems on specialized food recognition tasks. 3.3 Likely Ingredients Reasoning Beyond closed-vocabulary dish recognition, OliveGemma is trained to enumerate the likely ingredients of a plate, an auxiliary attribute that probes whether the model has acquired genuine food understanding rather than a surface image-to-label mapping. Each reference answer, distilled from the teacher model, is a structured list split into a visible- typical set (ingredients that should be discernible in the image) and a commonly-present but not visually confirmable set (ingredients implied by the dish but typically hidden, e.g. salt, yeast, or egg yolks). Because the answer is an unordered set rather than a single token, exact string match is uninformative. Therefore every prediction and reference is parsed 9 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet [SYSTEM INSTRUCTION -- sent with every dish_name call] You are a careful food-recognition expert. You will see one image of a plated dish and one question about it. Answer concisely and exactly in the requested format. TASK: identify the dish in the image by choosing from the CLOSED label set below. OUTPUT - follow EXACTLY: - Output ONE single line of JSON and NOTHING else. - Do NOT think out loud, do NOT explain, do NOT add any text, notes, or markdown fences before or after the JSON. - The JSON must be this exact schema: "candidates": ["<best>", "<2nd>", "<3rd>", "<4th>", "<5th>"] - âcandidatesâ is your ranked top-5: exactly 5 labels ordered from MOST to LEAST likely (use fewer only if the set has fewer than 5 labels). - The FIRST element is your single best answer. - Every element MUST be copied VERBATIM from the closed set below - same lowercase, same spelling, same punctuation. Do NOT invent, translate, split, or recombine labels. A label that reads as a comma-separated plate (e.g. âsalmon, leek mashed potatoesâ) is ONE indivisible label: copy it whole. - No duplicates in the list. CLOSED DISH LABEL SET (216 labels, lowercase): - almond cream cake - almonds - anthotyro - ... [ 213 further canonical labels ] ... - zucchini fritters [USER CONTENT] : <food image> + <question, e.g. "What is the name of this dish?"> Figure 3: System prompt and inputs given to Gemini, OpenAI and Anthropic models. The complete216-class closed vocabulary is appended to the system instruction while the model must return a ranked top-5of labels copied verbatim from this set. The same closed-vocabulary protocol is applied to all frontier models for fairness. into ingredient sets and grade the prediction against the reference with sample-averaged set-overlap metrics: precision, recall, F 1 , the Jaccard index, and the stricter exact-set rate (the fraction of images for which the predicted set matches the reference set exactly). Table 3 reports the results where, OliveGemma recovers the reference ingredient set with an F 1 of95.5± 0.6%and reproduces the entire set verbatim for90.8± 1.3%of images, with precision (95.7%) and recall (95.3%) closely balanced as the model neither systematically over nor under-generates ingredients. Table 3: Likely-ingredient reasoning of OliveGemma, graded as set overlap between the predicted and reference ingredient lists. Values are sample-averaged percentages per fold; the last row is the mean±standard deviation across the three folds. Exact-set is the fraction of images whose predicted ingredient set matches the reference exactly. FoldPrecisionRecallF 1 JaccardExact-set 096.2995.9796.1095.3691.98 195.6195.0895.3294.5090.98 295.2294.8294.9994.0189.41 Mean± std95.71±0.5495.29±0.60 95.47±0.5794.62±0.6890.79±1.30 Table 4 illustrates the behaviour qualitatively. For canonical, well-represented dishes the model is frequently verbatim- correct: for tiramisu it reproduces both the visible components (cocoa, mascarpone, ladyfingers, coffee-soaked sponge) and the hidden ones (espresso, sugar, egg yolks, Marsala) exactly. The partial cases are revealing as on the bacon- ham-pepper pizza the model substitutessliced mushroomsfor one of the peppers which is a visually plausible topping confusion while it recovers the crust, cheese, cured meats, and the hidden leavening agents. Most errors are of this substitution type rather than wholesale hallucination, and the commonly-present (hidden) set is recovered almost 10 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet perfectly even when the visible set drifts, as for the green apple where the model misreads the skin colour but still lists the correct fructose/malic-acid/fibre chemistry. The clearest failure mode is downstream of recognition whereas, when the dish itself is misjudged, i.e. russian salad read as a leafy green salad, the entire visible ingredient set follows the wrong dish, confirming that ingredient quality is bounded by recognition quality. Overall, these results indicate that the LoRA adaptation instils a structured, decompositional understanding of food rather than a flat label map. Table 4: Representativelikely_ingredientspredictions versus the distilled reference. âVisibleâ abbreviates visible- typical ingredients and âHiddenâ the commonly-present but not visually confirmable set.F 1 is the per-image set-overlap score. DishReferenceOliveGemma predictionF 1 TiramisuVisible: cocoa powder, mascarpone cream, ladyfinger biscuits, coffee-soaked sponge. Hidden:espresso, sugar, egg yolks, Marsala wine. Visible: cocoa powder, mascarpone cream, ladyfinger biscuits, coffee-soaked sponge. Hidden:espresso, sugar, egg yolks, Marsala wine. 1.00 Pizza (ba- con, ham, peppers) Visible: bacon bits, sliced ham, green bell peppers, mozzarella cheese, tomato sauce, pizza crust. Hidden: olive oil, salt, yeast, sugar. Visible:bacon bits, sliced ham, bell peppers,sliced mushrooms, mozzarella cheese, tomato sauce.Hidden: pizza dough, olive oil, yeast, salt. 0.70 Green ap- ple Visible: green skin, stem, flesh, seeds. Hid- den: fructose, malic acid, water, fiber. Visible:yellow skin, green stem, smooth waxy surface, fibrous interior. Hidden: fructose, malic acid, water, fiber. 0.50 Russian salad Visible: diced potatoes, green peas, carrots, boiled eggs, pickles, mayonnaise. Hidden: salt, black pepper, mustard, vinegar. Visible:lettuce leaves, cherry tomatoes, lemon wedges,cucumber slices, red onion strips, shredded carrots. Hidden: olive oil, vinegar, salt, black pepper. 0.30 4 Discussion This study presented OliveGemma, a domain-adapted version of PaliGemma-2-3B for fine-grained food recognition and visual food understanding. The proposed model was trained on a unified dataset comprising the MedGR, ODIN, and VIPPSTAR datasets, resulting in a canonical vocabulary of 216 Mediterranean and European food categories. Beyond food classification, OliveGemma was instruction tuned to perform complementary reasoning tasks, including ingredient identification, visual evidence generation, and discrimination between visually similar dishes. Experimental evaluation demonstrated that the proposed model consistently outperformed all evaluated CNN baselines as well as several contemporary proprietary vision-language models operating under the same closed-vocabulary evaluation protocol. The performance of OliveGemma should be interpreted in the context of both previous food recognition research and the evaluation setting adopted in this study. Conventional CNN based approaches evaluated on Food-101 [4] typically report top-1 accuracies from 85%-92%, while comparable performance has been reported on larger benchmarks such as Food2K [5]. However, these benchmarks primarily contain single-label dishes and comparatively homogeneous label spaces. In contrast, the task addressed here involves 216 multilingual food categories originating from three heterogeneous European datasets, introducing substantially greater visual variability, cultural diversity, and class imbalance. Within this considerably more challenging setting, OliveGemma achieved a mean top-1 accuracy of 92.96± 0.91%, demonstrating that parameter-efficient adaptation of a general-purpose vision-language model can achieve performance comparable to state-of-the-art CNNs while operating on a substantially richer and more complex label space. The comparison with the CNN baselines further illustrates the advantages of multimodal instruction tuning. All CNNs were evaluated under an identical evaluation scheme, where only the classification layer was optimized while the pretrained ImageNet feature extractor remained frozen. This design intentionally isolated the representational capabilities of each architecture from differences in optimization strategy. OliveGemma exceeded the strongest CNN baseline, DenseNet-121, by 7.31 percentage points in top-1 accuracy, although DenseNet-121 achieved marginally higher top-3 and top-5 accuracy. This observation suggests that CNN feature extractors often rank the correct class among their highest-confidence predictions but are less effective at selecting the correct class as the most probable output. In contrast, OliveGemma benefits from jointly modelling visual information and language representations, allowing predictions to be conditioned not only on image features but also on semantic relationships encoded during 11 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet instruction tuning. Similar observations have been reported in recent studies on instruction tuned vision-language models [7, 9]. The comparison with frontier vision-language models highlights the importance of domain adaptation. Although Gemini Flash 3, Gemini Flash 3.5, GPT-5.4 Mini, and Claude Haiku 4.6 are substantially larger and more general-purpose systems, they were consistently outperformed by OliveGemma under the closed-vocabulary evaluation protocol. The relatively small differences observed in top-3 and top-5 accuracy indicate that these models frequently identify the correct food among their highest-ranked predictions but struggle to distinguish visually similar food categories when a single prediction is required. These findings suggest that model scale alone does not guarantee superior performance in specialized domains. Instead, domain-specific adaptation through parameter-efficient fine-tuning remains essential for tasks involving subtle visual distinctions, multilingual label spaces, and culturally specific food categories. This observation is consistent with previous work demonstrating the effectiveness of LoRA for adapting large multimodal foundation models to specialized application domains [11, 31, 32]. Beyond recognition performance, OliveGemma offers several practical advantages. Unlike proprietary frontier models that are accessible only through cloud-based APIs, OliveGemma is fully reproducible, openly available, and can be deployed entirely on local infrastructure. This eliminates recurring inference costs while improving reproducibility and enabling inference without network connectivity. These characteristics are particularly important for clinical dietary assessment, where food images may contain patient-identifiable information and are subject to privacy regulations such as GDPR. The ability to perform inference entirely within institutional infrastructure makes OliveGemma a practical alternative for healthcare environments and contributes to the growing evidence that relatively small, open-weight foundation models can effectively address specialized biomedical tasks when combined with parameter-efficient adaptation. Aside quantitative improvements, these findings have important implications for automated dietary assessment. The improved performance of OliveGemma suggests that accurate food recognition benefits not only from visual feature extraction but also from the semantic knowledge acquired during multimodal pretraining. Unlike conventional CNN- based classifiers, which learn fixed mappings between images and predefined classes, the vision-language model can exploit relationships between visual appearance and textual concepts, enabling better discrimination of visually similar dishes. Furthermore, the strong performance achieved through parameter-efficient LoRA fine-tuning indicates that adapting a pretrained foundation model to a specialized food domain is more effective than relying solely on increasingly larger general-purpose models. This observation reinforces the idea that domain adaptation, rather than model scale alone, is an important factor for achieving robust performance in complex food recognition tasks. Furthermore, it is important of AI systems to combine high recognition accuracy with explainability, reproducibility, and privacy-preserving deployment for dietary assessment applications. Our findings demonstrate that an open-weight vision- language model can satisfy these requirements simultaneously, outperforming both conventional CNN architectures and several proprietary frontier VLMs while remaining suitable for local deployment. Consequently, OliveGemma provides a practical foundation for future clinical and nutritional applications, where reliable food recognition can improve downstream tasks such as nutritional analysis, dietary monitoring, and decision-support systems without requiring dependence on proprietary cloud-based services. The present study nevertheless has several limitations. The unified dataset covers 216 Mediterranean and European food categories, yet many regional cuisines remain unrepresented. Expanding the training corpus with additional datasets, such as the Central Asian Food Dataset [34], would increase both the diversity and complexity of the label space, particularly for visually similar carbohydrate-based dishes including rice and noodle varieties. Future work will also investigate extending OliveGemma beyond closed-vocabulary recognition toward open-vocabulary food understanding, nutritional estimation, and integration with clinical dietary assessment systems. Overall, the findings demonstrate that parameter-efficient adaptation of an open-weight vision-language model provides an effective solution for fine-grained food recognition. By combining multimodal instruction tuning with LoRA, OliveGemma achieves competitive recognition performance while providing explainable food understanding, local deployment, and reproducible experimentation. 5 Disclosure on AI Use In the current study we have employed Anthropic Claude Opus for english language refinements and OpenAI ChatGPT for Figure 1 stylistic modifications while this figure derived from originally designed graphics by the authors. All the english language and figures modifications were reviewed by the authors of this study to avoid hallucinated artifacts. 12 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet 6 Acknowledgments This work is supported by the Vippstar project, funded by the European Unionâs Horizon 2020 research and innovation program under grant agreement No. 101156763. It reflects only the authorâs view. The Commission is not responsible for any use that may be made of the information it contains. References [1]RamĂłn Estruch, Emilio Ros, Jordi Salas-SalvadĂł, Maria-Isabel Covas, Dolores Corella, Fernando ArĂłs, Enrique GĂłmez-Gracia, Valentina Ruiz-GutiĂ©rrez, Miquel Fiol, JosĂ© Lapetra, Rosa Maria Lamuela-Raventos, LluĂs Serra- Majem, Xavier PintĂł, Josep Basora, Miguel Angel Muñoz, JosĂ© V. SorlĂ, JosĂ© Alfredo MartĂnez, and Miguel Angel MartĂnez-GonzĂĄlez. Primary prevention of cardiovascular disease with a Mediterranean diet supplemented with extra-virgin olive oil or nuts. New England Journal of Medicine, 378(25):e34, 2018. [2] Wei Wang, Weiqing Min, Tianhao Li, Xiaoxiao Dong, Haisheng Li, and Shuqiang Jiang. A review on vision-based analysis for automatic dietary assessment. Trends in Food Science & Technology, 122:223â237, 2022. [3] Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, and Ramesh Jain. A survey on food computing. ACM Computing Surveys, 52(5):1â36, 2019. [4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 â mining discriminative components with random forests. In European Conference on Computer Vision (ECCV), volume 8694 of Lecture Notes in Computer Science, pages 446â461. Springer, 2014. [5]Weiqing Min, Zhiling Wang, Yuxin Liu, Mengjiang Luo, Liping Kang, Xiaoming Wei, Xiaolin Wei, and Shuqiang Jiang. Large scale visual food recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):9932â9949, 2023. [6]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023. arXiv:2301.12597. [7]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. arXiv:2304.08485. [8] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025. [9]Andreas Steiner, AndrĂ© Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alabdulmohsin, Lucas Beyer, and Xiaohua Zhai. PaliGemma 2: A family of versatile VLMs for transfer. arXiv preprint arXiv:2412.03555, 2024. [10] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521â3526, 2017. [11]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Represen- tations (ICLR), 2022. arXiv:2106.09685. [12]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023. arXiv:2305.14314. [13]Yoshiyuki Kawano and Keiji Yanai. Automatic expansion of a food image dataset leveraging existing categories with domain adaptation. In European Conference on Computer Vision (ECCV) Workshops â Transferring and Adapting Source Knowledge in Computer Vision (TASK-CV), pages 3â17. Springer, 2014. [14]Weiqing Min, Linhu Liu, Zhiling Wang, Zhengdong Luo, Xiaoming Wei, Xiaolin Wei, and Shuqiang Jiang. ISIA Food-500: A dataset for large-scale food recognition via stacked global-local attention network. In Proceedings of the 28th ACM International Conference on Multimedia (M), pages 393â401, 2020. [15]Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, and Antonio Torralba. Learning cross-modal embeddings for cooking recipes and food images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3020â3028, 2017. 13 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet [16]Fotios S Konstantakopoulos, Eleni I Georga, and Dimitrios I Fotiadis. An automated Image-Based dietary assessment system for mediterranean foods. IEEE Open J Eng Med Biol, 4:45â54, April 2023. [17]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770â778, 2016. [18]Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1â9, 2015. [19]Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4700â4708, 2017. [20]Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pages 6105â6114, 2019. [21]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. arXiv:2010.11929. [22]Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. Bilinear CNN models for fine-grained visual recognition. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1449â1457, 2015. [23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pages 8748â8763, 2021. [24]Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975â11986, 2023. [25] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. [26] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24185â24198, 2024. [27]Lucas Beyer, Andreas Steiner, AndrĂ© Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko BoĆĄnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. PaliGemma: A versatile 3b VLM for transfer. arXiv preprint arXiv:2407.07726, 2024. [28]Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pages 2790â2799, 2019. [29]Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP), pages 4582â4597, 2021. [30]Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3045â3059, 2021. [31]Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. LLaVA-Med: Training a large language-and-vision assistant for biomedicine in one day. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, volume 36, 2023. arXiv:2306.00890. 14 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet [32]Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abhijit Das, Salman Khan, and Fahad Shahbaz Khan. GeoChat: Grounded large vision-language model for remote sensing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27831â27840, 2024. [33]Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching for MobileNetV3. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1314â1324, 2019. [34] Aknur Karabay, Arman Bolatov, Huseyin Atakan Varol, and Mei-Yen Chan. A central asian food dataset for personalized dietary interventions. Nutrients, 15(7), 2023. 15 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet A Appendix I - Food Classes Here are presented the exact 216 food classes, where the models have been validated on. Table 5: List of 216 Classes almond cream cake, almonds, anthotyro, apple compote, apricot, asparagus, avocado, baked anchovies, baklava, banana, beef burger, beef in tomato sauce, beef stew, beef stew with celery, beef stew with onions, beef stew french fries, beef with artichokes in egg lemon sauce, beet salad, berries, biscotti, boiled antidia, boiled beef with vegetables, boiled beetroot, boiled cauliflower, boiled chicken, boiled greens, boiled octopus, boiled pork, boiled radikia, boiled stamnagathi, boiled vegetables, boiled vlita, breaded chicken, broccoli, bulbs, cabbage rolls, cabbage salad, calamari with spinach, canned peaches, carbonara, carrot, celery, cheese pie, cherries, chicken in red sauce, chicken souvlaki, chicken croquette applesauce, chicken rice currysauce, chickpea soup, chocolate bundt cake, chocolate cake, codfish plaki, cookies, corn, croissant with cheese, cucumber, cuttlefish with spinach, dolmades, dry fig, egg flan, eggplant salad, feta, fettine di manzo, firikia apples, french croissant, fresh garlic, fresh onion, fried anchovies, fried atherina, fried calamari, fried codfish, fried eggs, fried gopa, fried koutsomoura, fried marida, fried mushrooms, fried mussels, fried peppers, fried potatoes, fried rice, fried safridi, fried zucchini, frittata, galatopita, gelato, giant beans, gnocchi, grapefruit, graviera, greek artichoke stew, green apple, green bean and olive oil stew, green beans, green salad, grilled mushrooms, grilled quince, grilled sardines, hazelnut, honey rolls, imam bayildi, imam bayildi with mince, kalathaki lemnos, kefalograviera, kefalotyri, kiwi, ladotyri, lamb fricassee, lasagna, leek fritters, lemon, lentil soup, lettuce, lettuce salad, loquat, macaroni ham and cheese, mango, manouri, meat pie, meatball tomato sauce mashed potatoes, meatballs, mela, melon, metsovone, mezzo uovo, moustokouloura, nectarines, octopus in vinegar sauce, octopus with macaroni, okra stew, omelette, onion, orange, parsley, pasta bolognese, pasta souffle with cheese, pasta with cheese, pasta with eggplant sauce, pastitsio, peaches, peanuts arapiko, pear, peinirli, penne al ragu, penne al sugo, penne in bianco, pineapple, pineapple compote, pistachio, pizza, pizza margherita, pizza with bacon ham and peppers, plums, pomegranate, pork chop, pork chop fried potatoes chicory, pork in red sauce, potato fritters, potato salad, potatoes in red sauce, prosciutto cotto, prosciutto crudo, pumpkin, pumpkin seeds, purple cabbage, radish, raisins, ravioli, raw quince, raw turnip, red apple, red grapes, rice with pumpkin, risotto, roasted chicken, roasted vegetables, rocket salad, russian salad, salame, salmon leek mashed potatoes, salsiccia, sausage and peppers, sausage with leek, sausage carrot potato, semolina halvah, shrimp saganaki, shrimp salad, sole fillet, spaghetti al ragu, spaghetti al sugo, spaghetti in bianco, spinach, spinach and rice, spinach pie, steak mixed salad, strangolapretti, strawberries, sunflower seeds, sweet semolina cake, tangerine, tas kebab, telemes, tiramisu, tomato, tomato pasta, torta fetta, tuna salad, tzanera, tzatziki, uova strapazzate, walnuts, watermelon, white grapes, yellow pepper, zucchini, zucchini fritters, zucchini pasta. B Appendix I - Accessing & Inferring the Model OliveGemma could be accessed through HuggingFace repository under the following urlhttps://huggingface.co/ JamesZar/OliveGemma-3B. The following snippet presents the commands to download it and enable it locally. 16 OliveGemma: A 3 Billion VisualâLanguage Model for Recognising the Mediterranean & European Diet Table 6: OliveGemma Inference Code import torch from PIL import Image from transformers import AutoProcessor , PaliGemmaForConditionalGeneration REPO = "JamesZar/OliveGemma -3B" processor = AutoProcessor.from_pretrained(REPO) model = PaliGemmaForConditionalGeneration.from_pretrained( REPO , torch_dtype=torch.bfloat16 , device_map="auto" ).eval() image = Image.open("dish.jpg").convert("RGB") question = "What is the name of this dish?" # IMPORTANT: PaliGemma prompt format used in training: prompt = f"<image >answer en question " inputs = processor(text=prompt , images=image , return_tensors="pt").to(model.device ) in_len = inputs["input_ids"]. shape[-1] with torch.no_grad (): out = model.generate (** inputs , max_new_tokens =64, do_sample=False) print(processor.decode(out [0][ in_len:], skip_special_tokens=True).strip()) 17