Paper deep dive
MMTClinic: Multimodal, Multilingual Time Series Question Answering and Reasoning Benchmark for Clinical Domain
Sourav Malakar, Harshit Nigam, Akash Ghosh, Sriparna Saha, Amlan Chakrabarti, Saptarsi Goswami, Priti Singh
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 9/7/2026, 6:40:55 AM
Summary
The paper introduces MMTClinic, a multimodal and multilingual benchmark for evaluating Large Language Models (LLMs) on clinical time-series reasoning. The dataset comprises 30,000 question-answer pairs (15,000 MCQs and 15,000 open-ended) across five languages (English, Hindi, Bengali, Marathi, Tamil) and three clinical tasks: mortality prediction, heart rate forecasting, and SOFA score estimation. It integrates text, medical images, and multivariate physiological signals. The authors evaluate 13 state-of-the-art LLMs, revealing significant performance variations across modalities, languages, and reasoning types.
Entities (16)
Relation Signals (14)
MMTClinic → supportslanguage → Bengali
confidence 100% · across five languages: English, Hindi, Bengali, Marathi, and Tamil.
MMTClinic → supportslanguage → Tamil
confidence 100% · across five languages: English, Hindi, Bengali, Marathi, and Tamil.
MMTClinic → supportslanguage → Marathi
confidence 100% · across five languages: English, Hindi, Bengali, Marathi, and Tamil.
MMTClinic → supportslanguage → English
confidence 100% · across five languages: English, Hindi, Bengali, Marathi, and Tamil.
MMTClinic → supportslanguage → Hindi
confidence 100% · across five languages: English, Hindi, Bengali, Marathi, and Tamil.
Amlan Chakrabarti → affiliatedwith → University of Calcutta
confidence 95% · Amlan Chakrabarti Affiliation: University of Calcutta
Sourav Malakar → affiliatedwith → IIT Patna
confidence 95% · Sourav Malakar... Harshit Nigam Affiliation: IIT Patna
MMTClinic → containstask → Mortality Prediction
confidence 95% · These questions cover three important clinical tasks---mortality prediction, heart rate forecasting, and SOFA score estimation.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Time-series data in clinical settings is crucial for capturing dynamic changes in a patient's health over time, enabling timely diagnosis, personalized treatment, and early detection of critical events. However, the development of clinically reliable and linguistically inclusive medical AI systems remains a significant challenge, primarily due to the lack of multimodal, multilingual, and time-series-grounded benchmarks that reflect the complexity of real-world clinical scenarios. To fill this gap, we present MMTClinic, a benchmark designed to evaluate large language models (LLMs) on complex reasoning and question-answering tasks involving clinical time-series. MMTClinic combines text, medical images, and multivariate physiological signals and includes 30,000 QA pairs (15,000 multiple choice questions (MCQs) and 15,000 open-ended questions) across five languages: English, Hindi, Bengali, Marathi, and Tamil. These questions cover three important clinical tasks---mortality prediction, heart rate forecasting, and SOFA score estimation. We evaluate 13 state-of-the-art LLMs in zero-shot, few-shot, and chain-of-thought settings. Our evaluation reveals notable differences in model performance across tasks, languages, and modalities, highlighting current limitations in clinical reasoning capabilities. MMTClinic provides a valuable resource for advancing multilingual, multimodal, and time-series-aware medical AI research. The dataset will be made publicly available on successful acceptance of the work.
Tags
Links
- Source: https://arxiv.org/abs/2609.04842v1
- Canonical: https://arxiv.org/abs/2609.04842v1
Trouble viewing inline? Open PDF directly →
Full Text
134,776 characters extracted from source content.
Expand or collapse full text
MMTClinic: Multimodal, Multilingual Time Series Question Answering and Reasoning Benchmark for Clinical Domain Sourav Malakar Affiliation: Department of CSE Affiliation: Institute of Engineering and Management Affiliation: Kolkata, West Bengal, India Email: sourav.malakar@iem.edu.in Harshit Nigam Affiliation: Department of CSE Affiliation: IIT Patna Affiliation: Bihar, India Akash Ghosh Affiliation: Department of CSE Affiliation: IIT Patna Affiliation: Bihar, India Sriparna Saha Affiliation: Department of CSE Affiliation: IIT Patna Affiliation: Bihar, India Amlan Chakrabarti Affiliation: Department of A. K. Choudhury School of IT Affiliation: University of Calcutta Affiliation: Kolkata, West Bengal, India Saptarsi Goswami Affiliation: Department of Computer Science Affiliation: Bangabasi Morning College Affiliation: Kolkata, West Bengal, India Dr. Priti Singh Affiliation: Department of CSE Affiliation: IIT Patna Affiliation: Bihar, India Abstract Time-series data in clinical settings is crucial for capturing dynamic changes in a patient’s health over time, enabling timely diagnosis, personalized treatment, and early detection of critical events. However, the development of clinically reliable and linguistically inclusive medical AI systems remains a significant challenge, primarily due to the lack of multimodal, multilingual, and time-series-grounded benchmarks that reflect the complexity of real-world clinical scenarios. To fill this gap, we present MMTClinic, a benchmark designed to evaluate large language models (LLMs) on complex reasoning and question-answering tasks involving clinical time-series. MMTClinic combines text, medical images, and multivariate physiological signals and includes 30,000 QA pairs (15,000 multiple choice questions (MCQs) and 15,000 open-ended questions) across five languages: English, Hindi, Bengali, Marathi, and Tamil. These questions cover three important clinical tasks—mortality prediction, heart rate forecasting, and SOFA score estimation. We evaluate 13 state-of-the-art LLMs in zero-shot, few-shot, and chain-of-thought settings. Our evaluation reveals notable differences in model performance across tasks, languages, and modalities, highlighting current limitations in clinical reasoning capabilities. MMTClinic provides a valuable resource for advancing multilingual, multimodal, and time-series-aware medical AI research. The dataset will be made publicly available on successful acceptance of the work. A Preprint 1 Introduction Large language models (LLMs) and Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance in a variety of domains, including healthcareChan et al. (2024); Hu et al. (2023); Tu et al. (2024); Ye et al. (2021). State-of-the-art models, such as GPT OpenAI and others (2023), are progressively gaining popularity to support clinical decision making in the healthcare domain. However, the majority of studies in the healthcare domain have primarily focused on text or text–image combinations, leaving a notable gap in addressing more complex modalities such as time series data. The integration of time series, textual, and visual data is critical for understanding complex real-world phenomena, where numerical trends and contextual narratives jointly facilitate comprehensive analysis and decision-making. Although significant work exists in domains such as finance Jacob and Shasha (1999); Hu et al. (2025) and weather forecasting Rasp et al. (2020) illustrate the need for models that can jointly interpret numerical and textual data; however, this capability remains mostly underexplored in the healthcare domain. Research Gap. Existing clinical time-series reasoning benchmarks fall into three categories: (i) signal-only MCQs Oh et al. (2023); Wang et al. (2025); Pham and others (2025), which lack clinical text and imaging; (i) text + time-series datasets Kim and others (2025); Kweon et al. (2024), which omit visual data and MCQ formats; and (i) comprehensive multimodal datasets Bae et al. (2023); Hayat et al. (2022), which include text, signals, and images but rely on open-ended tasks and static radiographs. Specifically, none supports multilingual MCQ-based-reasoning with integrated clinical text, time-series, and images. To date, no benchmark unifies clinical text, images, and time-series data in an MCQ and reasoning-based framework in multiple languages. Present Work: To address the limitations of existing benchmarks, we propose MMTClinic a Multimodal and Multilingual Time-series question-answering and reasoning benchmark tailored for the healthcare domain. MMTClinic supports two complex downstream tasks: (a) multiple-choice question answering (MCQ-QA), and (b) open-ended clinical decision-making (Reasoning). Unlike previous efforts, our benchmark integrates multivariate clinical time-series data with both textual and visual modalities to capture and reason over temporal trends, critical for accurate disease progression analysis. The dataset is constructed using a semi-automated pipeline based on the 2012 PhysioNet/CinC Challenge dataset Silva et al. (2012), from which we generate a total of 15k MCQ and reasoning QA pairs, in total 30k samples. These samples are crafted across five linguistically diverse Indian languages: English, Bengali, Hindi, Marathi, and Tamil. We have evaluated a diverse set of 13 language models, encompassing LLMs–both proprietary and open-source—along with Small Language Models (SLMs), Reasoning Language Models (RLMs) and Multimodal LLMs (MLLMs). This broad evaluation setup enables a linguistically rich and modality-aware assessment across both MCQ and open-ended reasoning tasks involving text, images, and time-series data. The benchmark includes three real-world ICU-centered clinical tasks: (1) Predicting in-hospital mortality (binary classification), (2) Forecasting heart rate trends using the first 48 hours of ICU time-series data, and (3) Estimating the SOFA score from 48-hour multivariate physiological signals. Figures 1 illustrate the multimodal structure and overall design of MMTClinic. Contributions: In summary, the main contributions of the MMTClinic study are as follows. Comprehensive Evaluation: We perform joint reasoning over text, images, and time-series across two QA formats, four modality settings, three ICU tasks, and three prompting modes. Domain Specific Healthcare Coverage: The benchmark targets early ICU care (first 48 hours), capturing six vital signs in both numeric (.csv) and visual (.png) formats. Multilingual Coverage: We release 30,000 QA pairs (15k MCQ, 15k reasoning) in five languages: English, Hindi, Bengali, Marathi, and Tamil for fair multilingual evaluation. Diverse Model Benchmarking: 13 models, including general LLMs, reasoning-optimized LRMs, and MLLMs are benchmarked under all prompt settings, with analysis across tasks, languages, and modalities. Expert-Curated Quality: Data were generated semi-automatically and carefully refined by physicians and linguists. Specifically, approximately 70% of QA samples from each task and modality were independently reviewed and scored by four certified medical experts for clinical accuracy and reasoning quality. Additionally, native-speaking linguists checked all of the multilingual translations. Samples underwent quality control through expert scoring (shown in Table 7), targeted post-editing, and reasoning consistency checks. Figure 1: Overview of MMTClinic 2 Related Works In recent years, LLMs have been extensively used across diverse domains such as healthcare, education, law, finance Mariani et al. (2026) and scientific research, demonstrating remarkable adaptability and performance Brown et al. (2020). In spite of the growing impact of LLMs across various domains, their application in clinical time series analysis remains notably limited. Most existing LLMs are optimized for text data, and fail to effectively handle temporal physiological signals or multivariate time-series inputs crucial for clinical reasoning Hager et al. (2024); Jin et al. (2023); Brown et al. (2025); Mousavi and Termehchy (2023). In recent works, LLMs have mainly focused on English-centric clinical question-answering tasks, mostly ignoring regional and low-resource languages. This language bias raises concerns about ensuring fair access to reliable healthcare information for different communities. Benchmarks such as MedExpQA Alonso et al. (2024) and IndicQA Singh et al. (2024) demonstrate that current LLMs struggle with medical reasoning and accuracy in non-English settings. A growing body of work has begun to target multilingual and Indic-language medical understanding directly, including surveys of multilingual reasoning in language models Ghosh et al. (2025a), multilingual medical reasoning Onyame et al. (2026), trustworthiness evaluation for healthcare LLMs across languages Ghosh et al. (2025b), multimodal medical query analysis in Indian languages Ghosh et al. (2026), multi-agent multimodal medical reasoning in Indic languages Halder et al. (2026), and summarization of code-mixed clinical queries Ghosh et al. (2024b). However, these efforts largely address text or text-only reasoning and do not incorporate physiological time-series signals. Beyond the linguistic dimension, another line of research explores multimodal clinical understanding that jointly reasons over clinical text and medical images, such as multimodal question summarization Ghosh et al. (2024a), clinical document summarization Ghosh et al. (2024d); Ghosh et al. (2024c), and multimodal medical retrieval Acharya et al. (2025). While these approaches integrate visual and textual modalities, they rely on static images (e.g., radiographs) and do not model temporal trends in multivariate physiological data, leaving multimodal temporal reasoning largely unexplored. Recent research Singhal et al. (2025); Kim et al. (2025); Liévin et al. (2024); Zhou et al. (); Li et al. (2024) indicates that LLMs have mostly been evaluated using MCQ QAs in clinical tasks, which may not adequately assess their complex reasoning capabilities. Although LLMs reveal high precision on standardized MCQ benchmarks such as MedQA Yao et al. (2024) and MedMCQA Pal et al. (2022), they often struggle with tasks requiring nuanced clinical reasoning, such as differential diagnosis and treatment planning. Unlike prior work, this study unifies multivariate clinical time-series with textual and visual modalities, introduces open-ended reasoning tasks beyond MCQs, and evaluates a diverse set of LLMs across five languages, jointly addressing the temporal, multimodal, and multilingual gaps identified above. 3 Construction of MMTClinic The construction of MMTClinic is outlined through the following key components: dataset dimensions, detailed statistics, and the overall framework that guided the creation of MMTClinic. In Figure 2, the overall construction of MMTClinic is presented in detail. 3.1 Dataset Dimensions MMTClinic includes both multiple-choice (MCQ) and open-ended reasoning-based questions designed to evaluate clinical understanding from multimodal inputs. It is multilingual and covers five Indian languages, namely Hindi, Tamil, Marathi, English, and Bengali. It comprises three core tasks: Task 1: Given the physiological characteristics of a patient, such as heart rate, blood pressure, and respiratory rate, predict the in-hospital mortality outcome (0: survived, 1: died during hospital stay). Task 2: Given multivariate time-series data collected during the first 48 hours of ICU stay—including measurements like blood pressure, respiratory rate, and temperature-—forecast the patient’s heart rate (HR) as an early indicator of clinical deterioration. Task 3: Based on 48-hour physiological time-series data, including vital signs and lab measurements, estimate the Sequential Organ Failure Assessment (SOFA) score to quantify the level of organ dysfunction. Each task is evaluated under four distinct multimodal settings: (i) text and time-series (MCQ), (i) text, image, and time-series (MCQ), (i) text and time-series (reasoning), and (iv) text, image, and time-series (reasoning). Model Task1 Task2 Task3 Small LLMs Mistral-7B-Instruct 28.02 30.52 21.83 Qwen2.5-7B 6.97 14.83 11.02 DeepSeek-R1-Llama-8B 58.15 57.22 57.20 Large LLMs Llama-3.3-70B 31.12 27.71 23.24 Qwen3-30B-A3B 43.77 42.92 31.84 Qwen3-235B-A22B 52.10 51.88 50.28 QwQ-32B 52.37 50.72 50.58 DeepSeek-R1 66.0 65.1 65.3 Proprietary LLMs GPT-4.1-nano 45.27 45.75 39.73 Gemini-2-flash 27.4 35.53 30.47 Table 1: Task-wise average accuracy (MCQ, Text + Time Series) Model Task1 Task2 Task3 Small LLMs Mistral-7B-Instruct 35.72 31.97 24.80 Qwen2.5-7B 9.15 11.75 8.98 DeepSeek-R1-Llama-8B 71.38 64.19 65.74 Large LLMs Llama-3.3-70B 28.45 29.07 23.86 Qwen3-30B-A3B 35.99 37.02 26.06 Qwen3-235B-A22B 76.23 71.36 73.60 QwQ-32B 61.57 60.89 62.72 DeepSeek-R1 64.61 61.25 64.35 Proprietary LLMs GPT-4.1-nano 31.73 32.59 25.70 Gemini-2-flash 31.04 32.34 26.15 Table 2: Task-wise average accuracy (Reasoning, Text + Time Series) Model Task1 Task2 Task3 Small LLMs Qwen2.5-VL-7B 30.13 33.28 25.83 Large LLMs LLaMA-3.2-11B-Vision-Instruct 30.12 37.13 29.17 Qwen2.5-VL-72B 31.62 44.38 34.26 Gemma-3-27B-IT 26.91 29.68 42.91 Proprietary LLMs GPT-4.1-nano 46.50 46.26 42.48 Gemini-2-Flash 28.58 33.50 30.86 Table 3: Average MCQ Accuracy per Task (Text + Time Series + Image) Model Task1 Task2 Task3 Small LLMs Qwen2.5-VL-7B 33.07 38.76 42.39 Large LLMs LLaMA-3.2-11B-Vision-Instruct 29.56 34.72 24.71 Qwen2.5-VL-72B 29.57 42.85 50.29 Gemma-3-27B-IT 22.30 25.36 45.15 Proprietary LLMs GPT-4.1-nano 31.96 38.78 26.19 Gemini-2-Flash 24.54 31.14 23.42 Table 4: Average Reasoning Accuracy per Task (Text + Time Series + Image) 3.2 Statistics of MMTClinic The main statistics associated with MMTCLinic are shown below: i) The MMTClinic benchmark comprises a total of 30,000 questions. i) Two distinct types of MCQ and open-ended reasoning-based questions. i) The data set spans four unique clinical tasks, each designed to evaluate different aspects of multimodal and multilingual clinical reasoning. iv) MMTClinic supports 5 languages: English, Hindi, Bengali, Marathi, and Tamil. v) Each task is associated with an average of 7,500 questions. vi) On average, there are 6,000 questions available for each language. 3.3 Framework for developing MMTClinic The following steps are used to develop the MMTClinic : Step 1: Data collection: We begin by obtaining multivariate clinical time series for ICU patients from the 2012 PhysioNet/Computing in Cardiology Challenge dataset, which provides up to 42 physiological and laboratory variables (e.g. HR, BP, RespRate, Temp, Glucose) recorded during the first 48 hours of stay of each patient. For MMTClinic , we used hourly resampling and generated both numerical (.csv) and visual (.png) representations of six physiological signals. The scale, clinical richness, and structured signal data in this dataset make it an ideal choice for this benchmark. 2. Figure 2: Construction of MMTClinic. Step 1 indicates data collection. Step 2 implies the detailed pre-processing steps for extracting complete data of a patient. Step 3 focused on question answer generation in two experimental setup (a) Text + Time Series and (b) Text + Time Series + Image. Step 4 indicates the question answer validation by medical and linguistic expert. Step 2: Data Pre-processing: To ensure consistency, patients with more than 35% missing values are removed, and the remaining are imputed using multivariate polynomial interpolation. Signals originally sampled at irregular intervals are then uniformly resampled to 1-hour resolution. Finally, both the cleaned numerical series (.csv) and the corresponding multivariate line-plot images (.png) are generated using the matplotlib library in Python. Step 3: Initial Samples Generation: Our few-shot samples were carefully selected using a medical‑expert prompting strategy, ensuring at least one representative example from each task described in Appendix 9.9. Using these prompts, we generated 15,000 multiple-choice and 15,000 open-ended reasoning question-answer pairs in English from a randomly chosen subset of 2,000 patients. The Google Translate API was then used to translate these QA pairs into Bengali, Hindi, Marathi, and Tamil, after which professional linguists carefully reviewed and refined the translations to ensure accurate and high-quality outputs in all target languages. Importantly, the ground‑truth answers for both the MCQ and reasoning tasks were sourced directly from the PhysioNet dataset; no expert curation or guidance was used to derive these answers. Only task formation and samples question generation were done by experts. The entire dataset including images, .CSV files, task wise QA pairs, and prompts can be accessed at this link. Step 4: Samples Data Verification: To ensure the samples generated in Step 3 form a clinically relevant, high‑quality multilingual benchmark, we conduct a two‑stage evaluation: (1) clinical relevance review and (2) linguistic quality and fidelity assessment across languages. We discuss both of these aspects in detail in the following points: Clinical Relevance Review: We used GPT‑4o to generate question–answer samples with few‑shot prompts curated by medical experts, and we took all ground‑truth answers directly from the source dataset. We conducted a manual expert review on a stratified random subset that included about 70% of the samples from each task category. These categories were MCQ (Text + Time-Series), MCQ (Text + Time-Series + Images), Reasoning (Text + Time-Series), and Reasoning (Text + Images + Time-Series). Four medical experts independently reviewed each sample. The mean ratings were 4.2, 4.5, 4.4, and 4.5 for the respective tasks, indicating consistently high sample quality. This extensive study demonstrates that the carefully designed prompting strategy is effective in generating high-quality questions. Expert verification followed a standard set of guidelines to ensure consistency and objectivity. For each sample, reviewers evaluated (i) the clinical relevance of the question, (i) the correctness of the ground-truth answer based on the underlying physiological time series, (i) the clarity and plausibility of distractor options for multiple-choice questions, and (iv) the logical coherence of reasoning-based answers. Linguistic reviewers followed similar guidelines focusing on semantic accuracy, grammatical correctness, and the preservation of medical meaning across languages. Detailed verification instructions can be found in Appendix 9.7. Per task averages and inter‑annotator agreement are reported in the Appendix-9.6. Linguistic Quality: To ensure high-quality multilingual translations, the dataset was reviewed by native-speaking linguists who are fluent in the target languages. The linguistic evaluation followed a two-stage process: a) Each data point was rated on a scale from 0 to 5 based on linguistic accuracy and fluency. b) Samples scoring below 3.5 were flagged for expert post-editing. Only a small portion required further refinement—approximately 3.8% of samples in Tamil and Marathi, 2.1% in Bengali, and 1% in Hindi. Figure 3: * (a) Figure 4: * (b) Figure 5: Radar plot indicating language-specific accuracy of LLMs for Text and Time Series modality: (a) MCQ and (b) Reasoning. Figure 6: * (a) Figure 7: * (b) Figure 8: Radar plot indicating language-specific accuracy of LLMs for Text, Time Series, and Image modality: (a) MCQ and (b) Reasoning. 4 Experimental Section 4.1 Models To enable a comprehensive evaluation of diverse tasks in MMTClinic, which includes both text + time-series and text + time-series + image inputs, we benchmark a wide range of language models categorized into two primary groups: text-only language models and multimodal language models. Each group is further classified by scale (small vs. large) and accessibility (proprietary vs. open). For text-only models, we evaluate Small Language Models (SLMs) such as Mistral 7B Instruct Samo et al. (2024), Qwen 2.5 7B Yang et al. (2024), and DeepSeek R1-LLaMA 3 8B Guo et al. (2025a), and Large Language Models (LLMs) including LLaMA 3.1 70B Grattafiori et al. (2024), Qwen 3 30B Yang et al. (2025), Qwen 2.5 35B Hui et al. (2024), QWQ 32B Team (2025), and DeepSeek R1-Large Guo et al. (2025a). We also include proprietary models such as GPT-4.1 nano OpenAI and others (2023) and Gemini 2 Flash Team et al. (2023). For the multimodal setting, we assess Small Multimodal Models like Qwen 2.5 VL 7B Bai et al. (2025), and Large Multimodal Models including LLaMA 3.2 11B Vision Instruct Lee et al. (2025), Qwen 2.5 VL 72B Bai et al. (2025), and Gemma 3 27B Team et al. (2023). The same proprietary models—GPT-4.1 nano OpenAI (2025) and Gemini 2 Flash Google DeepMind (2025) are also considered in the multimodal evaluation. The model parameters and hyper–parameters have been presented in Appendix 9.2 and 9.3. 4.2 Evaluation Setup We have conducted zero-shot, few-shot, and CoT based evaluation across all four sub tasks defined in MMTClinic to authentically assess the LLM’s inherent medical knowledge and reasoning capabilities. This approach offers a more reliable and unbiased evaluation of the LLMs ability to generalize to unseen clinical questions. To comprehensively assess model performance, we divide the analysis into four distinct settings: (i) MCQ with Text + Time-Series, (i) Reasoning with Text + Time-Series, (i) MCQ with Text + Time-Series + Image, and (iv) Reasoning with Text + Time-Series + Image. In addition to these modality-based settings, we perform a fine-grained evaluation across all four types of question formats, further disaggregated by individual language, to examine multilingual generalization capabilities. 4.3 Discussion on Results Performance across different settings: Across the four evaluation settings, we see a clear hierarchy of model strengths and weaknesses as discussed in points below. a) MCQ + Text + Time-Series: From Table 2, Small LLMs like DeepSeek-R1 and LLaMA-8B show better performance, with average accuracies above 57% compared to Qwen 2.5-7B, which is around 7%, and Mistral 7B, which is about 28%. Among the larger models, Qwen 3-235B and QWQ-32B excel, achieving robust accuracy in the mid-60% range. They outperform proprietary models like GPT-4.1-Nano, which is roughly 45%, and Gemini 2-Flash by 27%. These proprietary models fall between the capabilities of small and large open-source LLMs. Figure 9: * (a) Figure 10: * (b) Figure 11: Box plot indicating few-shot and CoT specific accuracy of LLMs for Text and Time Series modality (a) MCQ (b) Reasoning Figure 12: The LHS displays the correctly answered questions, while the RHS highlights the incorrectly answered ones by the language models on the MMTClinic dataset. b) Reasoning + Text + Time-Series: In Table 2, the average accuracy for reasoning tasks using Text and Time-Series shows a growing performance gap. DeepSeek-R1-LLaMA-8B achieves about 71%, while Qwen 3-235B leads at around 76%. In contrast, Qwen 2.5-7B stays below 10%, and proprietary models fall to the low-30% range. This highlights how open-source reasoning LLMs better manage temporal clinical data. c) MCQ + Text + Time-Series + Image: Table 4 shows that adding images results in a notable performance drop. GPT-4.1-nano performs best at roughly 46%, followed by Qwen 2.5-VL-7B and 72B at 30–32%. Other vision-language models, including LLaMA-3.2 Vision and Gemma 3 27B, score lower at around 26–30%. This suggests that integrating visual elements remains difficult and proprietary models handle it better. d) Reasoning + Text + Time-Series + Image: This is the most challenging setting (Table 4). Qwen 2.5-VL 7B slightly outperforms GPT-4.1-nano (33% vs. 32%), while larger multimodal models and Gemini 2-Flash fall into the low-to-mid-20% range. Overall, models with specialized multimodal training do better on image-based MCQs, while temporal reasoning models such as DeepSeek variants and Qwen 3-235B excel in text-time tasks. Simpler models, such as Qwen 2.5-7B and Mistral 7B, consistently perform poorly. The key conclusion from the task-wise analysis is that open-source LLMs outperform in text-based MCQs and reasoning tasks, whereas proprietary models demonstrate superior performance on image-based MCQs and multimodal reasoning. Performance of models across languages: For the MCQ (Text + Time-Series) task in Figure 5 (a), English leads, followed closely by Bengali and Marathi. Large models like Qwen 3-235B and DeepSeek-R1-LLaMA-8B show strong multilingual consistency, losing only 5 to 10% in regional languages. In contrast, smaller models, such as Mistral 7B and Qwen 2.5-7B, drop over 30% in Tamil and Hindi. GPT-4.1-nano maintains stability across all five languages. In the Reasoning (Text + Time-Series) task (Figure 5 (b)), the overall precision decreases and the disparities grow. Top models still score at least 70% in English and Bengali, but they fall 15 to 20% in Tamil and Marathi. Mid-sized models experience sharper declines, while proprietary models remain stable at 30 to 35%. In the MCQ (Text + Time-Series + Image) setting (Figure 8 (a)), GPT-4.1-nano leads with scores of 45 to 50% in English, showing a moderate 10 to 12% drop in regional performance. Qwen 2.5-VL-7B performs moderately, scoring 30 to 35%, while other open-source vision-instruct models cluster between 25 to 30%. Simpler models fall below 20%. For Reasoning (Text + Time-Series + Image) in Figure 8 (b), accuracy drops another 5 to 8%. GPT-4.1-nano stays at 35 to 40%, while Qwen 2.5-VL-7B drops from about 38% in English to 20% in other languages. Gemini 2-Flash and vision-instruct models stabilize around 24 to 32%, reflecting limited but consistent multimodal reasoning. The key conclusion is that proprietary models are stronger in image-related tasks compared to open weight models. Models across all tiers experienced the worst performance in reasoning-based tasks, especially in languages like Tamil. Performance across model types: Small language models consistently perform poorly, rarely exceeding 30% in MCQ and falling below 10% in reasoning due to their limited capacity and lack of multimodal pretraining. LLMs do well on text and time-series tasks, with MCQ accuracy between 50% and 70% and reasoning up to 75%. However, they drop 10 to 20% when images are added, showing their limited ability to generalize to multimodal inputs. Proprietary Models (e.g., GPT-4.1-nano, Gemini 2-Flash) provide stable multilingual performance with narrow variance (about 10 to 12%) and lead in multimodal MCQs (about 45%) and reasoning (30 to 40%), although the top open-source LLMs outperform them in text-only settings. Multimodal Models (vision-instruct and VL variants) manage visual inputs the best, reaching 30 to 35% MCQ and 25 to 32% reasoning accuracy with images. They are the only group to outperform proprietary models in multimodal evaluations, even though they lag behind in pure time-series tasks. Multimodal models outperform others on image-based tasks, while proprietary LLMs lead in multilingual consistency and open-source LLMs excel in text-only reasoning. Performance across prompting techniques: Figures 11(a) and 11(b) compare Few-Shot and Chain-of-Thought (CoT) prompting across clinical tasks with increasing modality complexity—(a) text + time-series and (b) text + time-series + vision. DeepSeek-R1 consistently performs best, reaching up to 98.36% accuracy with negligible CoT gains, indicating strong inherent reasoning. GPT-4.1-nano shows limited improvement in MCQs but benefits modestly from CoT in multimodal settings (+3.96%). Gemini-2-Flash exhibits the largest CoT-driven gains in MCQs (+6.88%), suggesting high responsiveness to structured reasoning. In contrast, Gemma-3-27B-IT shows minimal adaptation. Vision-oriented models such as LLaMA-3.2-11B-Vision-Instruct and Qwen2.5-VL-72B perform poorly with negligible CoT benefits, highlighting shallow reasoning and weak generalization. Overall, the results indicate that only a small subset of models can effectively exploit CoT prompting across complex clinical modalities. DeepSeek-R1 consistently outperforms all models across both multimodal and multilingual clinical tasks with minimal reliance on Chain-of-Thought prompting, while others—especially proprietary and vision-centric models—exhibit limited generalization and rely heavily on CoT to boost reasoning. Extended experimental results and performance comparison of the MMTClinic compared to state-of-the-art benchmarks are shown in the Appendices 9.8 and 9.4. 5 Error Analysis To better understand the strengths and limitations of the best performing LLMs in the MMTClinic data set, we conducted an error analysis by categorizing the questions into correctly and incorrectly answered groups, as illustrated in Figure 12. The analysis reveals that questions on the left side tend to be well-structured and clearly framed, whereas those on the right suffer from issues such as limited subjectivity, insufficient contextual grounding in data, and semantic overlap. These shortcomings indicates that incorrectly answered questions typically require deeper temporal abstraction, longer-range trend integration, or joint reasoning across modalities. These errors highlight current model limitations rather than deficiencies in question construction, as all samples were clinically validated validated and based on original data. 6 Conclusion: We present MMTClinic, a clinically validated benchmark for evaluating multimodal and multilingual reasoning in large language models across text, physiological time-series, and visual data in five languages. Experiments on 13 state-of-the-art models show that while some open-source LLMs handle time-series reasoning well, proprietary models excel in complex multimodal settings. However, consistent performance drops in vision-language reasoning and regional languages reveal persistent challenges in multimodal integration and cross-lingual generalization. These gaps reflect genuine reasoning limitations rather than dataset artifacts, making MMTClinic a reliable foundation for advancing clinically grounded multimodal AI. 7 Limitations The limitations of this work are stated below : Limited Language Coverage: MMTClinic includes only five Indian languages (English, Hindi, Bengali, Marathi, Tamil), which may not generalize to other global or low-resource languages. Single Data Source: The benchmark relies solely on the 2012 PhysioNet/CinC ICU dataset, limiting evaluation to one institution’s patient population and measurement protocols . Static Visual Modality: Visual inputs are restricted to line‐plot images of time‐series trends, excluding richer imaging modalities (e.g., radiographs, CT scans) commonly encountered in clinical practice . Automatic Translation Artifacts: While translations into regional languages were reviewed by experts ‐, reliance on machine translation (Google Translate) may introduce certain subtle errors or biases in the phrasing of the questions that affect model performance. Accuracy Only Metrics: Evaluation uses accuracy for both MCQ and reasoning tasks, which does not capture other aspects of model performance such as calibration, reasoning depth, or clinical safety. Discrete Answer Formats: The current benchmark evaluates reasoning mainly through discrete outputs, such as multiple-choice questions and binary responses. While this approach allows for scalable and objective comparisons, it does not directly measure the quality of explanations or their interpretability. We see this as a trade-off between reliable evaluation and transparent reasoning. Incorporating free-form rationales and chain-of-thought supervision remains an important area for future work. 8 Ethical Considerations Use of Proprietary Models and Data Compliance: Each proprietary model was used carefully in accordance with the access guidelines that each provider had provided. Crucially, no personally identifiable or protected health information was accessed or shared at any point during the study’s use of the PhysioNet 2012 dataset, which is completely de-identified and openly accessible. In order to fully adhere to HIPAA compliance and data privacy standards, evaluations were conducted exclusively on derived QA content using synthetic prompts. 9 Appendix The Appendix provides additional details on Frequently Asked Questions 9.1, model parameters 9.2, model hyper–parameters 9.3, performance comparison compared to benchmarking datasets 9.4, performance of different prompting techniques across languages 9.5, questions 9.6, instructions for the doctor for Validation 9.7, results 9.8, and prompts 9.9. 9.1 Frequently Asked Questions (FAQ) Q1: How clinically realistic are the questions if they are generated using LLM prompting? Ans: Although the questions are generated using an LLM-based prompting pipeline, all ground-truth answers are directly sourced from the PhysioNet clinical dataset, and both MCQ and reasoning questions are curated to reflect realistic ICU scenarios. Additionally, a stratified subset of samples was reviewed by medical experts, who assigned high average clinical relevance scores, indicating that the generated questions are clinically meaningful rather than templated artifacts. Q2: Are the ground-truth answers independent of the LLM generation process? Ans: Yes. The LLM is used only to generate the question formulations, while all ground-truth answers are derived directly from the original physiological time-series and outcome files in the PhysioNet dataset. No expert curation or model-generated reasoning is used to derive answers, ensuring independence between question generation and answer supervision. Q3: Does reliance on a single ICU dataset limit generalizability? Ans: While MMTClinic is constructed using the PhysioNet 2012 ICU dataset, this choice enables controlled benchmarking with well-validated clinical signals. The benchmark is intended as an evaluation resource rather than a deployment dataset, and we view single-source construction as a common and acceptable trade-off for reproducibility and standardization in clinical benchmarks. Q4: What additional value do line-plot images provide beyond numeric time-series? Ans: The visual modality explicitly tests whether models can jointly reason over numerical trends and their visual abstractions. The observed performance drop when images are introduced highlights current limitations in multimodal temporal reasoning rather than redundancy, making this a deliberate stress test for multimodal models. Q5: Why are richer clinical imaging modalities not included? Ans: MMTClinic focuses on early ICU monitoring where physiological trends play a dominant role. Line-plot visualizations provide a controlled visual modality tightly aligned with time-series signals, allowing focused evaluation of temporal reasoning without confounding factors introduced by unrelated imaging tasks. Q6: Why were only five Indian languages selected? Ans: The selected languages represent linguistically diverse and underexplored Indic languages with substantial real-world clinical relevance. Our goal is not exhaustive multilingual coverage but to provide a fair and challenging testbed for low-resource India-specific multilingual clinical reasoning, which remains largely absent in existing benchmarks. Q7: Could machine translation artifacts affect multilingual evaluation? Ans: While initial translations were generated using Google Translate, all non-English samples were reviewed and refined by native-speaking linguists. Only a small fraction required post-editing, and expert ratings indicate consistently high linguistic quality across languages. Q8: Why is accuracy the primary evaluation metric for reasoning tasks? A: Accuracy enables objective and scalable comparison across models, languages, and modalities. Given the clinical domain, correctness of the final decision is critical, and accuracy provides a conservative baseline metric. We acknowledge that explanation quality is important and discuss it as future work. Q9: Why are free-form rationales not explicitly evaluated? A: Evaluating free-form explanations introduces subjectivity and scalability challenges. MMTClinic prioritizes reliable benchmarking of decision correctness. Incorporating rationale, faithfulness, and explanation quality is an important direction we leave for future extensions. Q10: How robust are the reported results statistically? A: All models are evaluated under identical inference settings across tasks and languages. While we report single-run inference results following standard benchmarking practice, the large evaluation scale (30K samples) mitigates variance and provides stable comparative trends. Q11: Are comparisons between open-source and proprietary models fair? A: We evaluate all models in inference-only mode using publicly available checkpoints or provider APIs with default settings. The goal is not to equalize training conditions, but to assess real-world performance differences under practical usage scenarios. Q12: Does the benchmark favor reasoning-optimized models? A: MMTClinic intentionally includes tasks that require temporal abstraction and cross-modal integration. Strong performance by reasoning-optimized models highlights genuine capability differences rather than benchmark bias, aligning with the benchmark’s stated objectives. Q13: Are models performing true reasoning or exploiting dataset patterns? A: Error analysis shows that failures predominantly occur on questions requiring long-range temporal integration and multimodal grounding, suggesting that models are not simply exploiting surface correlations. All questions are grounded in original clinical data rather than synthetic labels. Q14: How should MMTClinic be used by future researchers? A: MMTClinic is designed as an evaluation benchmark to diagnose multimodal, India specific, and time-series reasoning gaps. It should not be interpreted as a clinical decision-support system but as a controlled testbed for advancing clinically grounded AI research. Q15: What safeguards prevent misuse of the benchmark for unsafe clinical automation? A: The dataset is fully de-identified, and all evaluations are conducted on synthetic QA formulations derived from real data. We explicitly state that MMTClinic is not intended for deployment or automated decision-making without human oversight. 9.2 Model parameters Model parameters has been enlisted in Table 5. LLMs Parameters DeepSeek-R1 Guo et al. (2025a) Commercial GPT-4.1-nano Popov et al. (2025) Commercial gemini-2-flashRakshith et al. (2025) Commercial DeepSeek-R1-Llama-8B Guo et al. (2025b) 8B Gemma-3-27B-IT Team et al. (2025) 27B Llama-3.2-11B-Vision-Instruct Lee et al. (2025) 11B Llama-3.3-70B Wihl et al. (2025) 70B Mistral-7B-Instruct Samo et al. (2024) 7B QwQ-32B Zheng et al. (2024) 32B Qwen2.5-7B Yang et al. (2024) 7B Qwen2.5-VL-7B Bai et al. (2025) 7B Qwen3-235B-A22B Ji et al. (2025) 235B Qwen3-30B-A3BYang et al. (2025) 30B Table 5: List of LLMs with corresponding parameter sizes 9.3 Inference Setup and Hyperparameters All evaluated models operated in inference-only mode with publicly available or provider-hosted checkpoints. We did not perform any additional fine-tuning. For open-source models, we used PyTorch with HuggingFace Transformers and vLLM backends on NVIDIA A100 GPUs (80 GB VRAM) for inference. Unless stated otherwise, evaluations took place on a single GPU for each model instance. We used greedy decoding for all open-source models with a temperature of 0.0, top-p of 1.0, and a maximum output length of 512 tokens to guarantee consistent evaluation. We evaluated commercial models (like GPT-4o, Gemini, Claude) using their APIs with default inference settings. 9.4 Performance comparison compared to the state-of-the-art benchmarking datasets In Table 6, the proposed dataset is compared against the clinical QA and reasoning benchmarks across key dimensions. Table 6: Comparison of clinical QA and reasoning benchmarks across key dimensions. Dataset #Samples Modalities Question Types Multi. Reason. MCQ #Tasks #Langs. ECG-QA ∼ 2K ECG Signal MCQ No No Yes 1 1 ECG-Expert-QA 47K ECG Signal MCQ + Open-ended No Yes Yes Multi-cond. 1 Q-Heart 8.7K ECG + Text Open-ended No Yes No Multi-cond. 1 TIMER-Bench ∼ 10K Text + TS Instructional QA No Yes No Temporal QA 1 EHRNoteQA ∼ 2K Text + TS Open-ended No Yes No Patient QA 1 EHRXQA N/S Text + Image + TS Open-ended No Yes No EHR + X-ray 1 MedFuse Benchmark ∼ 35K TS + Image Classification No No No 3 1 MMTClinic 30K Text + TS + Image MCQ + Reasoning Yes Yes Yes 3 5 9.5 Performance of different prompting techniques across languages: Figures 15 (a) and (b) present a radar-based comparison of LLMs on multilingual reasoning tasks in five languages. They showcase model strength, cross-lingual generalization, and the effects of Few-Shot versus Chain-of-Thought prompting. Observations indicate that DeepSeek-R1 leads in all languages and setups, outperforming others by 8 to 20%. It displays consistent accuracy with a small CoT gain of about 1 to 2%. GPT-4.1-nano follows, lagging by 10 to 12%, but shows better Chain-of-Thought responsiveness, especially in Hindi and Bengali, where it gains 6 to 8%. This suggests a dependence on structured reasoning. Gemini-2-Flash exceeds lower-tier models by 5 to 10%, achieving Chain-of-Thought gains of 6 to 9% and showing moderate multilingual ability. Qwen2.5-VL-72B and LLaMA-3.2-11B-Vision-Instruct fall behind by 15 to 25%, with limited Chain-of-Thought benefits of about 2 to 4%. This points to weak multilingual reasoning. Gemma-3-27B-IT performs the worst, with a gap of 30 to 35% in Tamil and Marathi. This reveals a significant bias in language and reasoning. Overall, DeepSeek-R1 excels in language variety and reasoning depth, while others rely more on CoT to compensate for foundational weaknesses. Figure 13: * (a) Figure 14: * (b) Figure 15: Radial plot indicating language-specific accuracy of LLMs under few-shot and CoT setup for Text, Time Series, and Image modality (a) MCQ (b) Reasoning 9.6 Questions Table 7 presents detailed per-task evaluations from medical and linguistic experts, reporting high consistency across all QA types. Mean ratings ranged from 4.2 to 4.5, indicating strong clinical relevance and linguistic quality of the generated samples. Per-Task Expert Ratings (Medical, Linguistic, Mean) MCQ (Text + TimeSeries) •Medical Expert Rating: 4.3 •Linguistic Expert Rating: 4.1 •Mean Rating: 4.2 MCQ (Text + TimeSeries + Images) •Medical Expert Rating: 4.6 •Linguistic Expert Rating: 4.4 •Mean Rating: 4.5 Reasoning (Text + TimeSeries) •Medical Expert Rating: 4.5 •Linguistic Expert Rating: 4.3 •Mean Rating: 4.4 Reasoning (Text + TimeSeries + Images) •Medical Expert Rating: 4.6 •Linguistic Expert Rating: 4.4 •Mean Rating: 4.5 Table 7: Vertically formatted expert review scores for each task. 9.7 Instructions for the Doctor for Validation The clinical validation of MMTClinic was done by four medical professionals with significant experience in critical care medicine. They hold postgraduate medical degrees and have 8 to 12 years of clinical experience in intensive care units. They currently work as attending physicians at IIT Patna. Their daily responsibilities include monitoring patients in the ICU, and interpreting physiological time-series data. Linguistic validation was carried out by native-speaking language experts in Bengali, Hindi, Marathi, and Tamil. Each expert has professional experience in medical or technical translation and academic language review. This diverse involvement of experts ensures that MMTClinic is both clinically sound and linguistically accurate. For MCQ type questions: • Check Clinical Relevance: Confirm the question is clinically meaningful and reflects realistic ICU scenarios. • Validate Ground Truth Answer: Ensure the correct option matches the clinical data in the .csv or multivariate plot. • Temporal Consistency: Verify the time-frame referred to in the question aligns with data (e.g., ”first 48 hours”, ”at hour 03:45”). • No Ambiguity: Flag any question with ambiguous language or multiple interpretations. For reasoning type questions: • Clinical Soundness: Validate whether the reasoning steps reflect correct pathophysiological logic. • Data Justification: Check if the conclusions are logically derived from provided time-series values and trends. • Completeness: Ensure the reasoning includes all key variables that would influence a real clinical decision. • Avoid Overinterpretation: Ensure no conclusion is drawn beyond what the data supports (e.g., avoid diagnosis without enough evidence). • Consistency with SOFA / Mortality Logic: For subtasks involving SOFA scoring or survival prediction, verify alignment with ICU assessment guidelines. 9.8 Results In this section, empirical results have been presented in Table 8, 9, 10, 11, 12, 13, 17, 18, 19, 20 and 21. In Figure 16, some examples illustration of both MCQ and reasoning question predictions by LLMs have been presented. Modality Type Task Language Model Accuracy Small LLMs Text + Timeseries MCQ Task3 Bengali DeepSeek-R1-Llama-8B 53.3 Text + Timeseries MCQ Task1 Bengali DeepSeek-R1-Llama-8B 51.5 Text + Timeseries MCQ Task2 Bengali DeepSeek-R1-Llama-8B 52.21 Text + Timeseries MCQ Task1 Bengali Mistral-7B-Instruct 35.05 Text + Timeseries MCQ Task2 Bengali Mistral-7B-Instruct 34.38 Text + Timeseries MCQ Task3 Bengali Mistral-7B-Instruct 23 Text + Timeseries MCQ Task1 Bengali Qwen2.5-7B 8.85 Text + Timeseries MCQ Task2 Bengali Qwen2.5-7B 14.76 Text + Timeseries MCQ Task3 Bengali Qwen2.5-7B 7.95 Large LLMs Text + Timeseries MCQ Task3 Bengali QwQ-32B 44.7 Text + Timeseries MCQ Task1 Bengali QwQ-32B 43.4 Text + Timeseries MCQ Task2 Bengali QwQ-32B 41.2 Text + Timeseries MCQ Task1 Bengali Llama-3.3-70B 26.3 Text + Timeseries MCQ Task2 Bengali Llama-3.3-70B 19.32 Text + Timeseries MCQ Task3 Bengali Llama-3.3-70B 18.51 Text + Timeseries MCQ Task1 Bengali Qwen3-30B-A3B 43.34 Text + Timeseries MCQ Task2 Bengali Qwen3-30B-A3B 44.2 Text + Timeseries MCQ Task3 Bengali Qwen3-30B-A3B 32.9 Text + Timeseries MCQ Task3 Bengali Qwen3-235B-A22B 62.7 Text + Timeseries MCQ Task1 Bengali Qwen3-235B-A22B 61.2 Text + Timeseries MCQ Task2 Bengali Qwen3-235B-A22B 63.1 Text + Timeseries MCQ Task3 Bengali DeepSeek-R1 72.4 Text + Timeseries MCQ Task1 Bengali DeepSeek-R1 71.8 Text + Timeseries MCQ Task2 Bengali DeepSeek-R1 70.3 Commercial LLMs (Proprietary) Text + Timeseries MCQ Task1 Bengali GPT-4.1-nano 45.8 Text + Timeseries MCQ Task2 Bengali GPT-4.1-nano 39.8 Text + Timeseries MCQ Task3 Bengali GPT-4.1-nano 39 Text + Timeseries MCQ Task1 Bengali gemini-2-flash 19.1 Text + Timeseries MCQ Task2 Bengali gemini-2-flash 22.2 Text + Timeseries MCQ Task3 Bengali gemini-2-flash 21.7 Table 8: Model performance across tasks using different Bengali multimodal inputs Modality Type Task Language Model Accuracy Small LLMs Text + Timeseries MCQ Task3 English DeepSeek-R1-Llama-8B 59.6 Text + Timeseries MCQ Task1 English DeepSeek-R1-Llama-8B 61.24 Text + Timeseries MCQ Task2 English DeepSeek-R1-Llama-8B 58.6 Text + Timeseries MCQ Task1 English Mistral-7B-Instruct 45.99 Text + Timeseries MCQ Task2 English Mistral-7B-Instruct 44.55 Text + Timeseries MCQ Task3 English Mistral-7B-Instruct 32.02 Text + Timeseries MCQ Task1 English Qwen2.5-7B 2.3 Text + Timeseries MCQ Task2 English Qwen2.5-7B 18.64 Text + Timeseries MCQ Task3 English Qwen2.5-7B 15.65 Large LLMs (30B–70B) Text + Timeseries MCQ Task3 English QwQ-32B 51.2 Text + Timeseries MCQ Task1 English QwQ-32B 52.32 Text + Timeseries MCQ Task2 English QwQ-32B 47.23 Text + Timeseries MCQ Task1 English Llama-3.3-70B 47.91 Text + Timeseries MCQ Task2 English Llama-3.3-70B 34.42 Text + Timeseries MCQ Task3 English Llama-3.3-70B 31.19 Text + Timeseries MCQ Task1 English Qwen3-30B-A3B 47.27 Text + Timeseries MCQ Task2 English Qwen3-30B-A3B 44.98 Text + Timeseries MCQ Task3 English Qwen3-30B-A3B 36.76 Text + Timeseries MCQ Task3 English Qwen3-235B-A22B 59.1 Text + Timeseries MCQ Task1 English Qwen3-235B-A22B 52.0 Text + Timeseries MCQ Task2 English Qwen3-235B-A22B 51.1 Text + Timeseries MCQ Task3 English DeepSeek-R1 64.1 Text + Timeseries MCQ Task1 English DeepSeek-R1 65.2 Text + Timeseries MCQ Task2 English DeepSeek-R1 67.8 Commercial LLMs (Proprietary) Text + Timeseries MCQ Task1 English GPT-4.1-nano 58.6 Text + Timeseries MCQ Task2 English GPT-4.1-nano 48.4 Text + Timeseries MCQ Task3 English GPT-4.1-nano 54.4 Text + Timeseries MCQ Task1 English gemini-2-flash 19.8 Text + Timeseries MCQ Task2 English gemini-2-flash 32 Text + Timeseries MCQ Task3 English gemini-2-flash 27.6 Table 9: Model performance across tasks using different English multimodal inputs Modality Type Task Language Model Accuracy Small LLMs (≤7B) Text + Timeseries MCQ Task3 Hindi DeepSeek-R1-Llama-8B 55.2 Text + Timeseries MCQ Task1 Hindi DeepSeek-R1-Llama-8B 58.6 Text + Timeseries MCQ Task2 Hindi DeepSeek-R1-Llama-8B 54.1 Text + Timeseries MCQ Task1 Hindi Mistral-7B-Instruct 35.63 Text + Timeseries MCQ Task2 Hindi Mistral-7B-Instruct 40.27 Text + Timeseries MCQ Task3 Hindi Mistral-7B-Instruct 22.86 Text + Timeseries MCQ Task1 Hindi Qwen2.5-7B 7.19 Text + Timeseries MCQ Task2 Hindi Qwen2.5-7B 17.42 Text + Timeseries MCQ Task3 Hindi Qwen2.5-7B 14.5 Large LLMs (30B–70B) Text + Timeseries MCQ Task3 Hindi QwQ-32B 47.2 Text + Timeseries MCQ Task1 Hindi QwQ-32B 49.32 Text + Timeseries MCQ Task2 Hindi QwQ-32B 50.4 Text + Timeseries MCQ Task1 Hindi Llama-3.3-70B 29.46 Text + Timeseries MCQ Task2 Hindi Llama-3.3-70B 31.73 Text + Timeseries MCQ Task3 Hindi Llama-3.3-70B 23.08 Text + Timeseries MCQ Task1 Hindi Qwen3-30B-A3B 46.58 Text + Timeseries MCQ Task2 Hindi Qwen3-30B-A3B 43.22 Text + Timeseries MCQ Task3 Hindi Qwen3-30B-A3B 28.17 Text + Timeseries MCQ Task3 Hindi Qwen3-235B-A22B 43.7 Text + Timeseries MCQ Task1 Hindi Qwen3-235B-A22B 49.5 Text + Timeseries MCQ Task2 Hindi Qwen3-235B-A22B 45.3 Text + Timeseries MCQ Task3 Hindi DeepSeek-R1 60.3 Text + Timeseries MCQ Task1 Hindi DeepSeek-R1 61.7 Text + Timeseries MCQ Task2 Hindi DeepSeek-R1 62.2 Commercial LLMs (Proprietary) Text + Timeseries MCQ Task1 Hindi GPT-4.1-nano 45.2 Text + Timeseries MCQ Task2 Hindi GPT-4.1-nano 49.4 Text + Timeseries MCQ Task3 Hindi GPT-4.1-nano 37.2 Text + Timeseries MCQ Task1 Hindi gemini-2-flash 24.3 Text + Timeseries MCQ Task2 Hindi gemini-2-flash 37.6 Text + Timeseries MCQ Task3 Hindi gemini-2-flash 27.1 Table 10: Model performance across tasks using different Hindi multimodal inputs Modality Type Task Language Model Accuracy Small LLMs (≤7B) Text + Timeseries MCQ Task3 Marathi DeepSeek-R1-Llama-8B 58.7 Text + Timeseries MCQ Task1 Marathi DeepSeek-R1-Llama-8B 59.2 Text + Timeseries MCQ Task2 Marathi DeepSeek-R1-Llama-8B 52.6 Text + Timeseries MCQ Task1 Marathi Mistral-7B-Instruct 33.94 Text + Timeseries MCQ Task2 Marathi Mistral-7B-Instruct 32.86 Text + Timeseries MCQ Task3 Marathi Mistral-7B-Instruct 23.01 Text + Timeseries MCQ Task1 Marathi Qwen2.5-7B 6.77 Text + Timeseries MCQ Task2 Marathi Qwen2.5-7B 13.32 Text + Timeseries MCQ Task3 Marathi Qwen2.5-7B 10.61 Large LLMs (30B–70B) Text + Timeseries MCQ Task3 Marathi QwQ-32B 51.3 Text + Timeseries MCQ Task1 Marathi QwQ-32B 55.6 Text + Timeseries MCQ Task2 Marathi QwQ-32B 53.2 Text + Timeseries MCQ Task1 Marathi Llama-3.3-70B 26.08 Text + Timeseries MCQ Task2 Marathi Llama-3.3-70B 27.43 Text + Timeseries MCQ Task3 Marathi Llama-3.3-70B 23.24 Text + Timeseries MCQ Task1 Marathi Qwen3-30B-A3B 41.41 Text + Timeseries MCQ Task2 Marathi Qwen3-30B-A3B 39.55 Text + Timeseries MCQ Task3 Marathi Qwen3-30B-A3B 30.5 Text + Timeseries MCQ Task3 Marathi Qwen3-235B-A22B 40.1 Text + Timeseries MCQ Task1 Marathi Qwen3-235B-A22B 42.7 Text + Timeseries MCQ Task2 Marathi Qwen3-235B-A22B 48.4 Text + Timeseries MCQ Task3 Marathi DeepSeek-R1 61.2 Text + Timeseries MCQ Task1 Marathi DeepSeek-R1 65.2 Text + Timeseries MCQ Task2 Marathi DeepSeek-R1 61.7 Commercial LLMs (Proprietary) Text + Timeseries MCQ Task1 Marathi GPT-4.1-nano 37.8 Text + Timeseries MCQ Task2 Marathi GPT-4.1-nano 43.8 Text + Timeseries MCQ Task3 Marathi GPT-4.1-nano 29.8 Text + Timeseries MCQ Task1 Marathi gemini-2-flash 32.1 Text + Timeseries MCQ Task2 Marathi gemini-2-flash 39.4 Text + Timeseries MCQ Task3 Marathi gemini-2-flash 28.2 Table 11: Model performance across tasks using different Marathi multimodal inputs Small LLMs (≤7B) Text + Timeseries MCQ Task3 Tamil DeepSeek-R1-Llama-8B 61.2 Text + Timeseries MCQ Task1 Tamil DeepSeek-R1-Llama-8B 60.2 Text + Timeseries MCQ Task2 Tamil DeepSeek-R1-Llama-8B 56.6 Text + Timeseries MCQ Task1 Tamil Mistral-7B-Instruct 21.48 Text + Timeseries MCQ Task2 Tamil Mistral-7B-Instruct 31.56 Text + Timeseries MCQ Task3 Tamil Mistral-7B-Instruct 25.09 Text + Timeseries MCQ Task1 Tamil Qwen2.5-7B 9.73 Text + Timeseries MCQ Task2 Tamil Qwen2.5-7B 18.85 Text + Timeseries MCQ Task3 Tamil Qwen2.5-7B 8.42 Large LLMs (30B–70B) Text + Timeseries MCQ Task3 Tamil QwQ-32B 58.2 Text + Timeseries MCQ Task1 Tamil QwQ-32B 60.1 Text + Timeseries MCQ Task2 Tamil QwQ-32B 56.8 Text + Timeseries MCQ Task1 Tamil Llama-3.3-70B 26.88 Text + Timeseries MCQ Task2 Tamil Llama-3.3-70B 25.63 Text + Timeseries MCQ Task3 Tamil Llama-3.3-70B 23.4 Text + Timeseries MCQ Task1 Tamil Qwen3-30B-A3B 39.27 Text + Timeseries MCQ Task2 Tamil Qwen3-30B-A3B 39.64 Text + Timeseries MCQ Task3 Tamil Qwen3-30B-A3B 29.72 Text + Timeseries MCQ Task3 Tamil Qwen3-235B-A22B 48.3 Text + Timeseries MCQ Task1 Tamil Qwen3-235B-A22B 49.2 Text + Timeseries MCQ Task2 Tamil Qwen3-235B-A22B 51.4 Text + Timeseries MCQ Task3 Tamil DeepSeek-R1 63.5 Text + Timeseries MCQ Task1 Tamil DeepSeek-R1 67.2 Text + Timeseries MCQ Task2 Tamil DeepSeek-R1 68.1 Commercial LLMs (Proprietary) Text + Timeseries MCQ Task1 Tamil GPT-4.1-nano 33 Text + Timeseries MCQ Task2 Tamil GPT-4.1-nano 45 Text + Timeseries MCQ Task3 Tamil GPT-4.1-nano 34 Text + Timeseries MCQ Task1 Tamil gemini-2-flash 35.1 Text + Timeseries MCQ Task2 Tamil gemini-2-flash 42.6 Text + Timeseries MCQ Task3 Tamil gemini-2-flash 39.2 Table 12: Model performance across tasks using different Tamil multimodal inputs Modality Type Task Language Model Accuracy Small to Mid LLMs (≤32B) Text + Timeseries Reasoning Task3 English DeepSeek-R1-Llama-8B 60 Text + Timeseries Reasoning Task1 English DeepSeek-R1-Llama-8B 64.8 Text + Timeseries Reasoning Task2 English DeepSeek-R1-Llama-8B 53.2 Text + Timeseries Reasoning Task1 English Mistral-7B-Instruct 32.05 Text + Timeseries Reasoning Task2 English Mistral-7B-Instruct 31.23 Text + Timeseries Reasoning Task3 English Mistral-7B-Instruct 20 Text + Timeseries Reasoning Task1 English Qwen2.5-7B 9.73 Text + Timeseries Reasoning Task2 English Qwen2.5-7B 13.67 Text + Timeseries Reasoning Task3 English Qwen2.5-7B 8.32 Large LLMs (≥70B–235B) Text + Timeseries Reasoning Task3 English QwQ-32B 56 Text + Timeseries Reasoning Task1 English QwQ-32B 61.6 Text + Timeseries Reasoning Task2 English QwQ-32B 57.4 Text + Timeseries Reasoning Task1 English Llama-3.3-70B 27.8 Text + Timeseries Reasoning Task2 English Llama-3.3-70B 29.11 Text + Timeseries Reasoning Task3 English Llama-3.3-70B 24.8 Text + Timeseries Reasoning Task1 English Qwen3-30B-A3B 39.54 Text + Timeseries Reasoning Task2 English Qwen3-30B-A3B 40.8 Text + Timeseries Reasoning Task3 English Qwen3-30B-A3B 28.6 Text + Timeseries Reasoning Task3 English Qwen3-235B-A22B 60.2 Text + Timeseries Reasoning Task1 English Qwen3-235B-A22B 67.2 Text + Timeseries Reasoning Task2 English Qwen3-235B-A22B 60.2 Text + Timeseries Reasoning Task3 English DeepSeek-R1 77.6 Text + Timeseries Reasoning Task1 English DeepSeek-R1 75.2 Text + Timeseries Reasoning Task2 English DeepSeek-R1 72.4 Commercial LLMs (Proprietary) Text + Timeseries Reasoning Task1 English GPT-4.1-nano 53.3 Text + Timeseries Reasoning Task2 English GPT-4.1-nano 46.8 Text + Timeseries Reasoning Task3 English GPT-4.1-nano 40.9 Text + Timeseries Reasoning Task1 English gemini-2-flash 17.6 Text + Timeseries Reasoning Task2 English gemini-2-flash 24 Text + Timeseries Reasoning Task3 English gemini-2-flash 17.6 Table 13: Performance of models on Reasoning tasks using English multimodal inputs Modality Type Task Language Model Accuracy Small to Mid LLMs (≤32B) Text + Timeseries Reasoning Task1 Bengali DeepSeek-R1-Llama-8B 76.6 Text + Timeseries Reasoning Task2 Bengali DeepSeek-R1-Llama-8B 54.8 Text + Timeseries Reasoning Task3 Bengali DeepSeek-R1-Llama-8B 64.6 Text + Timeseries Reasoning Task1 Bengali Mistral-7B-Instruct 30.0 Text + Timeseries Reasoning Task2 Bengali Mistral-7B-Instruct 25.3 Text + Timeseries Reasoning Task3 Bengali Mistral-7B-Instruct 18.92 Text + Timeseries Reasoning Task1 Bengali Qwen2.5-7B 8.53 Text + Timeseries Reasoning Task2 Bengali Qwen2.5-7B 11.4 Text + Timeseries Reasoning Task3 Bengali Qwen2.5-7B 7.34 Large LLMs (≥70B–235B) Text + Timeseries Reasoning Task1 Bengali QwQ-32B 75.6 Text + Timeseries Reasoning Task2 Bengali QwQ-32B 61 Text + Timeseries Reasoning Task3 Bengali QwQ-32B 65.4 Text + Timeseries Reasoning Task1 Bengali Llama-3.3-70B 23.3 Text + Timeseries Reasoning Task2 Bengali Llama-3.3-70B 26.8 Text + Timeseries Reasoning Task3 Bengali Llama-3.3-70B 21.2 Text + Timeseries Reasoning Task1 Bengali Qwen3-30B-A3B 33.7 Text + Timeseries Reasoning Task2 Bengali Qwen3-30B-A3B 37.8 Text + Timeseries Reasoning Task3 Bengali Qwen3-30B-A3B 23.4 Text + Timeseries Reasoning Task1 Bengali DeepSeek-R1 84 Text + Timeseries Reasoning Task2 Bengali DeepSeek-R1 76.8 Text + Timeseries Reasoning Task3 Bengali DeepSeek-R1 80.8 Text + Timeseries Reasoning Task1 Bengali Qwen3-235B-A22B 75.8 Text + Timeseries Reasoning Task2 Bengali Qwen3-235B-A22B 63.53 Text + Timeseries Reasoning Task3 Bengali Qwen3-235B-A22B 68.4 Commercial LLMs (Proprietary) Text + Timeseries Reasoning Task1 Bengali GPT-4.1-nano 17.2 Text + Timeseries Reasoning Task2 Bengali GPT-4.1-nano 28.4 Text + Timeseries Reasoning Task3 Bengali GPT-4.1-nano 19.4 Text + Timeseries Reasoning Task1 Bengali gemini-2-flash 17.4 Text + Timeseries Reasoning Task2 Bengali gemini-2-flash 16.8 Text + Timeseries Reasoning Task3 Bengali gemini-2-flash 17.6 Table 14: Performance of models on Reasoning tasks using Bengali multimodal inputs Modality Type Task Language Model Accuracy Small to Mid LLMs (≤32B) Text + Timeseries Reasoning Task2 Hindi DeepSeek-R1-Llama-8B 57.6 Text + Timeseries Reasoning Task3 Hindi DeepSeek-R1-Llama-8B 66.6 Text + Timeseries Reasoning Task1 Hindi DeepSeek-R1-Llama-8B 75.6 Text + Timeseries Reasoning Task1 Hindi Mistral-7B-Instruct 28.6 Text + Timeseries Reasoning Task2 Hindi Mistral-7B-Instruct 24.7 Text + Timeseries Reasoning Task3 Hindi Mistral-7B-Instruct 17.43 Text + Timeseries Reasoning Task1 Hindi Qwen2.5-7B 8.42 Text + Timeseries Reasoning Task2 Hindi Qwen2.5-7B 10.6 Text + Timeseries Reasoning Task3 Hindi Qwen2.5-7B 7.12 Large LLMs (≥70B–235B) Text + Timeseries Reasoning Task2 Hindi QwQ-32B 62.8 Text + Timeseries Reasoning Task3 Hindi QwQ-32B 64.6 Text + Timeseries Reasoning Task1 Hindi QwQ-32B 74.8 Text + Timeseries Reasoning Task1 Hindi Llama-3.3-70B 24.2 Text + Timeseries Reasoning Task2 Hindi Llama-3.3-70B 24.2 Text + Timeseries Reasoning Task3 Hindi Llama-3.3-70B 23.2 Text + Timeseries Reasoning Task1 Hindi Qwen3-30B-A3B 32.6 Text + Timeseries Reasoning Task2 Hindi Qwen3-30B-A3B 38.4 Text + Timeseries Reasoning Task3 Hindi Qwen3-30B-A3B 25.1 Text + Timeseries Reasoning Task2 Hindi DeepSeek-R1 75.2 Text + Timeseries Reasoning Task3 Hindi DeepSeek-R1 83.2 Text + Timeseries Reasoning Task1 Hindi DeepSeek-R1 86.4 Text + Timeseries Reasoning Task2 Hindi Qwen3-235B-A22B 67.2 Text + Timeseries Reasoning Task3 Hindi Qwen3-235B-A22B 69 Text + Timeseries Reasoning Task1 Hindi Qwen3-235B-A22B 75.2 Commercial LLMs (Proprietary) Text + Timeseries Reasoning Task1 Hindi GPT-4.1-nano 17.6 Text + Timeseries Reasoning Task2 Hindi GPT-4.1-nano 32 Text + Timeseries Reasoning Task3 Hindi GPT-4.1-nano 17.8 Text + Timeseries Reasoning Task1 Hindi gemini-2-flash 16.8 Text + Timeseries Reasoning Task2 Hindi gemini-2-flash 32.2 Text + Timeseries Reasoning Task3 Hindi gemini-2-flash 18.4 Table 15: Performance of models on Reasoning tasks using Hindi multimodal inputs Modality Type Task Language Model Accuracy Small to Mid LLMs (≤32B) Text + Timeseries Reasoning Task1 Marathi DeepSeek-R1-Llama-8B 70.6 Text + Timeseries Reasoning Task2 Marathi DeepSeek-R1-Llama-8B 51.4 Text + Timeseries Reasoning Task3 Marathi DeepSeek-R1-Llama-8B 60 Text + Timeseries Reasoning Task1 Marathi Mistral-7B-Instruct 26.5 Text + Timeseries Reasoning Task2 Marathi Mistral-7B-Instruct 21.3 Text + Timeseries Reasoning Task3 Marathi Mistral-7B-Instruct 17.45 Text + Timeseries Reasoning Task1 Marathi Qwen2.5-7B 9.76 Text + Timeseries Reasoning Task2 Marathi Qwen2.5-7B 12.8 Text + Timeseries Reasoning Task3 Marathi Qwen2.5-7B 8.82 Large LLMs (≥70B–235B) Text + Timeseries Reasoning Task1 Marathi QwQ-32B 68.8 Text + Timeseries Reasoning Task2 Marathi QwQ-32B 60.6 Text + Timeseries Reasoning Task3 Marathi QwQ-32B 62.2 Text + Timeseries Reasoning Task1 Marathi Llama-3.3-70B 22.0 Text + Timeseries Reasoning Task2 Marathi Llama-3.3-70B 21.8 Text + Timeseries Reasoning Task3 Marathi Llama-3.3-70B 22.2 Text + Timeseries Reasoning Task1 Marathi Qwen3-30B-A3B 30.6 Text + Timeseries Reasoning Task2 Marathi Qwen3-30B-A3B 35.1 Text + Timeseries Reasoning Task3 Marathi Qwen3-30B-A3B 23.5 Text + Timeseries Reasoning Task1 Marathi DeepSeek-R1 80.4 Text + Timeseries Reasoning Task2 Marathi DeepSeek-R1 76 Text + Timeseries Reasoning Task3 Marathi DeepSeek-R1 81.2 Text + Timeseries Reasoning Task1 Marathi Qwen3-235B-A22B 73.2 Text + Timeseries Reasoning Task2 Marathi Qwen3-235B-A22B 65 Text + Timeseries Reasoning Task3 Marathi Qwen3-235B-A22B 70.8 Commercial LLMs (Proprietary) Text + Timeseries Reasoning Task1 Marathi GPT-4.1-nano 45.4 Text + Timeseries Reasoning Task2 Marathi GPT-4.1-nano 17.8 Text + Timeseries Reasoning Task3 Marathi GPT-4.1-nano 29.8 Text + Timeseries Reasoning Task1 Marathi gemini-2-flash 26.4 Text + Timeseries Reasoning Task2 Marathi gemini-2-flash 35.6 Text + Timeseries Reasoning Task3 Marathi gemini-2-flash 23.8 Table 16: Performance of models on Reasoning tasks using Marathi multimodal inputs Modality Type Task Language Model Accuracy Small to Mid LLMs (≤32B) Text + Timeseries Reasoning Task1 Tamil DeepSeek-R1-Llama-8B 80.8 Text + Timeseries Reasoning Task3 Tamil DeepSeek-R1-Llama-8B 65.2 Text + Timeseries Reasoning Task2 Tamil DeepSeek-R1-Llama-8B 56.2 Text + Timeseries Reasoning Task1 Tamil Mistral-7B-Instruct 23.87 Text + Timeseries Reasoning Task2 Tamil Mistral-7B-Instruct 23.43 Text + Timeseries Reasoning Task3 Tamil Mistral-7B-Instruct 18.2 Text + Timeseries Reasoning Task1 Tamil Qwen2.5-7B 8.32 Text + Timeseries Reasoning Task2 Tamil Qwen2.5-7B 11.3 Text + Timeseries Reasoning Task3 Tamil Qwen2.5-7B 7.5 Large LLMs (≥70B–235B) Text + Timeseries Reasoning Task1 Tamil QwQ-32B 78.8 Text + Timeseries Reasoning Task3 Tamil QwQ-32B 63.4 Text + Timeseries Reasoning Task2 Tamil QwQ-32B 65.6 Text + Timeseries Reasoning Task1 Tamil Llama-3.3-70B 23.4 Text + Timeseries Reasoning Task2 Tamil Llama-3.3-70B 23.2 Text + Timeseries Reasoning Task3 Tamil Llama-3.3-70B 25.7 Text + Timeseries Reasoning Task1 Tamil Qwen3-30B-A3B 35.7 Text + Timeseries Reasoning Task2 Tamil Qwen3-30B-A3B 31.6 Text + Timeseries Reasoning Task3 Tamil Qwen3-30B-A3B 21.7 Text + Timeseries Reasoning Task1 Tamil DeepSeek-R1 86.4 Text + Timeseries Reasoning Task3 Tamil DeepSeek-R1 85.2 Text + Timeseries Reasoning Task2 Tamil DeepSeek-R1 81.6 Text + Timeseries Reasoning Task1 Tamil Qwen3-235B-A22B 79.4 Text + Timeseries Reasoning Task3 Tamil Qwen3-235B-A22B 72 Text + Timeseries Reasoning Task2 Tamil Qwen3-235B-A22B 71.2 Commercial LLMs (Proprietary) Text + Timeseries Reasoning Task1 Tamil GPT-4.1-nano 35.4 Text + Timeseries Reasoning Task2 Tamil GPT-4.1-nano 44.1 Text + Timeseries Reasoning Task3 Tamil GPT-4.1-nano 36.2 Text + Timeseries Reasoning Task1 Tamil gemini-2-flash 36.2 Text + Timeseries Reasoning Task2 Tamil gemini-2-flash 43.6 Text + Timeseries Reasoning Task3 Tamil gemini-2-flash 37.2 Table 17: Performance of models on Reasoning tasks using Tamil multimodal inputs Modality Type Task Language Model Accuracy Text + Timeseries + Image MCQ Task1 Bengali Llama-3.2-11B-Vision-Instruct 30.05 Text + Timeseries + Image MCQ Task2 Bengali Llama-3.2-11B-Vision-Instruct 39.77 Text + Timeseries + Image MCQ Task3 Bengali Llama-3.2-11B-Vision-Instruct 27.81 Text + Timeseries + Image MCQ Task1 English Llama-3.2-11B-Vision-Instruct 34.48 Text + Timeseries + Image MCQ Task2 English Llama-3.2-11B-Vision-Instruct 39.26 Text + Timeseries + Image MCQ Task3 English Llama-3.2-11B-Vision-Instruct 30.96 Text + Timeseries + Image MCQ Task1 Hindi Llama-3.2-11B-Vision-Instruct 25.53 Text + Timeseries + Image MCQ Task2 Hindi Llama-3.2-11B-Vision-Instruct 35.81 Text + Timeseries + Image MCQ Task3 Hindi Llama-3.2-11B-Vision-Instruct 30.02 Text + Timeseries + Image MCQ Task1 Marathi Llama-3.2-11B-Vision-Instruct 32.07 Text + Timeseries + Image MCQ Task2 Marathi Llama-3.2-11B-Vision-Instruct 39.73 Text + Timeseries + Image MCQ Task3 Marathi Llama-3.2-11B-Vision-Instruct 27.52 Text + Timeseries + Image MCQ Task1 Tamil Llama-3.2-11B-Vision-Instruct 32.47 Text + Timeseries + Image MCQ Task2 Tamil Llama-3.2-11B-Vision-Instruct 33.07 Text + Timeseries + Image MCQ Task3 Tamil Llama-3.2-11B-Vision-Instruct 29.53 Text + Timeseries + Image MCQ Task1 Bengali Qwen2.5-VL-72B 33.16 Text + Timeseries + Image MCQ Task2 Bengali Qwen2.5-VL-72B 46.31 Text + Timeseries + Image MCQ Task3 Bengali Qwen2.5-VL-72B 33.78 Text + Timeseries + Image MCQ Task1 English Qwen2.5-VL-72B 37.34 Text + Timeseries + Image MCQ Task2 English Qwen2.5-VL-72B 46.31 Text + Timeseries + Image MCQ Task3 English Qwen2.5-VL-72B 38.07 Text + Timeseries + Image MCQ Task1 Hindi Qwen2.5-VL-72B 30.29 Text + Timeseries + Image MCQ Task2 Hindi Qwen2.5-VL-72B 48.35 Text + Timeseries + Image MCQ Task3 Hindi Qwen2.5-VL-72B 30.03 Text + Timeseries + Image MCQ Task1 Marathi Qwen2.5-VL-72B 30.29 Text + Timeseries + Image MCQ Task2 Marathi Qwen2.5-VL-72B 38.68 Text + Timeseries + Image MCQ Task3 Marathi Qwen2.5-VL-72B 37.27 Text + Timeseries + Image MCQ Task1 Tamil Qwen2.5-VL-72B 26.63 Text + Timeseries + Image MCQ Task2 Tamil Qwen2.5-VL-72B 44.27 Text + Timeseries + Image MCQ Task3 Tamil Qwen2.5-VL-72B 32.17 Text + Timeseries + Image MCQ Task1 Bengali Qwen2.5-VL-7B 32.67 Text + Timeseries + Image MCQ Task2 Bengali Qwen2.5-VL-7B 33.55 Text + Timeseries + Image MCQ Task3 Bengali Qwen2.5-VL-7B 25.86 Text + Timeseries + Image MCQ Task1 English Qwen2.5-VL-7B 33.33 Text + Timeseries + Image MCQ Task2 English Qwen2.5-VL-7B 38.25 Text + Timeseries + Image MCQ Task3 English Qwen2.5-VL-7B 27.31 Text + Timeseries + Image MCQ Task1 Hindi Qwen2.5-VL-7B 34.26 Text + Timeseries + Image MCQ Task2 Hindi Qwen2.5-VL-7B 32.51 Text + Timeseries + Image MCQ Task3 Hindi Qwen2.5-VL-7B 26.78 Text + Timeseries + Image MCQ Task1 Marathi Qwen2.5-VL-7B 25.13 Text + Timeseries + Image MCQ Task2 Marathi Qwen2.5-VL-7B 31.72 Text + Timeseries + Image MCQ Task3 Marathi Qwen2.5-VL-7B 24.27 Text + Timeseries + Image MCQ Task1 Tamil Qwen2.5-VL-7B 26.98 Text + Timeseries + Image MCQ Task2 Tamil Qwen2.5-VL-7B 32.38 Text + Timeseries + Image MCQ Task3 Tamil Qwen2.5-VL-7B 24.8 Text + Timeseries + Image MCQ Task1 Bengali Gemma-3-27B-IT 26.32 Text + Timeseries + Image MCQ Task1 Tamil Gemma-3-27B-IT 25.45 Text + Timeseries + Image MCQ Task1 Marathi Gemma-3-27B-IT 26.32 Text + Timeseries + Image MCQ Task1 English Gemma-3-27B-IT 27.13 Text + Timeseries + Image MCQ Task1 Hindi Gemma-3-27B-IT 29.53 Text + Timeseries + Image MCQ Task2 Hindi Gemma-3-27B-IT 21.60 Text + Timeseries + Image MCQ Task2 Bengali Gemma-3-27B-IT 34.65 Text + Timeseries + Image MCQ Task2 Tamil Gemma-3-27B-IT 43.23 Text + Timeseries + Image MCQ Task2 English Gemma-3-27B-IT 29.57 Text + Timeseries + Image MCQ Task2 Marathi Gemma-3-27B-IT 19.72 Text + Timeseries + Image MCQ Task3 Tamil Gemma-3-27B-IT 47.67 Text + Timeseries + Image MCQ Task3 Hindi Gemma-3-27B-IT 41.60 Text + Timeseries + Image MCQ Task3 English Gemma-3-27B-IT 51.70 Text + Timeseries + Image MCQ Task3 Marathi Gemma-3-27B-IT 43.3 Text + Timeseries + Image MCQ Task3 Bengali Gemma-3-27B-IT 46.26 Text + Timeseries + Image MCQ Task1 English GPT-4.1-nano 62.4 Text + Timeseries + Image MCQ Task1 Bengali GPT-4.1-nano 47.4 Text + Timeseries + Image MCQ Task1 Hindi GPT-4.1-nano 46.7 Text + Timeseries + Image MCQ Task1 Marathi GPT-4.1-nano 40.2 Text + Timeseries + Image MCQ Task1 Tamil GPT-4.1-nano 33.8 Text + Timeseries + Image MCQ Task2 English GPT-4.1-nano 50 Text + Timeseries + Image MCQ Task2 Bengali GPT-4.1-nano 48.8 Text + Timeseries + Image MCQ Task2 Hindi GPT-4.1-nano 43.6 Text + Timeseries + Image MCQ Task2 Marathi GPT-4.1-nano 44.5 Text + Timeseries + Image MCQ Task2 Tamil GPT-4.1-nano 46.4 Text + Timeseries + Image MCQ Task3 English GPT-4.1-nano 59.6 Text + Timeseries + Image MCQ Task3 Bengali GPT-4.1-nano 41.8 Text + Timeseries + Image MCQ Task3 Hindi GPT-4.1-nano 38.2 Text + Timeseries + Image MCQ Task3 Marathi GPT-4.1-nano 33.2 Text + Timeseries + Image MCQ Task3 Tamil GPT-4.1-nano 39.6 Text + Timeseries + Image MCQ Task1 English gemini-2-flash 21.6 Text + Timeseries + Image MCQ Task1 Bengali gemini-2-flash 18.2 Text + Timeseries + Image MCQ Task1 Hindi gemini-2-flash 26.4 Text + Timeseries + Image MCQ Task1 Marathi gemini-2-flash 36.8 Text + Timeseries + Image MCQ Task1 Tamil gemini-2-flash 38.3 Text + Timeseries + Image MCQ Task2 English gemini-2-flash 38.4 Text + Timeseries + Image MCQ Task2 Bengali gemini-2-flash 25.3 Text + Timeseries + Image MCQ Task2 Hindi gemini-2-flash 38 Text + Timeseries + Image MCQ Task2 Marathi gemini-2-flash 42.7 Text + Timeseries + Image MCQ Task2 Tamil gemini-2-flash 43.1 Text + Timeseries + Image MCQ Task3 English gemini-2-flash 28.3 Text + Timeseries + Image MCQ Task3 Bengali gemini-2-flash 25.3 Text + Timeseries + Image MCQ Task3 Hindi gemini-2-flash 27.4 Text + Timeseries + Image MCQ Task3 Marathi gemini-2-flash 32.2 Text + Timeseries + Image MCQ Task3 Tamil gemini-2-flash 39.1 Table 18: Performance of models on MCQ tasks with Text + Timeseries + Image inputs across multiple languages Modality Type Task Language Model Accuracy Text + Timeseries MCQ Task1 Bengali Llama-3.2-11B-Vision-Instruct 33.07 Text + Timeseries MCQ Task2 Bengali Llama-3.2-11B-Vision-Instruct 44.61 Text + Timeseries MCQ Task3 Bengali Llama-3.2-11B-Vision-Instruct 31.74 Text + Timeseries MCQ Task1 English Llama-3.2-11B-Vision-Instruct 39.07 Text + Timeseries MCQ Task2 English Llama-3.2-11B-Vision-Instruct 43.98 Text + Timeseries MCQ Task3 English Llama-3.2-11B-Vision-Instruct 34.92 Text + Timeseries MCQ Task1 Hindi Llama-3.2-11B-Vision-Instruct 29.23 Text + Timeseries MCQ Task2 Hindi Llama-3.2-11B-Vision-Instruct 41.08 Text + Timeseries MCQ Task3 Hindi Llama-3.2-11B-Vision-Instruct 33.93 Text + Timeseries MCQ Task1 Marathi Llama-3.2-11B-Vision-Instruct 35.78 Text + Timeseries MCQ Task2 Marathi Llama-3.2-11B-Vision-Instruct 45.45 Text + Timeseries MCQ Task3 Marathi Llama-3.2-11B-Vision-Instruct 31.37 Text + Timeseries MCQ Task1 Tamil Llama-3.2-11B-Vision-Instruct 35.44 Text + Timeseries MCQ Task2 Tamil Llama-3.2-11B-Vision-Instruct 38.00 Text + Timeseries MCQ Task3 Tamil Llama-3.2-11B-Vision-Instruct 33.28 Text + Timeseries MCQ Task1 Bengali Qwen2.5-VL-72B 36.76 Text + Timeseries MCQ Task2 Bengali Qwen2.5-VL-72B 52.17 Text + Timeseries MCQ Task3 Bengali Qwen2.5-VL-72B 39.45 Text + Timeseries MCQ Task1 English Qwen2.5-VL-72B 41.83 Text + Timeseries MCQ Task2 English Qwen2.5-VL-72B 52.40 Text + Timeseries MCQ Task3 English Qwen2.5-VL-72B 43.31 Text + Timeseries MCQ Task1 Hindi Qwen2.5-VL-72B 33.59 Text + Timeseries MCQ Task2 Hindi Qwen2.5-VL-72B 54.52 Text + Timeseries MCQ Task3 Hindi Qwen2.5-VL-72B 33.71 Text + Timeseries MCQ Task1 Marathi Qwen2.5-VL-72B 33.17 Text + Timeseries MCQ Task2 Marathi Qwen2.5-VL-72B 43.32 Text + Timeseries MCQ Task3 Marathi Qwen2.5-VL-72B 42.11 Text + Timeseries MCQ Task1 Tamil Qwen2.5-VL-72B 30.30 Text + Timeseries MCQ Task2 Tamil Qwen2.5-VL-72B 49.79 Text + Timeseries MCQ Task3 Tamil Qwen2.5-VL-72B 35.82 Text + Timeseries MCQ Task1 Bengali GPT-4.1-nano 70.20 Text + Timeseries MCQ Task2 Bengali GPT-4.1-nano 62.40 Text + Timeseries MCQ Task3 Bengali GPT-4.1-nano 61.60 Text + Timeseries MCQ Task1 English GPT-4.1-nano 73.80 Text + Timeseries MCQ Task2 English GPT-4.1-nano 67.40 Text + Timeseries MCQ Task3 English GPT-4.1-nano 68.20 Text + Timeseries MCQ Task1 Hindi GPT-4.1-nano 75.80 Text + Timeseries MCQ Task2 Hindi GPT-4.1-nano 62.10 Text + Timeseries MCQ Task3 Hindi GPT-4.1-nano 64.40 Text + Timeseries MCQ Task1 Marathi GPT-4.1-nano 74.30 Text + Timeseries MCQ Task2 Marathi GPT-4.1-nano 63.60 Text + Timeseries MCQ Task3 Marathi GPT-4.1-nano 66.30 Text + Timeseries MCQ Task1 Tamil GPT-4.1-nano 73.80 Text + Timeseries MCQ Task2 Tamil GPT-4.1-nano 61.40 Text + Timeseries MCQ Task3 Tamil GPT-4.1-nano 63.30 Text + Timeseries MCQ Task1 Bengali Gemini-2-Flash 51.40 Text + Timeseries MCQ Task2 Bengali Gemini-2-Flash 52.60 Text + Timeseries MCQ Task3 Bengali Gemini-2-Flash 50.20 Text + Timeseries MCQ Task1 English Gemini-2-Flash 53.20 Text + Timeseries MCQ Task2 English Gemini-2-Flash 54.10 Text + Timeseries MCQ Task3 English Gemini-2-Flash 52.40 Text + Timeseries MCQ Task1 Hindi Gemini-2-Flash 48.60 Text + Timeseries MCQ Task2 Hindi Gemini-2-Flash 46.20 Text + Timeseries MCQ Task3 Hindi Gemini-2-Flash 47.40 Text + Timeseries MCQ Task1 Marathi Gemini-2-Flash 47.30 Text + Timeseries MCQ Task2 Marathi Gemini-2-Flash 49.20 Text + Timeseries MCQ Task3 Marathi Gemini-2-Flash 48.62 Text + Timeseries MCQ Task1 Tamil Gemini-2-Flash 51.30 Text + Timeseries MCQ Task2 Tamil Gemini-2-Flash 50.30 Text + Timeseries MCQ Task3 Tamil Gemini-2-Flash 52.50 Text + Timeseries MCQ Task1 Bengali Gemma-3-27B-IT 26.32 Text + Timeseries MCQ Task2 Bengali Gemma-3-27B-IT 34.65 Text + Timeseries MCQ Task3 Bengali Gemma-3-27B-IT 46.26 Text + Timeseries MCQ Task1 English Gemma-3-27B-IT 27.13 Text + Timeseries MCQ Task2 English Gemma-3-27B-IT 29.57 Text + Timeseries MCQ Task3 English Gemma-3-27B-IT 51.70 Text + Timeseries MCQ Task1 Hindi Gemma-3-27B-IT 29.53 Text + Timeseries MCQ Task2 Hindi Gemma-3-27B-IT 21.60 Text + Timeseries MCQ Task3 Hindi Gemma-3-27B-IT 41.60 Text + Timeseries MCQ Task1 Marathi Gemma-3-27B-IT 26.32 Text + Timeseries MCQ Task2 Marathi Gemma-3-27B-IT 19.72 Text + Timeseries MCQ Task3 Marathi Gemma-3-27B-IT 43.30 Text + Timeseries MCQ Task1 Tamil Gemma-3-27B-IT 25.45 Text + Timeseries MCQ Task2 Tamil Gemma-3-27B-IT 43.23 Text + Timeseries MCQ Task3 Tamil Gemma-3-27B-IT 47.67 Text + Timeseries MCQ Task1 Bengali DeepSeek-R1 95.60 Text + Timeseries MCQ Task2 Bengali DeepSeek-R1 96.99 Text + Timeseries MCQ Task3 Bengali DeepSeek-R1 92.60 Text + Timeseries MCQ Task1 English DeepSeek-R1 93.40 Text + Timeseries MCQ Task2 English DeepSeek-R1 89.40 Text + Timeseries MCQ Task3 English DeepSeek-R1 88.20 Text + Timeseries MCQ Task1 Hindi DeepSeek-R1 96.00 Text + Timeseries MCQ Task2 Hindi DeepSeek-R1 95.80 Text + Timeseries MCQ Task3 Hindi DeepSeek-R1 91.30 Text + Timeseries MCQ Task1 Marathi DeepSeek-R1 92.40 Text + Timeseries MCQ Task2 Marathi DeepSeek-R1 89.70 Text + Timeseries MCQ Task3 Marathi DeepSeek-R1 90.50 Text + Timeseries MCQ Task1 Tamil DeepSeek-R1 95.80 Text + Timeseries MCQ Task2 Tamil DeepSeek-R1 94.30 Text + Timeseries MCQ Task3 Tamil DeepSeek-R1 91.00 Table 19: Performance of Few-Shot prompting for MCQ tasks across different languages and models Modality Type Task Language Model Accuracy Text + Timeseries MCQ Task1 Bengali Llama-3.2-11B-Vision-Instruct 38.63 Text + Timeseries MCQ Task2 Bengali Llama-3.2-11B-Vision-Instruct 50.72 Text + Timeseries MCQ Task3 Bengali Llama-3.2-11B-Vision-Instruct 38.39 Text + Timeseries MCQ Task1 English Llama-3.2-11B-Vision-Instruct 45.51 Text + Timeseries MCQ Task2 English Llama-3.2-11B-Vision-Instruct 50.28 Text + Timeseries MCQ Task3 English Llama-3.2-11B-Vision-Instruct 41.01 Text + Timeseries MCQ Task1 Hindi Llama-3.2-11B-Vision-Instruct 35.30 Text + Timeseries MCQ Task2 Hindi Llama-3.2-11B-Vision-Instruct 47.39 Text + Timeseries MCQ Task3 Hindi Llama-3.2-11B-Vision-Instruct 39.50 Text + Timeseries MCQ Task1 Marathi Llama-3.2-11B-Vision-Instruct 42.77 Text + Timeseries MCQ Task2 Marathi Llama-3.2-11B-Vision-Instruct 50.97 Text + Timeseries MCQ Task3 Marathi Llama-3.2-11B-Vision-Instruct 38.11 Text + Timeseries MCQ Task1 Tamil Llama-3.2-11B-Vision-Instruct 42.68 Text + Timeseries MCQ Task2 Tamil Llama-3.2-11B-Vision-Instruct 44.77 Text + Timeseries MCQ Task3 Tamil Llama-3.2-11B-Vision-Instruct 38.99 Text + Timeseries MCQ Task1 Bengali Qwen2.5-VL-72B 42.23 Text + Timeseries MCQ Task2 Bengali Qwen2.5-VL-72B 58.05 Text + Timeseries MCQ Task3 Bengali Qwen2.5-VL-72B 45.60 Text + Timeseries MCQ Task1 English Qwen2.5-VL-72B 46.81 Text + Timeseries MCQ Task2 English Qwen2.5-VL-72B 58.12 Text + Timeseries MCQ Task3 English Qwen2.5-VL-72B 49.03 Text + Timeseries MCQ Task1 Hindi Qwen2.5-VL-72B 38.86 Text + Timeseries MCQ Task2 Hindi Qwen2.5-VL-72B 60.51 Text + Timeseries MCQ Task3 Hindi Qwen2.5-VL-72B 39.69 Text + Timeseries MCQ Task1 Marathi Qwen2.5-VL-72B 38.20 Text + Timeseries MCQ Task2 Marathi Qwen2.5-VL-72B 48.69 Text + Timeseries MCQ Task3 Marathi Qwen2.5-VL-72B 48.70 Text + Timeseries MCQ Task1 Tamil Qwen2.5-VL-72B 35.82 Text + Timeseries MCQ Task2 Tamil Qwen2.5-VL-72B 55.29 Text + Timeseries MCQ Task3 Tamil Qwen2.5-VL-72B 41.45 Text + Timeseries MCQ Task1 Bengali GPT-4.1-nano 70.20 Text + Timeseries MCQ Task2 Bengali GPT-4.1-nano 62.40 Text + Timeseries MCQ Task3 Bengali GPT-4.1-nano 61.60 Text + Timeseries MCQ Task1 English GPT-4.1-nano 73.80 Text + Timeseries MCQ Task2 English GPT-4.1-nano 67.40 Text + Timeseries MCQ Task3 English GPT-4.1-nano 68.20 Text + Timeseries MCQ Task1 Hindi GPT-4.1-nano 75.80 Text + Timeseries MCQ Task2 Hindi GPT-4.1-nano 62.10 Text + Timeseries MCQ Task3 Hindi GPT-4.1-nano 64.40 Text + Timeseries MCQ Task1 Marathi GPT-4.1-nano 74.30 Text + Timeseries MCQ Task2 Marathi GPT-4.1-nano 63.60 Text + Timeseries MCQ Task3 Marathi GPT-4.1-nano 66.30 Text + Timeseries MCQ Task1 Tamil GPT-4.1-nano 73.80 Text + Timeseries MCQ Task2 Tamil GPT-4.1-nano 61.40 Text + Timeseries MCQ Task3 Tamil GPT-4.1-nano 63.30 Text + Timeseries MCQ Task1 Bengali Gemini-2-Flash 56.20 Text + Timeseries MCQ Task2 Bengali Gemini-2-Flash 58.48 Text + Timeseries MCQ Task3 Bengali Gemini-2-Flash 56.36 Text + Timeseries MCQ Task1 English Gemini-2-Flash 59.23 Text + Timeseries MCQ Task2 English Gemini-2-Flash 60.18 Text + Timeseries MCQ Task3 English Gemini-2-Flash 57.56 Text + Timeseries MCQ Task1 Hindi Gemini-2-Flash 53.78 Text + Timeseries MCQ Task2 Hindi Gemini-2-Flash 51.47 Text + Timeseries MCQ Task3 Hindi Gemini-2-Flash 53.28 Text + Timeseries MCQ Task1 Marathi Gemini-2-Flash 54.54 Text + Timeseries MCQ Task2 Marathi Gemini-2-Flash 57.14 Text + Timeseries MCQ Task3 Marathi Gemini-2-Flash 55.66 Text + Timeseries MCQ Task1 Tamil Gemini-2-Flash 56.54 Text + Timeseries MCQ Task2 Tamil Gemini-2-Flash 56.68 Text + Timeseries MCQ Task3 Tamil Gemini-2-Flash 58.34 Text + Timeseries MCQ Task1 Bengali Gemma-3-27B-IT 31.70 Text + Timeseries MCQ Task2 Bengali Gemma-3-27B-IT 42.23 Text + Timeseries MCQ Task3 Bengali Gemma-3-27B-IT 51.05 Text + Timeseries MCQ Task1 English Gemma-3-27B-IT 33.61 Text + Timeseries MCQ Task2 English Gemma-3-27B-IT 35.84 Text + Timeseries MCQ Task3 English Gemma-3-27B-IT 57.53 Text + Timeseries MCQ Task1 Hindi Gemma-3-27B-IT 34.33 Text + Timeseries MCQ Task2 Hindi Gemma-3-27B-IT 26.25 Text + Timeseries MCQ Task3 Hindi Gemma-3-27B-IT 47.84 Text + Timeseries MCQ Task1 Marathi Gemma-3-27B-IT 32.12 Text + Timeseries MCQ Task2 Marathi Gemma-3-27B-IT 27.22 Text + Timeseries MCQ Task3 Marathi Gemma-3-27B-IT 50.64 Text + Timeseries MCQ Task1 Tamil Gemma-3-27B-IT 30.89 Text + Timeseries MCQ Task2 Tamil Gemma-3-27B-IT 49.49 Text + Timeseries MCQ Task3 Tamil Gemma-3-27B-IT 53.93 Text + Timeseries MCQ Task1 Bengali DeepSeek-R1 07.91 Text + Timeseries MCQ Task2 Bengali DeepSeek-R1 97.88 Text + Timeseries MCQ Task3 Bengali DeepSeek-R1 97.48 Text + Timeseries MCQ Task1 English DeepSeek-R1 97.18 Text + Timeseries MCQ Task2 English DeepSeek-R1 95.26 Text + Timeseries MCQ Task3 English DeepSeek-R1 93.92 Text + Timeseries MCQ Task1 Hindi DeepSeek-R1 97.85 Text + Timeseries MCQ Task2 Hindi DeepSeek-R1 97.38 Text + Timeseries MCQ Task3 Hindi DeepSeek-R1 96.64 Text + Timeseries MCQ Task1 Marathi DeepSeek-R1 97.40 Text + Timeseries MCQ Task2 Marathi DeepSeek-R1 95.24 Text + Timeseries MCQ Task3 Marathi DeepSeek-R1 96.30 Text + Timeseries MCQ Task1 Tamil DeepSeek-R1 97.50 Text + Timeseries MCQ Task2 Tamil DeepSeek-R1 97.63 Text + Timeseries MCQ Task3 Tamil DeepSeek-R1 95.21 Table 20: Performance of CoT prompting for MCQ tasks across different languages and models Modality Type Task Language Model Accuracy Text + Timeseries + Image MCQ Task1 Bengali Llama-3.2-11B-Vision-Instruct 38.26 Text + Timeseries + Image MCQ Task2 Bengali Llama-3.2-11B-Vision-Instruct 52.51 Text + Timeseries + Image MCQ Task3 Bengali Llama-3.2-11B-Vision-Instruct 36.60 Text + Timeseries + Image MCQ Task1 English Llama-3.2-11B-Vision-Instruct 47.00 Text + Timeseries + Image MCQ Task2 English Llama-3.2-11B-Vision-Instruct 50.72 Text + Timeseries + Image MCQ Task3 English Llama-3.2-11B-Vision-Instruct 41.71 Text + Timeseries + Image MCQ Task1 Hindi Llama-3.2-11B-Vision-Instruct 33.38 Text + Timeseries + Image MCQ Task2 Hindi Llama-3.2-11B-Vision-Instruct 48.89 Text + Timeseries + Image MCQ Task3 Hindi Llama-3.2-11B-Vision-Instruct 40.83 Text + Timeseries + Image MCQ Task1 Marathi Llama-3.2-11B-Vision-Instruct 40.40 Text + Timeseries + Image MCQ Task2 Marathi Llama-3.2-11B-Vision-Instruct 51.17 Text + Timeseries + Image MCQ Task3 Marathi Llama-3.2-11B-Vision-Instruct 38.06 Text + Timeseries + Image MCQ Task1 Tamil Llama-3.2-11B-Vision-Instruct 42.65 Text + Timeseries + Image MCQ Task2 Tamil Llama-3.2-11B-Vision-Instruct 43.80 Text + Timeseries + Image MCQ Task3 Tamil Llama-3.2-11B-Vision-Instruct 38.48 Text + Timeseries + Image MCQ Task1 Bengali Qwen2.5-VL-72B 43.23 Text + Timeseries + Image MCQ Task2 Bengali Qwen2.5-VL-72B 61.29 Text + Timeseries + Image MCQ Task3 Bengali Qwen2.5-VL-72B 47.10 Text + Timeseries + Image MCQ Task1 English Qwen2.5-VL-72B 47.21 Text + Timeseries + Image MCQ Task2 English Qwen2.5-VL-72B 60.86 Text + Timeseries + Image MCQ Task3 English Qwen2.5-VL-72B 51.13 Text + Timeseries + Image MCQ Task1 Hindi Qwen2.5-VL-72B 40.01 Text + Timeseries + Image MCQ Task2 Hindi Qwen2.5-VL-72B 64.27 Text + Timeseries + Image MCQ Task3 Hindi Qwen2.5-VL-72B 40.63 Text + Timeseries + Image MCQ Task1 Marathi Qwen2.5-VL-72B 40.28 Text + Timeseries + Image MCQ Task2 Marathi Qwen2.5-VL-72B 50.84 Text + Timeseries + Image MCQ Task3 Marathi Qwen2.5-VL-72B 48.61 Text + Timeseries + Image MCQ Task1 Tamil Qwen2.5-VL-72B 36.26 Text + Timeseries + Image MCQ Task2 Tamil Qwen2.5-VL-72B 57.34 Text + Timeseries + Image MCQ Task3 Tamil Qwen2.5-VL-72B 41.99 Text + Timeseries + Image MCQ Task1 Bengali GPT-4.1-nano 82.46 Text + Timeseries + Image MCQ Task2 Bengali GPT-4.1-nano 74.82 Text + Timeseries + Image MCQ Task3 Bengali GPT-4.1-nano 72.41 Text + Timeseries + Image MCQ Task1 English GPT-4.1-nano 82.68 Text + Timeseries + Image MCQ Task2 English GPT-4.1-nano 78.78 Text + Timeseries + Image MCQ Task3 English GPT-4.1-nano 81.09 Text + Timeseries + Image MCQ Task1 Hindi GPT-4.1-nano 86.80 Text + Timeseries + Image MCQ Task2 Hindi GPT-4.1-nano 74.52 Text + Timeseries + Image MCQ Task3 Hindi GPT-4.1-nano 78.69 Text + Timeseries + Image MCQ Task1 Marathi GPT-4.1-nano 83.91 Text + Timeseries + Image MCQ Task2 Marathi GPT-4.1-nano 74.84 Text + Timeseries + Image MCQ Task3 Marathi GPT-4.1-nano 76.91 Text + Timeseries + Image MCQ Task1 Tamil GPT-4.1-nano 85.34 Text + Timeseries + Image MCQ Task2 Tamil GPT-4.1-nano 72.88 Text + Timeseries + Image MCQ Task3 Tamil GPT-4.1-nano 75.15 Text + Timeseries + Image MCQ Task1 Bengali Gemini-2-Flash 61.63 Text + Timeseries + Image MCQ Task2 Bengali Gemini-2-Flash 69.86 Text + Timeseries + Image MCQ Task3 Bengali Gemini-2-Flash 65.57 Text + Timeseries + Image MCQ Task1 English Gemini-2-Flash 66.44 Text + Timeseries + Image MCQ Task2 English Gemini-2-Flash 70.33 Text + Timeseries + Image MCQ Task3 English Gemini-2-Flash 68.53 Text + Timeseries + Image MCQ Task1 Hindi Gemini-2-Flash 57.87 Text + Timeseries + Image MCQ Task2 Hindi Gemini-2-Flash 59.46 Text + Timeseries + Image MCQ Task3 Hindi Gemini-2-Flash 60.66 Text + Timeseries + Image MCQ Task1 Marathi Gemini-2-Flash 59.71 Text + Timeseries + Image MCQ Task2 Marathi Gemini-2-Flash 63.33 Text + Timeseries + Image MCQ Task3 Marathi Gemini-2-Flash 64.90 Text + Timeseries + Image MCQ Task1 Tamil Gemini-2-Flash 62.43 Text + Timeseries + Image MCQ Task2 Tamil Gemini-2-Flash 61.60 Text + Timeseries + Image MCQ Task3 Tamil Gemini-2-Flash 64.72 Text + Timeseries + Image MCQ Task1 Bengali Gemma-3-27B-IT 41.49 Text + Timeseries + Image MCQ Task2 Bengali Gemma-3-27B-IT 50.52 Text + Timeseries + Image MCQ Task3 Bengali Gemma-3-27B-IT 59.64 Text + Timeseries + Image MCQ Task1 English Gemma-3-27B-IT 44.17 Text + Timeseries + Image MCQ Task2 English Gemma-3-27B-IT 44.76 Text + Timeseries + Image MCQ Task3 English Gemma-3-27B-IT 66.23 Text + Timeseries + Image MCQ Task1 Hindi Gemma-3-27B-IT 41.73 Text + Timeseries + Image MCQ Task2 Hindi Gemma-3-27B-IT 33.05 Text + Timeseries + Image MCQ Task3 Hindi Gemma-3-27B-IT 54.57 Text + Timeseries + Image MCQ Task1 Marathi Gemma-3-27B-IT 42.78 Text + Timeseries + Image MCQ Task2 Marathi Gemma-3-27B-IT 30.32 Text + Timeseries + Image MCQ Task3 Marathi Gemma-3-27B-IT 57.19 Text + Timeseries + Image MCQ Task1 Tamil Gemma-3-27B-IT 39.75 Text + Timeseries + Image MCQ Task2 Tamil Gemma-3-27B-IT 59.27 Text + Timeseries + Image MCQ Task3 Tamil Gemma-3-27B-IT 61.93 Text + Timeseries + Image MCQ Task1 Bengali DeepSeek-R1 97.60 Text + Timeseries + Image MCQ Task2 Bengali DeepSeek-R1 97.99 Text + Timeseries + Image MCQ Task3 Bengali DeepSeek-R1 96.60 Text + Timeseries + Image MCQ Task1 English DeepSeek-R1 96.40 Text + Timeseries + Image MCQ Task2 English DeepSeek-R1 97.40 Text + Timeseries + Image MCQ Task3 English DeepSeek-R1 97.20 Text + Timeseries + Image MCQ Task1 Hindi DeepSeek-R1 98.00 Text + Timeseries + Image MCQ Task2 Hindi DeepSeek-R1 98.80 Text + Timeseries + Image MCQ Task3 Hindi DeepSeek-R1 97.30 Text + Timeseries + Image MCQ Task1 Marathi DeepSeek-R1 97.40 Text + Timeseries + Image MCQ Task2 Marathi DeepSeek-R1 98.70 Text + Timeseries + Image MCQ Task3 Marathi DeepSeek-R1 98.50 Text + Timeseries + Image MCQ Task1 Tamil DeepSeek-R1 97.80 Text + Timeseries + Image MCQ Task2 Tamil DeepSeek-R1 97.30 Text + Timeseries + Image MCQ Task3 Tamil DeepSeek-R1 97.00 Table 21: Few-Shot Accuracy for reasoning tasks across different languages and models using Text + Timeseries + Image modality Modality Type Task Language Model Accuracy Text + Timeseries + Image MCQ Task1 Bengali Llama-3.2-11B-Vision-Instruct 45.63 Text + Timeseries + Image MCQ Task2 Bengali Llama-3.2-11B-Vision-Instruct 58.76 Text + Timeseries + Image MCQ Task3 Bengali Llama-3.2-11B-Vision-Instruct 39.82 Text + Timeseries + Image MCQ Task1 English Llama-3.2-11B-Vision-Instruct 48.82 Text + Timeseries + Image MCQ Task2 English Llama-3.2-11B-Vision-Instruct 55.39 Text + Timeseries + Image MCQ Task3 English Llama-3.2-11B-Vision-Instruct 47.87 Text + Timeseries + Image MCQ Task1 Hindi Llama-3.2-11B-Vision-Instruct 37.72 Text + Timeseries + Image MCQ Task2 Hindi Llama-3.2-11B-Vision-Instruct 49.38 Text + Timeseries + Image MCQ Task3 Hindi Llama-3.2-11B-Vision-Instruct 47.42 Text + Timeseries + Image MCQ Task1 Marathi Llama-3.2-11B-Vision-Instruct 47.33 Text + Timeseries + Image MCQ Task2 Marathi Llama-3.2-11B-Vision-Instruct 55.23 Text + Timeseries + Image MCQ Task3 Marathi Llama-3.2-11B-Vision-Instruct 44.23 Text + Timeseries + Image MCQ Task1 Tamil Llama-3.2-11B-Vision-Instruct 48.82 Text + Timeseries + Image MCQ Task2 Tamil Llama-3.2-11B-Vision-Instruct 46.78 Text + Timeseries + Image MCQ Task3 Tamil Llama-3.2-11B-Vision-Instruct 41.35 Text + Timeseries + Image MCQ Task1 Bengali Qwen2.5-VL-72B 48.22 Text + Timeseries + Image MCQ Task2 Bengali Qwen2.5-VL-72B 65.67 Text + Timeseries + Image MCQ Task3 Bengali Qwen2.5-VL-72B 49.62 Text + Timeseries + Image MCQ Task1 English Qwen2.5-VL-72B 49.44 Text + Timeseries + Image MCQ Task2 English Qwen2.5-VL-72B 67.56 Text + Timeseries + Image MCQ Task3 English Qwen2.5-VL-72B 57.65 Text + Timeseries + Image MCQ Task1 Hindi Qwen2.5-VL-72B 47.40 Text + Timeseries + Image MCQ Task2 Hindi Qwen2.5-VL-72B 69.54 Text + Timeseries + Image MCQ Task3 Hindi Qwen2.5-VL-72B 46.69 Text + Timeseries + Image MCQ Task1 Marathi Qwen2.5-VL-72B 45.27 Text + Timeseries + Image MCQ Task2 Marathi Qwen2.5-VL-72B 53.42 Text + Timeseries + Image MCQ Task3 Marathi Qwen2.5-VL-72B 50.77 Text + Timeseries + Image MCQ Task1 Tamil Qwen2.5-VL-72B 43.66 Text + Timeseries + Image MCQ Task2 Tamil Qwen2.5-VL-72B 59.86 Text + Timeseries + Image MCQ Task3 Tamil Qwen2.5-VL-72B 45.66 Text + Timeseries + Image MCQ Task1 Bengali GPT-4.1-nano 86.12 Text + Timeseries + Image MCQ Task2 Bengali GPT-4.1-nano 79.44 Text + Timeseries + Image MCQ Task3 Bengali GPT-4.1-nano 78.99 Text + Timeseries + Image MCQ Task1 English GPT-4.1-nano 88.73 Text + Timeseries + Image MCQ Task2 English GPT-4.1-nano 83.49 Text + Timeseries + Image MCQ Task3 English GPT-4.1-nano 88.26 Text + Timeseries + Image MCQ Task1 Hindi GPT-4.1-nano 89.50 Text + Timeseries + Image MCQ Task2 Hindi GPT-4.1-nano 78.27 Text + Timeseries + Image MCQ Task3 Hindi GPT-4.1-nano 83.73 Text + Timeseries + Image MCQ Task1 Marathi GPT-4.1-nano 89.56 Text + Timeseries + Image MCQ Task2 Marathi GPT-4.1-nano 79.05 Text + Timeseries + Image MCQ Task3 Marathi GPT-4.1-nano 79.30 Text + Timeseries + Image MCQ Task1 Tamil GPT-4.1-nano 89.59 Text + Timeseries + Image MCQ Task2 Tamil GPT-4.1-nano 78.18 Text + Timeseries + Image MCQ Task3 Tamil GPT-4.1-nano 79.43 Text + Timeseries + Image MCQ Task1 Bengali Gemini-2-Flash 69.87 Text + Timeseries + Image MCQ Task2 Bengali Gemini-2-Flash 74.92 Text + Timeseries + Image MCQ Task3 Bengali Gemini-2-Flash 69.72 Text + Timeseries + Image MCQ Task1 English Gemini-2-Flash 71.47 Text + Timeseries + Image MCQ Task2 English Gemini-2-Flash 75.32 Text + Timeseries + Image MCQ Task3 English Gemini-2-Flash 71.43 Text + Timeseries + Image MCQ Task1 Hindi Gemini-2-Flash 62.92 Text + Timeseries + Image MCQ Task2 Hindi Gemini-2-Flash 63.52 Text + Timeseries + Image MCQ Task3 Hindi Gemini-2-Flash 64.56 Text + Timeseries + Image MCQ Task1 Marathi Gemini-2-Flash 63.42 Text + Timeseries + Image MCQ Task2 Marathi Gemini-2-Flash 68.43 Text + Timeseries + Image MCQ Task3 Marathi Gemini-2-Flash 67.63 Text + Timeseries + Image MCQ Task1 Tamil Gemini-2-Flash 66.13 Text + Timeseries + Image MCQ Task2 Tamil Gemini-2-Flash 67.65 Text + Timeseries + Image MCQ Task3 Tamil Gemini-2-Flash 67.23 Text + Timeseries + Image MCQ Task1 Bengali Gemma-3-27B-IT 46.52 Text + Timeseries + Image MCQ Task2 Bengali Gemma-3-27B-IT 55.62 Text + Timeseries + Image MCQ Task3 Bengali Gemma-3-27B-IT 62.73 Text + Timeseries + Image MCQ Task1 English Gemma-3-27B-IT 48.34 Text + Timeseries + Image MCQ Task2 English Gemma-3-27B-IT 47.43 Text + Timeseries + Image MCQ Task3 English Gemma-3-27B-IT 69.28 Text + Timeseries + Image MCQ Task1 Hindi Gemma-3-27B-IT 44.34 Text + Timeseries + Image MCQ Task2 Hindi Gemma-3-27B-IT 38.34 Text + Timeseries + Image MCQ Task3 Hindi Gemma-3-27B-IT 58.63 Text + Timeseries + Image MCQ Task1 Marathi Gemma-3-27B-IT 47.42 Text + Timeseries + Image MCQ Task2 Marathi Gemma-3-27B-IT 35.63 Text + Timeseries + Image MCQ Task3 Marathi Gemma-3-27B-IT 63.24 Text + Timeseries + Image MCQ Task1 Tamil Gemma-3-27B-IT 43.82 Text + Timeseries + Image MCQ Task2 Tamil Gemma-3-27B-IT 63.56 Text + Timeseries + Image MCQ Task3 Tamil Gemma-3-27B-IT 63.67 Text + Timeseries + Image MCQ Task1 Bengali DeepSeek-R1 97.90 Text + Timeseries + Image MCQ Task2 Bengali DeepSeek-R1 98.70 Text + Timeseries + Image MCQ Task3 Bengali DeepSeek-R1 97.50 Text + Timeseries + Image MCQ Task1 English DeepSeek-R1 97.80 Text + Timeseries + Image MCQ Task2 English DeepSeek-R1 98.90 Text + Timeseries + Image MCQ Task3 English DeepSeek-R1 97.20 Text + Timeseries + Image MCQ Task1 Hindi DeepSeek-R1 98.70 Text + Timeseries + Image MCQ Task2 Hindi DeepSeek-R1 98.90 Text + Timeseries + Image MCQ Task3 Hindi DeepSeek-R1 97.90 Text + Timeseries + Image MCQ Task1 Marathi DeepSeek-R1 98.30 Text + Timeseries + Image MCQ Task2 Marathi DeepSeek-R1 98.90 Text + Timeseries + Image MCQ Task3 Marathi DeepSeek-R1 98.80 Text + Timeseries + Image MCQ Task1 Tamil DeepSeek-R1 98.20 Text + Timeseries + Image MCQ Task2 Tamil DeepSeek-R1 98.60 Text + Timeseries + Image MCQ Task3 Tamil DeepSeek-R1 98.00 Table 22: CoT Accuracy for reasoning tasks across different languages and models using Text + Timeseries + Image modality Modality Type Task Language Model Accuracy Text + Timeseries + Image Reasoning Task1 Bengali Llama-3.2-11B-Vision-Instruct 29.25 Text + Timeseries + Image Reasoning Task2 Bengali Llama-3.2-11B-Vision-Instruct 33.34 Text + Timeseries + Image Reasoning Task3 Bengali Llama-3.2-11B-Vision-Instruct 21.31 Text + Timeseries + Image Reasoning Task1 English Llama-3.2-11B-Vision-Instruct 30.12 Text + Timeseries + Image Reasoning Task2 English Llama-3.2-11B-Vision-Instruct 37.19 Text + Timeseries + Image Reasoning Task3 English Llama-3.2-11B-Vision-Instruct 23.8 Text + Timeseries + Image Reasoning Task1 Hindi Llama-3.2-11B-Vision-Instruct 23.12 Text + Timeseries + Image Reasoning Task2 Hindi Llama-3.2-11B-Vision-Instruct 33.43 Text + Timeseries + Image Reasoning Task3 Hindi Llama-3.2-11B-Vision-Instruct 23.9 Text + Timeseries + Image Reasoning Task1 Marathi Llama-3.2-11B-Vision-Instruct 32.07 Text + Timeseries + Image Reasoning Task2 Marathi Llama-3.2-11B-Vision-Instruct 35.33 Text + Timeseries + Image Reasoning Task3 Marathi Llama-3.2-11B-Vision-Instruct 25.31 Text + Timeseries + Image Reasoning Task1 Tamil Llama-3.2-11B-Vision-Instruct 31.22 Text + Timeseries + Image Reasoning Task2 Tamil Llama-3.2-11B-Vision-Instruct 32.31 Text + Timeseries + Image Reasoning Task3 Tamil Llama-3.2-11B-Vision-Instruct 27.21 Text + Timeseries + Image Reasoning Task1 Bengali Qwen2.5-VL-72B 30.73 Text + Timeseries + Image Reasoning Task1 Tamil Qwen2.5-VL-72B 32.03 Text + Timeseries + Image Reasoning Task1 Marathi Qwen2.5-VL-72B 29.17 Text + Timeseries + Image Reasoning Task1 English Qwen2.5-VL-72B 27.08 Text + Timeseries + Image Reasoning Task1 Hindi Qwen2.5-VL-72B 26.86 Text + Timeseries + Image Reasoning Task2 Hindi Qwen2.5-VL-72B 43.65 Text + Timeseries + Image Reasoning Task2 Bengali Qwen2.5-VL-72B 43.12 Text + Timeseries + Image Reasoning Task2 Tamil Qwen2.5-VL-72B 46.56 Text + Timeseries + Image Reasoning Task2 English Qwen2.5-VL-72B 39.15 Text + Timeseries + Image Reasoning Task2 Marathi Qwen2.5-VL-72B 41.80 Text + Timeseries + Image Reasoning Task3 Tamil Qwen2.5-VL-72B 50.26 Text + Timeseries + Image Reasoning Task3 Hindi Qwen2.5-VL-72B 47.89 Text + Timeseries + Image Reasoning Task3 English Qwen2.5-VL-72B 47.37 Text + Timeseries + Image Reasoning Task3 Marathi Qwen2.5-VL-72B 48.68 Text + Timeseries + Image Reasoning Task3 Bengali Qwen2.5-VL-72B 55.26 Text + Timeseries + Image Reasoning Task1 Bengali Qwen2.5-VL-7B 31.27 Text + Timeseries + Image Reasoning Task1 Tamil Qwen2.5-VL-7B 34.70 Text + Timeseries + Image Reasoning Task1 Marathi Qwen2.5-VL-7B 39.05 Text + Timeseries + Image Reasoning Task1 English Qwen2.5-VL-7B 29.16 Text + Timeseries + Image Reasoning Task1 Hindi Qwen2.5-VL-7B 36.15 Text + Timeseries + Image Reasoning Task2 Hindi Qwen2.5-VL-7B 36.13 Text + Timeseries + Image Reasoning Task2 Bengali Qwen2.5-VL-7B 36.93 Text + Timeseries + Image Reasoning Task2 Tamil Qwen2.5-VL-7B 41.20 Text + Timeseries + Image Reasoning Task2 English Qwen2.5-VL-7B 31.20 Text + Timeseries + Image Reasoning Task2 Marathi Qwen2.5-VL-7B 45.33 Text + Timeseries + Image Reasoning Task3 Tamil Qwen2.5-VL-7B 44.74 Text + Timeseries + Image Reasoning Task3 Hindi Qwen2.5-VL-7B 40.03 Text + Timeseries + Image Reasoning Task3 English Qwen2.5-VL-7B 41.51 Text + Timeseries + Image Reasoning Task3 Marathi Qwen2.5-VL-7B 42.45 Text + Timeseries + Image Reasoning Task3 Bengali Qwen2.5-VL-7B 41.51 Text + Timeseries + Image Reasoning Task1 Bengali Gemma-3-27B-IT 22.96 Text + Timeseries + Image Reasoning Task1 Tamil Gemma-3-27B-IT 21.77 Text + Timeseries + Image Reasoning Task1 Marathi Gemma-3-27B-IT 21.24 Text + Timeseries + Image Reasoning Task1 English Gemma-3-27B-IT 23.48 Text + Timeseries + Image Reasoning Task1 Hindi Gemma-3-27B-IT 22.03 Text + Timeseries + Image Reasoning Task2 Hindi Gemma-3-27B-IT 19.20 Text + Timeseries + Image Reasoning Task2 Bengali Gemma-3-27B-IT 30.93 Text + Timeseries + Image Reasoning Task2 Tamil Gemma-3-27B-IT 28.13 Text + Timeseries + Image Reasoning Task2 English Gemma-3-27B-IT 27.87 Text + Timeseries + Image Reasoning Task2 Marathi Gemma-3-27B-IT 17.87 Text + Timeseries + Image Reasoning Task3 Tamil Gemma-3-27B-IT 46.77 Text + Timeseries + Image Reasoning Task3 Hindi Gemma-3-27B-IT 43.80 Text + Timeseries + Image Reasoning Task3 English Gemma-3-27B-IT 49.60 Text + Timeseries + Image Reasoning Task3 Marathi Gemma-3-27B-IT 42.45 Text + Timeseries + Image Reasoning Task3 Bengali Gemma-3-27B-IT 45.15 Text + Timeseries + Image Reasoning Task1 English GPT-4.1-nano 56.4 Text + Timeseries + Image Reasoning Task1 Bengali GPT-4.1-nano 17.4 Text + Timeseries + Image Reasoning Task1 Hindi GPT-4.1-nano 19 Text + Timeseries + Image Reasoning Task1 Marathi GPT-4.1-nano 27.2 Text + Timeseries + Image Reasoning Task1 Tamil GPT-4.1-nano 36 Text + Timeseries + Image Reasoning Task2 English GPT-4.1-nano 49.7 Text + Timeseries + Image Reasoning Task2 Bengali GPT-4.1-nano 30.8 Text + Timeseries + Image Reasoning Task2 Hindi GPT-4.1-nano 29 Text + Timeseries + Image Reasoning Task2 Marathi GPT-4.1-nano 36.2 Text + Timeseries + Image Reasoning Task2 Tamil GPT-4.1-nano 46.2 Text + Timeseries + Image Reasoning Task3 English GPT-4.1-nano 41.2 Text + Timeseries + Image Reasoning Task3 Bengali GPT-4.1-nano 18.2 Text + Timeseries + Image Reasoning Task3 Hindi GPT-4.1-nano 14.16 Text + Timeseries + Image Reasoning Task3 Marathi GPT-4.1-nano 22.4 Text + Timeseries + Image Reasoning Task3 Tamil GPT-4.1-nano 37 Text + Timeseries + Image Reasoning Task1 English gemini-2-flash 19.7 Text + Timeseries + Image Reasoning Task1 Bengali gemini-2-flash 17.2 Text + Timeseries + Image Reasoning Task1 Hindi gemini-2-flash 19.2 Text + Timeseries + Image Reasoning Task1 Marathi gemini-2-flash 27.2 Text + Timeseries + Image Reasoning Task1 Tamil gemini-2-flash 39.4 Text + Timeseries + Image Reasoning Task2 English gemini-2-flash 28.4 Text + Timeseries + Image Reasoning Task2 Bengali gemini-2-flash 16.4 Text + Timeseries + Image Reasoning Task2 Hindi gemini-2-flash 27 Text + Timeseries + Image Reasoning Task2 Marathi gemini-2-flash 38.6 Text + Timeseries + Image Reasoning Task2 Tamil gemini-2-flash 45.2 Text + Timeseries + Image Reasoning Task3 English gemini-2-flash 20.4 Text + Timeseries + Image Reasoning Task3 Bengali gemini-2-flash 17.8 Text + Timeseries + Image Reasoning Task3 Hindi gemini-2-flash 17.4 Text + Timeseries + Image Reasoning Task3 Marathi gemini-2-flash 23.4 Text + Timeseries + Image Reasoning Task3 Tamil gemini-2-flash 38.1 Table 23: Performance of models on *Reasoning* tasks with Text + Timeseries + Image inputs across multiple languages. Figure 16: Example illustration of both MCQ and Reasoning predictions by LLM 9.9 Prompts A. Prompt for MCQ-QAs based on Text and Time Series: ⬇ You are a data assistant for ICU time-series analysis. You have access to: - (*@\patient\_id\.csv@*): (Time, Parameter, Value) - (*@Outcomes-a.csv@*): Patient-level in-hospital outcomes Focus only on: HR, DiasABP, SysABP, RespRate, Temp, Glucose Generate *2 distinct MCQs* for each of the following tasks: --- (*@Subtask 1: Predicting In-hospital Death@*) Answer is directly available from Outcomes-a.csv. Base questions on early (6–12h) stats or threshold values. Example format: Q. Based on vitals from the first 12 hours, what was the patient’s in-hospital outcome? A) Discharged to home B) Stable with ICU stay C) (*@Died in-hospital@*) D) Recovered without complications Correct Answer: C — Died in-hospital (Outcomes-a.csv = 1) --- (*@Subtask 2: Forecasting Heart Rate (HR)@*) Use 48-hour HR trends from patient_id.csv to compute mean/max/min. Example format: Q. What was the minimum HR observed in the first 48 hours? A) 85 bpm B) 98 bpm C) (*@76 bpm@*) D) 101 bpm Correct Answer: C — 76 bpm (observed in file) --- (*@Subtask 3: Estimating SOFA Score (Hypothetical)@*) Use parameter trends to infer plausible SOFA severity level. Base answers on clinical patterns from first 48h. Example format: Q. Based on sustained hypotension and respiratory distress, what is the likely SOFA status? A) Normal physiology B) Mild dysfunction (SOFA 1–3) C) (*@Moderate dysfunction (SOFA 4–6)@*) D) Multi-organ failure (SOFA >10) Correct Answer: C — Consistent with observed multi-parameter abnormality --- (*@Sample Data@*): First 20 rows from \patient\_id\.csv (*@ patient\_df.head(20).to\_string(index=False)@*) Outcome from Outcomes-a.csv: **in_hospital_death** Now generate **6 MCQs (2 per subtask)** with: - 4 options (A–D) - 1 clearly correct answer (justified by data or outcome) - 3 plausible distractors - No reuse of prior templates or examples Listing 1: ICU Task-based MCQ Generation Prompt B.Prompt for MCQ-QAs based on Text, Time Series, and Image: ⬇ You are a data assistant for ICU time-series analysis. You have access to: - <(*@\patient\_id\.csv@*)>: (Time, Parameter, Value) - <(*@\patient\_id\_main.png@*)>: Line plots of vitals - <(*@Outcomes-a.csv@*)>: In-hospital outcomes Focus only on these parameters: HR, DiasABP, SysABP, RespRate, Temp, Glucose Generate exactly **2 multiple-choice questions (MCQs)** for each subtask: --- (*@Subtask 1: Predicting In-hospital death@*) Use the first 6–12 hours of time-series data and compare with outcome in Outcomes-a.csv. Example format: Q. Based on the patient’s early vitals, what is the likely in-hospital outcome? A) Survived with stable vitals B) Unknown C) (*@Died in-hospital@*) D) Transferred to step-down Correct Answer: C — Died in-hospital (Outcomes-a.csv = 1) --- (*@Subtask 2: Forecasting HR during first 48 hours@*) Use HR trends from the CSV to calculate mean, max, min, or specific time-window comparisons. Example format: Q. What was the maximum HR recorded during the first 48 hours? A) 102 bpm B) (*@110 bpm@*) C) 95 bpm D) 88 bpm Correct Answer: B — 110 bpm (observed from patient_id.csv) --- (*@Subtask 3: Estimating SOFA score severity@*) Use physiological deviations to infer plausible SOFA-related conditions. Ground answer in trends from 48h data. Example format: Q. Based on sustained low blood pressure and elevated respiration, what is the plausible SOFA status? A) No organ dysfunction B) Minor dysfunction (SOFA < 2) C) (*@Moderate dysfunction (SOFA 4–6)@*) D) Severe failure (SOFA > 10) Correct Answer: C — Moderate dysfunction inferred from BP and RespRate patterns --- Below is a sample from patient_id’s ICU time-series data: (*@ patient\_df.head(20).to\_string(index=False)@*) Outcome: In-hospital death = (*@\in\_hospital\_death\@*) (*@Image: \patient\_id\_main.png@*) --- Now generate exactly **6 MCQs** (2 per subtask), with: - 4 answer options per question (A–D) - One correct and clearly supported/inferred answer - Three plausible but incorrect distractors - No reuse of examples or prior questions Listing 2: Multitask ICU MCQ Generation Prompt C. Prompt for Reasoning-QAs based on Text and Time Series: ⬇ You are a specialized data analyst for ICU time-series. You have access to: - (*@\patient\_id\.csv@*): Contains (Time, Parameter, Value) - (*@Outcomes-a.csv@*): Contains patient outcomes Use only these parameters: HR, DiasABP, SysABP, RespRate, Temp, Glucose Generate **one reasoning-style open-ended question per task**: - Subtask 1: Predicting in-hospital death (0: survived, 1: died) - Subtask 2: Forecasting HR using first 48h of HR time-series - Subtask 3: Predicting SOFA score using multivariate ICU data Each question must: 1. Use at least two parameters or timestamps 2. Require multi-hop numeric logic (trends, comparisons, averages, thresholds) 3. Include a brief answer using only (*@\patient\_id\.csv@*) and (*@Outcomes-a.csv@*) --- (*@Example Format@*) Subtask 1 Question: Check if the patient has ≥3 episodes (4h each) of hypotension (SysABP < 90, DiasABP < 60) and tachycardia (HR > 100) in the first 48h. Based on these, did the patient survive? Answer: Data shows ≥3 qualifying episodes. According to Outcomes-a.csv, in-hospital death = in_hospital_death. --- Subtask 2 Question: Does HR increase by >20% within 8h of Glucose >180 (compared to the 4h average before)? If yes, is this an early warning sign? Answer: Yes, at least two instances show this pattern in the 48h window. --- Subtask 3 Question: Average DiasABP and SysABP, RespRate >25 (hourly count), and Temp <36°C occurrences. Do these patterns indicate elevated SOFA risk? Answer: DiasABP is low, RespRate >25 in 15+ hours, Temp <36°C observed. SOFA score in Outcomes-a.csv = sofa_score. --- Patient snapshot (top 20 rows of \patient\_id\.csv): (*@ patient\_df.head(20).to\_string(index=False)@*) Outcome: In-hospital death = (*@\in\_hospital\_death\@*) (*@SOFA score = \sofa\_score\@*) if available Now generate one new question and answer per subtask (not reused from above). Listing 3: ICU Reasoning QA Generation Prompt D. Prompt for Reasoning-QAs based on Text, Time Series, and Image: ⬇ You are a data assistant for ICU time-series reasoning. You have access to: - <"patient_id.csv">: (Time, Parameter, Value) — ICU vitals - <"patient_id_main.png">: line plots of HR, DiasABP, SysABP, RespRate, Temp, Glucose - <"Outcomes-a.csv">: patient outcome data Only consider these parameters from patient_id.csv: HR, DiasABP, SysABP, RespRate, Temp, Glucose Generate **two reasoning questions per subtask** for: - Subtask 1: Predicting In-hospital death (0 = survived, 1 = died) - Subtask 2: Forecasting HR based on first 48h HR data - Subtask 3: Predicting SOFA score from multivariate time-series Each question must: 1. Involve at least two parameters or timestamps 2. Require multi-hop numeric logic (e.g., trends, comparisons, averages) 3. Include a brief **Answer** using only data from the three files --- (*@Example Format@*) Subtask 1 Question: Check for hypotension (SysABP < 90, DiasABP < 60) with HR > 100 for 3+ distinct 4h periods. Based on this, what does <"Outcomes-a.csv"> report for in-hospital death? Answer: patient_id.csv shows these conditions met 3+ times. In-hospital_death from Outcomes-a.csv is in_hospital_death. --- Subtask 2 Question: Does HR rise >20% within 8h after Glucose >180 compared to the 4h average before? If so, does this suggest early deterioration? Answer: Yes, in 2+ instances, HR spikes >20% post-glucose. Indicates deterioration. --- Subtask 3 Question: Avg DiasABP and SysABP in first 48h + count RespRate >25 + any Temp <36°C. Do these support high SOFA score? Answer: DiasABP low, RespRate >25 in 15+ hours, Temp <36°C twice. SOFA in Outcomes-a.csv is sofa_score. --- Sample from patient_id.csv (top 20 rows): patient_df.head(20).to_string(index=False) In-hospital death: **in_hospital_death** f"SOFA score: **sofa_score**" if sofa_score is not None else "" --- Now generate two novel reasoning-style Q&A pairs per subtask using the data. Do not reuse previous questions. Listing 4: Reasoning QA Generation Prompt E. Prompt for Reasoning-QA evaluation based on Text and Time Series: ⬇ You are a medical expert in ICU time-series interpretation. Input: - Patient CSV: (*@\PATIENT\_DF\@*) Vitals: HR, DiasABP, SysABP, RespRate, Temp, Glucose Your task: 1. Analyze the data 2. Answer the reasoning question 3. Return ‘1‘ if data clearly supports a conclusion 4. Return ‘0‘ if data is inconclusive Task: (*@\TASK\@*) --- Patient Data: (*@\PATIENT\_DF\@*) Question: (*@\QUESTION\@*) --- Instructions: - Use only the given data - No external medical knowledge - Return only: ‘1‘ or ‘0‘ - (*@No explanations or extra text@*) Listing 5: ICU Time-Series Reasoning Prompt F. Prompt for Reasoning-QA evaluation based on Text, Time Series, and Image: ⬇ You are a medical expert in ICU time-series and signal plot interpretation. Inputs: - CSV file: (*@\PATIENT\_DF\@*) - Plot image: (*@\IMAGE\@*) Parameters: HR, DiasABP, SysABP, RespRate, Temp, Glucose Your task: 1. Review the data and image 2. Assess the reasoning question 3. Return ‘1‘ if evidence supports a clear conclusion 4. Return ‘0‘ if evidence is insufficient Task: (*@\TASK\@*) --- Patient Data: (*@\PATIENT\_DF\@*) Image: (*@\IMAGE\@*) Question: (*@\QUESTION\@*) --- Instructions: - Use only the given data and image - Do not apply external medical knowledge - Return only: ‘1‘ or ‘0‘ - (*@No explanations or additional output@*) Listing 6: Multimodal Reasoning Prompt G. Prompt for MCQ-QA evaluation based on Text and Time Series: ⬇ You are an expert in interpreting ICU time-series data. The patient data is in CSV format with one row per timestamp. Parameters: HR, DiasABP, SysABP, RespRate, Temp, Glucose Use only the data below to answer the MCQ. Return (*@only one letter@*): A, B, C, or D. Do not explain or output anything else. Task: (*@\TASK\@*) --- Patient Data: (*@\PATIENT\_DF\@*) Question: (*@\QUESTION\@*) Options: (*@\OPTIONS\@*) Listing 7: ICU Time-Series MCQ Prompt H. Prompt for MCQ-QA evaluation based on Text, Time Series and Image: ⬇ You are an expert in interpreting ICU time-series data. Given patient vitals in CSV format and a corresponding image, use the information to answer the following multiple-choice question. Available parameters: - HR, DiasABP, SysABP, RespRate, Temp, Glucose Use only the data: (*@\PATIENT\_DF\@*) and image: (*@\IMAGE\@*) The question pertains to: (*@Task \TASK\@*) --- Patient Data: (*@\PATIENT\_DF\@*) Image: (*@\IMAGE\@*) Question: (*@\QUESTION\@*) Options: (*@\OPTIONS\@*) Return only one letter: A, B, C, or D. Listing 8: Multimodal ICU Prompt I. Few-shot prompt for MCQ-QA evaluation based on Text and Time Series: ⬇ You are an expert in interpreting ICU time-series data. You are given a CSV file containing vital signs for a patient. Each row corresponds to one timestamp, with parameters: - HR (Heart Rate) - DiasABP (Diastolic Arterial BP) - SysABP (Systolic Arterial BP) - RespRate (Respiration Rate) - Temp (Body Temperature) - Glucose (Blood Glucose Level) --- (*@Example 1:@*) Question: (*@ \QUESTION1\@*) Options: (*@ \OPTION1\@*) Answer: (*@ \ANSWER1\@*) --- (*@Example 2:@*) Question: (*@ \QUESTION2\@*) Options: (*@ \OPTION2\@*) Answer: (*@ \ANSWER2\@*) --- Use only the provided data to answer the following multiple-choice question. Strictly return (*@only the letter@*): A, B, C, or D. (*@No explanation. No extra text.@*) --- Task: (*@\TASK\@*) Patient Data: (*@ \PATIENT\_DF\@*) --- Question: (*@ \QUESTION\@*) Options: (*@ \OPTIONS\@*) Answer with only one letter: A, B, C, or D Listing 9: ICU Time-Series MCQ Prompt J. CoT prompt for MCQ-QA evaluation based on Text and Time Series: ⬇ You are a clinical decision support system specializing in interpreting ICU time-series physiological data to assist with evidence-based reasoning. Your task is to analyze patient vital trends and apply step-by-step clinical logic to answer a multiple-choice question. The time-series dataset includes: - HR: Heart Rate (bpm) - SysABP: Systolic Arterial BP (mmHg) - DiasABP: Diastolic Arterial BP (mmHg) - RespRate: Respiratory Rate (breaths/min) - Temp: Body Temperature (°C) - Glucose: Blood Glucose Level (mg/dL) You will be given a clinical task and patient data. Use the structured reasoning format below. --- (*@Example 1@*) Task: Clinical deterioration prediction Question: (*@ \QUESTION1\@*) Options: (*@ \OPTION1\@*) Answer Reasoning: Abnormal HR and RespRate with falling BP suggest decompensation. The likely interpretation is: (*@\ANSWER1\@*) --- (*@Example 2@*) Task: Detection of glycemic instability Question: (*@ \QUESTION2\@*) Options: (*@ \OPTION2\@*) Answer Reasoning: Glucose fluctuation with temp/HR change implies poor glycemic control. The correct conclusion is: (*@\ANSWER2\@*) --- (*@Now analyze the following ICU case using this reasoning template@*) Task: (*@\TASK\@*) --- Patient Data: (*@ \PATIENT\_DF\@*) --- Question: (*@ \QUESTION\@*) --- Options (choose one): (*@ \OPTIONS\@*) --- Step-by-Step Reasoning: (Interpret trends, flag abnormalities, and link findings to task context.) Final Answer: (*@A, B, C, or D@*) Listing 10: Clinical Reasoning Prompt with Step-by-Step Analysis K. Few-shot prompt for MCQ-QA evaluation based on Text, Time Series and Image: ⬇ You are an expert in interpreting ICU time-series and image-based physiological data. You are provided with: - Vital signs CSV data for a patient (timestamped time-series) - A corresponding line plot image visualizing parameter trends Available parameters include: - HR (Heart Rate) - DiasABP (Diastolic Arterial BP) - SysABP (Systolic Arterial BP) - RespRate (Respiratory Rate) - Temp (Body Temperature) - Glucose (Blood Glucose Level) --- (*@Example 1@*) Question: (*@ \QUESTION1\@*) Options: (*@ \OPTION1\@*) Answer: (*@\ANSWER1\@*) --- (*@Example 2@*) Question: (*@ \QUESTION2\@*) Options: (*@ \OPTION2\@*) Answer: (*@\ANSWER2\@*) --- Use only the provided time-series data and image to answer the MCQ below. Task: (*@\TASK\@*) --- Patient Data: (*@ \PATIENT\_DF\@*) Image: (*@ \IMAGE\@*) --- Question: (*@ \QUESTION\@*) Options: (*@ \OPTIONS\@*) Answer strictly with one letter only: (*@A, B, C, or D@*) Listing 11: ICU Multimodal MCQ Prompt (Time-Series + Image) L. CoT prompt for MCQ-QA evaluation based on Text, Time Series and Image: ⬇ You are a clinical decision support system specializing in analyzing ICU time-series data and corresponding line plot images. Your task is to evaluate physiological patterns and reason through multiple-choice questions using clinical logic. Input: - ICU patient vital signs (CSV time-series) - Line plot image showing trends in HR, SysABP, DiasABP, RespRate, Temp, Glucose --- (*@Example 1@*) Task: Predicting cardiovascular instability Question: (*@ \QUESTION1\@*) Options: (*@ \OPTION1\@*) Answer Reasoning: The image shows declining BP; CSV confirms tachycardia and rising RespRate. Together, this suggests early shock. Correct Answer: (*@\ANSWER1\@*) --- (*@Example 2@*) Task: Detecting metabolic abnormality Question: (*@ \QUESTION2\@*) Options: (*@ \OPTION2\@*) Answer Reasoning: Image shows glucose swings and dips; CSV aligns with temp spikes, indicating metabolic stress. Correct Answer: (*@\ANSWER2\@*) --- (*@Now apply the same clinical reasoning to the case below@*) Task: (*@\TASK\@*) --- Patient Data (CSV): (*@ \PATIENT\_DF\@*) Image (Line Plot): (*@ \IMAGE\@*) --- Question: (*@ \QUESTION\@*) Options (choose one): (*@ \OPTIONS\@*) --- Step-by-Step Reasoning: (Examine vitals + image. Identify abnormalities, infer clinical implications, and relate them to the question.) Final Answer: (*@A, B, C, or D@*) Listing 12: Multimodal Clinical Reasoning Prompt (Time-Series + Image) M. Few-shot prompt for Reasoning-QA evaluation based on Text, Time Series and Image: ⬇ You are a medical expert in interpreting ICU time-series data and physiological signal plots. You are given: - ICU patient data in CSV format - A time-series line plot image of vital parameters Parameters include: - HR (Heart Rate) - DiasABP (Diastolic Arterial BP) - SysABP (Systolic Arterial BP) - RespRate (Respiration Rate) - Temp (Body Temperature) - Glucose (Blood Glucose Level) You must: 1. Analyze patient data and line plot 2. Evaluate the reasoning question 3. Return ‘1‘ if the data supports the conclusion 4. Return ‘0‘ if the data is inconclusive --- (*@Example 1@*) Task: (*@ \EX\_TASK\_1\@*) Patient Data (CSV): (*@ \EX\_PATIENT\_DF\_1\@*) Vital Sign Image: (*@ \EX\_IMAGE\_1\@*) Reasoning Question: (*@ \EX\_QUESTION\_1\@*) Answer: (*@\EX\_ANSWER\_1\@*) --- (*@Example 2@*) Task: (*@ \EX\_TASK\_2\@*) Patient Data (CSV): (*@ \EX\_PATIENT\_DF\_2\@*) Vital Sign Image: (*@ \EX\_IMAGE\_2\@*) Reasoning Question: (*@ \EX\_QUESTION\_2\@*) Answer: (*@\EX\_ANSWER\_2\@*) --- (*@Now analyze the following patient case@*) Task: (*@ \TASK\@*) Patient Data (CSV): (*@ \PATIENT\_DF\@*) Vital Sign Image: (*@ \IMAGE\@*) Reasoning Question: (*@ \QUESTION\@*) --- Instructions: - Analyze trends, values, and correlations from both the CSV and the image - Do not use external medical knowledge - Return only: ‘1‘ or ‘0‘ - (*@No explanations. No extra text. Strictly one digit.@*) Listing 13: Binary Classification Prompt (Multimodal Clinical Reasoning) N. CoT prompt for Reasoning-QA evaluation based on Text, Time Series and Image: ⬇ You are a medical expert in interpreting ICU time-series physiological data and trend images. You are given: - ICU patient data in CSV format - A plot of time-series trends for: - HR (Heart Rate) - DiasABP (Diastolic Arterial BP) - SysABP (Systolic Arterial BP) - RespRate (Respiratory Rate) - Temp (Body Temperature) - Glucose (Blood Glucose Level) Your task is to: 1. Examine the data and visual trends 2. Decide whether the question can be definitively answered 3. Return ‘1‘ if the data supports a clear answer, or ‘0‘ if insufficient/inconclusive --- (*@Example 1@*) Task: (*@ \EX\_TASK\_1\@*) Patient Data (CSV): (*@ \EX\_PATIENT\_DF\_1\@*) Vital Sign Image: (*@ \EX\_IMAGE\_1\@*) Reasoning Question: (*@ \EX\_QUESTION\_1\@*) Step-by-Step Reasoning: (*@ \EX\_REASONING\_1\@*) Answer: (*@\EX\_ANSWER\_1\@*) --- (*@Example 2@*) Task: (*@ \EX\_TASK\_2\@*) Patient Data (CSV): (*@ \EX\_PATIENT\_DF\_2\@*) Vital Sign Image: (*@ \EX\_IMAGE\_2\@*) Reasoning Question: (*@ \EX\_QUESTION\_2\@*) Step-by-Step Reasoning: (*@ \EX\_REASONING\_2\@*) Answer: (*@\EX\_ANSWER\_2\@*) --- (*@Now analyze the following patient case using the same reasoning process@*) Task: (*@ \TASK\@*) Patient Data (CSV): (*@ \PATIENT\_DF\@*) Vital Sign Image: (*@ \IMAGE\@*) Reasoning Question: (*@ \QUESTION\@*) --- Step-by-Step Reasoning: (Identify and correlate physiological patterns from data and plot. Determine if they support a clear conclusion.) Final Answer: (*@Return only one digit: 1 or 0@*) Listing 14: Multimodal ICU Binary Reasoning Prompt References Acharya et al. (2025) A. Acharya, A. Ghosh, P. Verma, K. Pasupa, S. Saha, and P. Singh M3retrieve: benchmarking multimodal retrieval for medicine. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 15274–15287. Cited by: §2. Alonso et al. (2024) I. Alonso, M. Oronoz, and R. Agerri Medexpqa: multilingual benchmarking of large language models for medical question answering. Artificial intelligence in medicine 155, p. 102938. Cited by: §2. Bae et al. (2023) S. Bae, D. Kyung, J. Ryu, E. Cho, G. Lee, S. Kweon, J. Oh, L. Ji, E. I. Chang, T. Kim, and E. Choi EHRXQA: a multi-modal question answering dataset for electronic health records with chest x-ray images. arXiv preprint arXiv:2310.18652. External Links: Link Cited by: §1. Bai et al. (2025) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: §4.1, Table 5. Brown et al. (2025) K. E. Brown, C. Yan, Z. Li, X. Zhang, B. X. Collins, Y. Chen, E. W. Clayton, M. Kantarcioglu, Y. Vorobeychik, and B. A. Malin Large language models are less effective at clinical prediction tasks than locally trained machine learning models. Journal of the American Medical Informatics Association 32 (5), p. 811–822. Cited by: §2. Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877–1901. Cited by: §2. Chan et al. (2024) N. Chan, F. Parker, W. Bennett, T. Wu, M. Y. Jia, J. Fackler, and K. Ghobadi Medtsllm: leveraging llms for multimodal medical time series analysis. arXiv preprint arXiv:2408.07773. Cited by: §1. Ghosh et al. (2024a) A. Ghosh, A. Acharya, R. Jain, S. Saha, A. Chadha, and S. Sinha Clipsyntel: clip and llm synergy for multimodal question summarization in healthcare. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 22031–22039. Cited by: §2. Ghosh et al. (2024b) A. Ghosh, A. Acharya, P. Jha, S. Saha, A. Gaudgaul, R. Majumdar, A. Chadha, R. Jain, S. Sinha, and S. Agarwal Medsumm: a multimodal approach to summarizing code-mixed hindi-english clinical queries. In European Conference on Information Retrieval, p. 106–120. Cited by: §2. Ghosh et al. (2026) A. Ghosh, A. Acharya, M. Muhsin, A. Chadha, V. Jain, S. Sinha, and S. Saha IndicMedQA: multimodal medical query analysis in indian languages. ACM Transactions on Computing for Healthcare. Cited by: §2. Ghosh et al. (2024c) A. Ghosh, A. Acharya, S. Saha, G. Pandey, D. Raghu, and S. Sinha Healthalignsumm: utilizing alignment for multimodal summarization of code-mixed healthcare dialogues. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 11546–11560. Cited by: §2. Ghosh et al. (2025a) A. Ghosh, D. Datta, S. Saha, and C. Agarwal A survey of multilingual reasoning in language models. arXiv preprint arXiv:2502.09457. Cited by: §2. Ghosh et al. (2025b) A. Ghosh, S. Sridhar, R. K. Ravi, M. Muhsin, S. Saha, and C. Agarwal CLINIC: evaluating multilingual trustworthiness in language models for healthcare. arXiv preprint arXiv:2512.11437. Cited by: §2. Ghosh et al. (2024d) A. Ghosh, M. Tomar, A. Tiwari, S. Saha, J. Salve, and S. Sinha From sights to insights: towards summarization of multimodal clinical documents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 13117–13129. Cited by: §2. Google DeepMind (2025) Google DeepMind Gemini 2.0 flash. Note: Developer announcement via Google AI Studio / Vertex AI1 million‑token context, multimodal I/O, low‑latency workhorse model External Links: Link Cited by: §4.1. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §4.1. Guo et al. (2025a) D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §4.1, Table 5. Guo et al. (2025b) J. Guo, Y. Wu, J. Qiu, K. Huang, X. Juan, L. Yang, and M. Wang Temporal consistency for llm reasoning process error identification. arXiv preprint arXiv:2503.14495. Cited by: Table 5. Hager et al. (2024) P. Hager, F. Jungmann, R. Holland, K. Bhagat, I. Hubrecht, M. Knauer, J. Vielhauer, M. Makowski, R. Braren, G. Kaissis, et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature medicine 30 (9), p. 2613–2622. Cited by: §2. Halder et al. (2026) T. K. Halder, A. Ghosh, S. Baidya, A. Roy, and S. Saha ArogyaSutra: a multi-agent framework for multimodal medical reasoning in indic languages. arXiv preprint arXiv:2606.13572. Cited by: §2. Hayat et al. (2022) N. Hayat, K. Geras, and F. Shamout MedFuse: multi-modal fusion with clinical time-series data and chest x-ray images. In Proceedings of the Machine Learning for Health Conference, Vol. 182, p. 1–20. External Links: Link Cited by: §1. Hu et al. (2023) M. Hu, L. Wang, S. Yan, D. Ma, Q. Ren, P. Xia, W. Feng, P. Duan, L. Ju, and Z. Ge Nurvid: a large expert-level video database for nursing procedure activity understanding. Advances in Neural Information Processing Systems 36, p. 18146–18164. Cited by: §1. Hu et al. (2025) Y. Hu, Y. Li, P. Liu, Y. Zhu, N. Li, T. Dai, S. Xia, D. Cheng, and C. Jiang Fintsb: a comprehensive and practical benchmark for financial time series forecasting. arXiv preprint arXiv:2502.18834. Cited by: §1. Hui et al. (2024) B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. Cited by: §4.1. Jacob and Shasha (1999) K. J. Jacob and D. Shasha Fintime: a financial time series benchmark. ACM SIGMOD Record 28 (4), p. 42–48. Cited by: §1. Ji et al. (2025) Y. Ji, X. Tian, S. Zhao, H. Wang, S. Chen, Y. Peng, H. Zhao, and X. Li AM-thinking-v1: advancing the frontier of reasoning at 32b scale. arXiv preprint arXiv:2505.08311. Cited by: Table 5. Jin et al. (2023) M. Jin, Q. Wen, Y. Liang, C. Zhang, S. Xue, X. Wang, J. Zhang, Y. Wang, H. Chen, X. Li, et al. Large models for time series and spatio-temporal data: a survey and outlook. arXiv preprint arXiv:2310.10196. Cited by: §2. Kim et al. (2025) J. Kim et al. TIMER: temporal instruction modeling and evaluation for longitudinal clinical records. arXiv preprint arXiv:2503.04176. External Links: Link Cited by: §1. Kim et al. (2025) J. Kim, A. Podlasek, K. Shidara, F. Liu, A. Alaa, and D. Bernardo Limitations of large language models in clinical problem-solving arising from inflexible reasoning. arXiv preprint arXiv:2502.04381. Cited by: §2. Kweon et al. (2024) S. Kweon, J. Kim, H. Kwak, D. Cha, H. Yoon, K. H. Kim, J. Yang, S. Won, and E. Choi EHRNoteQA: a patient-specific question answering benchmark for evaluating large language models in clinical settings. PhysioNet. External Links: Link Cited by: §1. Lee et al. (2025) J. Lee, K. Song, S. Yang, D. Lim, J. Kim, W. Shin, B. Kim, Y. J. Lee, and T. Kim Efficient llama-3.2-vision by trimming cross-attended visual features. arXiv preprint arXiv:2504.00557. Cited by: §4.1, Table 5. Li et al. (2024) S. Li, V. Balachandran, S. Feng, J. Ilgen, E. Pierson, P. W. W. Koh, and Y. Tsvetkov Mediq: question-asking llms and a benchmark for reliable interactive clinical reasoning. Advances in Neural Information Processing Systems 37, p. 28858–28888. Cited by: §2. Liévin et al. (2024) V. Liévin, C. E. Hother, A. G. Motzfeldt, and O. Winther Can large language models reason about medical questions?. Patterns 5 (3). Cited by: §2. Mariani et al. (2026) M. C. Mariani, S. Malakar, A. Bagchi, S. Basu, S. Goswami, O. K. Tweneboah, S. Biswas, A. Dey, and A. Sinha Evaluating the efficacy of large language models in stock market decision-making: a decision-focused, price-only, multi-country analysis using historical price data. Machine Learning and Knowledge Extraction 8 (4), p. 104. Cited by: §2. Mousavi and Termehchy (2023) J. Mousavi and A. Termehchy Towards consistent language models using declarative constraints. arXiv preprint arXiv:2312.15472. Cited by: §2. Oh et al. (2023) J. Oh, S. Bae, G. Lee, J. Kwon, and E. Choi ECG-qa: a comprehensive question answering dataset combined with electrocardiogram. arXiv preprint arXiv:2306.15681. External Links: Link Cited by: §1. Onyame et al. (2026) E. Onyame, A. Ghosh, S. Baidya, S. Saha, X. Chen, and C. Agarwal CURE-med: curriculum-informed reinforcement learning for multilingual medical reasoning. arXiv preprint arXiv:2601.13262. Cited by: §2. OpenAI et al. (2023) OpenAI et al. GPT-4 technical report. External Links: 2303.08774, Link Cited by: §1, §4.1. OpenAI (2025) OpenAI GPT‑4.1 nano. Note: API announcementFastest, cheapest variant in GPT‑4.1 series; 1 million‐token context External Links: Link Cited by: §4.1. Pal et al. (2022) A. Pal, L. K. Umapathi, and M. Sankarasubbu Medmcqa: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, p. 248–260. Cited by: §2. Pham et al. (2025) H. M. Pham et al. Q-heart: ecg question answering via knowledge-informed multimodal llms. arXiv preprint arXiv:2505.06296. External Links: Link Cited by: §1. Popov et al. (2025) D. Popov, E. Terentev, D. Serenko, I. Sochenkov, and I. Buyanov Transferring natural language datasets between languages using large language models for modern decision support and sci-tech analytical systems. Big Data and Cognitive Computing 9 (5), p. 116. Cited by: Table 5. Rakshith et al. (2025) R. Rakshith, M. S. Khan, and A. Chopra AICOE at peranssumm 2025: an ensemble of large language models for perspective-aware healthcare answer summarization. In Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health), p. 398–408. Cited by: Table 5. Rasp et al. (2020) S. Rasp, P. D. Dueben, S. Scher, J. A. Weyn, S. Mouatadid, and N. Thuerey WeatherBench: a benchmark data set for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems 12 (11), p. e2020MS002203. Cited by: §1. Samo et al. (2024) H. Samo, K. Ali, M. Memon, F. A. Abbasi, M. Y. Koondhar, and K. Dahri Fine-tuning mistral 7b large language model for python query response and code generation: a parameter efficient approach. VAWKUM Transactions on Computer Sciences 12 (1), p. 205–217. Cited by: §4.1, Table 5. Silva et al. (2012) I. Silva, G. Moody, R. Mark, and L. A. Celi Predicting mortality of icu patients: the physionet/computing in cardiology challenge 2012. Predicting Mortality of ICU Patients: The PhysioNet/Computing in Cardiology Challenge 1. Cited by: §1. Singh et al. (2024) A. K. Singh, R. Murthy, J. Sen, A. Mittal, G. Ramakrishnan, et al. Indic qa benchmark: a multilingual benchmark to evaluate question answering capability of llms for indic languages. arXiv preprint arXiv:2407.13522. Cited by: §2. Singhal et al. (2025) K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, M. Amin, L. Hou, K. Clark, S. R. Pfohl, H. Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature Medicine, p. 1–8. Cited by: §2. Team et al. (2023) G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Cited by: §4.1. Team et al. (2025) G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. Cited by: Table 5. Team (2025) Q. Team Qwq-32b: embracing the power of reinforcement learning. March. Cited by: §4.1. Tu et al. (2024) T. Tu, S. Azizi, D. Driess, M. Schaekermann, M. Amin, P. Chang, A. Carroll, C. Lau, R. Tanno, I. Ktena, et al. Towards generalist biomedical ai. Nejm Ai 1 (3), p. AIoa2300138. Cited by: §1. Wang et al. (2025) X. Wang, J. Kang, and P. Han ECG-expert-qa: a benchmark for evaluating medical large language models in heart disease diagnosis. arXiv preprint arXiv:2502.17475. External Links: Link Cited by: §1. Wihl et al. (2025) J. Wihl, E. Rosenkranz, S. Schramm, C. Berberich, M. Griessmair, P. Woźnicki, F. Pinto, S. Ziegelmayer, L. C. Adams, K. K. Bressem, et al. Data extraction from free-text stroke ct reports using gpt-4o and llama-3.3-70b: the impact of annotation guidelines. medRxiv, p. 2025–01. Cited by: Table 5. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4.1, Table 5. Yang et al. (2024) A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. Cited by: §4.1, Table 5. Yao et al. (2024) Z. Yao, Z. Zhang, C. Tang, X. Bian, Y. Zhao, Z. Yang, J. Wang, H. Zhou, W. S. Jang, F. Ouyang, et al. Medqa-cs: benchmarking large language models clinical skills using an ai-sce framework. arXiv preprint arXiv:2410.01553. Cited by: §2. Ye et al. (2021) Q. Ye, C. Hsieh, Z. Yang, Y. Kang, J. Chen, D. Cao, S. He, and T. Hou A unified drug–target interaction prediction framework based on knowledge graph and recommendation system. Nature communications 12 (1), p. 6775. Cited by: §1. Zheng et al. (2024) C. Zheng, Z. Zhang, B. Zhang, R. Lin, K. Lu, B. Yu, D. Liu, J. Zhou, and J. Lin Processbench: identifying process errors in mathematical reasoning. arXiv preprint arXiv:2412.06559. Cited by: Table 5. [60] Y. Zhou, X. Liu, C. Ning, X. Zhang, C. Yan, X. Fu, and J. Wu Revisiting the scaling effects of llms on medical reasoning capabilities. Cited by: §2.