Paper deep dive
Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languages
Mohammadreza Ghaffarzadeh-Esfahani, Nahid Yousefian, Ebrahim Heidari-Farsani, Ali Akbar Omidvarian, Sepehr Ghahraei, Atena Farangi, AmirBahador Boroumand
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 1:00:08 PM
Summary
This study evaluates a two-step pipeline for privacy-preserving clinical information extraction from low-resource Persian transcripts using a translation model (Aya-expanse-8B) and five small language models (SLMs). The pipeline processes 1,221 anonymized palliative care transcripts to extract 13 binary clinical features. Qwen2.5-7B-Instruct achieved the highest overall performance (macro-F1: 0.899), while larger models generally outperformed smaller ones in sensitivity and Matthews Correlation Coefficient. Translating Persian to English improved sensitivity and reduced missing outputs but slightly lowered specificity. Physiological symptoms were reliably extracted, whereas psychological and administrative features remained challenging.
Entities (16)
Relation Signals (11)
Aya-Expanse 8B → translatesto → English
confidence 98% · Persian-to-English translation model
Aya-Expanse 8B → usedfortranslation → Persian
confidence 98% · Aya-expanse-8B as a Persian-to-English translation model
Qwen2.5-7B-Instruct → achievedhighestperformance → Macro-averaged F1 score
confidence 95% · Qwen2.5-7B-Instruct achieved the highest overall performance (median macro-F1: 0.899)
Larger models → outperform → Smaller models
confidence 95% · Larger models (7B--8B parameters) consistently outperformed smaller counterparts in sensitivity and MCC
Persian transcripts → processedby → Aya-Expanse 8B
confidence 95% · direct Persian processing ... in the native language
Gemma 3 1B-IT → showedweakestresults → Performance
confidence 95% · Gemma-3-1B-it showed the weakest results
Psychological complaints → challengingfor → Most models
confidence 90% · psychological complaints ... remained challenging
Translation to English → improves → Sensitivity
confidence 90% · translating Persian transcripts to English improved sensitivity
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Extracting clinical information from medical transcripts in low-resource languages remains a significant challenge in healthcare natural language processing (NLP). This study evaluates a two-step pipeline combining Aya-expanse-8B as a Persian-to-English translation model with five open-source small language models (SLMs) -- Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, and Gemma-3-1B-it -- for binary extraction of 13 clinical features from 1,221 anonymized Persian transcripts collected at a cancer palliative care call center. Using a few-shot prompting strategy without fine-tuning, models were assessed on macro-averaged F1-score, Matthews Correlation Coefficient (MCC), sensitivity, and specificity to account for class imbalance. Qwen2.5-7B-Instruct achieved the highest overall performance (median macro-F1: 0.899; MCC: 0.797), while Gemma-3-1B-it showed the weakest results. Larger models (7B--8B parameters) consistently outperformed smaller counterparts in sensitivity and MCC. A bilingual analysis of Aya-expanse-8B revealed that translating Persian transcripts to English improved sensitivity, reduced missing outputs, and boosted metrics robust to class imbalance, though at the cost of slightly lower specificity and precision. Feature-level results showed reliable extraction of physiological symptoms across most models, whereas psychological complaints, administrative requests, and complex somatic features remained challenging. These findings establish a practical, privacy-preserving blueprint for deploying open-source SLMs in multilingual clinical NLP settings with limited infrastructure and annotation resources, and highlight the importance of jointly optimizing model scale and input language strategy for sensitive healthcare applications.
Tags
Links
- Source: https://arxiv.org/abs/2602.21374v1
- Canonical: https://arxiv.org/abs/2602.21374v1
Trouble viewing inline? Open PDF directly →
Full Text
44,218 characters extracted from source content.
Expand or collapse full text
Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languages Mohammadreza Ghaffarzadeh-Esfahani 1,† , Nahid Yousefian 1 , Ebrahim Heidari-Farsani 1 , Ali Akbar Omidvarian 1 , Sepehr Ghahraei 1 , Atena Farangi 1 , AmirBahador Boroumand 2,† 1 Student Research Committee, Isfahan University of Medical Sciences, Isfahan, Iran 2 Department of Emergency Medicine, Isfahan University of Medical Sciences, Isfahan, Iran † Correspondence: Mohammadreza Ghaffarzadeh-Esfahani: mreghafarzadeh@gmail.com, Tel/Fax: +98-3136700479, ORCID: 0009-0009-9322-5471 AmirBahador Boroumand: ab.boroumand@med.mui.ac.ir, Tel/Fax: +98-36688597, ORCID: 0000-0001-5055-6881 Abstract Extracting clinical information from medical transcripts in low-resource languages re- mains a significant challenge in healthcare natural language processing (NLP). This study evaluates a two-step pipeline combining Aya-expanse-8B as a Persian-to-English transla- tion model with five open-source small language models (SLMs) — Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, and Gemma-3-1B-it — for binary extraction of 13 clinical features from 1,221 anonymized Persian transcripts collected at a cancer palliative care call center. Using a few-shot prompting strategy with- out fine-tuning, models were assessed on macro-averaged F1-score, Matthews Correlation Coefficient (MCC), sensitivity, and specificity to account for class imbalance. Qwen2.5-7B- Instruct achieved the highest overall performance (median macro-F1: 0.899; MCC: 0.797), while Gemma-3-1B-it showed the weakest results. Larger models (7B–8B parameters) con- sistently outperformed smaller counterparts in sensitivity and MCC. A bilingual analysis of Aya-expanse-8B revealed that translating Persian transcripts to English improved sensi- tivity, reduced missing outputs, and boosted metrics robust to class imbalance, though at the cost of slightly lower specificity and precision. Feature-level results showed reliable ex- traction of physiological symptoms across most models, whereas psychological complaints, administrative requests, and complex somatic features remained challenging. These find- ings establish a practical, privacy-preserving blueprint for deploying open-source SLMs in multilingual clinical NLP settings with limited infrastructure and annotation resources, and highlight the importance of jointly optimizing model scale and input language strategy for sensitive healthcare applications. Keywords: Natural language processing (NLP); Small language model (SLM); Few-shot prompting; Clinical information extraction; Low-resource languages; Machine translation 1 arXiv:2602.21374v1 [cs.CL] 24 Feb 2026 1. Introduction In the era of digital health, natural language processing (NLP) has emerged as a transformative tool for data extraction from unstructured clinical text [1]. This extraction is crucial in medicine, as it converts raw descriptions into structured, quantifiable information, facilitating advanced analytics like predictive modeling for disease progression [2] and evidence-based decision support systems that enhance patient outcomes and operational efficiency [3]. Particularly in palliative oncology, where patients often report multifaceted symptoms, automated information extraction from patient-provider interactions can alleviate clinician workload, reduce diagnostic delays, and optimize support, especially in resource-constrained settings [4]. However, the practical implementation of NLP solutions, especially across diverse languages and settings, faces technical and methodological hurdles [5]. Traditional NLP pipelines for medical entity recognition and data extraction have relied on rule-based systems [6] or supervised machine learning models trained on annotated corpora [7], but these approaches face challenges like training data scarcity and struggle to understand lin- guistic nuances [8]. Recently, the advent of large language models (LLMs) like GPT-5 [9] has promised capabilities for cross-lingual tasks [10], yet their proprietary nature, reliant on API access, raises critical data safety and security concerns, risking patient privacy [11]. Meanwhile, their large open-source counterparts demand substantial computational resources, posing bar- riers to deployment in low-infrastructure environments [12]. This led to a growing interest in small language models (SLMs) with compact architectures (1–10 billion parameters), fine-tuned for instruction-following, which provide a compelling balance of efficiency, accessibility, and per- formance [13, 14]. Recent benchmarks, such as those on the MedNLI and MIMIC-I datasets, demonstrate SLMs’ viability for English-centric clinical tasks, achieving high F1-scores for symp- tom detection [15]. Yet, empirical evidence on their efficacy for medical extraction in low-resource languages, especially Persian, remains understudied. Significant gaps persist in understanding how translation artifacts, class imbalance, and language-specific prompting affect performance in real-world clinical settings. In this study, we address these gaps by designing a two-step pipeline containing a transla- tor model, Aya-expanse-8B [16], plus five open-source SLMs, Qwen2.5-7B-Instruct [17], Llama- 3.1-8B-Instruct [18], Llama-3.2-3B-Instruct [19], Qwen2.5-1.5B-Instruct [20], and Gemma-3-1B- it [21], on a novel dataset of 1,221 anonymized Persian transcripts from a cancer palliative care call center. Leveraging a few-shot prompting strategy [22] for binary extraction of 13 clinical fea- tures, we assess model performance across metrics robust to imbalance, including macro-averaged F1-score, Matthews Correlation Coefficient (MCC), sensitivity, and specificity. We addition- ally conducted a bilingual assessment of Aya-expanse-8B, contrasting direct Persian processing against English-translated inputs to quantify translation’s trade-offs. Our findings reveal that while larger SLMs, such as Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, delivered superior performance for symptom detection, the use of original Persian inputs in the Aya-expanse-8B model notably enhanced the recognition of more subjective conditions, such as psychological complaints. These insights not only benchmark SLMs’ potential for equitable, low-resource healthcare NLP but also inform hybrid strategies that combine native-language inference with translation safeguards to enhance robustness in sensitive domains. 2. Results Our evaluation of the proposed two-step pipeline, which combines the Aya-expanse-8B transla- tor model with five open-source SLMs, demonstrated variability in the extraction of structured clinical information from Persian palliative oncology call transcripts. Benchmarking against a manually annotated dataset of 1,221 calls revealed that model capability was strongly influ- enced by parameter count and input language, with no single model dominating across all 13 clinical features. To provide a comprehensive overview of the models’ capabilities, the results 2 are presented across four key dimensions: the overall comparative performance of models, their feature-specific analysis, a focused analysis on the impact of language translation for the mul- tilingual model, and an evaluation of model robustness through imbalance-aware performance metrics and error analysis. This multifaceted presentation reveals critical insights into models’ behavior, the influence of parameter scale and language, and the practical implications of de- ploying such systems in sensitive domains such as palliative care. A visual summary illustrating the key aspects of the study is presented in Figure 1. Tabular Data Multi-facet Analysis Manual Extraction of 13 Features 1,221 phone calls In Palliative care center (in Persian) DatasetPre-processing System Prompt 3 Input and Output Examples Original Transcripts Translate to English Small Language Models English English & Persian Aya-expanse-8b Qwen2.5-7b-Instruct Llama-3.2-3B-Instruct Qwen2.5-1.5B-Instruct Gemma-3-1b-it Llama-3.1-8B-Instruct Accuracy, Sensitivity Specificity, Precision, F1 score Robustness: MCC and Missing Data Impact Sensitivity-Specificity Translation Effects Post-processing Models’ Structured Outputs Data extraction Figure 1: Schematic overview of the study. The upper panel shows the dataset prepro- cessing, inference generation, and postprocessing, starting from 1,221 Persian palliative care phone-call transcripts, followed by translation into English, prompt construction with input– output examples, and inference using multiple small language models (SLMs). The models’ structured outputs are then post-processed to extract tabular data. The lower panel illustrates the multi-facet analysis framework, comparing manual extraction of 13 reference features with model-derived features through performance metrics (accuracy, sensitivity, specificity, precision, F1-score), assessment of translation effects, crobustness analysis (Matthews correlation coeffi- cient (MCC), missing values), and sensitivity–specificity trade-offs. 2.1. Qwen2.5-7B-Instruct demonstrated the highest overall performance among the evaluated models Evaluation of the five SLMs demonstrated heterogeneous performance in extracting binary clini- cal features from the translated English transcripts. Overall, Qwen2.5-7B-Instruct exhibited the strongest balanced performance, achieving the highest median specificity (0.987 [0.975, 0.992]), macro-averaged F1-score (0.899 [0.832, 0.908]), precision (0.814 [0.759, 0.878]), and accuracy (0.96 [0.947, 0.984]) across all features. This model demonstrated robustness in handling class imbalance (Figure 2A and Supplementary File 1). In contrast, Gemma-3-1B-it showed the lowest median sensitivity (0.613 [0.294, 0.735]) and macro-averaged F1-score (0.74 [0.676, 0.79]), reflecting challenges in detecting positive instances. Additionally, the multilingual Aya-expanse-8B model performed comparably when using differ- ent prompt variants (macro-averaged F1-score: English 0.855 [0.804, 0.87] and Persian 0.842 [0.753, 0.868]). Finally, larger models like Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct gen- erally outperformed their smaller counterparts (e.g., Llama-3.2-3B-Instruct and Qwen2.5-1.5B- Instruct) in sensitivity and overall accuracy, with median accuracies exceeding 0.90 in most cases. 3 Accuracy Precision Sensitivity Specificity F1-score A Evaluation metrics B C MCC Missing counts Figure 2: Comparative performances of different models on validation metrics. (A) The median value for 5 metrics of accuracy, sensitivity, specificity, macro-averaged F1 score, and precision among 13 extracted features compared to the manually-extracted ground truth. (B) Matthews Correlation Coefficient (MCC) values for each evaluated model across the 13 extracted clinical features, comparing model-generated outputs with the manually extracted ground-truth annotations. (C) Total number of missing counts for each model among different extracted features. 2.2. Extraction performance was highly dependent on the type of clinical feature To gain deeper insight into model performance, we conducted a feature-wise analysis comparing model predictions with manually extracted data. The results revealed substantial variability in 4 how each model identified specific clinical features from patients’ phone calls. The following section presents a comparison of macro-averaged F1 scores for each feature across all evalu- ated models (Table 1). Other evaluation metrics, including accuracy, precision, sensitivity, and specificity, are provided in Supplementary File 1. Table 1: Macro-averaged F1 scores for the binary extraction of 13 clinical features from Persian palliative care transcripts across the evaluated small language mod- els. Values are reported for Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Llama-3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, Gemma-3-1B-it, and Aya-expanse-8B (English-translated and direct Per- sian variants). Feature Llama-3.1 Qwen2.5 Llama-3.2 Qwen2.5 Gemma-3Aya-expanse-8B -8B-Inst. -7B-Inst. -3B-Inst. -1.5B-Inst. -1B-itEnglish Persian Doctor’s visit request0.8320.6780.7230.7330.6330.4930.411 Psychological complaints0.8250.8320.8180.8000.7820.8350.859 Sleep disorders0.8040.8380.7660.7880.7400.7390.753 Loss of appetite0.9050.9150.8950.8740.7320.8650.874 Seizures0.8690.9080.8800.8190.7900.8690.798 Weakness and fatigue0.8920.9070.8560.9210.8650.7530.742 Decreased level of consciousness0.7920.9090.7690.8020.7480.8430.804 Fever0.9070.9240.9070.9210.8670.8910.918 Respiratory complaints0.9180.8910.9080.8830.655 0.9080.918 Insurance/treatment cost issues0.8160.8990.8660.7910.6890.8700.674 Urinary tract issues0.8700.7950.8660.7660.5980.8880.842 Pain0.9300.9030.8930.8730.8280.8550.868 Gastrointestinal issues0.8740.8180.8480.8280.676 0.8040.849 Median [IQR 1 , IQR 3 ] 0.870 [0.825, 0.905] 0.899 [0.832, 0.908] 0.866 [0.818, 0.893] 0.819 [0.791, 0.874] 0.74 [0.676, 0.79] 0.855 [0.804, 0.87] 0.842 [0.753, 0.868] 2.2.1. Physiological symptoms were reliably detected across most models Several features were consistently identified with a high macro-averaged F1 score (>0.85) by most models. Pain was the top-performing feature overall, with Llama-3.1-8B-Instruct achiev- ing the highest score (0.93). Fever and respiratory complaints also showed robust performance, with multiple models (Qwen2.5-7B-Instruct, Qwen2.5-1.5B-Instruct, Aya-expanse-8B (Persian) for Fever; Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Aya-expanse-8B (Persian) for respira- tory complaints) scoring above 0.91. Similarly, loss of appetite and seizures were well-detected, particularly by the Qwen2.5-7B-Instruct model (0.915 and 0.908, respectively). 2.2.2. Complex somatic and psychological features showed inconsistent extraction Performance was more varied for features like weakness and fatigue, decreased level of conscious- ness, and psychological complaints. While some models excelled (for example, Qwen2.5-7B- Instruct on decreased level of consciousness (0.909) and Aya-expanse-8B (Persian) on psycholog- ical complaints (0.859)), others, particularly smaller models like Gemma-3-1B-it, struggled more on these conceptually related symptoms. 2.2.3. Administrative requests and multifaceted complaints remained difficult for all models Certain features proved challenging across the board. The doctor’s visit request was notably diffi- cult, with the highest score (0.832 from Llama-3.1-8B-Instruct) and the lowest score (0.410 from Aya-expanse-8B (Persian)). Sleep disorders and insurance/treatment cost issues also showed lower and more inconsistent scores. Features related to urinary tract and gastrointestinal issues revealed a performance gap with larger models like Llama-3.1-8B-Instruct handled them reason- 5 ably well (0.870 and 0.874, respectively), but the Gemma-3-1B-it model performed very poorly (0.598 and 0.676, respectively). Ultimately, while all models reliably identified overt physiological symptoms, their perfor- mance diverged significantly for administrative requests, complex somatic complaints, and psy- chological expressions. Notably, Aya-expanse-8B demonstrated language-dependent variability, performing differently across English and Persian transcripts. This suggests that the choice of input language may influence extraction accuracy and completeness, highlighting the importance of evaluating multilingual models in both original and translated contexts. The following sec- tion examines how language translation affects Aya-expanse-8B’s performance across the same clinical features. 2.3. Translating Persian transcripts to English enhanced sensitivity and reduced missing outputs To assess the influence of translating Persian transcripts to English on the performance of the multilingual Aya-expanse-8B model, we compared its outputs when processing original Persian text directly (Persian version) versus English-translated inputs (English version). This analysis focused on the 13 binary features extracted from the 1,221 transcripts, using the evaluation metrics described in the methods section. Overall, translation to English resulted in a modest improvement in performance metrics, such as macro-averaged F1 score and MCC (Figure 2B and Supplementary file 1). Notably, the English version produced fewer missing predictions, suggesting enhanced output completeness (Figure 2C and Supplementary File 1). The English version achieved a macro-averaged F1 score of 0.855 [0.804–0.870] compared to 0.842 [0.753–0.868] for the Persian version. Similarly, the MCC was higher in English (0.724 [0.634–0.757]) than in Persian (0.686 [0.584–0.757]), reflecting a stronger overall correla- tion between predictions and ground truth labels. These gains were driven by higher sensitivity in English (0.901 [0.842–0.911]) versus Persian (0.893 [0.818–0.952]), which is particularly valuable for detecting rare positive cases in this imbalanced dataset. However, precision was lower in En- glish (0.625 [0.597–0.671]) than in Persian (0.672 [0.413–0.745]), leading to a higher rate of false positives and a corresponding slight drop in accuracy (0.951 [0.905–0.973] vs. 0.955 [0.912–0.963]) and specificity (0.955 [0.906–0.976] vs. 0.960 [0.919–0.975]). Feature-level analysis revealed that the English version outperformed the Persian version in macro-averaged F1 for 6 features: doctor’s visit request (0.493 vs. 0.411), seizures (0.869 vs. 0.798), weakness and fatigue (0.753 vs. 0.742), decreased level of consciousness (0.843 vs. 0.804), insurance/treatment cost issues (0.870 vs. 0.674), and urinary tract issues (0.888 vs. 0.842). Conversely, the Persian version excelled in 7 features, including psychological complaints (0.859 vs. 0.835), sleep disorders (0.753 vs. 0.739), loss of appetite (0.874 vs. 0.865), fever (0.918 vs. 0.891), respiratory complaints (0.918 vs. 0.908), pain (0.868 vs. 0.855), and gastrointestinal issues (0.849 vs. 0.804). A key drawback of using Persian transcripts was the higher incidence of missing values (38 [38, 39]) compared to 28 [27, 30] in English, potentially arising from the model’s challenges in adhering to the structured output template in the native language. This incompleteness disproportionately affected features such as fever and insurance/treatment cost issues, where Persian missing outputs reached 40. In summary, while direct processing of Persian transcripts yielded marginally superior accu- racy and specificity, translating to English enhanced the model’s robustness for imbalanced med- ical extraction tasks by improving sensitivity, reducing missing outputs, and boosting balanced metrics such as F1 and MCC. These findings underscore the potential benefits of English-centric prompting for multilingual SLMs in low-resource clinical settings. 6 2.4. Larger models demonstrated superior performance under class imbalance To assess the robustness of the evaluated SLMs in handling class-imbalanced medical extraction tasks, we analyzed the MCC across all seven configurations, emphasizing model-level stabil- ity rather than isolated feature performance. Qwen2.5-7B-Instruct demonstrated the highest median MCC of 0.797 [0.67, 0.819], indicating superior overall correlation between predictions and true labels, even under severe imbalance, with consistent high scores for physiological fea- tures like fever (0.8502) and seizures (0.8165). Llama-3.1-8B-Instruct demonstrated a median MCC of 0.749 [0.67, 0.818], indicating strong and balanced predictive performance. It achieved particularly high MCCs for symptoms such as pain (0.864) and respiratory complaints (0.842). However, slightly broader interquartile ranges were observed, reflecting moderate variability in features like doctor’s visit requests (0.6635). In contrast, smaller models such as Gemma-3- 1B-it exhibited a lower median MCC of 0.502 [0.398, 0.581], indicating reduced overall relia- bility. Intermediate-sized models, including Llama-3.2-3B-Instruct and Qwen2.5-1.5B-Instruct, achieved median MCCs of 0.734 [0.656, 0.787] and 0.654 [0.589, 0.753], respectively, indicating a scale-dependent trend in which larger parameter counts are associated with improved overall concordance with ground truth labels. The multilingual Aya-expanse-8B variants demonstrated nuanced performance, with the English-translated input achieving a median MCC of 0.724 [0.634, 0.757], surpassing the Persian direct input at 0.686 [0.584, 0.757]. These results suggest that translation can enhance predictive correlation for specific classes without compromising over- all model stability. Collectively, this metric affirms that mid-to-large SLMs maintain superior performance in resource-constrained and non-English environments. 2.5. Larger models favored sensitivity while smaller models achieved comparable or superior specificity To further evaluate model performance, sensitivity and specificity were computed for each of the 13 binary features, with corresponding correlations presented for each model in Figure 3A–G. In terms of sensitivity, Llama-3.1-8B-Instruct emerged as the top performer with a me- dian sensitivity of 0.909 [0.842–0.941], closely followed by Aya-expanse-8B (English) at 0.901 [0.842–0.911] and Aya-expanse-8B (Persian) at 0.893 [0.818–0.952]. Qwen2.5-7B-Instruct and Llama-3.2-3B-Instruct exhibited solid but more variable sensitivities of 0.818 [0.784–0.848] and 0.842 [0.762–0.893], respectively, with the former excelling in features such as insurance/treatment cost issues (0.824) but lagging on respiratory complaints (0.728). In contrast, smaller models ex- hibited markedly lower sensitivity. Qwen2.5-1.5B-Instruct achieved a median sensitivity of 0.737 [0.700–0.859], whereas Gemma-3-1B-it demonstrated the lowest sensitivity at 0.613 [0.294–0.735], with pronounced deficiencies in detecting features such as urinary tract issues (0.140) and doc- tor’s visit requests (0.215). Regarding specificity, Qwen2.5-7B-Instruct achieved the highest median specificity of 0.987 [0.975–0.992], demonstrating exceptional precision in features such as seizures (0.998) and in- surance issues (0.997), which is crucial for avoiding unnecessary escalations in resource-limited environments. Gemma-3-1B-it and Qwen2.5-1.5B-Instruct achieved median specificities of 0.986 [0.969–0.995] and 0.982 [0.951–0.988], respectively. Gemma-3-1B-it exhibited high specificity for psychological complaints (0.968), whereas Qwen2.5-1.5B-Instruct showed similarly high speci- ficity for loss of appetite (0.987). Gemma-3-1B-it demonstrated particularly high specificity for psychological complaints (0.968), while Qwen2.5-1.5B-Instruct showed comparable specificity for loss of appetite (0.987). The larger Llama models also maintained high specificity, with Llama-3.1-8B-Instruct at 0.958 [0.951–0.982] and Llama-3.2-3B-Instruct at 0.951 [0.937–0.985], though slightly lower than the top-performing models and reduced for features such as gastrointestinal issues (0.938 and 0.937, respectively). The Aya-expanse-8B variants demonstrated balanced specificity, with the English version at 0.955 [0.906–0.976] and the Persian version at 0.960 [0.919–0.975]. Notably, the Persian 7 1 : Doctor’s Visit Request 2 : Sleep Disorders 3 : Psychological Complaints 4 : Loss Of Appetite 5 : Seizures 6 : Weakness And Fatigue 7 : Decreased Level Of Consciousness 8 : Fever 9 : Respiratory Complaints 10 : Insurance/Treatment Cost Issues 11 : Urinary Tract Issues 12 : Pain 13 : Gastrointestinal Issues A G EF BC Specificity vs Sensitivity by Model D Figure 3: Sensitivity–specificity trade-offs across evaluated small language models. Each panel depicts sensitivity (y-axis) versus specificity (x-axis) for 13 binary clinical features (points labeled 1–13; see legend) for (A) Llama-3.1-8B-Instruct, (B) Qwen2.5-7B-Instruct, (C) Llama-3.2-3B-Instruct, (D) Qwen2.5-1.5B-Instruct, (E) Gemma-3-1B-it, (F) Aya-expanse-8B (English), and (G) Aya-expanse-8B (Persian). The dashed diagonal line denotes the locus where sensitivity equals specificity. input achieved higher specificity for pain (0.902) but showed reduced detection performance for features like doctor’s visit requests (0.342). Overall, these model-wise differences indicate that increasing model size enhances sensitivity and supports more balanced coverage across features, while smaller models maintain comparable or even higher specificity. 3. Discussion This study demonstrates that open-source SLMs, when applied within a translation-augmented two-step pipeline, can effectively extract structured clinical information from Persian palliative care transcripts without the need for fine-tuning. Model performance varied considerably, with larger instruction-tuned models demonstrating superior MCC and sensitivity, particularly un- der class imbalance conditions. Feature-wise, common physiological symptoms were extracted reliably, whereas complex somatic, psychological, and administrative features posed challenges across all models. Translating Persian transcripts into English enhanced sensitivity and reduced missing outputs, albeit with a modest decrease in specificity, highlighting a trade-off with prac- tical implications for pipeline design in low-resource language contexts. 8 Building upon these findings, this study aligns with and extends several concepts identified in previous works. Similar to the study by Akcali et al. [23] on Turkish mammography reports, we demonstrate the efficacy of prompt-engineered language models for clinical information extraction in a low-resource language context. However, while their approach utilized a massive, proprietary model (Gemini 1.5 Pro) in a many-shot setting, our research pivots to evaluating smaller, open- source models in a few-shot paradigm. This shift addresses the critical need for patient privacy and accessible tools highlighted in the Navarro et al. study [24], which noted a scarcity of freely available models and their rare translation to clinical practice. Our work directly responds to this call by benchmarking models that can run on desktop-grade hardware, similar to the Strata framework described by Liu et al. [25], which achieved human-level performance with a fine-tuned Llama-3.1-8B model on a curated dataset of pathology reports. Additionally, our analysis of translation effects supports the findings of Balk et al. [26], confirming that machine translation can serve as an effective preprocessing strategy to reduce language bias. Specifically, we observed enhanced sensitivity and data completeness when employing English prompts with the multilingual Aya-expanse-8B model. In comparison with recent methodological advances, our findings highlight the capabilities of general-purpose SLMs. Unlike the hybrid, multi-stage frameworks combining fine-tuned BERT with retrieval-augmented generation [27] or the sophisticated multi-granularity embedding archi- tectures proposed for psychomedical named entity recognition (NER) [28], our approach evalu- ates the out-of-the-box few-shot capability of general-purpose SLMs. This simplicity is a strate- gic advantage for rapid deployment. Collectively, while specialized, finely-tuned architectures may achieve better accuracy on benchmark datasets, moderately sized, open-source SLMs with careful prompting present a compelling and immediately practical pathway for building privacy- preserving automated extraction systems in real-world, multilingual clinical settings with limited annotation resources and infrastructure. While this study demonstrates the potential of SLMs for clinical information extraction in a low-resource language, several limitations exist. These constraints include the relatively small scale of the dataset and the inherent noise introduced by machine translation, which may alter subtle semantic meanings. Also, our evaluation remains within a single, specific domain of palliative care oncology, limiting generalizability. Furthermore, the persistent underperformance on administrative and psychological features suggests that current SLMs lack the deep contextual and sociolinguistic understanding required for fully reliable autonomous extraction. Building on these limitations, future studies should prioritize several key directions, including using large- scale, multi-center validation across diverse medical specialties and languages to establish true generalizability and foster clinical adoption. In conclusion, this study establishes a meaningful proof of concept for deploying open-source SLMs in privacy-preserving clinical NLP pipelines beyond high-resource linguistic settings. By demonstrating that a translation-augmented, few-shot prompting strategy enables structured information extraction from Persian palliative oncology transcripts without model fine-tuning, we offer a practical blueprint for healthcare systems operating under infrastructure and data governance constraints. The consistent advantage of 7B–8B parameter models in sensitivity and imbalance-aware performance metrics, combined with the trade-offs observed between native- language and translated inference, underscores that a thoughtful combination of model scale and language strategy is essential for deployment success. Ultimately, integrating such systems into clinical workflows may reduce clinician documentation burden and expedite symptom-based triage for underserved populations. However, achieving this potential will require sustained interdisciplinary collaboration to develop linguistically diverse corpora, establish domain-specific evaluation frameworks, and implement robust human oversight mechanisms that satisfy the safety and ethical standards required in sensitive clinical settings. 9 4. Methods This study aimed to evaluate and compare the performance of various SLMs against a manually annotated dataset for the task of medical information extraction. Utilizing a dataset of 1,221 transcribed phone calls from a palliative care center, we benchmarked multiple open-source SLMs. The process involved translating the original Persian transcripts, employing a structured few- shot prompting strategy for feature extraction, and conducting a comprehensive quantitative assessment using metrics tailored for imbalanced binary classification. The following sections detail the ethical approval, dataset construction, preprocessing pipeline, model inference, and evaluation framework. 4.1. Ethical consideration The study was approved by the Institutional Review Board (IRB) of Isfahan University of Medical Sciences (IR.ARI.MUI.REC.1404.321). The IRB granted this study a consent waiver. To protect patient confidentiality, all data were pseudonymized before analysis. The study adhered to the guidelines of the Helsinki Declaration (2013) and all applicable regulations of the Iranian Ministry of Health. 4.2. Data collection and dataset To establish a robust baseline for assessing SLM performance, we compiled a dataset comprising 1,221 transcribed phone calls to a cancer-focused palliative care call center. The transcripts, recorded in Persian by operators, were anonymized by a single author (A.O.). They were then divided into two groups, with two authors (S.G. and A.F.) independently extracting medical in- formation from each patient’s reported complaints. To ensure consistency, a blinded cross-review was conducted between S.G. and A.F. All discrepancies were resolved by the corresponding au- thor (M.G.), ensuring a consistently annotated dataset for comparative analysis. The extracted features are: doctor’s visit request, psychological complaints, sleep disorders, loss of appetite, seizures, weakness and fatigue, decreased level of consciousness, fever, respiratory complaints, in- surance/treatment cost issues, urinary tract issues, pain, and gastrointestinal issues. Collectively, they are essential for assessing patient needs and guiding resource allocation. 4.3. Data preprocessing To preprocess our dataset of 1,221 phone calls in Persian, we first used a translation model to convert each sample into English. Then, we applied a few-shot setting to each sample to generate the final input prompts for five different models with varying numbers of parameters. We used the Aya-expanse-8B model, a multilingual SLM supporting 23 languages, including Persian, as our translator. The generation configuration was set with a temperature of 0.3 and a maximum of 2048 new tokens. For each sample in our dataset, we prompted the model with the instruction “Translate the following text from Persian to English:” to obtain its English translation. To prepare the textual data for structured information extraction, we employed a few-shot prompting strategy. This approach involved providing a clear system prompt alongside three examples of input and output data from the dataset to guide the model in generating outputs in a predefined format. The full input prompt, including few-shot examples, is presented in Supplementary File 2. The system prompt defined the model’s role and constraints as follows: You are an expert in data extraction specializing in medical information. You are provided with clinical data about patients with cancer who require palliative care. Your 10 task is to read the patient’s condition and extract ONLY complications strictly into the predefined format below. Specific instructions were included to ensure consistency: 1. Output exactly the same structure and order of fields. 2. Fill each field with True or False only. 3. Do not infer or assume any information that is not explicitly stated. 4. Do not output any text outside the provided template. The output template comprised 13 binary fields covering a range of potential complications and concerns, including: patient-requested visits, psychiatric complaints, sleep disorders, ap- petite loss, seizures, weakness/fatigue, decreased consciousness, fever, respiratory complaints, insurance/cost issues, urinary tract issues, pain, and gastrointestinal issues. In addition, for extra evaluation of the translator model, we used an identical system prompt in Persian for processing the original transcripts. 4.4. Generation and post-processing We evaluated five SLMs with varying parameter counts for text generation. The models in- cluded: Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Gemma-3-1B-it, and Qwen2.5-1.5B-Instruct, alongside Aya-expanse-8B as the translator model. All experiments were conducted using HuggingFace’s Transformers library (version 4.57.3) [29] with sampling disabled and a limit of 512 new tokens per generation. Following the generation phase, a cus- tom function was employed to detect and extract each precise feature from the model outputs. The resulting feature sets were subsequently structured into tables to enable comparison with the manually extracted ground truth data. Additionally, we quantified missing predictions— cases where models failed to output a valid binary value—and treated these as negative (False) instances in final analyses, reflecting the model’s inability to detect the feature. 4.5. Assessment To evaluate each model’s performance, we employed a suite of complementary metrics capturing global accuracy, class-wise behavior, and robustness to class imbalance. Specifically, we computed accuracy, specificity, sensitivity, precision, the macro-averaged F1 score, and MCC. Accuracy was used to summarize the overall proportion of correct predictions. Sensitivity and precision were analyzed to assess the models’ ability to detect positive cases. Sensitivity measured the proportion of true positives correctly identified, reflecting the model’s capacity to avoid false negatives, while precision quantified the proportion of predicted positives that were correct, indicating how well the model minimizes false positive predictions. Specificity quantified the correct rejection of negative cases, which is essential in imbalanced settings. To provide a class-balanced measure of predictive quality, we computed the macro-averaged F1 score, preventing inflation from majority classes. To quantify global agreement between predicted and true labels, we used the MCC, defined as: MCC = T P× T N− F P× F N p (T P + F P)(T P + F N)(T N + F P)(T N + F N) (1) which provides a stable summary even under severe class imbalance. All statistical analyses and metric computations were performed in Python (version 3.13.5) [30] using pandas (version 2.3.3) [31], NumPy (version 2.3.5) [32], SciPy (version 1.16.3) [33], and scikit-learn (version 1.7.2) [34]. 11 4.6. Computational environment All inferences for the five SLMs and the translator model were performed locally using a single L4 GPU (24 GB VRAM) and a two-core CPU with 8 GB of RAM, with no external API calls, which preserves privacy. 12 Data availability The data generated and/or utilized in this study can be obtained from the corresponding author upon reasonable request (mreghafarzadeh@gmail.com). Code availability The code for all experiments and analyses is publicly available at https://github.com/mohammad-gh009/ Small-language-models-on-clinical-data-extraction.git. Acknowledgments The authors would like to express their sincere gratitude to Ali Motahharynia for providing insightful comments that greatly improved the quality of this paper. Funding No funding was received for this study or its publication. Competing interests The authors declare no competing interests. Author contributions Conceptualization: M.G, A.B. Supervision: M.G. Dataset preparation: M.G, A.O, S.G, A.F. Model development: M.G. Model assessment: M.G, N.Y, E.H. Data interpretation: All authors. Drafting original manuscript: M.G, N.Y, E.H. Revising the manuscript: M.G, A.B. All the authors have read and approved the final version for publication and agreed to be responsible for the integrity of the study. 13 References [1] Maguluri K. Natural language processing in healthcare: Unlocking insights from clinical data. Deep Science Publishing. Available from: https://deepscienceresearch.com/dsr/ catalog/book/43/chapter/137 [2] Starling MS, Kehoe L, Burnett BK, Green P, Venkatakrishnan K, Madabushi R. The Poten- tial of Disease Progression Modeling to Advance Clinical Development and Decision Making. Clin Pharmacol Ther. 2024;117(2):343–352. [3] Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. npj Digit Med. 2020;3(1):17. [4] Stapleton SJ, Holden J, Epstein J, Wilkie DJ. Symptom Clusters in Patients with Cancer in the Hospice/Palliative Care Setting. Support Care Cancer. 2016;24(9):3863–3871. [5] Baclic O, Tunis M, Young K, Doan C, Swerdfeger H, Schonfeld J. Challenges and oppor- tunities for public health made possible by advances in natural language processing. Can Commun Dis Rep. 2020;46(6):161–168. [6] Bartoli R, Dreyer AG, Schmidt D. A rule-based approach for automatically extracting data from systematic reviews and their updates to model the risk of conclusion change. PubMed. 2020. Available from: https://pubmed.ncbi.nlm.nih.gov/33350584/ [7] Adamson B, Waskom M, Blarre A, Kelly J, Krismer K, Nemeth S, et al. Approach to machine learning for extraction of real-world data variables from electronic health records. Front Pharmacol. 2023;14:1180962. [8] Durango MC, Torres-Silva EA, Orozco-Duque A. Named Entity Recognition in Electronic Health Records: A Methodological Review. Healthc Inform Res. 2023;29(4):286–300. [9] Singh A, Fry A, Perelman A, et al. OpenAI GPT-5 System Card. arXiv; 2025. Available from: http://arxiv.org/abs/2601.03267 [10] Rohanian M, Mehra T, Miglino N, Nooralahzadeh F, Krauthammer M, Wicki A. Towards scalable and cross-lingual specialist language models for oncology. Sci Rep. 2025;15(1):35480. [11] Kim M, Kim Y, Kang HJ, Seo H, Choi H, Han J, et al. Fine-Tuning LLMs with Medical Data: Can Safety Be Ensured? NEJM AI. 2025;2(1):AIcs2400390. [12] DeepSeek-AI, Liu A, Feng B, et al. DeepSeek-V3 Technical Report. arXiv; 2025. Available from: http://arxiv.org/abs/2412.19437 [13] Wang F, Zhang Z, Zhang X, et al. A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness. ACM Trans Intell Syst Technol. 2025;16(6):145:1–145:87. [14] Belcak P, Heinrich G, Diao S, et al. Small Language Models are the Future of Agentic AI. arXiv; 2025. Available from: http://arxiv.org/abs/2506.02153 [15] Garg M, Raza S, Rayana S, Liu X, Sohn S. The Rise of Small Language Models in Health- care: A Comprehensive Survey. arXiv; 2025. Available from: https://arxiv.org/abs/ 2504.17119v1 [16] CohereLabs. aya-expanse-8b [model card]. Hugging Face; 2025. Available from: https: //huggingface.co/CohereLabs/aya-expanse-8b 14 [17] Qwen, Yang A, Yang B, et al. Qwen2.5 Technical Report. arXiv; 2025. Available from: http://arxiv.org/abs/2412.15115 [18] Grattafiori A, Dubey A, Jauhri A, et al. The Llama 3 Herd of Models. arXiv; 2024. Available from: http://arxiv.org/abs/2407.21783 [19] Meta. Llama-3.2-3B-Instruct [model card]. Hugging Face; 2024. Available from: https: //huggingface.co/meta-llama/Llama-3.2-3B-Instruct [20] Qwen. Qwen2.5-1.5B-Instruct [model card]. Hugging Face; 2025. Available from: https: //huggingface.co/Qwen/Qwen2.5-1.5B-Instruct [21] Google. gemma-3-1b-it [model card]. Hugging Face; 2025. Available from: https:// huggingface.co/google/gemma-3-1b-it [22] Brown TB, Mann B, Ryder N, et al. Language Models are Few-Shot Learners. arXiv; 2020. Available from: https://arxiv.org/abs/2005.14165v4 [23] Akcali Z, Cubuk HS, Oguz A, Kocak M, Farzaliyeva A, Guven F, et al. Automated Ex- traction of Key Entities from Non-English Mammography Reports Using Named Entity Recognition with Prompt Engineering. Bioengineering. 2025;12(2):168. [24] Fraile Navarro D, Ijaz K, Rezazadegan D, Rahimi-Ardabili H, Dras M, Coiera E, et al. Clinical named entity recognition and relation extraction using natural language processing of medical free text: A systematic review. Int J Med Inf. 2023;177:105122. [25] Liu L, Lian L, Hao Y, Pace A, Kim E, Homsi N, et al. Human level information extraction from clinical reports with finetuned language models. Sci Rep. 2025. Available from: https: //w.nature.com/articles/s41598-025-28767-z [26] Balk EM, Chung M, Chen ML, Chang LKW, Trikalinos TA. Data extraction from machine- translated versus original language randomized trial reports: a comparative study. Syst Rev. 2013;2(1):97. [27] Sreenivas SC, Chowdhury S, Masum M. Enhancing Clinical Named Entity Recognition via Fine-Tuned BERT and Dictionary-Infused Retrieval-Augmented Generation. Electronics. 2025;14(18):3676. [28] Liu Z, Zhang G, Shen Y. Psychomedical named entity recognition method based on multi- level feature extraction and multi-granularity embedding fusion. Sci Rep. 2025;15(1):16927. [29] Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, et al. HuggingFace’s Transform- ers: State-of-the-art Natural Language Processing [Internet]. arXiv; 2020 [cited 2026 Feb 24]. Available from: http://arxiv.org/abs/1910.03771doi:10.48550/arXiv.1910.03771 [30] Python.org [Internet]. [cited 2026 Feb 24]. Python Release Python 3.13.5. Available from: https://w.python.org/downloads/release/python-3135/ [31] team T pandas development. pandas-dev/pandas: Pandas [Internet]. Zenodo; 2020. Avail- able from: https://doi.org/10.5281/zenodo.3509134 doi:10.5281/zenodo.3509134 [32] Harris CR, Millman KJ, Walt SJ van der, Gommers R, Virtanen P, Cournapeau D, et al. Array programming with NumPy. Nature. 2020 Sep;585(7825):357–62. doi:10.1038/s41586- 020-2649-2 [33] Virtanen P, Gommers R, Oliphant TE, Haberland M, Reddy T, Cournapeau D, et al. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nat Methods. 2020;17:261–72. doi:10.1038/s41592-019-0686-2 15 [34] Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: Machine Learning in Python. CoRR [Internet]. 2012;abs/1201.0490. Available from: http: //arxiv.org/abs/1201.0490 16