Paper deep dive
TharuChat: Bootstrapping Large Language Models for a Low-Resource Language via Synthetic Data and Human Validation
Prajwal Panth, Agniva Maiti
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/22/2026, 5:50:19 AM
Summary
The paper introduces Tharu-LLaMA (3B), an instruction-following model for the low-resource Tharu language, developed using a novel 'LLM-to-Human' bootstrapping pipeline. By creating the TharuChat dataset (3,955 instruction-response pairs) and employing LoRA fine-tuning on consumer-grade hardware (NVIDIA T4), the authors demonstrate that small-scale, high-quality synthetic data can effectively adapt large models to indigenous languages, reducing perplexity from 88 to 2.88.
Entities (5)
Relation Signals (3)
Tharu-LLaMA â trainedon â TharuChat
confidence 100% ¡ We release a parameter-efficient fine-tuned version of Metaâs LLaMA-3.2-3B-Instruct, trained on the TharuChat dataset.
Tharu-LLaMA â usestechnique â LoRA
confidence 100% ¡ We employed Low-Rank Adaptation (LoRA) to fine-tune the model.
TharuChat â generatedby â Gemini 2.5 Pro
confidence 95% ¡ We utilized prompt-engineered Gemini models... to synthesize training data.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The rapid proliferation of Large Language Models (LLMs) has created a profound digital divide, effectively excluding indigenous languages of the Global South from the AI revolution. The Tharu language, an Indo-Aryan vernacular spoken by approximately 1.7 million people across the Terai belt of Nepal and India, exemplifies this crisis. Despite a rich oral tradition, Tharu suffers from severe data scarcity and linguistic fragmentation, causing state-of-the-art multilingual models to routinely "hallucinate" or default to dominant high-resource neighbors like Hindi and Nepali due to contamination in pre-training corpora. This paper presents Tharu-LLaMA (3B), a specialized instruction-following model designed to address this exclusion. We introduce TharuChat, a novel dataset constructed via a LLM-to-Human bootstrapping pipeline. We utilized prompt-engineered Gemini models, fed with Rana Tharu grammar and folklore, to synthesize training data. Unlike curated gold-standard corpora, TharuChat reflects the noisy, heterogeneous linguistic reality of the region: it is predominantly anchored in Rana Tharu (~70%) while integrating elements of Dangaura and Kochila dialects. We provide a transparent analysis of the dataset's limitations, including dialectal code-mixing and residual Awadhi/Hindi influence. Through a rigorous empirical ablation study, we demonstrate that despite these imperfections, small-scale synthetic data is highly effective, increasing the dataset volume from 25% to 100% results in a linear reduction in perplexity from 6.42 to 2.88. The resulting model serves as a proof-of-concept for the preservation of under-resourced Himalayan languages via generative AI, achievable on consumer-grade hardware.
Tags
Links
- Source: https://arxiv.org/abs/2603.17220v1
- Canonical: https://arxiv.org/abs/2603.17220v1
Trouble viewing inline? Open PDF directly â
Full Text
26,467 characters extracted from source content.
Expand or collapse full text
TharuChat: Bootstrapping Large Language Models for a Low-Resource Language via Synthetic Data and Human Validation Prajwal Panth School of Computer Engineering KIIT Deemed to be University Bhubaneswar, India prajwal.panth21@gmail.com Agniva Maiti School of Computer Engineering KIIT Deemed to be University Bhubaneswar, India maitiagniva@gmail.com AbstractâThe rapid proliferation of Large Language Models (LLMs) has created a profound "digital divide," effectively excluding indigenous languages of the Global South from the AI revolution. The Tharu language, an Indo-Aryan vernacular spoken by approximately 1.7 million people across the Terai belt of Nepal and India, exemplifies this crisis. Despite a rich oral tradition, Tharu suffers from severe data scarcity and linguistic fragmentation, causing state-of-the-art multilingual models to routinely "hallucinate" or default to dominant high-resource neighbors like Hindi and Nepali due to contamination in pre- training corpora. This paper presents Tharu-LLaMA (3B), a specialized instruction-following model designed to address this exclusion. We introduce TharuChat, a novel dataset constructed via a "LLM-to-Human" bootstrapping pipeline. We utilized prompt- engineered Gemini models, fed with Rana Tharu grammar and folklore, to synthesize training data. Unlike curated gold- standard corpora, TharuChat reflects the noisy, heterogeneous linguistic reality of the region: it is predominantly anchored in Rana Tharu (approx. 70%) while integrating elements of Dangaura and Kochila dialects. We provide a transparent analysis of the datasetâs limitations, including dialectal code-mixing and residual Awadhi/Hindi influence. Through a rigorous empirical ablation study, we demonstrate that despite these imperfections, small-scale synthetic data is highly effective; increasing the dataset volume from 25% to 100% results in a linear reduction in perplexity from 6.42 to 2.88. The resulting model serves as a proof-of-concept for the preservation of under-resourced Himalayan languages via generative AI, achievable on consumer- grade hardware. IndexTermsâLow-ResourceNLP,TharuLanguage, Instruction-Tuned LLMs, Synthetic Data Generation, LoRA Fine-Tuning, Indigenous Language Preservation I. INTRODUCTION The democratization of Natural Language Processing (NLP) remains a largely unfulfilled promise for the linguistic com- munities of the Himalayas. While foundational models such as LLaMA-3, GPT-4, and Claude achieve near-human proficiency in English and major European languages, their performance degrades precipitously when applied to "low-resource" lan- guagesâthose lacking the massive, digitized web corpora required for standard self-supervised learning. This disparity creates a "digital cliff," where indigenous communities are excluded from the benefits of the AI revolution, ranging from automated translation to accessible governance. This exclusion is particularly acute for Tharu, an indige- nous language native to the Terai lowlands of Nepal and India. Although spoken by a significant population, Tharu is linguistically complex, consisting of a dialect continuum (Rana, Dangaura, Kochila) that shares the Devanagari script with Hindi and Nepali but retains distinct verbal morphology, honorific systems, and vocabulary. Current commercial LLMs, having seen negligible Tharu text during pre-training, treat the language as a noisy variation of Hindi. Consequently, they exhibit "catastrophic code-switching," often beginning a sentence in Tharu only to revert to Hindi grammar mid-way, essentially erasing the linguistic identity of the user. To address this "cold start" problemâwhere no data exists to train a model, and no model exists to generate dataâwe adopt and refine the bootstrapping methodology. Rather than relying on noisy web scraping or expensive manual transcrip- tion, we employ a "Human-in-the-Loop" synthetic generation pipeline. By utilizing a high-capacity teacher model (Gem- ini 2.5) to elicit grammatical rules and generate seed data, followed by verification by native speakers, we cultivate a training corpus from scratch. Our contributions are threefold: 1) The TharuChat Dataset: We release a curated corpusof3,955instructionâresponsepairs (prajwal-panth/tharu-chat)[1],ofwhich approximately 3,100 were employed in our research. We frankly acknowledge the "silver" quality of this dataset; it does not force a single standard dialect but embraces the linguistic mix of the Terai, featuring a majority of Rana Tharu content supplemented by Dangaura and Kochila samples, alongside corrected synthetic noise. 2) Tharu-LLaMA (3B): We release a parameter-efficient arXiv:2603.17220v1 [cs.CL] 17 Mar 2026 fine-tuned version of Metaâs LLaMA-3.2-3B-Instruct, trained on the TharuChat dataset. The model implemen- tation is publicly available [2]. To enhance accessibility, we selected the 3B parameter class, ensuring deploya- bility on consumer-grade hardware (e.g., NVIDIA T4 GPUs) and thereby lowering the barrier to entry for researchers and developers in Nepal and India. 3) Empirical Validation of Data Scaling: We conduct a detailed ablation study to quantify the relationship between synthetic data volume and model fluency. Our results show a strong linear scaling law: small quantities of verified data can reduce perplexity by an order of magnitude (from >88 zero-shot to 2.88), suggesting that the "data wall" for low-resource languages is surmount- able without requiring millions of examples. I. BACKGROUND AND RELATED WORK A. NLP for Low-Resource Languages The field of low-resource NLP has historically been dom- inated by transfer learning approaches utilizing large-scale multilingual encoders such as mBERT [3] and XLM-R [4]. While effective for discriminative tasks like Named Entity Recognition (NER) or Part-of-Speech (POS) tagging, these encoder-only architectures lack the generative capabilities re- quired for conversational agents or question-answering sys- tems. Furthermore, these massive multilingual models suffer from the "curse of multilinguality," where the representation of low-resource languages is diluted by the dominance of high- resource languages (e.g., English, Hindi) in the pre-training corpus. For languages like Tharu, which exhibit distinct morpholog- ical agglutination and honorific systems not present in Hindi, zero-shot transfer from Hindi-centric models often results in poor syntactic coherence. Recent advancements have shifted the paradigm towards Parameter-Efficient Fine-Tuning (PEFT) of decoder-only Large Language Models (LLMs). Techniques such as Low-Rank Adaptation (LoRA) [5] allow for the adaptation of massive models to new linguistic domains by updating less than 1% of the parameters, making it compu- tationally feasible to train dedicated models for indigenous languages without industrial-scale infrastructure. B. The NagaNLP Methodology This work builds directly upon the NagaNLP Frame- work (2025), which introduced the concept of the "LLM- as-Elicitor." Instead of asking the model to simply "translate to target language X," which usually results in poor qual- ity output, the framework employs a multi-turn interactive pipeline. The model first "learns" the grammar through few- shot prompting with authentic texts, consolidates this knowl- edge into a style guide, and only then generates synthetic samples. We extend this methodology to address the specific chal- lenge of the Tharu dialect continuum. Unlike the NagaNLP work which focused on a Creolized language, our work must navigate distinct dialects. We intentionally move beyond single-dialect generation to a mixed-dialect corpus, accepting the resulting noise as a necessary trade-off for broader cover- age. C. Linguistic Complexity of Tharu Tharu is not a monolith. It belongs to the Indo-Aryan family but is heavily influenced by Tibeto-Burman languages due to geographic proximity. The Rana Tharu dialect, which forms the backbone of our dataset, shares significant lexical overlap with Awadhi but maintains a unique phonology and verbal structure. For example, the plural marking in verbs and the specific ergative constructions differ from standard Hindi. A major challenge in automated processing is the "fluidity" of the language; spelling is not standardized, and speakers often code-mix with Nepali or Hindi depending on the formality of the context. This necessitates a dataset that is not rigid but flexible enough to encompass these variations. I. METHODOLOGY: THE THARUCHAT DATASET A central contribution of this work is the release of prajwal-panth/tharu-chat. Constructing this re- source required navigating the complex sociolinguistic land- scape of the Terai region, where no single "standard" Tharu exists. A. Synthetic Generation Pipeline We utilized the advanced reasoning capabilities of Gemini- series models (specifically prompt-engineering Gemini 2.5 Pro) to synthesize the dataset. The process involved three distinct phases: 1) Phase 1: Grammar Injection and Context Loading: Commercial LLMs do not "know" Tharu natively. To condition the model, we utilized a technique called Context Loading. We fed the model with: ⢠Grammar Rules: Explicit rules regarding Tharu sentence structure (SOV), gender agreement, and tense markers specific to Rana Tharu. ⢠Folklore and Stories: We digitized childrenâs stories and oral folk tales (The "Alha" and "Sorathi" traditions) to provide the model with examples of authentic diction and vocabulary. ⢠Dialect Markers: We explicitly prompted the model to distinguish between "Dangaura" style (Western Nepal) and "Rana" style (Kailali/Kanchanpur). 2) Phase 2: Domain-Specific Elicitation: Once condi- tioned, we instructed the model to generate Question-Answer pairs. We focused on domains relevant to rural life to ensure the utility of the resulting model: ⢠Agriculture: Crop cycles for rice and wheat, pest control methods suitable for the Terai climate. ⢠Civic Life: Citizenship documentation, land registry in- teractions, and local governance. ⢠Cultural Knowledge: Festivals (Maghi), traditional attire, and food habits. 3) Phase 3: Human Validation and Cleaning: The raw output from the LLM was substantial but noisy. We engaged in a validation process to filter the data. The primary issues encountered were: ⢠Hindi-fication: The model frequently defaulted to Hindi grammar while using Tharu vocabulary. ⢠Dialect Confusion: The model occasionally mixed Rana and Dangaura morphologies within a single sentence. ⢠Regional Contamination: Due to the linguistic proxim- ity, some generated samples contained Awadhi or Bho- jpuri phrases. Human validation fixed these issues to a large extent. Obvious Hindi sentence structures were rewritten, and vocabulary was standardized where possible. However, we retained a degree of dialect mixing to reflect the natural speech patterns of the region. B. Dataset Composition and Quality Assessment We make available the tharu-chat dataset consisting of 3,955 instructionâresponse pairs; for this study, we draw upon roughly 3,116 pairs in our experimentation. Dialect Distribution: ⢠Rana Tharu (â70%): The dominant dialect in the dataset. This bias is due to the availability of Rana Tharu folk tales used for prompting. ⢠Dangaura Tharu (â20%): Integrated to support users from Western Nepal. ⢠Kochila/Others (â10%): Minor representation from Eastern dialects. Quality Note: We explicitly acknowledge that the dataset is "Silver Standard." It is not a linguistically perfect corpus. The presence of mixed dialects means the model trained on this data effectively learns a "Pan-Tharu" representation rather than a strictly prescriptive grammar of any single dialect. This was a deliberate choice to maximize coverage given the extreme scarcity of data. IV. THARU-LLAMA MODEL ARCHITECTURE We selected the Meta-Llama-3.2-3B-Instruct as our base model. While larger models (8B, 70B) offer superior rea- soning, the 3B parameter class represents a "sweet spot" for deployment in resource-constrained environments typical of the Global South. A. Hardware Constraints and Accessibility A key motivation for this work is accessibility. High-end H100 or A100 clusters are inaccessible to most researchers in Nepal and India. We designed our training pipeline to run entirely on a single NVIDIA T4 GPU (16 GB VRAM), which is available via free or low-cost cloud tiers (e.g., Google Colab, Kaggle). B. Fine-Tuning Configuration (LoRA) We employed Low-Rank Adaptation (LoRA) to fine-tune the model. This technique freezes the pre-trained weights W 0 and injects trainable rank decomposition matrices A and B, significantly reducing memory usage. The update rule is defined as: h = W 0 x + Îą r BAx(1) Our specific configuration targeted deep adaptation to cap- ture Tharu syntax: ⢠Target Modules: We applied LoRA adapters to all lin- ear layers: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj. This com- prehensive targeting was necessary to overwrite the strong Hindi priors in the feed-forward networks (FFNs). ⢠Rank (r) and Alpha (Îą): We set r = 16 and Îą = 32. A higher alpha scaling factor (Îą/r = 2) was chosen to increase the influence of the LoRA updates, forcing the model to prioritize the new Tharu syntax over its pre- trained multilingual knowledge. ⢠Precision: The training pipeline utilized native fp16 (16-bit floating point). We avoided bf16 to ensure com- patibility with the older T4 architecture. V. EXPERIMENTS: ABLATION STUDY To empirically validate the hypothesis that "small, high- quality synthetic data is sufficient for dialect adaptation," we conducted a rigorous ablation study focusing on data scaling laws. A. Experimental Setup We partitioned the verified training dataset into four incre- mental subsets: ⢠25% Data: 779 samples. ⢠50% Data: 1,558 samples. ⢠75% Data: 2,337 samples. ⢠100% Data: 3,116 samples. A fixed, held-out validation set of 390 samples was used to compute Perplexity (PPL). Perplexity measures how well the probability distribution predicted by the model matches the actual text; lower values indicate better performance. To ensure experimental validity on limited hardware, we implemented an aggressive memory cleanup protocol. Python garbage collection (gc.collect()) and CUDA cache clear- ing were triggered before every new initialization to prevent memory fragmentation from influencing training dynamics. B. Quantitative Results The results of the scaling experiments are presented in Table I. TABLE I ABLATION STUDY RESULTS: IMPACT OF DATA SCALE ON PERPLEXITY (PPL). Data %SamplesTrain LossVal LossPPL Base Model0--> 88.0 25%7791.08151.85926.42 50%1,5580.83641.46584.33 75%2,3370.66521.22413.40 100%3,1160.60661.05712.88 255075100 2 3 4 5 6 7 Training Data Fraction (%) Perplexity (Lower is Better) Validation PPL Fig. 1.The Tharu Data Scaling Law. The linear descent in perplexity demonstrates that the model effectively generalizes from limited data. C. Analysis of Scaling Laws The base LLaMA-3.2 model, when prompted in Tharu zero- shot, yields a Perplexity of > 88, producing output that is effectively random or completely reverts to Hindi. 1) Initial Adaptation (0-25%): The most dramatic shift occurs with the first 779 samples. The PPL drops to 6.42. This suggests that the model primarily needs to "unlock" the Tharu token space. Since Tharu uses Devanagari, the tokens already exist in the modelâs vocabulary; the instruction tuning simply realigns the probability weights to favor Tharu sequences over Hindi ones. 2) Linear Improvement (25-100%): As illustrated in Fig- ure 1, the improvement in perplexity follows a strictly monotonic, linear trend. The drop from 4.33 (50%) to 2.88 (100%) is significant. A Perplexity of 2.88 is comparable to the performance of base models on high- resource languages, indicating a surprising degree of fluency given the datasetâs size and "silver" quality. 3) Absence of Overfitting: Despite the small dataset size, the validation loss continues to decrease alongside train- ing loss (0.6066 vs 1.0571). This indicates that the aggressive dropout (0.05) and the regularization effect of LoRA (r = 16) effectively prevented rote memorization. D. Reproducibility To ensure transparency and reproducibility of our results, we make all core components of this work publicly available. The TharuChat dataset, training configurations, and model weights can be accessed through publicly hosted repositories. This allows other researchers to replicate our training pipeline, validate findings, and extend the work to other low-resource languages. VI. QUALITATIVE ANALYSIS While Perplexity provides a statistical measure of predictive power, it cannot fully capture the semantic coherence or cultural nuance of the generated text. To assess the modelâs pragmatic utility, we subjected Tharu-LLaMA to a series of inference tests. A. Inference Configuration To ensure reproducible and stable generation, we utilized a deterministic sampling strategy. 1 generation_config = 2 "do_sample": True, 3 "temperature": 0.3, # Low temp for stability 4 "top_k": 15, # Restrict sampling pool 5 "top_p": 0.3, # Nucleus sampling 6 "repetition_penalty": 1.2, 7 "max_new_tokens": 100 8 Listing 1. Inference Hyperparameters B. Case Studies a) Scenario 1: Banking and Technical Literacy: Prompt: ATM machine kaisik kam karat hae? (How does an ATM machine work?) Model Output: "ATM me card dhalo aur pin number daeke, paisa nikal jaat hae. Mastercard se bhi paisa nikalat haen. Bank jaeke check karo." Analysis: The model demonstrates a strong grasp of pro- cedural logic. Syntactically, the use of "nikalat haen" (they come out/emerge) reflects the specific plural/honorific marking characteristic of the Rana Tharu dialect. The inclusion of "Mastercard" indicates the modelâs ability to handle English loanwords within the Tharu grammatical structure (Code- Mixing), a critical feature for digital literacy in the region. b) Scenario 2: Abstract Concepts: Prompt: Machine Learning ko ho? (What is Machine Learning?) Model Output: "Machine Learning (ML) computer ko sikhawat hae ki naya data deo, tab u kaisik samajh lehi." Analysis: This example highlights the modelâs capacity for abstraction. It correctly treats "Machine Learning" as a proper noun. The explanationâ"computer ko sikhawat hae" (it teaches the computer)âis a semantically accurate simplifica- tion. The phrase "tab u kaisik samajh lehi" uses the future/- conditional marker -lehi, demonstrating correct morphological inflection for hypothetical scenarios. c) Scenario 3: Civic Processes (Governance): Prompt: Nagarikta banaye khatir kahan jaeke?? (Where to go to make Citizenship?) Model Output: "Jilla Prashasan Karyalaya (CDO Office) me Nagarikta banawe ko form bharo. Sarkari school se padhne wale logan khatir iskul me master form bhar sakat haen." Analysis: This output is the most significant indicator of the datasetâs quality. It correctly identifies the Jilla Prashasan Karyalaya (CDO Office) and provides culturally relevant advice regarding government schools, reflecting the specific content present in the training data. VII. CHALLENGES AND LIMITATIONS It is imperative to maintain a realistic perspective on the modelâs capabilities. Despite the successful reduction in per- plexity, Tharu-LLaMA and the TharuChat dataset possess inherent limitations. A. Dialectal Noise and Consistency As noted in the Methodology, the dataset is a mix of approx- imately 70% Rana, 20% Dangaura, and 10% Kochila/Others. While this aids in generalization, it also introduces inconsis- tency. The model may answer a query phrased in Dangaura Tharu with a response using Rana Tharu morphology. For users expecting a "pure" dialect response, this can be jarring. This is a direct consequence of the data scarcity; we simply do not possess enough separated data to train distinct models for each dialect. B. Residual Hindi/Awadhi Influence Despite human validation, the synthetic origin of the data means that some "Hindi-fication" persists. The Gemini models used for generation have a strong prior for Hindi. Conse- quently, complex sentence structures in Tharu-LLaMA some- times drift into Hindi word order (SOV), or utilize specific post-positions that are more common in Awadhi or Hindi than in Tharu. The model is "Tharu-dominant" but not "Tharu- exclusive." C. Domain Specificity The model excels in the domains it was trained on: agricul- ture, basic civics, and health. However, if prompted on topics outside this distributionâsuch as global history, advanced physics, or international politicsâthe model is likely to hallu- cinate or revert to English/Hindi. This is a known limitation of small-scale instruction tuning; the model has learned a "style" and specific facts, but has not fundamentally acquired a new world knowledge base in the Tharu language. VIII. DISCUSSION A. The "Pan-Tharu" Generalization Effect A primary concern in training on the tharu-chat dataset was its linguistic heterogeneity. In high-resource NLP, mixing dialects is often considered "noise" that degrades performance. However, our results suggest a different phenomenon: Dialect Generalization. Tharu-LLaMA appears to have learned a "Koine" or common-denominator representation. For instance, the use of auxiliary verbs like sakat hun (can) in the modelâs output is intelligible to speakers of both Rana and Dangaura dialects, even if it is not perfectly prescriptive in either. This suggests that for low-resource languages, aggregating dialects into a single model is not only an economic necessity but a viable linguistic strategy to create tools accessible to a broader population. B. Implications for the Global South The successful fine-tuning of a 3B model on a single T4 GPU with onlyâ3,000 examples challenges the prevailing narrative that AI requires millions of dollars in compute and billions of tokens. ⢠FP16 Stability: We utilized float16 precision, avoid- ing hardware limitations of older GPUs. ⢠Gradient Accumulation: By setting the per-device batch size to 2 and gradient accumulation to 8, we achieved an effective batch size of 16 without exceeding 16GB VRAM. This proof of low-compute feasibility implies that researchers in the Global South can build, fine-tune, and serve their own language models, effectively democratizing the production of AI for indigenous languages. IX. CONCLUSION This paper has presented Tharu-LLaMA, a 3B pa- rameter instruction-following model that establishes a new baseline for the Tharu language in the digital domain. By eschewing the traditional reliance on massive web- scraped corpora, which are virtually non-existent for Tharu, we successfully demonstrated the efficacy of the "LLM- to-Human" bootstrapping pipeline. The creation of the prajwal-panth/tharu-chat datasetâa mixed-dialect corpus anchored in Rana Tharu but inclusive of other va- rietiesâproves that indigenous linguistic complexity can be preserved rather than erased in the age of AI. Our empirical results lead to three critical conclusions: 1) Data Density over Volume: The linear reduction in per- plexity from 6.42 to 2.88 suggests that model alignment for distinct dialects is less about the quantity of data and more about the quality and syntactic density of the examples. A mereâ3,100 verified pairs were sufficient to override the modelâs strong pre-trained bias towards Hindi. 2) Viability of Small Models: We demonstrated that a 3B parameter model, when fine-tuned with high-rank LoRA adapters (r = 16), possesses sufficient capacity to model the complex verbal morphology of Tharu. This is pivotal for the "Green AI" movement and for accessibility in the Global South. 3) Acceptance of Imperfection: We argue that in the con- text of endangered or low-resource languages, waiting for "perfect" or "pure" datasets is a privilege we cannot afford. A "silver-standard" mixed dataset that enables functional communication is superior to no dataset at all. We release the Tharu-LLaMA weights and the TharuChat dataset to the open-source community. We hope this work serves as a reproducible blueprint for language activists and researchers working to digitize the diverse linguistic tapestry of the Himalayan region. REFERENCES [1] P.Panth,âTharu-chat[dataset],âhttps://huggingface.co/datasets/ prajwal-panth/tharu-chat, 2026, hugging Face. [2] A. Maiti, âTharullama-3.2-3b-instruct [program],â https://huggingface.co/ agnivamaiti/TharuLLaMA-3.2-3B-Instruct, 2025, hugging Face. [3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, âBert: Pre-training of deep bidirectional transformers for language understanding,â in Pro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), 2019, p. 4171â4186. [4] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. GuzmĂĄn, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, âUnsu- pervised cross-lingual representation learning at scale,â in Proceedings of the 58th annual meeting of the association for computational linguistics, 2020, p. 8440â8451. [5] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., âLora: Low-rank adaptation of large language models.â ICLR, vol. 1, no. 2, p. 3, 2022. APPENDIX A. Training Hyperparameters The final model was trained using the configuration detailed in Table I. We emphasize the use of fp16 and high gradient accumulation to fit within the 16GB VRAM constraint of the Tesla T4. B. Training Logs The training progression showed consistent convergence. The rapid drop in loss during the first epoch of the 100% data run confirms the high information density of the dataset. TABLE I FINAL TRAINING HYPERPARAMETERS HyperparameterValue Base ModelLlama-3.2-3B-Instruct Precisionfp16 (Mixed Precision) OptimizerAdamW Learning Rate 2Ă 10 â4 LR SchedulerLinear Decay Num Epochs3 Batch Size2 Grad Accumulation8 Effective Batch Size16 Max Seq Length512 Warmup Ratio0.03 LoRA Config Rank (r)16 Alpha (Îą)32 Dropout0.05 BiasNone Target ModulesAll Linear Layers 1 Run Name: run_100pct (3,116 Samples) 2 [Epoch 1] 3 Training Loss: 1.5085 4 Validation Loss: 1.4344 5 [Epoch 2] 6 Training Loss: 0.9083 7 Validation Loss: 1.1245 8 [Epoch 3] 9 Training Loss: 0.6066 10 Validation Loss: 1.0571 11 12 Final Metrics: 13 Eval Loss: 1.0571 14 Perplexity: 2.8780 Listing 2. Training Log Extract: 100% Data Run