Paper deep dive
From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text
Van-Truong Le
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 99%
Last extracted: 4/27/2026, 7:41:44 PM
Summary
This paper presents a dual-aspect evaluation framework to assess Large Language Models (LLMs) on their ability to simplify complex Vietnamese legal texts. The researchers conducted a large-scale benchmark of four models (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1) across three dimensions: Accuracy, Readability, and Consistency. Using a curated dataset of 60 articles from the Penal Code 2015, Civil Code 2015, and Land Law 2024, the study also implemented a novel nine-category error typology to diagnose reasoning failures. The findings reveal that while models like Claude 3 Opus achieve high accuracy, they often suffer from subtle misinterpretations, whereas models like Grok-1 excel in readability and consistency but struggle with complex application. The research highlights a fundamental gap between linguistic fluency and true legal reasoning capabilities in current LLMs.
Entities (10)
Relation Signals (5)
GPT-4o â exhibitserror â Oversimplification
confidence 100% ¡ GPT-4oâs defining flaw, observed consistently, was Oversim- plification (48 errors).
Claude 3 Opus â exhibitserror â Misinterpretation
confidence 100% ¡ Its high frequency of Misinterpretation errors (41) was particularly alarming.
Grok-1 â exhibitserror â Incorrect Example
confidence 100% ¡ Its primary failuresâIrrelevant and Incorrect Examplesâdemonstrate a clear trade-off
Gemini 1.5 Pro â exhibitserror â Internal Contradiction
confidence 100% ¡ It produced many perfect, error-free responses but also exhibited significant logical failures, including instances of Internal Contradiction
Penal Code 2015 â partofdataset â Legal Reasoning Error Typology
confidence 90% ¡ The articles were selected from three core legislative texts: the Penal Code 2015...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The complexity of Vietnam's legal texts presents a significant barrier to public access to justice. While Large Language Models offer a promising solution for legal text simplification, evaluating their true capabilities requires a multifaceted approach that goes beyond surface-level metrics. This paper introduces a comprehensive dual-aspect evaluation framework to address this need. First, we establish a performance benchmark for four state-of-the-art large language models (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1) across three key dimensions: Accuracy, Readability, and Consistency. Second, to understand the "why" behind these performance scores, we conduct a large-scale error analysis on a curated dataset of 60 complex Vietnamese legal articles, using a novel, expert-validated error typology. Our results reveal a crucial trade-off: models like Grok-1 excel in Readability and Consistency but compromise on fine-grained legal Accuracy, while models like Claude 3 Opus achieve high Accuracy scores that mask a significant number of subtle but critical reasoning errors. The error analysis pinpoints \textit{Incorrect Example} and \textit{Misinterpretation} as the most prevalent failures, confirming that the primary challenge for current LLMs is not summarization but controlled, accurate legal reasoning. By integrating a quantitative benchmark with a qualitative deep dive, our work provides a holistic and actionable assessment of LLMs for legal applications.
Tags
Links
- Source: https://arxiv.org/abs/2604.16270v1
- Canonical: https://arxiv.org/abs/2604.16270v1
Trouble viewing inline? Open PDF directly â
Full Text
32,396 characters extracted from source content.
Expand or collapse full text
From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text Van-Truong Le University of Science, VNUHCM Vietnam National University Hanoi Open University Ho Chi Minh, Vietnam 23120181@student.hcmus.edu.vn AbstractâThe complexity of Vietnamâs legal texts presents a significant barrier to public access to justice. While Large Language Models offer a promising solution for legal text simpli- fication, evaluating their true capabilities requires a multifaceted approach that goes beyond surface-level metrics. This paper introduces a comprehensive dual-aspect evaluation framework to address this need. First, we establish a performance benchmark for four state-of-the-art large language models (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1) across three key dimensions: Accuracy, Readability, and Consistency. Second, to understand the "why" behind these performance scores, we conduct a large-scale error analysis on a curated dataset of 60 complex Vietnamese legal articles, using a novel, expert-validated error typology. Our results reveal a crucial trade-off: models like Grok-1 excel in Readability and Consistency but compromise on fine-grained legal Accuracy, while models like Claude 3 Opus achieve high Accuracy scores that mask a significant number of subtle but critical reasoning errors. The error analysis pinpoints Incorrect Example and Misinterpretation as the most prevalent failures, confirming that the primary challenge for current LLMs is not summarization but controlled, accurate legal reasoning. By integrating a quantitative benchmark with a qualitative deep dive, our work provides a holistic and actionable assessment of LLMs for legal applications. Index TermsâLarge Language Models, Legal Reasoning, Er- ror Analysis, Text Simplification, Benchmarking, Vietnamese Law I. INTRODUCTION The advent of Large Language Models (LLMs) offers a transformative promise: to democratize access to justice by translating complex, codified legislation into language the public can understand. In civil law systems like Vietnamâs, this promise is particularly profound. The nationâs laws, while comprehensive, are often shrouded in âlegaleseââa dense legal terminology that creates a significant barrier for citizens seeking to understand their fundamental rights and obligations [8]. LLMs present a potential tool to remove this barrier. However, this promise is shadowed by a significant dan- ger: the risk of generating fluent, plausible, yet inaccurate legal simplifications [3]. To harness the potential of LLMs Accepted at the FISU Joint Conference on Artificial Intelligence (FJCAI 2026), Vietnam. responsibly, we must first be able to measure their capabilities with nuance and depth. Initial evaluations, including our own preliminary work, have focused on setting performance bench- marks using surface-level metrics such as legal accuracy, user- perceived readability, and output consistency [5]. Although these metrics provide a valuable âwhatââquantifying which models perform better on the surfaceâthey fundamentally fail to explain the âwhy.â A model might achieve a high accuracy score by correctly summarizing the general rule for inheritance, yet completely miss a critical exception for a specific circumstance, a subtle but catastrophic reasoning error that superficial scores would mask. This limitation was a key piece of feedback on our initial research, which was criticized for its small sample size and lack of novel insight into the modelsâ failure modes. This highlights a critical gap in the literature: the absence of a holistic evaluation framework that marries large-scale quantitative benchmarking with deep qualitative diagnostics of legal reasoning [5]. To fill this gap, this paper introduces a comprehensive, dual-aspect evaluation framework designed to be both broad and deep. We combine two distinct but complementary research thrusts: 1) A Large-Scale Performance Benchmark: We expand upon our previous methodology, applying the metrics of Accuracy, Readability, and Consistency to a large and diverse dataset of 60 complex Vietnamese legal articles to ensure the generalizability and statistical significance of our findings. 2) An In-Depth Legal Reasoning Error Analysis: We introduce and apply a novel, expert-validated error typol- ogy to the same outputs. This allows us to move beyond scores, dissecting each modelâs performance to identify and categorize the specific root causes of their reasoning failures. By integrating these quantitative and qualitative approaches, we provide more than just a leaderboard of models. Our work offers a detailed diagnostic of the current capabilities and, more importantly, the systemic weaknesses of state-of- arXiv:2604.16270v1 [cs.CL] 17 Apr 2026 the-art LLMs in the Vietnamese legal domain. The findings reveal crucial trade-offs between performance aspectsâsuch as readability versus factual fidelityâand pinpoint the core challenges that must be overcome to build truly reliable and trustworthy legal AI. I. RELATED WORK The application of LLMs in the legal domain has rapidly evolved, yet significant gaps remain, particularly for non- English languages and civil law systems. Our work is situated within several key research streams. A. Domain Adaptation and Pre-trained Legal Models A foundational line of research demonstrates that domain- specific pre-training significantly improves performance on le- gal tasks. Models like LEGAL-BERT [2] show that continued pre-training on large legal corpora enables models to better understand legal vocabulary and context compared to general- purpose models. This highlights the importance of domain- specific data, a challenge for languages like Vietnamese where large, digitized legal corpora are less abundant. B. Legal Text Simplification Simplifying legal texts is a recognized task aimed at im- proving access to justice. Research in this area has explored various techniques, from unsupervised methods combining lexical replacement and sentence splitting [1] to supervised approaches. However, a recurring challenge, as noted by Garimella et al. [4], is the scarcity of parallel complex- simple legal corpora, making supervised training difficult. This scarcity elevates the importance of evaluating the zero- shot simplification capabilities of modern LLMs, which is a focus of our study. C. Evaluation and Reliability of Legal LLMs As LLMs become more powerful, evaluating their reliability in high-stakes domains is paramount. Surveys like Lai et al. [6] provide a broad overview of LLM applications in law, from drafting assistance to legal Q&A, while also outlining risks related to reasoning, transparency, and factual accuracy. A critical aspect of evaluation is understanding and mitigating âhallucinations.â Dahl et al. [3], for instance, developed a typology for legal hallucinations focused primarily on factual inaccuracies and citation errors in case law summarization and legal Q&A tasks. While such work is foundational for identifying fact-based errors, our research addresses a different, yet equally critical, challenge: reasoning failures within the context of text sim- plification. Our nine-category error typology is specifically designed to capture nuances unique to this task. Unlike typolo- gies focused on hallucinations (i.e., fabricating non-existent facts), our framework also categorizes subtle but critical errors like Oversimplification, Misinterpretation of legal terms, and failure to apply principles correctly in Incorrect Examples. This focus on simplification-specific reasoning errors, rather than just factual correctness, represents a key contribution that complements existing evaluation frameworks. It underscores the inadequacy of surface-level accuracy metrics and motivates our deep dive into error analysis. D. Vietnamese Legal Language Processing In the context of Vietnamese, efforts to develop specialized legal AI resources are accelerating. On the research front, projects like ViGPT-Law [9] and legal Q&A datasets such as VLQA and VNLAWQC [7] are building foundational mod- els and benchmarks, primarily for information retrieval and question answering. In parallel, the Vietnamese government has deployed practical tools, such as the Ministry of Justiceâs âVirtual Legal Assistant,â aimed at improving public access to legal information. However, a critical gap exists between creating these ap- plications and fundamentally understanding their reliability. While existing efforts often emphasize factual correctness, they may overlook subtle but critical reasoning failures in gen- erative tasks like simplification. Our work directly addresses this gap. While applied tools provide valuable services, our research offers a fundamental, diagnostic evaluation of the un- derlying LLM capabilities, identifying systemic failure modes through a novel error typology. This large-scale, systematic evaluation of generative simplification, viewed through the lens of legal reasoning, is a contribution that complements existing work. It not only advances technical benchmarks but also provides an actionable framework for auditing and improving the safety and reliability of public-facing legal AI systems in Vietnam. I. A DUAL-ASPECT EVALUATION FRAMEWORK To provide a holistic assessment, we designed a framework that evaluates LLM performance from two complementary per- spectives: high-level performance metrics and low-level error analysis. The entire framework was applied to a large dataset to ensure statistical significance and generalizability, while also enabling a deeper understanding of how different models succeed or fail across various aspects of legal reasoning. A. Data and Model Selection Models: We selected four state-of-the-art LLMs represent- ing the current forefront of commercial AI: GPT-4o (OpenAI), Claude 3 Opus (Anthropic), Gemini 1.5 Pro (Google), and Grok-1 (xAI). These models were chosen based on three criteria: (1) Market Dominance: They represent the flagship models from the leading AI laboratories; (2) Architectural Diversity: They embody different training philosophies and context window capabilities (e.g., Geminiâs long context vs. Grokâs MoE architecture); and (3) Accessibility: They are cur- rently the most accessible high-performance tools for general users, making their evaluation highly relevant for public legal access. Dataset: We curated a dataset of 60 legal articles via purposive sampling. This large-scale approach prioritizes both depth and breadth. The articles were selected from three core legislative texts: the Penal Code 2015 (20 articles), the Civil Code 2015 (20 articles), and the Land Law 2024 (20 articles). The selection criteria focused on articles known for their complexity, including those with: (a) multiple exceptions and conditions, (b) abstract legal terminology (e.g., âtrĂĄi Äấo Äᝊc xĂŁ háťiâ), and (c) requirements for sequential logical reasoning (e.g., âthᝍa káşż tháşż váťâ). This ensures a challenging and representative testbed for the LLMs. By combining civil, criminal, and land law, the dataset reflects a wide range of legal reasoning challenges encountered in both daily life and specialized legal contexts. Moreover, this carefully balanced corpus provides sufficient variation to stress-test the adaptabil- ity of different model families under comparable conditions. Task: For each article, models were given a consistent, zero-shot prompt asking them to act as a legal assistant and explain the law in simple terms with a practical example for a layperson. The specific prompt used was: âAct as a legal assistant. Explain the following article [Article Content] in simple terms for a layperson and provide a practical exampleâ. Each model generated two responses per article (at temperature 0.2), creating a corpus of 480 outputs (4 modelsĂ 60 articles Ă 2 runs) for evaluation. This experimental design enables us to evaluate not only the accuracy and accessibility of the generated simplifications but also the stability of model behav- ior across repeated generations. Notably, this dual-component promptârequiring both explanation (a summarization task) and example generation (an application task)âallowed us to observe that models consistently struggled more with the latter, highlighting a core challenge in legal reasoning over mere text simplification. B. Phase 1: Overall Performance Benchmarking This phase quantifies the overall quality of the generated texts across three user-centric dimensions, providing a high- level comparison of the models. 1) Metric 1: Legal Accuracy: This metric assessed the legal precision of the outputs. It was rated by a team of five trained law students on a 1-5 Likert scale. To ensure reliability, each output was rated by at least two students, and any significant discrepancies were resolved through discussion. The metric is a weighted average of four sub-criteria: ⢠Content Preservation (40%): Fidelity to the core legal rule. ⢠Completeness (30%): Inclusion of all essential elements and conditions. ⢠Clarity (20%): How easy the legal definition is to understand. ⢠Example Relevance (10%): Appropriateness of the pro- vided example. The weighting scheme was established based on the principle of âSafety Firstâ in legal advice. Content Preservation is assigned the highest weight (40%) because a simplification that omits core legal meaning poses the greatest risk of misinformation. Completeness (30%) follows, ensuring no critical conditions are missed. Clarity (20%) and Example Relevance (10%) are weighted lower, as a legally accurate but dry explanation is preferable to a fluent but incorrect one. 2) Metric 2: Readability: This metric assessed the ease of understanding for a non-expert audience. It was rated by a diverse group of 253 non-expert participants recruited via university mailing lists and social media. Each participant rated a random subset of 10-15 outputs on a 1-5 Likert scale across three weighted sub-criteria: ⢠Ease of Language (40%): Simplicity of vocabulary and sentence structure. ⢠Structural Coherence (30%): Logical flow, formatting, and organization. ⢠Utility of Examples (30%): How helpful the example was in clarifying the concept. For readability, Ease of Language is weighted highest (40%) as it directly addresses the barrier of âlegaleseâ for laypeople, followed equally by Structural Coherence (30%) and Utility of Examples (30%), which support comprehension and retention. 3) Metric 3: Consistency: This metric measured the sta- bility of a modelâs output across two independent runs. This composite metric combined automated scores and human judgment: ⢠Semantic Consistency (30%): Cosine similarity be- tween sentence embeddings (using paraphrase- multilingual-MiniLM-L12-v2). ⢠Linguistic Consistency (30%): Sentence-level BLEU score to measure phrasing overlap. ⢠Length Consistency (10%): Stability in output length (word count). ⢠Manual Consistency Score (30%): Human evaluator rating of meaning preservation across runs. Length Consistency (10%): While output length is not a direct measure of reasoning quality, significant variance in length for identical prompts (e.g., producing a brief summary in one run and a detailed essay in another) indicates system instability. This metric serves as a proxy for the modelâs determinism and reliability in a production environment. C. Phase 2: In-Depth Legal Reasoning Error Analysis This phase moves beyond scores to identify the root causes of failure. 1) Error Typology: We developed a nine-category error typology, validated and refined in collaboration with legal experts from Hanoi Open University. This framework, detailed in Table I, allows for a consistent and granular assessment of model outputs. The typology is designed to capture a wide range of potential failures, from simple omissions to complex reasoning flaws. 2) Annotation Process: The same team of five law students who rated for accuracy also performed the error annotation. To ensure high-quality annotations, a rigorous two-stage process was employed. First, two raters independently annotated each of the 480 outputs. The initial inter-rater agreement was mea- sured using Cohenâs Kappa, yielding a score of Îş = 0.72. This value indicates substantial agreement between the annotators and validates the clarity of our error categories. Second, all discrepancies were flagged and resolved in a group session moderated by the lead researcher to produce the annotated dataset of annotated errors. IV. EXPERIMENTS AND RESULTS Our dual-aspect evaluation yielded a rich dataset, allowing us to both rank the models on performance and diagnose their underlying weaknesses. A. Phase 1: Overall Performance Results The large-scale benchmark revealed clear trade-offs between the modelsâ capabilities. The aggregated scores, presented in Table I, provide a high-level view of their strengths. Interestingly, while Claude 3 Opus had a slightly lower total score, Grok-1 emerged as the top performer by a narrow margin when all factors were considered. Grok-1 demon- strated a compelling profile, dominating in both Readability and Consistency. Claude 3 Opus was the clear leader in Legal Accuracy but was held back by slightly lower scores in other areas. Gemini 1.5 Pro showed strong accuracy but was significantly hampered by poor readability, often producing large, unstructured blocks of text. GPT-4o consistently trailed the other models. B. Phase 2: Legal Reasoning Error Analysis Results The error analysis, summarized in Table I, provides a deeper understanding of the scores from Phase 1. The large-scale data reinforce the distinct âpersonalitiesâ of each model: a) Grok-1: The Cautious Complier: With the lowest error count by a significant margin (89), Grok-1 proved to be the safest model. It committed zero errors of Omission of Core Elements, Misinterpretation, or Oversimplification. Its failures were almost exclusively concentrated in generative tasks, such as fabricating details in examples (Error 1.3) or providing irrelevant analogies (Error 3.1). b) Claude 3 Opus: The Ambitious but Risky Lawyer: Claude 3 Opus consistently attempted the most sophisticated analyses and, as a result, made the most nuanced and severe errors (196 total). Its high frequency of Misinterpretation errors (41) was particularly alarming. It repeatedly failed on articles involving subtle legal concepts, such as the distinction between different types of invalid civil transactions or the precise requirements for âurgent situations.â These tendencies indicate that Claude 3 Opus aims high but often overreaches in ways that compromise reliability. c) GPT-4o: The Eager but Oversimplifying Teacher: GPT-4oâs defining flaw, observed consistently, was Oversim- plification (48 errors). In numerous instances, it reduced com- plex, multi-condition rules into simplistic, often dangerously misleading, maxims. This highlights GPT-4oâs difficulty in balancing accessibility with fidelity to the original legal text. d) Gemini 1.5 Pro: The Inconsistent Performer: Gem- iniâs performance was marked by high variance. It produced many perfect, error-free responses but also exhibited signif- icant logical failures, including instances of Internal Con- tradiction and a high number of Irrelevant Examples. Such inconsistency makes Gemini less predictable and therefore harder to trust in sensitive legal contexts. C. Synthesis: Connecting Performance Metrics to Error Types By combining both phases, we can move beyond a simple leaderboard to draw a richer set of conclusions about the modelsâ underlying behaviors. ⢠The âAccuracyâ Illusion: Claude 3 Opusâs top score in Accuracy is misleading when viewed in isolation. Its high score is achieved by correctly handling the simpler components of legal articles, which masks a high propensity to fail on the most complex parts. Its leading count of critical Misinterpretation errors reveals a fragility in abstract reasoning that aggregate scores alone cannot capture. ⢠The Readability vs. Fidelity Trade-off: Grok-1âs top scores in Readability and Consistency are a direct result of its âcautious,â source-adherent nature. This strategy ensures high stability but hampers its ability to generate novel, accurate applications of the law. Its primary fail- uresâIrrelevant and Incorrect Examplesâdemonstrate a clear trade-off: it excels at faithful paraphrasing at the cost of being a less capable reasoner. ⢠Oversimplification as a Double-Edged Sword: GPT- 4oâs performance demonstrates how prioritizing accessi- bility can backfire. Its dominant error, Oversimplification, is the very mechanism it uses to achieve readable outputs. By stripping away essential legal nuances for the sake of simplicity, it creates distortions that undermine legal ac- curacy and render its simplifications potentially harmful. V. DISCUSSION Our large-scale findings provide robust evidence that current LLMs, while fluent, still struggle systematically with the core demands of legal reasoning. The consistent prevalence of Incorrect Example and Misinterpretation errors across a broad dataset confirms that models are significantly more capable at rephrasing existing text than at applying legal principles to novel scenarios. This highlights a fundamental gap between linguistic competence and abstract reasoning capabilities in the legal domain. A. Impact of Model Architecture and Alignment Strategies The distinct error profiles observed in Phase 2 suggest that performance differences are driven not just by model size, but by underlying architectural choices and alignment philosophies. Grok-1 and the âAlignment Taxâ Hypothesis: Grok-1âs un- expected dominance in Readability and Consistency, coupled with its lack of critical interpretation errors, can be attributed to its unique design. As a massive Mixture-of-Experts (MoE) model with 314 billion parameters, Grok-1 possesses a high capacity for nuance. More critically, unlike models from Ope- nAI or Anthropic which undergo aggressive Reinforcement Learning from Human Feedback (RLHF) focused on safety and refusal, Grok-1 appears to operate with looser alignment Fig. 1: Stacked Bar Chart of Error Distribution per LLM on 60 Articles. The chart clearly shows Grok-1âs lower total error count and its unique profile, which lacks errors in categories 1.1, 1.4, and 2.1. TABLE I: The Nine Categories of the Legal Reasoning Error Typology IDError NameDefinition 1.1Omission of Core ElementsFails to mention a core condition, subject, right, or obligation. 1.2Omission of ExceptionsFails to mention exceptions or limitations. 1.3HallucinationFabricates information or legal consequences. 1.4MisinterpretationIncorrectly explains the meaning of a legal term. 2.1OversimplificationSimplifies to the point of losing critical legal nuance. 2.2Cross-reference FailureFails to incorporate info from a referenced article. 2.3Internal ContradictionProvides contradictory statements in the same response. 3.1Irrelevant ExampleProvides an example that does not match the provision. 3.2Incorrect ExampleProvides a relevant example but draws a legally incorrect conclusion. TABLE I: Overall Performance Scores (Scaled 1â5) MetricGPT-4oClaude 3 Gemini 1.5Grok-1 Accuracy3.784.714.654.39 Readability4.554.784.414.93 Consistency3.493.813.944.28 Total4.024.554.424.57 constraints. We hypothesize that Grok-1 suffers less from the âalignment taxââa phenomenon where excessive safety tuning degrades performance on specific downstream tasks. This allows it to adhere faithfully to the simplification instruction without over-filtering or sanitizing the output, resulting in a âcautious complierâ behavior that prioritizes fidelity to the source text over risky inferential leaps. The Paradox of Safety in GPT-4o: Conversely, GPT-4oâs lower ranking reveals a tension between safety and legal nuance. Its primary failure mode, Oversimplification (Error 2.1), likely stems from robust safety guardrails and instruction tuning that favor conciseness and neutrality. When processing complex penal or civil codes, GPT-4o exhibits a conservative behavior, stripping away necessary but intricate conditions to provide a safe, high-level summary. While this reduces the risk of generating specific hallucinations, it renders the simplification legally shallow and less useful for analyzing edge cases, effectively trading accuracy for safety. Claude 3 Opus and Inference-Heavy Reasoning: Claude 3 Opusâs performance reflects the âhelpfulnessâ objective of its Constitutional AI (CAI) training. The model frequently attempts to make abstract connections to provide insightful explanations. While this ambition yields high accuracy on standard interpretations, it leads to a high rate of Misinter- TABLE I: Error Frequency Matrix by Model and Error Type (based on 480 responses) Model1.11.21.31.42.12.22.33.13.2Total GPT-4o312211234800429168 Claude 3 Opus28192341220111339196 Gemini 1.5 Pro21239343091824141 Grok-1021280000221889 Fig. 2: Radar Chart of Overall Performance Scores. The chart visualizes the distinct strengths of each model: Claude 3 Opusâs strength in Accuracy, Grok- 1âs dominance in Readability and Consistency, and Geminiâs imbalance. pretation when the law requires strict literalism. This suggests that optimization for âhelpfulâ conversation may inadvertently encourage the model to overreach, generating plausible but legally incorrect inferences. B. Contextualizing Findings within Broader Literature Our results corroborate and extend recent findings in Legal NLP. Similar to Dahl et al. [3], who identified pervasive hallucinations in case law generation, we found that even when models have access to the correct statute, they struggle to apply it accurately (Error 3.2: Incorrect Example). However, our study reveals a distinct failure mode specific to simplification: the fluency-accuracy trade-off. While Dahl et al. focused on factual fabrication, our findings suggest that models can generate highly fluent, logically sounding legal reasoning that contains subtle interpretation errors. This confirms that high linguistic competence often masks deficits in logical reason- ing, creating an âillusion of competenceâ that is particularly risky for lay users. C. Practical Implications: Towards Risk-Aware Human-in-the- Loop The scale of this study strengthens the conclusion that deploying LLMs for public-facing legal aid without robust human oversight is premature. However, our error typology offers a blueprint for more efficient oversight. Rather than treating all outputs as equally suspect, a system could apply our findings as a risk-assessment framework. For example, outputs from âinference-heavyâ models like Claude 3 Opus could be flagged for interpretation review, while example generation from all models should be routed to legal professionals, given the high failure rate in Error 3.2. This shifts human oversight from a passive safeguard to an active, risk-aware process. D. Limitations and Future Work While this study establishes a rigorous framework, we acknowledge specific methodological limitations that affect the interpretation and generalizability of our results. First, regarding data scale, our dataset consists of 60 legal articles. Although these were purposively sampled to ensure complexity, this sample size is relatively small compared to the vastness of the Vietnamese legal corpus. However, this scale was necessary to ensure the depth and quality of our rigorous manual annotation process, which would be unfeasible with a larger dataset. Consequently, the findings may not fully capture the performance variability across other specialized legal domains, such as intellectual property or administrative law. Second, regarding evaluator expertise, the reliance on senior law students rather than practicing attorneys is a limita- tion. While we mitigated this through rigorous training and achieved substantial inter-rater agreement (Îş = 0.72), we acknowledge that students may lack the practical experience required to detect the most subtle nuances in legal application errors compared to professional experts. Third, the experimental design was restricted to a zero-shot setting. We deliberately chose this to benchmark the modelsâ inherent, unassisted capabilities. However, this approach likely establishes a performance âlower bound.â We recognize that advanced techniques such as Few-Shot Learning or Chain- of-Thought (CoT) prompting could significantly mitigate spe- cific reasoning failures, particularly Oversimplification, which remains an open question for future research. Furthermore, this study represents a static snapshot of rapidly evolving models; newer iterations released after our evaluation period may exhibit different behaviors. Finally, the study relies exclusively on commercial closed- source models. Future benchmarks should include open-weight models (e.g., Llama 3, Qwen) to enhance reproducibility and allow for a deeper investigation into how training data transparency correlates with legal reasoning performance. VI. CONCLUSION This study provides the first large-scale, dual-aspect evalua- tion of leading LLMs on complex Vietnamese legal texts. By integrating a quantitative benchmark with a qualitative error analysis across 60 articles, we demonstrated that each model exhibits a unique and stable âerror profile.â We found that no model is flawlessly reliable and that the most significant, consistent challenges lie in applying legal rules to new ex- amples and correctly interpreting nuanced legal concepts. Our research contributes a reusable error typology and provides robust empirical evidence that, while promising, LLM tech- nology requires significant advancements in reasoning and a framework of expert human oversight before it can be safely deployed to simplify law for the public. Ultimately, our dual- aspect framework serves as a critical blueprint for evaluating high-stakes AI applications not only in Vietnam but also in other civil law jurisdictions and low-resource languages facing similar challenges. ACKNOWLEDGMENT We would like to express our sincere gratitude to the team of lecturers and law students from Hanoi Open University for their invaluable participation and expertise in the evaluation process. Their insights were crucial to the validation of the error typology and the annotation of the large-scale dataset. REFERENCES [1] Mert Cemri, Tolga C ̧ ukur, and Aykut Koc ̧. Unsupervised simplification of legal texts. arXiv preprint arXiv:2209.00557, 2022. [2] Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. Legal-bert: The muppets straight out of law school. arXiv preprint arXiv:2010.02559, 2020. [3] Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E Ho. Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16(1):64â93, 2024. [4] Aparna Garimella, Abhilasha Sancheti, Vinay Aggarwal, Ananya Ganesh, Niyati Chhaya, and Nanda Kambhatla. Text simplification for legal domain: Insights and challenges. In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 296â304, 2022. [5] Joshua Kelsall, Xingwei Tan, Aislinn Bergin, Jiahong Chen, Maria Waheed, Tom Sorell, Rob Procter, Maria Liakata, Jenny Chim, and Serene Chi. A rapid evidence review of evaluation techniques for large language models in legal use cases: trends, gaps, and recommendations for future research. AI & SOCIETY, pages 1â19, 2025. [6] Jinqi Lai, Wensheng Gan, Jiayang Wu, Zhenlian Qi, and Philip S Yu. Large language models in law: A survey. AI Open, 5:181â196, 2024. [7] Tan-Minh Nguyen, Hoang-Trung Nguyen, Trong-Khoi Dao, Xuan-Hieu Phan, Ha-Thanh Nguyen, and Thi-Hai-Yen Vuong. Vlqa: The first com- prehensive, large, and high-quality vietnamese dataset for legal question answering. arXiv preprint arXiv:2507.19995, 2025. [8] Pip Nicholson. Access to justice in Vietnam: State supplyâprivate distrust. In Legal Reforms in China and Vietnam, pages 188â215. Routledge, 2010. [9] Huy Quang Pham, Quan Van Nguyen, Dan Quang Tran, Thang Kien- Bao Nguyen, and Kiet Van Nguyen. Top 2 at alqac 2024: Large language models (llms) for legal question answering. International Journal of Asian Language Processing, 35(01):2450010, 2025.