Paper deep dive
Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task
Jungmin Choi, Keisuke Sakaguchi, Hiroaki Yamada
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 6/21/2026, 7:19:57 AM
Summary
This paper introduces a new dataset designed to evaluate the open-ended legal reasoning capabilities of Large Language Models (LLMs) specifically within the Japanese jurisdiction, using the writing component of the Japanese Bar Examination. The study evaluates models like GPT-4o, o3, and Claude 3 Opus across different prompting settings (Zero-Shot, Few-Shot, and Few-Shot with Law). Results indicate that current LLMs struggle significantly with complex, structured legal argumentation, with an average score of only 25.6/100. The research also highlights a high rate of hallucinations, particularly regarding legal precedents, and demonstrates that providing explicit statutory references (FS+Law) can improve performance and reduce errors.
Entities (8)
Relation Signals (5)
Japanese Bar Examination â covers â Civil Code
confidence 100% ¡ The examination consists of a multiple-choice test component and a writing test component covering core legal areas such as Civil Code, Penal Code...
GPT-4o â evaluatedby â Japanese Bar Examination
confidence 100% ¡ Specifically, our experiments include the following models: GPT-4o... The dataset is based on the writing component of the Japanese bar examination
o3 â evaluatedby â Japanese Bar Examination
confidence 100% ¡ o3(o3-2025-04-16)... The dataset is based on the writing component of the Japanese bar examination
Claude 3 Opus â evaluatedby â Japanese Bar Examination
confidence 100% ¡ Claude 3 Opus (claude-3-opus-20240229)... The dataset is based on the writing component of the Japanese bar examination
LLMs â exhibit â Hallucination
confidence 100% ¡ Moreover, we conducted a manual analysis of hallucinations to characterize when and how the models introduce content not supported by precedent or law.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) have shown strong performance on legal benchmarks, including multiple-choice components of bar exams. However, their capacity for generating open-ended legal reasoning in realistic scenarios remains insufficiently explored. Notably, to our best knowledge, there are no prior studies or datasets addressing this issue in the Japanese context. This study presents the first dataset designed to evaluate the open-ended legal reasoning performance of LLMs within the Japanese jurisdiction. The dataset is based on the writing component of the Japanese bar examination, which requires examinees to identify multiple legal issues from long narratives and to construct structured legal arguments in free text format. Our key contribution is the manual evaluation of LLMs' generated responses by legal experts, which reveals limitations and challenges in legal reasoning. Moreover, we conducted a manual analysis of hallucinations to characterize when and how the models introduce content not supported by precedent or law. Our real exam questions, model-generated responses, and expert evaluations reveal the milestones of current LLMs in the Japanese legal domain. Our dataset and relevant resources will be available online.
Tags
Links
- Source: https://arxiv.org/abs/2604.23730v1
- Canonical: https://arxiv.org/abs/2604.23730v1
Trouble viewing inline? Open PDF directly â
Full Text
29,946 characters extracted from source content.
Expand or collapse full text
Expert Evaluation of LLMâs Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task Jungmin Choi RIKEN Tokyo, Japan jungmin.choi@riken.jp Keisuke Sakaguchi Tohoku University Miyagi, Japan keisuke.sakaguchi@tohoku.ac.jp Hiroaki Yamada Tokyo Metropolitan University Tokyo, Japan ymd@tmu.ac.jp Abstract Large language models (LLMs) have shown strong performance on legal benchmarks, including multiple-choice components of bar ex- ams. However, their capacity for generating open-ended legal rea- soning in realistic scenarios remains insufficiently explored. No- tably, to our best knowledge, there are no prior studies or datasets addressing this issue in the Japanese context. This study presents the first dataset designed to evaluate the open-ended legal reasoning performance of LLMs within the Japan- ese jurisdiction. The dataset is based on the writing component of the Japanese bar examination, which requires examinees to iden- tify multiple legal issues from long narratives and to construct structured legal arguments in free text format. Our key contribu- tion is the manual evaluation of LLMsâ generated responses by le- gal experts, which reveals limitations and challenges in legal rea- soning. Moreover, we conducted a manual analysis of hallucina- tions to characterize when and how the models introduce content not supported by precedent or law. Our real exam questions, model-generated responses, and ex- pert evaluations reveal the milestones of current LLMs in the Japan- ese legal domain. Our dataset and relevant resources will be avail- able online. CCS Concepts â˘Computing methodologiesâNatural language processing; Neural networks;Information extraction; â˘Applied computingâ Law. Keywords Large language models; Legal reasoning; Bar exam questions; Ex- pert annotation; Hallucinations 1 Introduction Large language models (LLMs) have recently achieved strong re- sults on a variety of LegalNLP benchmarks, many of which take the form of classification or short-form question answering (e.g., entailment-style tasks, label prediction, and short retrieval-based QA) [ 2,7]. Such gains have fueled interest in deploying LLMs in high-stakes legal workflows. However, practical legal reasoning rarely consists of selecting a predefined option. This process involves identifying issues, articu- lating applicable laws or norms, applying statutes and precedents to facts, and justifying conclusions in a structured written argu- mentation. Furthermore, LLMs may generate legally or factually inaccurate or unsupported claims. Taken together, these consid- erations motivate the development of benchmarks that (i) require free-form legal argumentation over complex fact patterns and (i) provide expert-grounded evaluation beyond a single overall score. Professional qualification examinations provide a natural source of evaluation tasks because they are explicitly designed to assess application of laws under realistic scenarios. Recent work has used bar exams to assess frontier LLMs in the U.S. context, reporting strong performance on the Uniform Bar Exam, including its multiple- choice and essay writing components [ 9,12]. However, they do not address open-ended written reasoning under controlled prompt- ing interventions or in the Japanese jurisdiction. Figure1shows a translated excerpt of a Japanese bar exam writing-test question, illustrating the long-form fact narrative and open-ended prompt that require structured legal analysis rather than option selection. In Japan, entry into legal professions, such as judges, public prosecutors, and attorneys, is nationally regulated through this ex- amination. Passing the examination is mandatory for admission to the unified post-exam judicial training program administered by the Legal Training and Research Institute of the Supreme Court. The examination consists of a multiple-choice test component and a writing test component covering core legal areas such as Civil Code, Penal Code, their corresponding procedural codes, the Con- stitution of Japan, Commercial Code, and administrative laws. The writing component requires examinees to analyze factually com- plex prompts by identifying relevant legal issues, interpreting and applying statutory provisions and precedents, and producing co- herent, structured arguments. Thus, this examination provides a suitable foundation for evaluating open-ended legal reasoning that current benchmarks often fail to capture. We introduce a new dataset derived from the writing test com- ponent of the Japanese bar exam. Each instance in the dataset con- sists of an exam question, a response generated by an LLM, and de- tailed manual evaluations by legal experts. The expert evaluation assesses model outputs across several dimensions, including legal correctness, identification of issues, and argumentative structure. Additionally, we conduct a post-hoc analysis of hallucinations, sys- tematically identifying instances where models introduce content that is not supported by applicable laws or precedents. Overall, by combining real exam questions, model-generated answers, and ex- pert assessments, our dataset enables a detailed and realistic eval- uation of open-ended legal reasoning. Our contributions are threefold. 1) We constructed a dataset comprising the Japanese bar exam writing tests and their answers generated by LLMs. 2) We perform manual evaluation of model arXiv:2604.23730v1 [cs.AI] 26 Apr 2026 Choi et al. Facts 1. A operated a sole proprietorship using a single truck (hereinafter, the âTruckâ) to purchase fresh produce from growers and sell it to retailers and restaurants. 2. On September 10, 2017, A entered into a contract with B to purchase 5 kilograms of matsutake mushrooms for a price of 500,000 yen (here- inafter, the âSales Contractâ). Under the Sales Contract, delivery of the matsutake was to be made on the night of September 21, 2017 at Bâs warehouse (the â Warehouseâ) located near Bâs apple orchard,concur- rently with payment(i.e., delivery against payment). ⎠Question 1 (prompt) Assuming Facts 1 through 9 above, is Bâs claim against A for payment of the purchase price under the Sales Contract (as stated in Fact 9) legally justified? Answer with reasons. Figure 1: Excerpt from a writing-test question, machine- translated and manually corrected. This subquestion is about 700â800 words in English. outputs with expert legal annotators, providing fine-grained as- sessments of legal correctness, reasoning quality, and argumenta- tive coherence. 3) We conduct a hallucination analysis on the out- puts from models, offering insight into the types of failure modes that present in open-ended legal generation tasks. 2 Related Work Various datasets and benchmarks have been introduced in the do- main of legal NLP, including information retrieval, legal textual entailment, sentence classification, and question answering over statutes and case law. With the rise of LLMs, the trend has shifted from classification-style evaluation (e.g., selecting a label) toward assessing generation-style legal reasoning (e.g., producing struc- tured arguments). Our dataset lies at the intersection of them, tar- geting realistic, open-ended legal analysis, and supports expert- grounded evaluation of model outputs. 2.1 Legal Benchmark Datasets There are major benchmarks employing various tasks in legal lan- guage understanding and reasoning. LexGLUE [2] is a benchmark suite comprising multiple English legal datasets and tasks, enabling systematic comparisons across domains such as case law, legis- lation, and contracts. More recently, LegalBench [7] extends the scope of evaluation toward legal reasoning, reflecting the emer- gence of LLMs by compiling a large set of tasks constructed with legal expertise, with the explicit goal of measuring legally meaning- ful reasoning skills rather than narrow text classification accuracy. Another line of work introduces tasks from legal education and professional qualification exams. Such sources are attractive be- cause they are intentionally designed to assess whether examinees can arrange facts and apply laws to them. For example, JEC-QA [ 17] is constructed from Chinaâs national judicial examination, and Choi et al. [ 4] similarly evaluates GPT-family models on the multiple- choice component of the Japanese bar examination. In the common- law context, CaseHOLD [ 16] provides multiple-choice questions derived from judicial opinions, focusing on identifying the correct holding given a scenario. These datasets are useful for benchmark- ing legal knowledge and the retrieval ability of precedents and statutes. However, their answer formats often constrain evaluation to selection among predefined options, making it difficult to assess whether a model can produce coherent legal arguments. Shared tasks also play an important role in the legal domain. The COLIEE competition includes tasks such as case law retrieval and statute-based entailment [8]. COLIEE demonstrated how real exams can be utilized in standardized evaluation tasks. However, their primary evaluation settings remain classification-based eval- uations for retrieval and entailment performance, which do not require free-form legal argumentation. Given the limitations of multiple-choice and short-answer for- mats, there are datasets designed to assess the long-form answers and reasoning. LLeQA [ 10] provides a long-form legal question an- swering task with sample answers written by experts. LEXam [6] utilizes exam questions in the law domain. They include open-ended questions that require structured argumentation based on multi- step legal reasoning. GreekBarBench [3] aligns closely with our work on motivation. They utilize bar exam style questions and pro- vide scores of LLM-generated answers by human experts, focusing on legal analysis beyond surface correctness. In spite of these advances, manual evaluations by experts for model-generated answers remain limited. Many datasets provide (i) questions along with reference answers, or (i) scores on final outcomes for automatic scoring. However, they do not provide an expert assessment. This gap prevents us from analyzing where models fail in legal reasoning and intermediate steps. 2.2 Hallucinations Reliability is the major concern in applying LLMs in the legal do- main. Legal problems are highly sensitive to factual details and le- gal interpretations. LLMs are known to produce fluent but unsup- ported arguments. Recent work reports the prevalence of legal hal- lucinations and their risks, including false citations and confidently stated but incorrect legal claims [ 5]. The hallucination problems are reported even in legal domain-specific research tools. Magesh et al. [11] report that even popular RAG-based systems suffer from hallucination at considerable rates (17-33%). These issues motivate evaluation frameworks that explicitly separate reasoning quality and legal soundness from hallucination, rather than conflating these aspects into a single overall score. 2.3 Our Dataset Our dataset has advantages beyond the previous datasets in three ways. First, our dataset provides a task, targeting open-ended, exam- style legal reasoning in a setting that naturally elicits structured argumentation rather than simple classification tasks. Second, our dataset provides pairs of questions and their LLM-generated an- swers, enabling analysis of model behavior in legal reasoning. Third, our dataset additionally provides expert evaluation results. They include not only overall scoring about legal reasoning, but also hallucinations. Our dataset enables a fine-grained empirical study of LLMsâ legal reasoning capabilities and failures in a high-stakes, professionally aligned evaluation setting. Expert Evaluation of LLMâs Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task 3 Dataset Construction 3.1 Source Our dataset is constructed from official questions of the writing test of the Japanese National Bar Examination for the years 2017â 2023. For each year, we include all subjects published in the exam, yielding 161 questions in total (23 per year). In the writing test, each subject typically contains two to three independent questions. Each question presents a detailed factual scenario as a prompt, and poses one or more subquestions that ask the examinee to identify and analyze the legally relevant issues and articulate reasoned views on the applicable law. These questions are designed to elicit practical legal reasoning: examinees must se- lect and interpret statutory provisions and precedents, distinguish among competing interpretations, and organize their analysis into coherent, well-structured arguments. 3.2 Data Scheme We provide a structured representation of the writing test ques- tions for reproducible evaluation and controlled prompting. Each instance corresponds to a single question and contains question data and statutory references. Question data comprises (i) a fact-rich narrative prompt, (i) one or more subquestions, and (i) any accompanying materials refer- enced in the prompt (e.g., statutory excerpts, tables/figures). Statutory references are a collection of relevant statutes not ex- plicitly cited in the questions. One legally trained author manually compiled them with reference to third-party answers. Each entry contains the law identifier, law name, and the full text of the refer- enced article in a lawtext-like 1 format. 4 Experimental Settings 4.1 Models and Parameters We evaluate several state-of-the-art large language models repre- senting different model families and training paradigms. Specifi- cally, our experiments include the following models:GPT-4o(gpt- 4o-2024-11-20) [ 13,14],o3(o3-2025-04-16) [15],Claude3Opus (claude-3-opus-20240229) [1]. Unless otherwise specified, all models were evaluated using their default inference and decoding configurations. GPT-4o was run with standard sampling (temperature = 1.0, top-í= 1.0). The o3 model does not provide explicit sampling parameters for users. For the reasoning effort, we used a default of medium. Unless other- wise stated, no explicit maximum output token limit was imposed. For all models, system prompts assign the role of a Japanese legal expert and require that outputs be based on Japanese law and written in a casual form (non-polite form) Each question instance is input as a part of a user prompt. The question statement is presented without modification, following an instruction specifying the temporal scope of applicable law (e.g., âAnswer based on laws and regulations in force as of [the date of the examination]â). This instruction ensures that models reason within a consistent legal temporal constraint. 1 https://github.com/yamachig/lawtext 4.2 Approaches In the experiments, we prepare three different settings for each LLM.ZS: The zero-shot (ZS) setting is the most straightforward approach, involving no specialized prompts or in-context learning configurations. Models receive only the target questions as input. FS: The few-shot (FS) employs in-context learning. The prompt includes example bar exam problems and corresponding sample answers. These examples were drawn from the 2018 and 2019 ex- ams, which do not overlap with our evaluation set used in the ex- periment.FS+Law: Building on FS, this setting supplements the prompt with relevant legal articles, sourced from statutory refer- ences from our dataset. 4.3 Data All experiments were conducted using bar exam questions from 2021, 2022, and 2023. From the full dataset described in Section 3, only questions from the seven compulsory subjects were utilized: The Constitution of Japan, Civil Code, Penal Code, administrative laws, Code of Civil Procedure, Code of Criminal Procedure, and commercial laws. In total, we evaluate21questions (7 per year across 2021â2023), producing189model-generated answers under three prompting settings. 4.4 Evaluation Metrics Our experiments relied on expert manual evaluation to assess out- puts involving complex, extended legal reasoning. The evaluation was conducted by legal experts from the faculty of law at a Japan- ese university. Each expert was assigned to a specific subject, for a total of 7 evaluators. Experts assessed answers holistically, consid- ering issue identification, legal analysis, application of law to facts, logical structure, and clarity of writing, in line with the written exam scoring philosophy. They could also optionally leave free- form comments. The outputs were assessed using both categorical gradings and numerical scores. The categorical gradings consist of four categories that generally correspond to the predefined score ranges in the bar exam grading policy:Excellent(100â75),Good(74â58),Adequate (57â42), andPoor(41â0). We treat 25 as an important threshold: in the real exam, a score below 25% is an instant failure, so an answer scoring at least 25 meets the minimum requirement. 5 Results and Discussion 5.1 Overall Performance The left side of Table1shows scores from each approach. The scores are averages of scores over the three years. The right side of Table 1shows the number of answers that received the corre- sponding categorical grade for each approach. Across all answers, the average score is 25.6 out of 100, and only9.0%of answers are Adequate;5.3%reachGood. No answer is labeled asExcellent. When looking at the overall average scores, providing expert- curated statutes (FS+Law) improves outcomes compared to zero- shot (ZS) and few-shot without statutes (FS). However, the mag- nitude of the effect differs across models. Claude 3 Opus benefits substantially from FS+Law, increasing its score from30.4%in FS to37.1%, whereas GPT-4o consistently performsPoorly across all Choi et al. Table 1: Expert evaluation results ApproachesScores by subjects, averagedCategorical grades for each setting ModelPrompting Penal Criminal Proc. Const. Admin. Civil Comm. Civil Proc. Avg. Exc. Good Adeq.Poor o3ZS18.731.351.334.7 10.740.721.3 29.803018 o3FS20.326.344.035.3 14.045.021.0 29.402118 o3FS+Law20.328.750.733.7 18.339.022.7 30.501317 GPT-4oZS7.024.017.023.30.726.315.0 16.200021 GPT-4oFS9.716.723.318.32.728.318.3 16.800021 GPT-4oFS+Law9.017.325.022.35.326.717.7 17.600021 Claude 3 Opus ZS18.331.320.725.77.333.719.0 22.300021 Claude 3 Opus FS26.330.750.739.09.332.024.7 30.401416 Claude 3 Opus FS+Law31.338.749.751.0 14.747.027.3 37.10399 Table 2: Hallucinations by Subjects SubjectHallucination Not hallucinationSum Civil133 (24.7%)406 (75.3%) 539 (100%) Commercial117 (24.8%)354 (75.2%) 471 (100%) Criminal Proc.115 (41.4%)163 (58.6%) 278 (100%) Civil Proc.88 (21.2%)328 (78.8%) 416 (100%) Constitution84 (20.6%)323 (79.4%) 407 (100%) Penal72 (16.0%)379 (84.0%) 451 (100%) Administrative67 (14.8%)387 (85.2%) 454 (100%) settings. Scores from o3 are relatively stable across prompting set- tings, though its FS+Law still shows a better score. 5.2 Expert Comments Analysis We analyze the expert comments available for104/189answers. Among the 104 answers, 36, 32, and 36 answers are from ZS, FS, and FS+Law promptings, respectively. We summarize and report the remarkable comments, focusing on recurring keywords and phrases related to errors. There are comments about the inappropriate format of answers, such as bullet fragments. These types of comments are more com- mon in ZS (found in 10 out of 36 answers,27.8%) than in FS (2/32, 6.3%) or FS+Law (2/36,5.6%). In contrast, comments indicating weak fact-sensitive application remain common across settings: ZS (13/36,36.1%), FS (10/32,31.3%), FS+Law (10/36,27.8%). These ob- servations suggest that in-context learning enhances consistency in answer style compliance; however, it does not necessarily im- prove the mapping of complex facts to legal requirements during practical reasoning. Expert comments noted missing or incorrect mentions of statutes. This type of comment decreased in FS+Law (6/36,16.7%) from FS (11/32,34.3%). Providing gold statutes reduces statute-selection or citation errors, as we intended. Moreover, there are 7 comments on answers from o3 models regarding hallucinations of legal re- sources, such as laws and precedents. 5.3 Hallucination Check To systematically assess the extent to which the LLMs generate hallucinated legal content, we performed a post-hoc analysis on the hallucinated content. The analysis focuses on hallucinations Table 3: Hallucinations by Approaches ModelPrompting Hallucination Not hallucination o3ZS236 (34.7%)445 (65.3%) o3FS177 (30.8%)397 (69.2%) o3FS+Law126 (22.8%)426 (77.2%) GPT-4oZS12 (11.7%)91 (88.3%) GPT-4oFS16 (11.5%)123 (88.5%) GPT-4oFS+Law17 (10.2%)149 (89.8%) Claude 3 Opus ZS21 (14.6%)123 (85.4%) Claude 3 Opus FS 53 (17.6%) 248 (82.4%) Claude 3 Opus FS+Law18 (5.1%)338 (94.9%) involving laws and precedents, as they are principal components of legal reasoning and can be clearly distinguished. We employed a law school graduate of a Japanese university to annotate the hallucinated, false laws and precedents. The an- notator was instructed to first identify every mention of laws or precedents, and classify each as either a hallucination or not. If a mention consisted solely of a name or a number identifier of a law or precedent, and if the mention is inconsistent with actual laws or precedents, the annotator labeled the mention as a hallucination. If a mention included a quotation of a part of or a whole law article or a precedent, and if its quoted content contradicted its original source, the annotator flagged the mention as hallucination. We used all of 189 answers from the LLMs for the analysis. As a result, there are 2,575 mentions of laws and 441 mentions of prece- dents in total. Out of them, 381 mentions of laws and 295 mentions of precedents are hallucinations. The hallucination rate for prece- dents (66.9%) is substantially higher than that for laws (14.8%), in- dicating that the modelâs knowledge is less reliable for precedents than for law articles. Table 2presents the distribution of hallucinations by subject. The Civil Code showed the highest number of hallucinated men- tions (133), while Administrative laws had the lowest (67). The to- tal number of mentions to laws and precedents for the Code of Criminal Procedure was the lowest among all subjects (278), and its hallucination rate was the highest (41.4%). The LLMs are less likely to reference legal resources in the subject and more likely to make incorrect citations. Expert Evaluation of LLMâs Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task Comparing the results across models (Table3), approaches with o3 resulted in more frequent hallucinations than approaches with the other models. The differences in prompting also impacted the hallucinations. FS+Law consistently showed lower hallucination rates than those with FS or ZS across all the models. 6 Conclusion We introduced a dataset for open-ended legal reasoning based on the writing test of the Japanese bar exam. Using a subset, we manu- ally evaluated 189 LLM-generated answers. Performance remained limited. The average score stayed at 25.6/100. These results reveal a substantial gap between prior LegalNLP benchmark performance and the ability to produce legally coherent, well-structured argu- ments. Furthermore, our analysis of hallucinated mentions of legal resources highlights substantial room for improvement in Japan- ese legal reasoning. 7 Ethical Considerations and Limitations We summarize key ethical considerations and practical limitations of our dataset construction and evaluation protocol. â˘Expert grading and hallucination annotation were conducted by human evaluators who have expert knowledge in the Japanese legal domain. Annotators were compensated for their time. â˘There are limitations of reliability in expert scoring. Each subject was evaluated by a single expert. This design matches practical constraints (domain expertise, financial and time cost); however, it limits the extent to which we can quantify inter-rater variabil- ity. Also, we cannot perfectly replicate the true scoring protocol in the real exam. Thus, one should be careful when comparing our results with actual human examineesâ scores and statistics. â˘We evaluated answers generated by specific API snapshots of GPT-4o, o3, and Claude 3 Opus. Because proprietary models evolve over time, results may not transfer directly to future versions, open-source models, or systems with different tool access. We mitigate this by reporting the exact model identifiers and set- tings, but readers should interpret the findings as a snapshot of model behavior under the specified experimental conditions. Acknowledgments This work was supported by JST PRESTO Grant Number JPMJPR236B. References [1]Anthropic. 2024.The Claude 3 Model Family: Opus, Son- net, Haiku (Model Card).https://w-cdn.anthropic.com/ de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf. Accessed 2026-01-30. [2]Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androut- sopoulos, Daniel Katz, and Nikolaos Aletras. 2022. LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, 4310â4330. doi:10.18653/v1/2022.acl-long.297 [3]Odysseas S. Chlapanis, Dimitrios Galanis, Nikolaos Aletras, and Ion Androut- sopoulos. 2025. GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations. InFindings of the Association for Computational Lin- guistics: EMNLP 2025. Association for Computational Linguistics, Suzhou, China, 25099â25119. doi:10.18653/v1/2025.findings-emnlp.1368 [4]Jungmin Choi, Jungo Kasai, and Keisuke Sakaguchi. 2023. Nihon no Shi- houshiken wo Daizai toshita GPTmoderu no hyouka (Evaluation of GPT Models on the Japanese Bar Examination).Jxiv Preprint(2023). doi:10.51094/jxiv.559In Japanese. [5]Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E Ho. 2024. Large Le- gal Fictions: Profiling Legal Hallucinations in Large Language Models.Journal of Legal Analysis16, 1 (Jan. 2024), 64â93.doi:10.1093/jla/laae003 [6]Yu Fan, Jingwei Ni, Jakob Merane, Yang Tian, Yoan HermstrĂźwer, Yinya Huang, Mubashara Akhtar, Etienne Salimbeni, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, Mrinmaya Sachan, Alexander Stremitzer, Christoph Engel, Elliott Ash, and Joel Niklaus. 2025. LEXam: Benchmarking Legal Rea- soning on 340 Law Exams. arXiv:2505.12864[cs.CL]https://arxiv.org/abs/2505. 12864 [7]Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher RĂŠ, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rock- more, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fa- gan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jes- sica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. 2023. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. arXiv:2308.11462[cs.CL] [8]Rabelo Juliano, Kim Mi-Young, Goebel Randy, Yoshioka Masaharu, Kano Yoshi- nobu, and Satoh Ken. 2021. COLIEE 2020: Methods for Legal Document Re- trieval and Entailment.Lecture Notes in Computer Science(2021), 196â210. doi:10.1007/978-3-030-79942-7_13 [9]Daniel Martin Katz, Michael J. Bommarito, Shanqing Gao, et al. 2023. GPT-4 Passes the Bar Exam.Philosophical Transactions of the Royal Society A(2023). [10]Antoine Louis, Gijs van Dijck, and Gerasimos Spanakis. 2024. Interpretable long-form legal question answering with retrieval-augmented large language models. InProceedings of the Thirty-Eighth AAAI Conference on Artificial Intel- ligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intel- ligence and Fourteenth Symposium on Educational Advances in Artificial Intelli- gence (AAAIâ24/IAAIâ24/EAAIâ24). AAAI Press, Article 2484, 10 pages.doi:10. 1609/aaai.v38i20.30232 [11]Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Man- ning, and Daniel E. Ho. 2024. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. arXiv: 2405.20362[cs.CL]https://arxiv.org/ abs/2405.20362 [12]OpenAI. 2024. GPT-4 Technical Report. arXiv:2303.08774[cs.CL]https://arxiv. org/abs/2303.08774 [13]OpenAI. 2024. GPT-4o model documentation (snapshot: gpt-4o-2024-11-20). OpenAI documentation.https://platform.openai.com/docs/models/gpt-4o? snapshot=gpt-4o-2024-11-20 Accessed 2026-01-30. [14]OpenAI. 2024. GPT-4o System Card. OpenAI documentation.https://openai. com/index/gpt-4o-system-card/Accessed 2026-01-30. [15]OpenAI. 2025. o3 model documentation.https://platform.openai.com/docs/ models/o3. Accessed 2026-01-30. [16]Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. 2021. When does pretraining help? assessing self-supervised learning for law and the CaseHOLD dataset of 53,000+ legal holdings. InProceedings of the Eighteenth International Conference on Artificial Intelligence and Law(SĂŁo Paulo, Brazil)(ICAIL â21). Association for Computing Machinery, New York, NY, USA, 159â168. doi:10.1145/3462757.3466088 [17]Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. JEC-QA: A Legal-Domain Question Answering Dataset. InThe Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelli- gence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press, 9701â 9708. doi:10.1609/AAAI.V34I05.6519