Paper deep dive
BenGER: A Collaborative Web Platform for End-to-End Benchmarking of German Legal Tasks
Sebastian Nagl, Matthias Grabmair
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 99%
Last extracted: 4/18/2026, 1:28:59 AM
Summary
BenGER is an open-source, collaborative web platform designed for end-to-end benchmarking of legal tasks, specifically targeting German law. It integrates task creation, collaborative annotation, configurable LLM execution, and multi-metric evaluation into a single, role-aware workflow to improve transparency, reproducibility, and accessibility for non-technical legal experts.
Entities (7)
Relation Signals (4)
Sebastian Nagl → affiliatedwith → Technical University of Munich
confidence 100% · Sebastian Nagl Technical University of Munich
Matthias Grabmair → affiliatedwith → Technical University of Munich
confidence 100% · Matthias Grabmair Technical University of Munich
BenGER → developedby → Sebastian Nagl
confidence 100% · Sebastian Nagl and Matthias Grabmair. 2026. BenGER: A Collaborative Web Platform
BenGER → usestechnology → Next.js
confidence 100% · BenGER uses a modular service architecture with a Next.js (TypeScript) frontend
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Evaluating large language models (LLMs) for legal reasoning requires workflows that span task design, expert annotation, model execution, and metric-based evaluation. In practice, these steps are split across platforms and scripts, limiting transparency, reproducibility, and participation by non-technical legal experts. We present the BenGER (Benchmark for German Law) framework, an open-source web platform that integrates task creation, collaborative annotation, configurable LLM runs, and evaluation with lexical, semantic, factual, and judge-based metrics. BenGER supports multi-organization projects with tenant isolation and role-based access control, and can optionally provide formative, reference-grounded feedback to annotators. We will demonstrate a live deployment showing end-to-end benchmark creation and analysis.
Tags
Links
- Source: https://arxiv.org/abs/2604.13583v1
- Canonical: https://arxiv.org/abs/2604.13583v1
Trouble viewing inline? Open PDF directly →
Full Text
11,451 characters extracted from source content.
Expand or collapse full text
BenGER: A Collaborative Web Platform for End-to-End Benchmarking of German Legal Tasks Sebastian Nagl Technical University of Munich Munich, Germany sebastian.nagl@tum.de Matthias Grabmair Technical University of Munich Munich, Germany matthias.grabmair@tum.de Abstract Evaluating large language models (LLMs) for legal reasoning re- quires workflows that span task design, expert annotation, model execution, and metric-based evaluation. In practice, these steps are split across platforms and scripts, limiting transparency, re- producibility, and participation by non-technical legal experts. We present the BenGER 1 (Benchmark for German Law) framework, an open-source web platform that integrates task creation, collabora- tive annotation, configurable LLM runs, and evaluation with lexical, semantic, factual, and judge-based metrics. BenGER supports multi- organization projects with tenant isolation and role-based access control, and can optionally provide formative, reference-grounded feedback to annotators. We will demonstrate a live deployment showing end-to-end benchmark creation and analysis. Keywords Legal NLP, Benchmarking, Large Language Models, Annotation Systems, Legal AI ACM Reference Format: Sebastian Nagl and Matthias Grabmair. 2026. BenGER: A Collaborative Web Platform for End-to-End Benchmarking of German Legal Tasks. In Proceedings of International Conference on Artificial Intelligence and Law (ICAIL 2026). ACM, New York, NY, USA, 2 pages. 1 Introduction and Motivation Legal AI benchmarking is costly and technically demanding, espe- cially in jurisdictions such as Germany where high-quality legal expertise is scarce. In many projects, the pipeline is fragmented: experts provide materials or it is taken from existing sources, re- searchers translate them into benchmark instances, optionally an- notations are collected in separate tooling, model runs are executed via ad-hoc scripts, and evaluation code is reimplemented per study [1,2]. This introduces avoidable handoffs, reduces expert oversight, and makes collaboration and reproduction difficult. BenGER ad- dresses this by providing a unified, browser-based workflow that domain experts can operate end-to-end, from task definition and annotation through model execution and evaluation. 2 System Overview BenGER is a production-ready web application for end-to-end benchmarking of legal tasks (initially focused on German law, but not jurisdiction-bound). It supports multiple task formats (free-text 1 The full code is available here: https://github.com/SebastianNagl/benger-platform. A public instance of the application is accessible at https://what-a-benger.net. The authors thank the Daimler Benz Foundation for generous funding of the TITAN project which allowed us to do this work. reasoning, multiple choice, span annotation), collaborative annota- tion, batch execution of arbitrary LLMs, and result analysis using a broad set of metrics. BenGER is released as open-source software and can be deployed locally or institutionally while we also offer a securely hosted version for our community work. 3 Supported Workflow and Use Cases BenGER explicitly models the full legal benchmarking workflow that we will also show in our demonstration: (1) Task Creation: Legal experts define tasks and reference solutions directly in the platform. (2)Annotation: Human annotators submit solutions using a collaborative web interface. (3)Formative Feedback (Optional): Annotators may receive LLM-based feedback comparing their answers to reference solutions, providing constructive guidance. (4) Model Execution: Selected LLMs are executed on the same tasks using configurable API keys. (5)Evaluation: Results are evaluated using lexical, semantic, factual, classification, and LLM-as-a-judge metrics. (6)Analysis and Export: Results can be analyzed within the platform or exported for publication. 4 Technical Architecture and Security BenGER uses a modular service architecture with a Next.js (Type- Script) frontend and a FastAPI (Python) backend backed by Post- greSQL. Redis and Celery workers support scalable background execution for model runs and evaluations. The system is fully con- tainerized and deployable via Docker Compose or Kubernetes and is designed for collaborations involving potentially sensitive legal materials. Tenant isolation and access control. Organizations are isolated at the data layer and via role-based permissions (e.g., administrators, contributors, annotators). Project-level access controls allow fine- grained sharing while preventing accidental cross-organization data exposure. API key handling and operational boundaries. Model execution can be configured per user or per project, enabling contributors to use their own API credentials when appropriate. This reduces centralized credential management and helps align usage with institutional policies. Human oversight. Optional LLM feedback is designed to be sup- portive and educational rather than authoritative. Projects can disable feedback entirely or restrict it to specific tasks and roles, arXiv:2604.13583v1 [cs.CL] 15 Apr 2026 ICAIL 2026, June 08-12, SingaporeSebastian Nagl and Matthias Grabmair ensuring that expert governance remains the primary mechanism for benchmark quality assurance. 5 Positioning and Benefits over Existing Tooling BenGER targets a common gap: existing solutions often cover sin- gle steps of the benchmarking pipeline - either annotation/data management (e.g. DeepWrite [3]) or model evaluation, but do not provide an integrated, role-aware workflow that domain experts can run without scripting. Compared to general-purpose annota- tion platforms (such as LabelStudio or Doccano [4,5]), BenGER adds multi-organization isolation, configurable LLM execution, and standardized evaluation runs within one system and cost-free. Com- pared to ad-hoc evaluation scripts, it turns tasks, model configu- rations, and metrics into reusable, auditable artifacts, improving reproducibility for groups spanning universities, public institutions, and NGOs. Beyond general-purpose annotation platforms. General legal an- notation systems (e.g. Lawnotation [6]) offer flexible labeling UIs and dataset export, but they generally require additional infrastruc- ture to (a) enforce clean separation between multiple contributing organizations, (b) connect to heterogeneous LLM providers, and (c) execute and track evaluation runs with pre-defined and therefore consistent metric definitions. BenGER closes this gap by combining native legal-task annotation with model execution and standard- ized evaluation, allowing domain experts to retain control over task definition, reference answers, and quality assurance throughout the lifecycle. Beyond evaluation scripts and ad-hoc pipelines. In many research projects, evaluation is implemented as project-specific code: prompt templates, model calls, and metrics are encoded in notebooks or scripts that are hard to reuse across organizations and tasks. BenGER externalizes these steps into a shared platform: tasks, model config- urations, and metrics become explicit artifacts that can be reused, audited, and compared across groups, improving reproducibility and lowering onboarding costs. 6 Benefits for Annotators and Human Baselines To improve incentives and learning value, the platform can option- ally provide reference-grounded, constructive feedback to annota- tors - typical for german legal education (from a private lecturer, the ’Repetitor’), highlighting missing reasoning steps and common pitfalls while keeping expert governance in control. Human baselines with quality signals. The platform supports quality monitoring at the annotation level (e.g., progress tracking and agreement/consistency indicators), enabling project leads to manage baseline construction systematically. In practice, this re- duces the risk that benchmark conclusions are driven by noisy annotations or inconsistent task interpretation. 7 Community Impact and Reproducibility As open-source software, BenGER lowers barriers for non-technical contributors by making task contribution, annotation, model exe- cution, and evaluation accessible via a browser. It promotes repro- ducibility by storing task definitions, reference solutions, model configurations, and metric choices as explicit artifacts that can be shared, audited, and rerun. The integration layers are designed to be extensible for new tasks, model providers, and metrics. Lowering barriers for public institutions and NGOs. Legal datasets are often constrained by capacity and governance requirements. By providing organization-aware data separation and a browser-based workflow for task creation, annotation, and evaluation, BenGER en- ables institutions to contribute tasks and obtain model performance analyses without handing off raw materials to external engineers. Reproducible evaluation artifacts. BenGER encourages evaluation configurations to be stored as explicit, shareable artifacts: task definitions, reference solutions, model configurations, and chosen metrics. This supports transparent reporting and makes it easier to reproduce experimental results across research groups and over time. Extensibility. The metric and model integration layers are de- signed for incremental extension. New tasks, model providers, or scoring methods can be added without rewriting end-to-end evalu- ation pipelines, making the platform suitable for long-lived bench- mark initiatives. 8 Conclusion BenGER demonstrates how legal AI benchmarking can be made more transparent, collaborative, and accessible by integrating an- notation, model evaluation, and analysis into a single platform. We make legal experts control the full evaluation pipeline, which makes the system supports more reliable and can lead to scalable research on LLM capabilities in law. 9 AI Usage We used OpenAI GPT-5 for Latex, grammar and spelling tasks on the manuscript as well as Anthropic Claude Sonnet 4 and Claude Opus 4 and Google Gemini 2.5 Pro on code tasks. References [1]Yu Fan, Jingwei Ni, Jakob Merane, Etienne Salimbeni, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, Mrinmaya Sachan, Alexander Stremitzer, Christoph Engel, Elliott Ash, and Joel Niklaus. 2025. LEXam: Benchmarking Legal Reasoning on 340 Law Exams. arXiv:2505.12864 [cs] doi:10.48550/arXiv.2505.12864 [2]Neel Guha, Daniel E. Ho, Julian Nyarko, and Christopher Ré. 2022. LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoning. arXiv:2209.06120 [cs] [3]Urs Kramer, Michael Granitzer, and Johann Graf Lambsdorff. 2024. DeepWrite: Annotation and Extraction of Legal Texts. https://extract-annotations.deepwrite. pads.fim.uni-passau.de/. [4]Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018. doccano: Text Annotation Tool for Human. https://github.com/ doccano/doccano Software available from https://github.com/doccano/doccano. [5]Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Li- ubimov. 2020-2025.Label Studio: Data labeling software.https:// github.com/HumanSignal/label-studio Open source software available from https://github.com/HumanSignal/label-studio. [6]Gijs van Dijck, Carlos Aguilera, Chris van der Lans, Shashank Chakravarthy, and Sander van Essel. 2022. Lawnotation: A Formal Language for Legal Rules. https://w.lawnotation.org/.