Paper deep dive
MERCI Cards: An LLM Evaluation and Deployment Framework for High-Stakes Domains
Aparna Komarla, Annalisa Szymanski
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As LLMs are increasingly deployed in high-stakes professional workflows, engineers and researchers require principled protocols to systematically track, monitor, and improve model performance across deployment cycles. We present a mathematical framework for iterative LLM evaluation and deployment, and demonstrate its application to AI systems used in criminal justice. Our framework formalizes LLM integration in high-stakes, high-risk, and resource-constrained domains across model selection, rubric design, evaluations and deployment via a weighted multi-objective optimization. We demonstrate that MERCI Cards can guide improvements of the system across deployment iterations, direct developer attention toward under-performing areas, and focus user attention on validation and error-correction in the LLM's outputs.
Tags
Links
- Source: https://arxiv.org/abs/2610.04430v1
- Canonical: https://arxiv.org/abs/2610.04430v1
Trouble viewing inline? Open PDF directly →
Full Text
54,782 characters extracted from source content.
Expand or collapse full text
MERCI Cards: An LLM Evaluation and Deployment Framework for High-Stakes Domains Aparna Komarla Note: Aparna is the founder of Redo.io (https://getredo.io), a legal AI nonprofit building technology solutions for resentencing implementation in California; a Responsible AI Fellow at the Duke University School of Law; and a graduate student in Symbolic Systems at Stanford University. email: akomarla@stanford.edu Affiliation: Redo.io (https://getredo.io), Berkeley, California, USA Affiliation: Stanford University, Palo Alto, California, USA and Annalisa Szymanski Note: Annalisa Szymanski is a PhD candidate in the Human-Computer Interaction (HCI) Lab at the University of Notre Dame email: aszyman2@nd.edu Affiliation: University of Notre Dame, Notre Dame, Indiana, USA Abstract. As LLMs are increasingly deployed in high-stakes professional workflows, engineers and researchers require principled protocols to systematically track, monitor, and improve model performance across deployment cycles. We present a mathematical framework for iterative LLM evaluation and deployment, and demonstrate its application to AI systems used in criminal justice. Our framework formalizes LLM integration in high-stakes, high-risk, and resource-constrained domains across model selection, rubric design, evaluations and deployment via a weighted multi-objective optimization. We demonstrate that MERCI Cards can guide improvements of the system across deployment iterations, direct developer attention toward under-performing areas, and focus user attention on validation and error-correction in the LLM’s outputs. Keywords: AI for Access to Justice, Socio-Technical Systems, Fairness and Transparency, Public Interest Technology 1. Introduction Deploying LLMs in professional workflows demands ongoing judgment: when are outputs reliable enough, when should human reviewers be involved, and when does a newer model justify switching costs? LLM judges offer scalable evaluation but are susceptible to self-preference bias and rubric ceiling effects (Spiliopoulou et al., 2025; Wataoka et al., 2025; Bavaresco et al., 2025; Gu and Jiang, 2024; Yancey et al., 2023). Human evaluators offer domain expertise but are costly and may encode interpretive biases that are difficult to tease out (Chen et al., 2024; Shankar et al., 2024). Few established protocols exist that govern how to combine evaluator types, select general-purpose AI models, and make principled tradeoffs between quality, cost, and human involvement. The absence of guidance for building effective systems, and transparency about deployed systems, has made some practitioners skeptical of AI integrations overall, and sometimes opt to avoid AI adoption (Chappidi and Singh, 2026). This information gap is particularly consequential in domains where access to technology is unequal and the stakes of error are high. For instance, the cost of repeating an evaluation with human experts is amplified in organizations with funding constraints for core operations. But without a disciplined evaluation framework, it is difficult to know whether a given system is ready for deployment and how to improve it. We propose a mathematical framework for LLM evaluation and deployment: Model-Evaluation-Rubric-Iterate, MERCI Cards for short—a framework for AI deployment and integration for transparency and observability. MERCI Cards are inspired by Model Cards among other documentation tools for machine learning and automated decision making systems (Mitchell et al., 2019). The framework covers model selection, evaluation rubric design, and human-LLM evaluation feedback across multiple iterations. Section 2 presents the formal design and MERCI Card and Section 3 introduces an application of the framework to the RJA and reports findings from real-world deployments. 1.1. Research Questions Our work addresses three questions: RQ1. How should model selection, evaluation, and iteration be formalized for LLM integration in high-stakes, high-risk and resource-constrained settings? RQ2. How much does protocol for model selection and evaluation influence the accuracy of system outputs, and the cost and time required to generate them? RQ3. What system design choices—model family, reasoning level, context, and rubric—most influence system performance? 2. Framework Design AI-assisted generation encodes choices about the model, the prompt, and the supporting context. Evaluation encodes choices about the rubric, the evaluating model, and the degree and timing of expert involvement. Each choice may be revised in light of the outcomes of prior evaluation, producing an iterative pipeline. We formalize this pipeline for domains where quality, cost, latency, and output integrity all matter, so that each deployment cycle produces an output paired with the diagnostics needed to decide whether to release it, how to improve it, and whether a new model release warrants adoption. The framework is agnostic to the underlying system. The output may be an answer in a question-answering system, a synthesized report, a document summary, a drafted communication, or any other model-produced artifact for an end-user. It makes three commitments that distinguish it from ad hoc evaluation practice. Cost and latency are selection criteria rather than post hoc considerations, which matters where repeating an expert evaluation is itself a binding constraint. Human and machine evaluation are kept as separate signals with separate roles, so scarce expert attention can be directed where it has the most value. And every configuration choice and constraint violation is recorded on a versioned artifact, the MERCI card, issued once per iteration. We refer throughout to the deploying team: whoever is accountable for putting the system in front of users, whether that is a product owner, an engineering team, a research group, or an organization adopting a tool built elsewhere. The framework does not tell this team what to choose. It fixes the structure of the pipeline and the set of choices that structure exposes, and it requires that each choice, and the reasoning behind it, be written down where a later reader can find it. What counts as an acceptable rubric, a trustworthy reviewer panel, or a releasable score is a judgment about the domain, and the people who understand the domain are the ones positioned to make it. What the framework supplies is the obligation to make those judgments explicit rather than incidental. 2.1. Setup Task and outputs. Let T be a generation task. At each iteration τ the task is run on N inputs, producing task results 1,…,NTR_1,…,TR_N. A task result is the user-facing artifact and the unit of evaluation throughout. The inputs are assumed similar enough that one configuration is expected to serve all of them, which is what licenses averaging scores across them. Iterations. Iterations are indexed τ=0,1,2,τ=0,1,2, and so on. A configuration is the tuple (G,P,C,R,W)(G,P,C,R,W): generation model, prompt, context, rubric, and rubric weights. Every input below belongs to some iteration; we suppress the superscript (τ)(τ) for readability except where two iterations are compared. Rubric and scores. Quality is assessed along a total of D rubric dimensions R=d1,…,dDR=\d_1,…,d_D\. Each dimension is scored on an ordinal scale such as [0,1][0,1], where 11 denotes full agreement with the ideal response and 00 none. The framework fixes the range and the direction; the deploying team sets the granularity and records it on the card.11 1 Scores can be continuous, discrete or categorical (e.g., full credit mapped to 1, partial credit mapped to 0.5 and no credit mapped to 0) all resulting in outputs on a [0,1][0,1] scale. Evaluators and scores. Two evaluator types may be activated. The LLM judge J scores each task result TR a total of ℓ times, since judge scores are stochastic and their disagreements can be a diagnostic. A total of V human reviewers are recruited to score a task result once. Aggregating over task results and over runs or reviewers gives two scores per dimension: JdJ_d, the aggregate judge score on dimension d, and HdH_d, the aggregate human score on dimension d. These combine into the quality vector Q=[Q1,…,QD]Q= [\,Q_1,…,Q_D\, ] through another aggregation function the deploying team chooses and records on the card: (1) Qd=agg(Jd,Hd)Q_d=agg\! (J_d,\,H_d ) The choice of aggagg encodes an opinion on whose judgment counts, and different organizations will reasonably take different positions. A team using both evaluator types might take the mean, agg=12(Jd+Hd)agg= 12(J_d+H_d), treating the two as equally informative about the performance of the system; a weighted mean λJd+(1−λ)Hdλ J_d+(1-λ)H_d to let expert judgment dominate without discarding the judge signal. A team may choose to admit HdH_d only if they pass a reliability threshold explained in Step F, so aggagg may reduce to JdJ_d at an iteration if such a threshold is not met. Q is the feedback signal: it drives the revisions in Steps A–C, the model selection in Step D, and the release decision in Step H. Cost and latency. Mean cost c and mean latency t per task result are recorded alongside quality but are not part of Q. They enter model selection in Step D. Models. A model m is characterized by its family ℱ(m)F(m) and its reasoning level ρ(m)∈0,1,2,3,4ρ(m)∈\0,1,2,3,4\. Families expose this control under different names and native scales, so the deploying team states the mapping from a family’s settings onto 0,…,4\0,…,4\ on the MERCI card. For example, where ℱ(m)F(m) is Anthropic, the native setting takes values in none,adaptive,max\none,\ adaptive,\ max\, and the team declares which normalized level each corresponds to. The mapping is a stated assumption rather than a measurement, which is why it is recorded: a reader who disagrees with it can see exactly what was assumed and recompute accordingly. Table 1. Notation. Symbol Meaning T, nTR_n, N Task; n-th task result; number of results τ Iteration index (suppressed when unambiguous) E Domain expert input G, P, C, R, W Configuration: model, prompt, context, rubric, weights D, d Number of rubric dimensions; dimension index J, ℓ LLM judge; judge runs per task result V Number of human reviewers JdJ_d, HdH_d Mean judge and human score on dimension d aggagg, λ Aggregation of JdJ_d and HdH_d into QdQ_d; its weight, where weighted QdQ_d, Q Quality score on d; quality vector [Q1,…,QD][\,Q_1,…,Q_D\,] ϕφ, δϕ _φ Inter-rater reliability and its threshold ℳM, m Candidate model set; a candidate ℱ(m)F(m), ρ(m)ρ(m) Model family; reasoning level 00–44 cmc_m, tmt_m Mean cost and latency per task result wdw_d, wcw_c, wtw_t Dimension, cost, and latency weights BcB_c, BtB_t Cost and latency budgets, where used Qd(m)Q_d(m) Quality score for candidate m during selection S^m S_m Composite selection score δJ _J, δH _H Overall judge and human release thresholds δJ,d _J,d, δH,d _H,d Per-dimension release floors Table 2. Choices the framework leaves to the deploying team. The framework requires that each be recorded on the MERCI card with its rationale; it does not prescribe the value. Decision Symbol Step Scoring granularity per dimension — Setup Aggregation of judge and human scores aggagg, λ Setup Rubric dimensions R A Dimension weights W A Whether context is supplied, and whether it is withheld during selection C C Candidate model set ℳM D Cost and latency weights, or hard budgets wcw_c, wtw_t, BcB_c, BtB_t D Judge model, and whether to accept a constraint violation J E Judge runs per task result ℓ E Whether to use human review, and panel size V F Reliability metric and threshold ϕφ, δϕ _φ F Release thresholds and per-dimension floors δJ _J, δH _H, δ⋅,d _·,d H 2.2. Objective The pipeline seeks a configuration that maximizes quality while keeping cost and latency low, with all weights non-negative: (2) max∑d=1DG,P,C,R,WwdQd−wcc−wtt _G,P,C,R,W\;\; _d=1^Dw_d\,Q_d\;-\;w_c\,c\;-\;w_t\,t Equation (2) is not solved jointly: the search space is combinatorial, the quality term is observable only by running the pipeline, and the prompt, context, and rubric are artifacts rather than optimizable parameters. The pipeline instead improves one input at a time. The expert revises R, W, P, and C using the previous iteration’s quality vector (Steps A–C), and the model coordinate is re-optimized by an explicit search over a finite candidate set with the rest held constant (Step D). The composite score S^m S_m of Step D is Equation (2) restricted to the G coordinate, with Qd(m)Q_d(m) the same quality score measured for a single candidate. Iteration therefore produces a documented improvement record rather than a model convergence, and Step H defines what counts as good enough to stop. The weights wdw_d, wcw_c, and wtw_t are set by the deploying team and are where domain priorities enter the arithmetic. A team that treats latency as a convenience and a team that treats it as a determinant of whether the tool gets used at all will set wtw_t very differently, and both are defensible given their circumstances. Recording the weights alongside the outcome is what lets a later reader distinguish a model that won on quality from one that won on price. 2.3. Pipeline Steps A–D fix the configuration: the rubric and weights, the prompt, the context, and the generation model. Steps E–F set up evaluation, Step G runs the task and scores the results, and Step H decides whether to release. Model selection at Step D is itself an evaluation pass, using the judge of Step E and the scoring procedure of Step G applied to each candidate in turn. expert EEConfigure A–DRR, W, P, C, G=argmaxmS^mG= _m S_mSet up evaluation E–Fjudge J, reviewers, ϕ≥δϕφ≥ _φRun and score G1,…,NTR_1,…,TR_N → JdJ_d, HdH_d, QQRelease or iterate HQQ Figure 1. Framework pipeline. The expert configures the task, rubric, and generation model, evaluation is set up, the task is run and scored, and a release decision is made. The quality vector Q returns to the configuration stage. 2.4. Step A. Rubric and Weights The rubric defines the criteria against which each task result is scored, translating output quality into discrete, scoreable dimensions. At the first iteration, τ=0τ=0, it is written by domain experts; in subsequent iterations it is revised using the previous quality vector: (3) R(0)=R(E),R(τ)=R(E,Q(τ−1)) R^(0)=R(E), R^(τ)=R\! (E,\,Q^(τ-1) ) (4) W(0)=W(E),W(τ)=W(E,Q(τ−1)) W^(0)=W(E), W^(τ)=W\! (E,\,Q^(τ-1) ) Weights are revised on the same schedule and for the same reason. A dimension that scored poorly at τ−1τ-1 may be overweighted so that the next iteration’s model selection and prompt revision are pulled toward it. Weight changes are recorded on the card with their rationale, so that a selection outcome can be traced to a stated priority grounded in prior results rather than to an unexplained retuning. Dimensions themselves may be added, removed, or respecified as understanding of the task deepens (Szymanski et al., 2026b; Chiang et al., 2026; Szymanski et al., 2026a). Because a rubric change alters the measurement instrument, the deploying team should note which dimensions carried over unchanged; scores are comparable across iterations only on those. 2.5. Step B. Prompt The prompt is the primary instruction to G, specifying what the task result should contain, how it should be structured, and what constraints it must satisfy: (5) P(0)=P(E),P(τ)=P(E,Q(τ−1))P^(0)=P(E), P^(τ)=P\! (E,\,Q^(τ-1) ) Later iterations use Q(τ−1)Q^(τ-1) to direct targeted revisions at the dimensions that scored lowest. 2.6. Step C. Context The context supplies G with domain knowledge beyond the prompt: reference materials, worked examples, data constraints, or precedents. It is optional, and is revised on the same schedule: (6) C(τ)=C(E,Q(τ−1)),C∈⋅,∅C^(τ)=C\! (E,\,Q^(τ-1) ), C∈\·, \ One option during model selection (Step D) is to withhold context, setting C=∅C= across all candidates, and measure performance using zero-shot prompts. Whichever the deploying team chooses, the card records it, since the two conditions answer different questions: withheld context compares the models, supplied context compares the deployments. 2.7. Step D. Generation Model Selection The generation model is chosen from a candidate set ℳM by running the evaluation of Steps E–G for each candidate, with P, R, W held fixed and C handled as described in Step C. This gives, for each candidate m, a quality score Qd(m)Q_d(m) per dimension plus mean cost cmc_m and latency tmt_m. Each dimension is min-max normalized across candidates before being combined.22 2 x~m=(xm−minm′xm′)/(maxm′xm′−minm′xm′) x_m= (x_m- _m x_m )/ ( _m x_m - _m x_m ), applied across all m∈ℳm . The composite score and selection rule mirror Equation (2): (7) S^m=∑d=1DwdQ~d(m)−wcc~m−wtt~m S_m= _d=1^Dw_d\, Q_d(m)-w_c\, c_m-w_t\, t_m (8) G=argmaxm∈ℳS^m G= _m S_m The weights make domain priorities explicit and auditable. Raising wdw_d on a dimension that underperformed at τ−1τ-1 lets a selection decision be justified by pointing to a stated priority grounded in prior results rather than to an opaque aggregate. Where the deploying team treats cost or latency as a hard limit rather than something to trade off, the weighted form can be replaced by maximizing ∑dwdQd(m) _dw_dQ_d(m) subject to cm≤Bcc_m≤ B_c and tm≤Btt_m≤ B_t, with the budgets recorded on the card. 2.8. Step E. LLM Judge A second model is designated as the judge J. Where G produces a task result, J scores it against the rubric and weights. To limit self-preference bias, the tendency of a model to favor outputs from its own family (Spiliopoulou et al., 2025; Wataoka et al., 2025; Chen et al., 2025), the judge should satisfy: (9) ℱ(J)≠ℱ(G),ρ(J)≥ρ(G)F(J) (G), ρ(J)≥ρ(G) The first condition puts the judge in a different family from the generator. The second keeps the judge from being less equipped to catch errors than the generator was to produce them. Neither is enforceable, and a deploying team may have good reason to violate either, most often cost or availability. Judge scores are stochastic, so the judge is run ℓ times per task result and the spread across runs is reported alongside JdJ_d. A cache-busting protocol ensures repeated runs are not served from cache. 2.9. Step F. Human Review and Rubric Reliability Human reviewers may score task results in addition to or in place of J. With more than one reviewer, inter-rater reliability ϕφ determines whether their scores are trustworthy enough to use. The choice of metric follows from the scale set in Step A rather than being fixed by the framework. For a dimension scored on defined ordinal levels, ordinal Krippendorff’s α is used since it tolerates a varying number of reviewers per item. Reliability is computed per dimension and overall, and the deploying team may set an optional threshold δϕ _φ gating entry into Q: (10) Hd is admitted to agg⇔ϕ≥δϕH_d is admitted to agg φ≥ _φ Below the threshold, reviewers might be applying the rubric inconsistently. A development team may view this as evidence of rubric ambiguity rather than genuine expert disagreement, and revise the rubric before admitting human scores; per-dimension ϕφ identifies which dimensions need clarification. The opposite reading is also available and may be the right one: low agreement can mean the dimension is genuinely contested among experts, in which case reporting the dispersion is more faithful than revising it away. Which reading applies is a domain judgment, and the card records the interpretation taken alongside the figure itself. When ϕφ is undefined for a dimension because every score is identical, the development team might check for a ceiling effect. A constant score may reflect a fixed evaluator response rather than true agreement, and such a dimension carries no signal, so it is flagged for revision rather than counted as perfect reliability. 2.10. Step G. Run and Score At each iteration G runs the task on all N inputs: (11) n=G(P,C,R,Q(τ−1)),n=1,…,NTR_n=G\! (P,\,C,\,R,\,Q^(τ-1) ), n=1,…,N Each result is scored by the judge, by human reviewers, or both, and the scores are combined as described in Section 2.1 to give JdJ_d, HdH_d, and Q. Four diagnostics are reported per dimension and carried onto the card: • JdJ_d and HdH_d, with their standard deviations across judge runs and across reviewers respectively; • the judge–human gap Jd−HdJ_d-H_d; • the change from the previous iteration, ΔJd J_d and ΔHd H_d, defined only on dimensions carried unchanged from τ−1τ-1 (Step A). The gap is the quantity that directs expert effort. A dimension where the judge tracks human scores closely can be delegated to J in later iterations; a dimension with a large gap needs continued human review. Judge spread separates an unstable judge from a genuinely difficult dimension. 2.11. Step H. Release or Iterate Release requires that quality clear a threshold on every dimension, not just on average. Writing J¯=1D∑dJd J= 1D _dJ_d and H¯=1D∑dHd H= 1D _dH_d for the overall scores: (12) release⇔J¯≥δJ and H¯≥δH J≥ _J\; and \; H≥ _H andJd≥δJ,d and Hd≥δH,dfor all d J_d≥ _J,d\; and \;H_d≥ _H,d\;\;\;for all d The deploying team may choose to apply the human conditions only when HdH_d passes the Step F gate. Otherwise the pipeline advances to τ+1τ+1 and Q informs revisions to P, C, R, and W. The thresholds themselves are the team’s to set, and setting them is the point at which an abstract commitment to quality becomes a number somebody has to defend. What the framework fixes is the shape: floors apply per dimension, not only to the average. A threshold on the average alone lets a system release while failing on one dimension. Floors may differ across dimensions, and a dimension needing no floor is assigned 00 and recorded as such. Termination. The loop is not guaranteed to reach a releasable configuration. A deploying team may stop iterating for reasons the framework does not model, such as an exhausted budget or a judgment that the remaining gap will not close with further prompt and rubric revision. Stopping without release is a legitimate outcome rather than a failure of the framework, and the final card documents it: the configuration reached, which dimensions failed their floors, and the team’s diagnosis of why. Recorded that way, the decision can be revisited when a more capable candidate appears, without the reasoning having to be reconstructed from memory. 2.12. The MERCI Card Each iteration closes by issuing a MERCI ((Model-Evaluation-Rubric-Continuous-Improvement) card: a single versioned record of the configuration that was run, the scores it produced, and the decision that followed. The card is the framework’s enforcement mechanism. Section 2 leaves the substance of every configurable choice to the deploying team, so what remains to be required is that the choices be legible afterwards, to a colleague inheriting the system, to a reviewer assessing a claim made on its output, or to the team itself deciding a year later whether a new model release is worth adopting. Table 3 gives the specification. Each field is labelled by what kind of thing it is. A choice is a decision the team makes, and recording it means recording both the value and the reasoning behind it; these are the fields of Table 2. A derived field is computed from the run and is not open to interpretation at recording time. A record is a factual matter such as a date, a model identifier, or a prior card reference. Optional fields improve the card’s usefulness without being necessary to interpret the iteration. The card is written per iteration rather than per release, which means iterations that end without release generate cards too. A sequence of cards is therefore a history of what was tried, not a record of what succeeded: a configuration abandoned at τ=2τ=2 is worth knowing about when a similar one is proposed at τ=5τ=5. Table 3. MERCI Card specification. One card per iteration τ. choice fields are decisions the deploying team makes: record the value and the reasoning. derived fields are computed from the run. record fields are factual. optional fields improve the card without being necessary to interpret the iteration. Section Field Kind Identity System name, domain, iteration τ record Card version, date issued, prior card reference (τ−1τ-1) record Deploying team and accountable contact record Task T: task description and what a task result is (e.g., answer, report, summary, drafted document) record N: number of inputs the task was run on; how they were selected record Scoring granularity per dimension: continuous, discrete, or categorical, with the mapping onto [0,1][0,1] choice aggagg: how JdJ_d and HdH_d combine into QdQ_d; λ if weighted choice Rubric and Weights R=d1,…,dDR=\d_1,…,d_D\: dimensions with descriptions choice W=w1,…,wDW=\w_1,…,w_D\: dimension weights choice E: who defined R, and their domain expertise record Dimensions added, removed, or respecified since τ−1τ-1, and which carried over unchanged record Prompt P: prompt text or a reference to the versioned artifact choice Revisions since τ−1τ-1, and which dimensions of Q(τ−1)Q^(τ-1) motivated each record Context C: what was supplied, or ∅ ; provenance of each source choice Whether C was withheld during model selection, and why choice Model Selection ℳM: candidate set in full, listed individually choice Mapping from each family’s native settings onto ρ(m)∈0,…,4ρ(m)∈\0,…,4\ choice wdw_d, wcw_c, wtw_t, or budgets BcB_c, BtB_t under the constrained form choice Aggregation used for selection, where it differs from the deployed aggagg (e.g., Qd(m)=Jd(m)Q_d(m)=J_d(m)), and the share of the sweep that received human review choice Qd(m)Q_d(m), cmc_m, tmt_m for every m∈ℳm derived S^m S_m for every m∈ℳm , post-normalization; metrics on which all candidates tied and were dropped derived G=argmaxmS^mG= _m S_m, with the margin over the runner-up derived LLM Judge J: judge model, ℱ(J)F(J), ρ(J)ρ(J) choice ℓ : judge runs per task result; cache-busting protocol choice ℱ(J)≠ℱ(G)F(J) (G): satisfied or violated; if violated, the implication for the scores record ρ(J)≥ρ(G)ρ(J)≥ρ(G): satisfied or violated, with implication record JdJ_d per dimension, with standard deviation across the ℓ runs derived Human Review V: number of reviewers; expertise and recruitment method choice Scoring protocol: independence, blinding, order effects optional ϕφ: reliability metric chosen, and why it fits the scale choice δϕ _φ: threshold, if the team set one choice ϕφ per dimension and overall; dimensions where ϕφ is undefined for want of variance derived HdH_d per dimension, with inter-reviewer standard deviation derived Where ϕ<δϕφ< _φ: whether read as rubric ambiguity or as a genuinely contested dimension, and what followed choice Run and Score 1,…,NTR_1,…,TR_N: task results, or a reference to where they are stored record Q=[Q1,…,QD]Q=[\,Q_1,…,Q_D\,]: quality vector derived Jd−HdJ_d-H_d per dimension: judge–human gap derived ΔJd J_d, ΔHd H_d: change from τ−1τ-1, on carried-over dimensions only derived Release Decision δJ _J, δH _H: overall thresholds choice δJ,d _J,d, δH,d _H,d: per-dimension floors, 00 where none applies; whether the human conditions were applied at all choice Decision: release, or advance to τ+1τ+1; dimensions that failed their floors derived Rationale for the decision record Planned changes to R, W, P, C, G for τ+1τ+1; null on release or on stopping without release optional Limitations Release scope, and the human oversight required before a task result is relied on record Judge constraint violations carried from the judge section, and their implications read together record Dimensions showing ceiling effects, distinguished from dimensions with genuine agreement record Data constraints and known exclusions affecting task result quality record Where the team stopped iterating without release: the diagnosis of why the gap did not close optional 3. Example Use Cases 3.1. Redo.io Background. The California Racial Justice Act allows a person to challenge a conviction or sentence by showing racial disparity in how similar cases were charged or sentenced. At the prima facie stage the showing is statistical, which means a defender must assemble sentencing data, select appropriate tests, and present the results in a form a court will credit. Public defender offices carry heavy caseloads and rarely have statistical staff, so the effort a single filing requires is often the binding constraint on whether a claim is brought at all (Ball, 2025; Bazelon et al., 2025; Kuluk, 2024). The system. Redo.io is a nonprofit platform that produces statistical evidence reports for these filings. Disparity statistics are computed upstream by conventional code: odds ratios, relative risks, chi-square tests, with their confidence intervals, p-values, and sample sizes. The language model’s role is interpretive rather than analytical. It receives the computed figures and drafts a narrative that states what they show, where they disagree, and what the underlying data cannot support. Every report requires attorney review before it is filed. Task and task result. The task T is to synthesize a set of pre-computed statistics into an evidence narrative. A task result TR is one such report, covering one sentencing county, sentencing outcome, and ethnicity pair, with cohort controls on prior record. Reports are the unit of evaluation throughout. 3.2. Instantiating the Framework We report the second iteration, τ=1τ=1. A first iteration established the rubric from expert input alone, without model selection or human review, and produced the quality vector Q(0)Q^(0) that drove the revisions below. Appendices B, C, and J give the prompt, rubric, and reliability analysis; source materials are at https://github.com/redoio/ai_evals. Rubric and weights. The rubric has D=9D=9 dimensions (Table 4): six on correct statistical contextualization, two on stating what the data cannot support, and one, Attribution, on whether disparities are attributed to systemic factors rather than to the propensity of any racial group (Angwin et al., 2016; Rudin, 2020; Rudin, 2019; Miron et al., 2020). All nine carried over unchanged from τ=0τ=0, so scores are comparable across iterations. Weights did not: cross-method comparison was weakest at τ=0τ=0, so we set wCmp=5w_Cmp=5, with wd=1w_d=1 otherwise and wc=wt=1w_c=w_t=1. Table 4. Examples of evaluation rubric dimensions Dimension Tests whether the report OR CIscore contextualizes odds ratios with CI width and whether the interval spans 1 R CIscore does the same for relative risk OR Smpscore contextualizes odds ratios with sample size R Smpscore does the same for relative risk χ2χ^2 Smpscore does the same for chi-square ppscore contextualizes p-values against the significance threshold Cmpscore compares the statistical methods and reconciles methods that diverge Limscore states dataset limitations and their effect on the findings Attscore attributes disparities to systemic factors, not inherent criminality Prompt and context. The prompt directs G to interpret the supplied figures across the three statistical methods, surface disagreement between them, and attribute any disparity to systemic factors; it states explicitly that G must perform no computation. The context supplies excerpts from prior legal filings and documentation of known gaps in the underlying sentencing dataset. Model selection. We evaluated |ℳ|=11|M|=11 configurations over N=18N=18 reports at ℓ=15 =15 judge runs each, judge scores only, context withheld. Table 5 declares the mapping from each family’s native setting onto ρ∈0,…,4ρ∈\0,…,4\, so a reader who would map them differently can see what was assumed. The selected configuration is Opus at ρ(G)=3ρ(G)=3, S^G=11.94 S_G=11.94. Judge and human review. The judge is GPT at ρ(J)=3ρ(J)=3, satisfying both conditions. It was fixed across the sweep while ℳM spans two families, so three of eleven candidates were scored within family and eight were not. Each report is scored ℓ=15 =15 times with a cache-busting protocol.33 3 Each request carries a fresh UUID for user and prompt_cache_key, routing repeated requests to different cache buckets. Three domain experts then scored the reports independently. We set no δϕ _φ in advance. Table 5. Model selection sweep: |ℳ|=11|M|=11 configurations over N=18N=18 reports and ℓ=15 =15 judge runs. Native settings appear with the normalized reasoning level ρ the team declared for them. JdJ_d is the mean judge score across the nine dimensions; S^m S_m is the composite including cost and latency (wCmp=5w_Cmp=5, wd=1w_d=1 otherwise, wc=wt=1w_c=w_t=1). Sorted by JdJ_d. Highlighted row is the selected G. Generation model Judge ℱ(G)F(G) native ρ ℱ(J)F(J) ρ JdJ_d S^m S_m t (s) c ($) Opus none 0 GPT 3 0.92 11.60 40 0.06 Opus adapt / med 2 GPT 3 0.92 10.92 53 0.08 Opus adapt / high 3 GPT 3 0.92 11.94 55 0.08 Opus adapt / max 4 GPT 3 0.91 10.31 96 0.16 Sonnet adapt / high 3 GPT 3 0.90 9.91 180 0.21 Sonnet adapt / med 2 GPT 3 0.90 9.10 166 0.19 Sonnet adapt / max 4 GPT 3 0.88 8.48 274 0.34 Sonnet none 0 GPT 3 0.85 8.19 38 0.04 GPT high 3 Opus 4 0.82 8.12 96 0.09 GPT med 2 Opus 3 0.81 8.63 54 0.05 GPT none 0 Opus 3 0.67 1.52 25 0.02 Table 6. Scores per rubric dimension at τ=1τ=1 (N=30N=30, ℓ=15 =15, V=3V=3), with the change from the prior iteration. Judge sd is the mean standard deviation across the ℓ runs per report; human sd is the inter-reviewer standard deviation. Judge Human Δ from τ=0τ=0 Dimension JdJ_d sd HdH_d sd ΔJd J_d ΔHd H_d OR CIscore 0.94 0.03 0.89 0.16 ++0.15 ++0.09 OR Smpscore 1.00 0.00 0.80 0.13 ++0.51 ++0.19 R CIscore 0.86 0.04 0.87 0.18 ++0.10 ++0.07 R Smpscore 1.00 0.00 0.80 0.13 ++0.51 ++0.19 Cmpscore 0.74 0.01 0.76 0.35 ++0.05 −-0.02 χ2χ^2 Smpscore 1.00 0.02 0.80 0.14 ++0.50 ++0.21 ppscore 1.00 0.00 0.99 0.01 ++0.11 ++0.15 Limscore 0.75 0.00 0.96 0.07 ++0.00 ++0.02 Attscore 0.96 0.02 1.00 0.00 −-0.04 ++0.00 Overall 0.92 0.01 0.88 0.13 ++0.21 ++0.10 3.3. The Resulting MERCI Card Table 7 is the card for this iteration: not a claim that the system works, but a statement of what was configured, what it scored, what was wrong with the measurement, and what followed. Two entries do work the score tables cannot. The fivefold weight sits beside the modest gain it produced, making the intervention assessable rather than merely reported. And the reliability figure sits beside the reading taken from it, so a later reader can disagree with the reading without re-deriving the number. The card records no release. We set no per-dimension floors in advance, which is itself a finding: the framework asks for them before the scores arrive, precisely so a threshold cannot be chosen to fit a result. Under a uniform 0.800.80 floor this configuration would have failed on cross-method comparison and on limitations. Reporting that is more useful than reporting the 0.920.92 average alone, and it is what the per-dimension rule exists to surface. Table 7. MERCI Card for τ=1τ=1. Section Field Entry Identity Prior card τ=0τ=0 N inputs 30 reports; 18 in the selection sweep Task Granularity any value in the range [0,1][0,1] aggagg Qd=12(Jd+Hd)Q_d= 12(J_d+H_d) for deployment evaluation, i.e. both humans and LLM judges are used; Qd=JdQ_d=J_d for model selection, i.e. no humans are used Rubric R R(E,Q(0))R(E,Q^(0)), D=9D=9 W wCmp=5w_Cmp=5, wd=1w_d=1 otherwise, wc=wt=1w_c=w_t=1 Rationale Cmp was weakest at τ=0τ=0 (Jd=0.69J_d=0.69) so additional weights applied Prompt, Context P, C P(E,Q(0))P(E,Q^(0)); C(E,Q(0))C(E,Q^(0)), context on data limitations and comparison for statistical methods expanded since these were lowest-scoring dimensions in τ=0τ=0 In selection context withheld, C=∅C= Model Selection ℳM 11 configurations, listed in Table 5 ρ mapping declared in Table 5 G Opus, native adapt / high, ρ(G)=3ρ(G)=3 S^G S_G 11.9411.94 out of 1414, the best possible score LLM Judge J, ℓ GPT, ρ(J)=3ρ(J)=3; ℓ=15 =15; cache-busting per run Constraints ℱ(J)≠ℱ(G)F(J) (G) satisfied; ρ(J)≥ρ(G)ρ(J)≥ρ(G) satisfied (3≥33≥ 3) Human Review V, ϕφ 3 expert reviewers; ordinal Krippendorff’s α δϕ _φ not set in advance Result and reading α=0.08α=0.08 overall, negative for 2 of 3 pairs; read as rubric ambiguity, calibration required before τ=2τ=2 Run and Score J¯ J, H¯ H 0.920.92; 0.880.88 (Δ from τ=0τ=0: +0.21+0.21, +0.10+0.10 respectively) Gap Jd−HdJ_d-H_d +0.04+0.04 overall; −0.21-0.21 on Limitations, −0.20-0.20 on the sample size dimensions Weakest d Cmp (0.740.74), Limitations (0.750.75) Release Floors δ⋅,d _·,d not set in advance Decision iterate Rationale Cmp and Limitations dimensions perform poorly; human reliability unusable due to poor agreement Limitations Ceiling dimensions 4 of 9 show zero judge variance across 450 evaluations Measurement human scores not admitted as a reliable signal this iteration Oversight required attorney review before any report is filed 3.4. Framework Scope and Transferability The framework presented here is domain-agnostic. The core requirements for adoption are a rubric definable and refinable by domain experts, a generation task for which LLM judges can approximate human judgment, and a cost and latency budget that makes exhaustive human evaluation impractical at every iteration. The framework is particularly well-suited to under-resourced legal or related settings where the weighted composite scores makes cost and time selection criteria rather than afterthoughts. Our findings point to two transferable insights. First, the separation of roles between LLM judges as a scalable, reproducible evaluation instrument and human evaluators as a signal allows practitioners to concentrate human effort on dimensions where it has the highest marginal value—specifically those with high inter-rater spread or large human-machine divergence. Second, the weight vector provides an explicit, auditable mechanism for encoding domain priorities: model selection decisions can be justified by pointing to weight choices grounded in prior evaluation outcomes rather than opaque aggregate scores. In the Redo.io case study, elevating wCmpw_Cmp at τ=1τ=1 directly reflected a practitioner judgment derived from τ=0τ=0, demonstrating that the MERCI Cards framework can support evidence-based configuration decisions rather than arbitrary tuning. More broadly, the pace of general-purpose model releases means that any deployed pipeline faces continuous pressure to re-evaluate whether the current generation task remains optimal. The iterative structure of the framework anchored by the feedback loop provides a stable procedure within which new models can be tested and integrated without redesigning the evaluation pipeline from scratch. This property is especially valuable in high-stakes settings, where the frontier of model capabilities can translate directly into improved outcomes for those the system is designed to serve but resources for end-to-end evaluation are severely constrained. 4. Limitations of MERCI Cards While MERCI Cards provide a structured artifact for documenting iterative LLM deployment, three limitations warrant acknowledgment. First, the card is designed for developers and researchers familiar with the underlying framework: many fields in the card require fluency with the mathematical notation and pipeline design to interpret correctly. Practitioners without a technical background (including the attorneys and advocates who are the intended beneficiaries of systems like Redo.io) cannot readily read a MERCI Card to assess whether a system is ready for use. A practitioner-facing summary layer, analogous to the "intended use" and "out-of-scope use" sections of model cards, would need to accompany the card for it to serve a broader audience (Mitchell et al., 2019). Second, the card records what happened at each iteration but cannot establish why. A low inter-rater reliability score may reflect ambiguous rubric language, insufficient reviewer training, genuine domain disagreement among experts, or reviewers with varying levels of expertise—and the card does not distinguish between these explanations. Similarly, a gain in ΔL L across iterations may be attributable to the generation model upgrade, the addition of context C(τ)C^(τ), or rubric revision, or all three simultaneously. Without a controlled study, the card captures correlation across iterations but not causal attribution. Third, the card structure reflects the priorities of the development team at the time of its design, which may shift as the system matures. Rubric dimensions that are critical early in deployment may become less informative as performance ceilings are reached, while new dimensions reflecting evolving legal standards or practitioner needs may need to be introduced. The card must therefore be treated as a living document, versioned alongside the pipeline it documents, rather than a fixed specification applied uniformly across iterations. 5. Conclusion We present a mathematical framework for iterative LLM integration in high-stakes, resource-constrained domains, formalizing model selection, rubric design, and evaluation feedback as a principled loop in which outcomes drive configuration decisions at each iteration. Applied to RJA evidence synthesis on the Redo.io platform, the framework yields substantial performance improvements across iterations. References Angwin et al. (2016) J. Angwin, J. Larson, S. Mattu, and L. Kirchner Machine bias: there’s software used across the country to predict future criminals. and it’s biased against blacks.. ProPublica. External Links: Link Cited by: §3.2. Ball (2025) W. D. Ball Modeling meaning: causal inference under the california racial justice act. Santa Clara Law Review 65 (1). External Links: Link Cited by: §3.1. Bavaresco et al. (2025) A. Bavaresco, R. Bernardi, L. Bertolazzi, and D. Elliott LLMs instead of human judges? a large scale empirical study across 20 NLP evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Vienna, Austria, p. 238–255. External Links: Link, Document, ISBN 979-8-89176-252-7 Cited by: §1. Bazelon et al. (2025) L. Bazelon, B. Redbird, and B. Yan Ground Rules: Giving Meaning and Effect to Key Contested Terms in the California Racial Justice Act. Santa Clara Law Review 65 (1). External Links: Link Cited by: §3.1. Chappidi and Singh (2026) S. Chappidi and J. Singh To build or not to build? Factors that lead to non-development or abandonment of AI systems. In Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’26, Montreal, QC, Canada, p. 1–25. External Links: Document, Link Cited by: §1. Chen et al. (2024) G. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang Humans or LLMs as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 8301–8327. External Links: Link, Document Cited by: §1. Chen et al. (2025) Z. Chen, H. Wang, X. Zhang, E. Hu, and Y. Lin Beyond the surface: measuring self-preference in LLM judgments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 1653–1672. Cited by: §2.8. Chiang et al. (2026) C. Chiang, S. A. Gebreegziabher, A. Szymanski, Y. Yang, H. J. Do, Z. Ashktorab, W. Geyer, T. J. Li, and D. Gómez-Zará MultEval: collaboratively creating criteria for llm-as-a-judge systems. In Companion Proceedings of the 31st International Conference on Intelligent User Interfaces, IUI ’26 Companion, New York, NY, USA, p. 180–184. External Links: ISBN 9798400719851, Link, Document Cited by: §2.4. Gu and Jiang (2024) J. Gu and X. Jiang A survey on llm-as-a-judge. arXiv preprint. External Links: Link Cited by: §1. Kuluk (2024) E. Kuluk Disparate Racial Impact of Discretionary Prosecutorial Charging Decisions in Gang-Related Murder Cases: Litigating the Racial Justice Act in People v. Windom. Berkeley Journal of Criminal Law 29 (1), p. 71–80. External Links: Document, Link Cited by: §3.1. Miron et al. (2020) M. Miron, S. Tolan, E. Gomez, and C. Castillo Evaluating causes of algorithmic bias in juvenile criminal recidivism. Artificial Intelligence and Law, p. 111–147. External Links: Document, Link Cited by: §3.2. Mitchell et al. (2019) M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, New York, NY, USA, p. 220–229. External Links: ISBN 9781450361255, Link, Document Cited by: §1, §4. Rudin (2019) C. Rudin Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1 (5), p. 206–215. External Links: Document, Link Cited by: §3.2. Rudin (2020) C. Rudin The age of secrecy and unfairness in recidivism prediction. Harvard Data Science Review 2 (1). External Links: Document, Link Cited by: §3.2. Shankar et al. (2024) S. Shankar, J.D. Zamfirescu-Pereira, B. Hartmann, A. Parameswaran, and I. Arawjo Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, UIST ’24, New York, NY, USA. External Links: ISBN 9798400706288, Link, Document Cited by: §1. Spiliopoulou et al. (2025) E. Spiliopoulou, R. Fogliato, H. Burnsky, T. Soliman, J. Ma, G. Horwood, and M. Ballesteros Play favorites: a statistical method to measure self-bias in llm-as-a-judge. External Links: 2508.06709, Link Cited by: §1, §2.8. Szymanski et al. (2026a) A. Szymanski, O. Anuyah, T. J. Li, and R. Metoyer Key considerations for domain expert involvement in llm design and evaluation: an ethnographic study. In Proceedings of the 31st International Conference on Intelligent User Interfaces, IUI ’26, New York, NY, USA, p. 617–630. External Links: ISBN 9798400719844, Link, Document Cited by: §2.4. Szymanski et al. (2026b) A. Szymanski, S. Araya Gebreegziabher, O. Anuyah, R. A. Metoyer, and T. Jia-Jun Li Designing staged evaluation workflows for llms: integrating domain experts, lay users, and model-generated evaluation criteria. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, CHI ’26, New York, NY, USA. External Links: ISBN 9798400722783, Link, Document Cited by: §2.4. Wataoka et al. (2025) K. Wataoka, T. Takahashi, and R. Ri Self-preference bias in llm-as-a-judge. External Links: 2410.21819, Link Cited by: §1, §2.8. Yancey et al. (2023) K. P. Yancey, G. Laflair, A. Verardi, and J. Burstein Rating short L2 essays on the CEFR scale with GPT-4. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Toronto, Canada, p. 576–584. External Links: Link, Document Cited by: §1. Appendix A GenAI Use Disclosure Authors leveraged Claude and GPT to assist with generating LaTeX code for tables, and grammar and cohesion checks. Appendix B Example of AI-Powered Evidence Report Example of a report generated using https://app.getredo.io/bias_analysis: Executive Summary The sentencing outcome analyzed is sentence type "DSL" and the comparison is between White and Hispanic individuals. There is evidence of a sentencing disparity that disproportionately results in Hispanic individuals receiving DSL; the disparity is moderate in magnitude and warrants further scrutiny of systemic sentencing processes. Findings What statistical methods were used? The odds ratio was used to compare the odds that White individuals receive DSL relative to Hispanic individuals; it expresses the ratio of odds of the outcome across the two groups. The relative risk was used to compare the probability (risk) of receiving DSL for White individuals relative to Hispanic individuals; it provides an intuitive measure of how much more or less likely the outcome is in one group versus the other. The chi-square test of independence was used to evaluate whether an association exists between ethnic group and sentencing outcome, testing whether the observed distribution of cases across groups departs from what would be expected under independence. What meaning and trends did we find? The odds ratio of 0.521 with 95% confidence interval (0.364, 0.745) indicates that the odds of White individuals receiving DSL are lower than the odds for Hispanic individuals, and the confidence interval excludes 1.0, indicating statistical significance at the conventional 0.05 level. The relative risk of 0.69 with 95% confidence interval (0.566, 0.857) indicates that the risk of receiving DSL is lower for White individuals compared to Hispanic individuals, and the confidence interval excludes 1.0, indicating statistical significance. The chi-square test produced a statistic of 12.26 with p-value 0.0005, providing strong evidence of an association between ethnic group and the DSL sentencing outcome that is unlikely to be due to chance. Taken together, the three methods are concordant: they show a statistically significant disparity in DSL sentencing that disfavors Hispanic individuals; the odds ratio shows a larger numerical contrast than the relative risk (a pattern expected when the outcome is relatively common), and all measures point to a consistent direction and statistically significant magnitude of disparity in sentencing outcomes. What ethical considerations are necessary when interpreting this result? Observed disparities in sentencing outcomes should be interpreted as evidence of differences in how sentences are imposed rather than as evidence of inherent differences in criminal propensity among ethnic groups. Explanations for disparities should emphasize systemic factors, such as prosecutorial discretion, implicit bias, differential charging practices, and structural inequalities, rather than characteristics of the groups themselves. Care must be taken to avoid attributing causation to race without further case-level analysis and to ensure that interpretations focus on sentencing disparities alone. Analytical Constraints and Considerations What is the input dataset unable to tell us and why? The input dataset does not by itself explain case-level drivers of sentencing decisions such as offense severity, plea bargaining details, prosecution charging rationales, judge-level discretion, or validated measures of prior criminal history; absence of these covariates limits ability to isolate whether observed disparities are attributable to race or to other case-specific factors. The data are anonymized and may lack certain variables excluded under disclosure rules, which prevents follow-up verification of individual case contexts and constrains the ability to control for potentially confounding factors. Statistical association observed in this aggregated data cannot establish causal attribution and may be influenced by measurement error, missing information, or systemic processes earlier in the criminal legal pipeline. Key Terms and Methodology The odds ratio is a ratio comparing the odds of an event occurring in one group to the odds in another group; it is commonly used to quantify relative differences in odds between two groups. Relative risk is the ratio of the probability of an event in one group to the probability in another group and conveys how much more or less likely the event is in one group compared with the other. The chi-square test is a statistical test that assesses whether two categorical variables are associated by comparing observed cell counts to expected counts under independence. A confidence interval is a range of values, derived from the data, that is believed to contain the true population parameter with a specified level of confidence. A p-value is the probability of observing data as extreme as those observed, or more so, if the null hypothesis of no association were true. The balance ratio is a descriptive measure indicating the relative sample sizes or proportions between comparison groups and is used to assess whether groups are similar in size for comparative analysis. Read the full report here: https://github.com/redoio/ai_evals/tree/main/report_001