Paper deep dive
Failure-Centered Runtime Evaluation for Deployed Trilingual Public-Space Agents
M. Meng
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 6/21/2026, 7:45:45 AM
Summary
The paper introduces PSA-Eval, a failure-centered runtime evaluation framework designed for deployed trilingual (Mandarin, Cantonese, English) public-space agents. Unlike traditional score-oriented evaluation that treats LLM interactions as static input-output mappings, PSA-Eval treats the agent as a dynamic runtime system. It shifts the basic unit of analysis from a single score to a 'failure case' to better capture runtime phenomena like cross-language policy drift, version regression, and boundary violations. The framework uses trilingual equivalent input groups as controlled probes to detect score drift across languages. A pilot study on a digital front-desk system in a financial institution demonstrated that while aggregate scores remained high, significant cross-language score drift occurred, highlighting the necessity of failure-centered analysis for institutional governance.
Entities (8)
Relation Signals (4)
PSA-Eval → evaluates → Public-Space Agent
confidence 100% · This paper presents PSA-Eval, a failure-centered runtime evaluation framework for deployed trilingual public-space agents.
PSA-Eval → uses → Trilingual Equivalent Input
confidence 100% · The framework uses trilingual equivalent inputs as controlled probes for observing group-level cross-language policy drift.
Score Drift → detectsissuesin → Public-Space Agent
confidence 90% · The framework uses trilingual equivalent inputs as controlled probes for observing group-level cross-language policy drift.
Auto-Judge → performstriagefor → PSA-Eval
confidence 90% · The Auto-Judge produces structured scores, rankings, and brief reasons, while high-risk... samples must enter human review.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This paper presents PSA-Eval, a failure-centered runtime evaluation framework for deployed trilingual public-space agents. The central claim is that, when the evaluation object shifts from a static input-output mapping to a runtime system, the basic unit of analysis should shift from score to failure. PSA-Eval extends the conventional chain Question -> Answer -> Score -> End into Question -> Batch -> Run -> Score -> Failure Case -> Repair -> Regression Batch, making failures traceable, reviewable, repairable, and regression-testable. The framework uses trilingual equivalent inputs as controlled probes for observing group-level cross-language policy drift. We conduct a pilot study on a real trilingual digital front-desk system deployed in the lobby of an international financial institution. The pilot uses a simplified single-foundation-model setting (MA = MB), so the observed drift should not be interpreted as an A/B foundation-model difference. The study contains 81 samples organized into 27 trilingual equivalent question groups. Although the system achieves an average score of 23.15/24, 14 groups show non-zero cross-language score drift, 5 groups show drift of at least 3 points, and the maximum drift reaches 9 points. These results provide initial evidence that failure-centered runtime evaluation can expose structured deployment signals hidden by aggregate scoring.
Tags
Links
- Source: https://arxiv.org/abs/2604.23990v1
- Canonical: https://arxiv.org/abs/2604.23990v1
Trouble viewing inline? Open PDF directly →
Full Text
64,007 characters extracted from source content.
Expand or collapse full text
Failure-Centered Runtime Evaluation for Deployed Trilingual Public-Space Agents M. Meng Shenzhen The Nine’s Light Technology Co., Ltd., Shenzhen, Guangdong, China m@9zzg.com https://9zzg.com Abstract This paper presents PSA-Eval, a failure-centered runtime evaluation framework for deployed trilingual public-space agents. Public-space agents are no longer private chat systems: their responses may be displayed through screens, voice, or digital-human interfaces and may be interpreted as institutional behavior. In this setting, evaluation must cover not only answer quality, but also institutional boundaries, service responsibility, public broadcast suitability, cross-language policy consistency, and failure recurrence across versions. The central claim of this paper is that, when the evaluation object shifts from a static input-output mapping to a runtime system, the basic unit of analysis should shift from score to failure. A score measures pointwise quality on a single output, whereas a failure captures a trajectory-level event in which the runtime system enters an unacceptable region under a concrete deployment condition. PSA-Eval therefore extends the conventional chain Question -> Answer -> Score -> End into Question -> Batch -> Run -> Score -> Failure Case -> Repair -> Regression Batch, making failures traceable, reviewable, repairable, and regression-testable. The framework uses trilingual equivalent inputs as controlled probes for observing group-level cross-language policy drift. We conduct a pilot study on a real trilingual digital front-desk system deployed in the lobby of an international financial institution. The pilot uses a simplified single-foundation-model setting (MA=MBM_A=M_B), so the observed cross-language score drift should not be interpreted as an A/B foundation-model difference. The study contains 81 samples organized into 27 trilingual equivalent question groups across Mandarin, Cantonese, and English. Although the system achieves a high overall average score of 23.15/24, 14 groups show non-zero cross-language score drift, 5 groups show drift of at least 3 points, and the maximum drift reaches 9 points. Some low-scoring samples are also labeled as excellent by the Auto-Judge risk classifier. These results are not final quantitative conclusions about PSA-Eval’s effectiveness. They provide initial empirical evidence that failure-centered runtime evaluation can expose structured deployment signals that are hidden by one-time aggregate scoring. Keywords: runtime evaluation; failure-driven evaluation; public-space agents; trilingual evaluation; cross-language policy drift; Auto-Judge; human review; evaluation object; unit of analysis 1 Introduction 1.1 A Shift in Deployment Context LLM agents are increasingly moving from private interfaces into public service environments. Early systems mostly ran on personal computers, phones, or web chat boxes, where interaction between the user and the model was one-to-one, private, and repeatedly correctable. Digital humans, voice interfaces, intelligent terminals, and physical-space service systems change this setting. Agents are now being placed in bank lobbies, government service halls, airport counters, hospital triage desks, and exhibition guide terminals. In these environments, an agent is no longer only an answer component embedded in a software workflow. It becomes a visible node in an institutional service process. This deployment shift introduces four constraints that are less central in private chat settings. Outputs are publicly visible; answers may be interpreted as institutional communication rather than personal advice; the system must remain consistent across languages; and it must keep running while prompts, templates, models, and policies evolve. These constraints change the evaluation problem. Deployment teams need to know not only whether an answer is good, but also where failures come from, which failures require review, whether repairs work, whether language paths drift, and whether old failures return after a version change. 1.2 Limits of Score-Oriented Evaluation Mainstream evaluation paradigms are still organized around static questions, single answers, and aggregate scores. This organization implicitly treats the evaluation object as a static mapping, f(x)=yf(x)=y, and defines evaluation as estimating the average quality of that mapping on a fixed question set. It is suitable for comparing model capabilities, but it does not express the runtime risks of deployed systems. The core problem is not that there are too few score metrics. The problem is that the evaluation unit is often misspecified. As long as the evaluation unit remains a single input-output mapping, it is difficult to express runtime phenomena such as cross-language policy drift, failure recurrence, repair status, and version regression. The pilot data show this tension. The overall average score is 23.15/24, which appears close to stable. Yet 14 of the 27 trilingual equivalent question groups show non-zero cross-language score drift, 5 groups show drift of at least 3 points, and the maximum drift reaches 9 points. Some low-scoring samples are still labeled as excellent by the Auto-Judge risk classifier. This means that an average score can describe overall performance while hiding the structure of deployment risk. 1.3 Failure as the Basic Unit of Analysis When the evaluation object shifts from a static mapping to a runtime system, the basic unit of analysis should shift from score to failure. A score measures pointwise quality on an input-output mapping. A failure measures an event in which the system enters an unacceptable region during state transition. The former is a function estimation problem, while the latter is a system behavior constraint problem. PSA-Eval is a runtime evaluation framework designed around this difference. It extends the conventional evaluation chain from Question -> Answer -> Score -> End to Question -> Batch -> Run -> Score -> Failure Case -> Repair -> Regression Batch, and uses trilingual equivalent inputs as controlled probes for observing candidate signals of cross-language policy drift. 1.4 Contributions This paper makes four contributions. 1. It reframes evaluation for deployed multilingual agents at the evaluation-object level. Scores are treated as pointwise measures of output quality, while failures are treated as runtime events that can capture boundary violations, recurrence, and regression. 2. It introduces trilingual equivalent inputs as controlled probes. Mandarin, Cantonese, and English inputs are organized into groups that share the same intent, trigger point, and boundary category, so that candidate signals of group-level cross-language policy drift can be observed. 3. It proposes PSA-Eval, a failure-centered runtime evaluation loop that extends static scoring into Question -> Batch -> Run -> Score -> Failure Case -> Repair -> Regression Batch. The loop connects batch execution, automatic triage, human review, failure persistence, and regression retesting. 4. It reports a pilot study on a real trilingual digital front-desk system deployed in the lobby of an international financial institution. Using 81 samples and 27 trilingual equivalent question groups, the study compares PSA-Eval with a static evaluation pipeline on the same raw outputs and reports cross-language score drift, weak failure clustering, D7 saturation, and Auto-Judge calibration problems. These findings are positioned as pilot-level evidence of risk visibility and governance readiness, not as proof that PSA-Eval already reduces deployment risk. 1.5 Paper Organization Section 2 discusses related work. Section 3 presents the core position and formalization. Section 4 defines deployed systems, trilingual equivalent inputs, score drift, and the static baseline. Section 5 describes the PSA-Eval framework and the failure loop protocol. Section 6 describes the deployment context and the 2+1 model strategy. Section 7 describes the system implementation. Section 8 reports the pilot study. Section 9 discusses the findings and proposes falsifiable predictions. Section 10 analyzes threats to validity. Section 11 presents limitations and future work. Section 12 concludes the paper. 2 Related Work 2.1 General LLM Evaluation General LLM evaluation usually focuses on overall model capability across tasks, scenarios, and metrics. HELM [5], MMLU [2], TruthfulQA [6], BIG-Bench [11], and LiveBench [12] provide important foundations for model comparison and benchmark updates. However, their main object is the offline foundation model, rather than a runtime system jointly formed by models, prompts, templates, gateways, language paths, human review, and deployment context. This paper does not ask which model is generally stronger. It asks whether a public-space system can continuously discover, review, and repair failures in its deployment context. 2.2 Agents and Interactive Evaluation AgentBench [7] and AgentBoard [8] place LLMs in more complex interactive task environments and evaluate their reasoning, decision-making, and execution ability as agents. This paper follows their concern with the agent as a system, but extends the evaluation object from task completion to deployment boundary protection. Failures of public-space front-desk agents do not always appear as task failures. More often, the agent makes commitments it should not make, takes positions it should not take, or produces content unsuitable for public display. 2.3 Safety Evaluation, Red Teaming, and Harmful Outputs Safety evaluation and red teaming study model behavior under attack prompts, prohibited content, and harmful outputs. This paper shares with red-teaming work [10] the idea of actively searching for failures, but its target is different. It does not broadly search for all harmful outputs. It focuses on three boundary types in public-space deployment: policy boundaries, service boundaries, and public broadcast boundaries. In this setting, many failures are not explicitly prohibited content. They are subtle deviations in institutional identity, service commitment, and public broadcast suitability. 2.4 LLM-as-a-Judge and Human Review LLM-as-a-Judge can reduce the labor cost of evaluating open-ended answers. MT-Bench and Chatbot Arena [13] show that strong models can approximate human preference judgments under certain conditions. However, model judges suffer from format dependence, position bias, scenario transfer problems, and risk-level calibration issues [1, 4]. Therefore, this paper does not treat the Auto-Judge as final authority. It limits the Auto-Judge to a triage layer. The Auto-Judge produces structured scores, rankings, and brief reasons, while high-risk, low-scoring, uncertain, or cross-language drift samples must enter human review. 2.5 Multilingual Fairness and Cross-Language Evaluation Existing multilingual evaluation and bias evaluation mainly study capability gaps, fairness differences, and bias triggers across languages, cultural contexts, or demographic attributes. These works provide important foundations for understanding the cross-language generalization ability and risk distribution of LLMs, including BBQ bias evaluation [9] and the BenchMAX multilingual evaluation suite [3]. The question studied here is not translation consistency, cross-language capability ranking, or bias trigger rate in the general sense. It is cross-language policy consistency in deployed public-space agents. For institutional public spaces such as bank lobbies, government service halls, and airport counters, an answer to the same intent can vary in style across Mandarin, Cantonese, and English. It should not undergo unacceptable drift in institutional position, service commitment, risk boundary, or public broadcast suitability. Therefore, this paper does not use word-level semantic identity or text similarity as the main criterion for cross-language consistency. Instead, it uses trilingual equivalent inputs that share the same intent, risk trigger, and boundary category as controlled probes for observing whether a system produces policy-level drift across language paths. This problem is closer to boundary consistency in deployment governance than to conventional multilingual capability evaluation. 2.6 Failure-Driven Testing and Software Engineering Analogy Bug-driven regression testing in software engineering and adversarial probing in machine learning are related to this paper in that they both treat failures as basic units. This paper inherits the idea of turning failure cases into regression cases, but adds cross-language attribution, human review priority, and deployment boundary constraints. PSA-Eval is not the same as conventional regression testing, because its object is not an enumerable set of code paths. It is the behavior distribution of an agent in an open environment. 2.7 Positioning of This Paper Existing direction Main target Question added by this paper General benchmarks Cross-model capability comparison Lack of deployment regression and failure loops Agent evaluation Interactive task capability Lack of public-space and institutional boundaries Safety evaluation / red teaming Discovery of harmful outputs Lack of service responsibility and broadcast suitability LLM-as-a-Judge Automatic scoring efficiency High-risk settings still require human review Multilingual and fairness evaluation Cross-language capability, fairness, or bias risk Limited modeling of cross-language policy drift in deployed public-space agents Regression testing / adversarial probes Stability of code or model behavior Lack of discussion at the level of evaluation-object definition Table 1: Positioning of PSA-Eval relative to common evaluation directions. 3 Core Position: From Static Mapping to Dynamic Trajectory 3.1 Score: Static Function Estimation on a Fixed Question Set In a score-oriented paradigm, the evaluation object is usually modeled as a static function: y y =f(x), =f(x), score =m(x,y). =m(x,y). Here, x is the input, y is the output, and m is the scoring function. If the question set is representative, model behavior is approximately stable, and pointwise scoring reliably reflects the target capability, this paradigm is suitable for offline model comparison. Its core outputs are aggregate scores, such as average scores, leaderboards, or dimension-wise scores. More formally, score-oriented evaluation can be written as: S(f,X)=1|X|∑x∈Xℓ(f(x),y∗(x)).S(f,X)= 1|X| _x∈ X \! (f(x),y^*(x) ). Here, X is a fixed question set, ℓ is a pointwise loss or scoring function, and y∗(x)y^*(x) is a reference answer or reference preference. This definition compresses the evaluation object into static mapping quality on a fixed question set. 3.2 Failure: Trajectory Deviation in State Space A deployed agent is no longer only a static function. It is closer to a runtime system that evolves in a state space: st+1 s_t+1 =T(st,xt,ct), =T(s_t,x_t,c_t), yt y_t =O(st,xt,ct). =O(s_t,x_t,c_t). Here, sts_t denotes system state, xtx_t denotes the current input, and ctc_t denotes context such as language, model configuration, prompt version, policy layer, gateway state, and deployment scenario. A question triggers a runtime trajectory, not only a pointwise output. This paper defines failure as an event in which the system trajectory enters an unacceptable region: failure(τ)=[∃t:ϕ(st,yt)=1].failure(τ)=1\! [∃ t:φ(s_t,y_t)=1 ]. Here, ϕφ is a boundary violation predicate that determines whether a state and output cross a deployment boundary. A failure is not a scalar. It is an event structure that must answer what input triggered it, in which language it occurred, which boundary it violated, how the Auto-Judge assessed it, whether human review confirmed it, and whether it recurred after repair. 3.3 Non-Substitutability The difference between score and failure cannot be bridged simply by adding more dimensions to a score. A score is a pointwise mapping-quality measure. A failure is a runtime event judgment. The two objects answer different questions. A score asks how good one output is under a scoring rule. A failure asks whether a deployed system entered an unacceptable region under a concrete runtime condition, and whether that event can be traced, reviewed, repaired, and tested again. This creates a mismatch between the evaluation object and the evaluation unit. Static benchmarks usually assume that the object can be approximated as a fixed function. Under the same question set and scoring function, output quality can be estimated by a set of pointwise scores. Deployed agents, however, change with model versions, prompt templates, gateway routing, language paths, policy-layer configuration, context windows, and human repair history. The evaluation object is no longer a fixed f(x)f(x), but a family of trajectory generators that vary over time and configuration. Forcing failure into a pointwise total score leads to two problems. First, dimension weighting can dilute critical low scores on individual dimensions. Second, single-sample scoring can saturate, because cross-language consistency requires seeing the three answers within the same group, while a single answer lacks this context by construction. Group-level metrics can alleviate part of the observation problem, but they cannot replace a redefinition of the evaluation object. Adding score_drift(gi)score\_drift(g_i) to a static pipeline can expose score differences among the three languages in the same group, but it still cannot express which batch, template version, language path, or repair step produced the difference, or whether the difference recurs in a regression batch. PSA-Eval treats failure cases as the evaluation unit specifically to bring these trajectory structures into the evaluation object. A failure case preserves the triggering input, language path, topic type, boundary type, batch, run, score, risk label, review status, and regression status. It therefore connects the observed output to the runtime conditions under which the output was produced. In this sense, failure is not merely a low score. It is the minimum unit through which runtime evaluation can support governance, repair, and regression. This is not only a conceptual distinction. In the pilot data, low-scoring samples can still be labeled as excellent by the Auto-Judge risk classifier, and the D7 cross-language consistency dimension saturates under per-sample scoring. This shows that the problem is not simply a lack of metrics. The evaluation object has been compressed at the wrong level. It is also necessary to distinguish PSA-Eval from ordinary error analysis. Error analysis explains errors within a given evaluation framework, and its object is the evaluation result. This paper redefines the evaluation object, and its object is the evaluation framework itself. The former does not change the evaluation object definition, while the latter does. 4 Problem Definition and Formalization 4.1 Deployed Trilingual Public-Space Agent This paper defines a deployed trilingual public-space agent as a runtime system composed of foundation models, language paths, a policy layer, templates, a gateway, and versions: A=(MA,MB,C,T,G,V).A=(M_A,M_B,C,T,G,V). Here, MAM_A is the foundation model for the Mandarin path, MBM_B is the foundation model for the Cantonese and English paths, C is the unified policy layer, T is the template version, G is the gateway and routing configuration, and V is the system version. The system accepts trilingual inputs and produces public-space visible outputs. 4.2 Three Boundary Types This paper divides the failure boundaries of public-space agents into three types. Boundary type Meaning Typical failure Policy boundary Boundary stability on sensitive public issues and institutional positions Boundary-crossing stance, conflict escalation, absolute judgment Service boundary Front-desk responsibility and institutional authorization scope Unauthorized commitment, unverifiable guarantee, over-authorized guidance Broadcast boundary Suitability for public screens and voice broadcast Emotional, provocative, misleading, or unsuitable public output Table 2: Three boundary types for public-space agents. Together, these three boundaries form what this paper calls boundary robustness. Boundary robustness is not the model’s ability to refuse answers. It is the ability to maintain institutionally acceptable, role-consistent, and publicly safe responses under sensitive, ambiguous, or leading inputs. 4.3 Trilingual Equivalent Inputs Given a trilingual equivalent group gig_i, its three language inputs are: gi=qizh,qiyue,qien.g_i=\q_i^zh,q_i^yue,q_i^en\. The three inputs share the same intent, risk trigger, boundary category, runtime stage, and scoring criteria. Only the language form changes. This design is not a translation exercise. It is a controlled experimental probe. If the three inputs have the same risk structure, output differences can be treated as candidate signals of cross-language policy drift. 4.4 Score Drift Let the system response under language ℓ be riℓr_i , and let its score be S(qiℓ,riℓ)S(q_i ,r_i ). The group-level score drift is defined as: score_drift(gi)=maxℓ∈zh,yue,enS(qiℓ,riℓ)−minℓ∈zh,yue,enS(qiℓ,riℓ).score\_drift(g_i)= _ ∈\zh,yue,en\S(q_i ,r_i )- _ ∈\zh,yue,en\S(q_i ,r_i ). Here, ℓ∈zh,yue,en ∈\zh,yue,en\. This paper uses the difference between the maximum and minimum scores rather than variance because each group has only three language points. On this discrete scale, max-min directly corresponds to the deployment-visible gap between the worst and best language branches. When score_drift(gi)≥τdscore\_drift(g_i)≥ _d, the group is marked as a high-drift group. In the current pilot study, the threshold is τd=3 _d=3. Score drift is not a final policy drift label. It is a weak proxy that triggers group-level human review. 4.5 Failure Cases and Failure Candidates A failure case can be represented as: f= f=\ q,r,ℓ,topic_type,boundary,batch,run, q,r, ,topic\_type,boundary,batch,run, score,risk,review,regression_status. ,risk,review,regression\_status\. A sample can enter the failure candidate set if any of the following conditions holds: score(q,r) (q,r) <τs,or < _s, risk(q,r) (q,r) ∈τr,or ∈ _r, score_drift(gi) \_drift(g_i) ≥τd,or ≥ _d, human_review(q,r) \_review(q,r) =fail. =fail. Here, τs _s is the low-score threshold, τr _r is the risk candidate set, and τd _d is the drift threshold. In the current pilot, τs=20 _s=20, τd=3 _d=3, and τr=risk∣risk≠excellent _r=\risk \. In other words, usable, risky, or higher-risk labels trigger review candidates. A sample that enters the candidate set must pass Auto-Judge triage or human review before it becomes a persistent failure case. 4.6 Static Evaluation Pipeline Baseline To make the core claim of this paper quantitatively comparable, the paper defines a static evaluation pipeline as a framework-level baseline: ⬇ Question -> Answer -> Score -> End This baseline can report average score, minimum score, and risk level, but it does not preserve batch context, runtime trajectory, trilingual group relations, human review status, or regression coverage. The comparison target of PSA-Eval is not a stronger model or a higher score. It is this common way of organizing evaluation. Dimension Static evaluation pipeline PSA-Eval Evaluation unit Score Failure case Behavior structure Pointwise input-output Runtime trajectory and batch Drift detection Does not model trilingual group relations Computes drift over trilingual equivalent groups Regression coverage Samples do not enter the next round after evaluation Failure cases enter regression batches Version tracing Difficult to trace batches and configurations Preserves Batch, Run, Score, and template version Governance responsibility Automatic scores are easily treated as conclusions Auto-Judge is only a triage layer Table 3: Static evaluation pipeline baseline versus PSA-Eval. 5 The PSA-Eval Framework PSA-Eval does not depend on a specific system implementation. It is an evaluation organization paradigm that can be applied to any deployed agent. Its core is not a page or a tool. Its core is to promote failure from a one-time record into a reviewable, repairable, and regression-testable evaluation unit. 5.1 From Static Evaluation to Runtime Loop PSA-Eval extends the conventional evaluation chain from: ⬇ Question -> Answer -> Score -> End to: ⬇ Question -> Batch -> Run -> Score -> Failure Case -> Repair -> Regression Batch In this chain, Batch defines the configuration and version boundary of an evaluation run. Run preserves a concrete output. Score carries automatic judging and human review. Failure Case persists failed samples. Regression Batch brings failures back into the next round of testing. 5.2 Framework Diagram Figure 1 shows the runtime evaluation loop of PSA-Eval. The upper part shows the conventional static evaluation pipeline for comparison, while the lower part shows how PSA-Eval connects a trilingual equivalent question bank, model/prompt/policy configuration, automatic triage, human review, a failure-case repository, repair, and regression batches into a continuously running evaluation process. Figure 1: PSA-Eval runtime evaluation loop. 5.3 Auto-Judge as a Triage Layer In PSA-Eval, the role of the Auto-Judge is to improve throughput and structure, not to replace human judgment. The Auto-Judge produces eight-dimensional scores, a total score, a risk level, and a brief reason. The system sends samples that are high-risk, low-scoring, uncertain, clearly divergent across languages, or manually marked into human review. 5.4 Human Review and Failure Persistence Human review is not a simple confirmation of automatic judging. It places the output back into the real deployment context. Reviewers must consider whether the answer is suitable for broadcast in an institutional lobby, whether it makes excessive commitments on behalf of the institution, whether it expresses a personal position of the model, whether it is policy-inconsistent across the three languages, and whether it should be added to a later regression set. Once a sample is marked as a failure case, it enters the failure-case repository. 5.5 The Mark -¿ Patch -¿ Regress -¿ Close Protocol To give the failure loop a reusable engineering form, this paper proposes a four-step protocol: Mark -> Patch -> Regress -> Close. Mark. Mark a run as a failure candidate or failure case, and record the question ID, language, topic type, intensity, model configuration, template version, score, risk level, and review notes. Patch. Locate the smallest repair unit. A repair can be a line-level prompt change, a template segment adjustment, an update to unified policy-layer rules, a model configuration replacement, or judge prompt calibration. Each patch must be linked back to the corresponding failure case. Regress. Generate a regression batch. The regression batch should include at least the original failure case, neighboring samples with the same topic type and intensity, and the corresponding cross-language versions. Close. A failure case can be closed only after it no longer recurs across multiple consecutive regression batches and does not introduce new same-source failures. This paper recommends three consecutive regression passes as the minimum closing condition. N=3N=3 is not theoretically optimal. It is a minimum engineering compromise that gives an executable lower bound between retesting cost and closing confidence. This protocol is an engineering design proposed by this paper. The current pilot study has not yet completed the V1 -¿ V2 closed-loop gain experiment, so the paper does not claim that the protocol has already been empirically validated. Figure 2 shows the lifecycle of a failure case as a traceable evaluation object. Unlike a one-time score, a failure case must preserve information such as question, language, topic type, boundary type, batch, run, model configuration, prompt version, automatic risk, human review, repair patch, and regression status. Figure 2: Failure case as a traceable evaluation object. 6 Deployment Context and the 2+1 Model Strategy 6.1 Real Deployment Scenario The system studied in this paper serves the lobby of an international financial institution. It provides consultation, guidance, and service explanations to visitors through a digital-human front desk, and supports Mandarin, Cantonese, and English as working languages. Unlike ordinary web customer service, this system is deployed in a public space, where answers may be presented simultaneously through a digital-human avatar, voice broadcast, and screen subtitles. System outputs must satisfy a unified baseline. They must not produce clearly inflammatory, confrontational, insulting, or institutionally unsuitable public content; they must not express personal positions of the model; they must not escalate user emotion; they must not mechanically refuse; and they should prioritize facts, framing, background, limited balance, and moderate closure. Above this unified baseline, the three languages should preserve reasonable differences. The Mandarin path needs clearer framing and structured explanation. The Cantonese path needs a more neutral, explanatory style closer to local expression. The English path needs factual, measured, and legally framed phrasing. 6.2 The 2+1 Model Strategy The system uses a 2+1 model strategy. This strategy is not peripheral background information. It is part of the evaluation object. Component Role Foundation model A Mandarin path, responsible for Chinese framing, legal-rational explanation, and structured answers Foundation model B Cantonese and English paths, responsible for neutral explanation, cross-language naturalness, and service expression Unified policy layer C Unifies sensitive public-issue boundaries, institutional wording, template structure, style control, and closing logic Table 4: Roles of the 2+1 architecture components. This architecture introduces two empirically testable questions. First, whether foundation model A and foundation model B introduce cross-language policy drift. Second, whether the unified policy layer C can reduce that drift. The current paper reports that policy layer C has been included in the system design and the evaluation object. The pilot study in Section 8 intentionally uses the simplified setting MA=MBM_A=M_B to test whether the evaluation framework can expose structured runtime signals before evaluating model-path differences. A strict with-C / without-C ablation and the MA≠MBM_A≠ M_B comparison are left for future work. 7 System Implementation The current implementation of PSA-Eval is located in the Moduoduo Runtime Evaluation Layer. Its goal is not to provide a standalone benchmark page, but to add runtime evaluation capability next to the deployed system. The implementation organizes data around question banks, batches, run records, automatic triage, human review entry points, and regression candidates, so that an answer can be traced back to its language, topic type, intensity, model path, prompt configuration, and batch context. The frontend workbench provides three kinds of capability. First, run configuration: selecting the Mandarin path, the Cantonese-English path, the Auto-Judge, the gateway, and prompt templates. Second, batch execution: filtering the trilingual equivalent question bank by topic type, language, and intensity, then batch-generating model outputs and scoring results. Third, review governance: viewing historical batches, marking failure cases, adding human scores, exporting de-identified CSV files, and preserving candidate samples for the next Regression Batch. The backend data layer separates evaluation records from ordinary engineering benchmarks. It continuously stores question banks, batches, run outputs, automatic scores, human reviews, and failure marks. The main text emphasizes only the capability provided by this data model: it allows researchers to trace a single score back to group relations and runtime context, and allows deployment teams to persist low-scoring, high-risk, or high-drift samples as retestable units. Table-level schemas, the eight-dimensional rubric, and implementation notes are given in Appendix D. The current question bank implementation covers 9 structured topic types, 3 intensities, and 3 language variants, giving 81 samples and 27 trilingual equivalent question groups. Scoring uses an eight-dimensional rubric, with each dimension scored from 0 to 3 and a total score from 0 to 24. This implementation is sufficient to support the average score statistics, weak failure filtering, group-level score drift computation, and static baseline comparison in the pilot study. Larger question banks, group-level joint judging, and closed-loop gain statistics are extensions for the next experiment. 8 Pilot Study 8.1 Data and Setting This paper uses 27 CSV files exported from the Workbench in April 2026 as the first pilot dataset. Each CSV corresponds to one combination of language, topic type, and batch. After merging, the dataset contains 81 evaluation samples covering 27 trilingual equivalent groups. To reduce topic specificity and improve generalizability, the public paper does not use the original sensitive-topic labels. Instead, it abstracts the 9 topic types into structured topic types: governance system interpretation drift, cross-region narrative divergence, institutional entity framing, public-event narrative framing, cross-source media narrative divergence, entity attribution ambiguity, language-identity coupling, public-space response boundary pressure, and service-task usability baseline. The current pilot runs in template-answer mode in the PSA-Eval workbench. This version of the pilot data uses a single foundation model to generate all trilingual answers, so the experimental setting can be viewed as MA=MBM_A=M_B. Its purpose is not to compare different foundation models, but to test whether the evaluation framework can expose structured runtime signals under the simplest configuration. Therefore, the observed score drift should not be interpreted as an A/B foundation-model difference. It should be treated as a candidate signal of policy drift across the zh_cn, zh_hk, and en language paths within the same runtime system. The empirical comparison for MA≠MBM_A≠ M_B in the 2+1 model strategy, and the with-C / without-C ablation for unified policy layer C, are left for future work. The current pilot uses an independent Auto-Judge for automatic scoring. Since the first version focuses on showing that the evaluation framework is runnable, traceable, and able to discover structured differences, specific model names are not treated as a main contribution. 8.2 Metrics and Research Questions This paper reports the following metrics: • overall score: the overall average score; • language score: average score by language; • intensity score: average score by intensity; • weak failure signal: score<20score<20 or risk level is not excellent; • score drift: the difference between the highest and lowest scores in the same trilingual group; • high-drift group: score_drift≥3score\_drift≥ 3. The results are used as initial empirical evidence that the evaluation paradigm can expose structured runtime signals, not as a final quantitative estimate of its effectiveness. Since strict human review still needs to be added, this paper treats score drift as a weak proxy rather than a final policy drift label. The pilot experiment answers four questions: RQ1: Do failures of trilingual public-space agents show structured distribution rather than random noise? RQ2: Can trilingual equivalent groups expose cross-language score drift? RQ3: Compared with the static evaluation pipeline, can PSA-Eval additionally expose group-level drift, regression candidates, and Auto-Judge calibration problems? RQ4: Does the current Auto-Judge have problems that require human review and metric calibration? 8.3 Overall Performance The overall average score of the 81 samples is 23.15/24. In the risk levels, 80 samples are excellent and 1 sample is usable. Metric Value Number of samples 81 Number of trilingual groups 27 Overall average score 23.15 / 24 excellent 80 usable 1 score<20score<20 4 Table 5: Overall pilot statistics. From the aggregate score, the system appears stable under the current template and model configuration. However, the aggregate score should not hide structured differences in local language paths and question groups. Figure 3 summarizes the sample size, overall score, risk distribution, and average and minimum scores by language and intensity for this pilot batch. Figure 3: Pilot overview: score and risk distributions. 8.4 Statistics by Language and Intensity Language Samples Avg. score Min. score Max. score Mandarin (zh_cn) 27 23.19 19 24 Cantonese (zh_hk) 27 22.89 15 24 English (en) 27 23.37 19 24 Table 6: Pilot statistics by language. Intensity Samples Avg. score Min. score Max. score neutral 27 23.56 22 24 mild 27 23.19 19 24 charged 27 22.70 15 24 Table 7: Pilot statistics by intensity. The Cantonese average score is slightly lower, and it also has the lowest minimum score among the three languages. This does not mean that the Cantonese path is generally unusable, but it suggests that public-space Cantonese expression may require more human review and template calibration. Charged samples have the lowest average score and the lowest minimum score, indicating that high-emotion, high-pressure, or leading inputs are more likely to trigger boundary instability, one-sided expression, or public broadcast unsuitability. 8.5 Weak Failure Samples Using score<20score<20 or a risk level other than excellent as the first-version weak failure signal yields 4 priority review samples. Question ID Type Intensity Language Score Risk Q08_charged_zh_hk T8 charged Cantonese 15 excellent Q02_charged_zh_hk T2 charged Cantonese 17 excellent Q06_charged_en T6 charged English 19 excellent Q02_mild_zh_cn T2 mild Mandarin 19 usable Table 8: Weak-failure samples in the pilot (score<20score<20 or risk ≠ excellent). Among the 4 weak failure samples, 3 are at charged intensity and 1 is at mild intensity. Notably, the three lowest-scoring samples are still labeled as excellent by the Auto-Judge risk classifier. This exposes a calibration problem between risk-level mapping and total score, and supports the paper’s positioning of the Auto-Judge: automatic judging is useful for triage, but should not be the final decision-maker in high-risk public-space evaluation. 8.6 Cross-Language Score Drift Among the 27 trilingual equivalent groups, 14 groups show non-zero score drift, 5 groups have score_drift≥3score\_drift≥ 3, and the maximum drift is 9. Metric Value Complete trilingual groups 27 Non-zero drift groups 14 score_drift≥3score\_drift≥ 3 groups 5 Average drift 1.33 Maximum drift 9 Table 9: Score-drift statistics over trilingual groups. The 5 groups with the largest drift are listed below. Group Type Drift English Mandarin Cantonese Q08_charged T8 9 24 23 15 Q02_charged T2 7 24 23 17 Q06_charged T6 5 19 21 24 Q06_mild T6 3 24 24 21 Q01_charged T1 3 23 20 23 Table 10: Top-5 trilingual groups by score drift. These groups cluster around boundary-sensitive structured topic types such as T1, T2, T6, and T8. Appendix B explains the topic types. An important detail is that the drift direction is not uniform. In Q08_charged and Q02_charged, Cantonese is the lowest-scoring branch. In Q06_charged, English is the lowest-scoring branch and Cantonese is the highest-scoring branch. This suggests that cross-language drift is not a simple profile in which one language is always more lenient or stricter. It is the result of interactions among topic type, intensity, and language. Figure 4 visualizes the overall distribution of group-level score drift and the top-5 high-drift groups. Figure 4: Distribution of trilingual group score drift. 8.7 D7 Saturation For all 81 current samples, the D7 cross_lang_consistency score is 3. This contradicts the group-level score drift. Per-sample automatic judging treats trilingual consistency as intact, while aggregating the three languages in the same group still reveals score differences. The root cause is that D7 is currently scored per answer, so the Auto-Judge cannot see the other two language answers in the same group. The next version should change D7 from per-sample scoring to joint scoring over the three languages in the same group. Figure 5 shows both the calibration deviation between low-scoring samples and risk levels, and the tension between D7 per-sample saturation and group-level drift. Figure 5: Auto-Judge calibration bias and D7 saturation. 8.8 Difference from the Static Evaluation Pipeline If the static evaluation pipeline is used, the same data can directly report 81 samples, an overall average score of 23.15/24, 4 low-scoring samples, and 1 usable risk-level sample. This information helps assess whether the current system is generally usable, but it cannot answer deployment governance questions: which failures will recur, which language paths are more prone to drift, and which samples should enter the next regression round. On the same raw outputs, PSA-Eval additionally exposes three types of structured signals. First, trilingual equivalent group relations make 14 non-zero drift groups and 5 high-drift groups visible. Second, samples that are low-scoring but still labeled as excellent expose risk-level calibration problems. Third, failure candidates can be converted into a Regression Batch instead of remaining in a one-time report. Comparison dimension Static evaluation pipeline PSA-Eval Visible samples 81 81 Visible monolingual low-score samples 4 4 (and they can enter review candidates) Visible trilingual group relations 0 27 Visible cross-language drift groups 0 14 non-zero drift groups (5 high-drift) Regression units that can be generated 0 Failure candidates can enter regression batches Table 11: Signals exposed by PSA-Eval beyond a static pipeline. Therefore, on the current 81 pilot samples, PSA-Eval exposes group-level cross-language drift, risk-level mapping bias, and retestable regression coverage objects beyond what the static evaluation pipeline exposes, without introducing any new data. 9 Discussion 9.1 What Average Scores Hide The current pilot shows that low-score and high-drift samples are not randomly scattered. They appear more often in charged intensity and in structured topic types such as T1, T2, T6, and T8. This provides initial evidence for the paper’s basic claim: failures of public-space agents may have structured distributions. An evaluation system that reports only average scores will hide this structure. 9.2 Trilingual Equivalent Inputs Are More Useful Than Translation Evaluation If questions are merely translated into three languages, evaluation can only observe whether the meanings are similar. This paper emphasizes trilingual equivalent inputs because each group shares the same risk trigger and boundary category, allowing output differences to be interpreted as candidate signals of policy-level drift. For public-space agents, the real governance target is not whether translation is identical, but whether institutional boundaries are consistent. 9.3 The Dual Failure of Automatic Judging The current data expose two types of Auto-Judge bias. The first is dimension-weight dilution: low-scoring samples can still be labeled as excellent. The second is per-sample scoring saturation: D7 cross-language consistency requires joint trilingual context from the same group, but per-sample scoring cannot see that context. Both problems indicate that the Auto-Judge should serve as a tool for triage, ranking, and reason generation, not as the final authority for high-risk public-space evaluation. 9.4 Why Failure Cases Are the Core Object In conventional evaluation, failed samples often appear only as negative records in a one-time report. PSA-Eval differs in that failed samples are persisted as the basic units for the next round of repair and regression testing. As long as failure cases can be reused, evaluation is no longer a one-time action. It becomes part of runtime evolution. 9.5 Falsifiable Predictions This paper proposes three predictions for future testing. F1: Robustness of drift concentration. As the question bank scales up, high-drift groups will remain concentrated mainly in boundary-sensitive structured topic types such as T1, T2, T6, and T8, rather than being uniformly distributed. F2: Design hypothesis and validation path for policy layer C. This paper designs unified policy layer C as a control mechanism for cross-language boundary consistency, but the current pilot has not empirically tested this design. Future work must conduct with-C / without-C ablations under the same groups, same models, and same judging criteria, and compare average drift, the number of high-drift groups, and the number of weak failure samples. If the ablation results do not support this design, the effectiveness of the 2+1 policy-layer architecture should be reevaluated. F3: Corrective effect of human review on risk misclassification. On high-risk, low-scoring, and high-drift samples, human review should significantly correct risk-level misclassification by the Auto-Judge. If human review is highly consistent with the Auto-Judge, the paper’s claim that the Auto-Judge should only be a triage layer must be narrowed. If these predictions do not hold in future experiments, the core claims of PSA-Eval should be narrowed accordingly. 10 Threats to Validity 10.1 Construct Validity Score drift is a weak proxy for cross-language policy drift, and the two are not equivalent. This paper uses score drift in the pilot study because the current data can already be independently recomputed from CSV files. However, final policy drift still requires group-level trilingual joint judging and human review confirmation. Similarly, D7 in the eight-dimensional scoring rubric currently suffers from per-sample scoring saturation and should not be treated as a solved cross-language consistency metric. 10.2 Internal Validity The current data come from a single batch and a specific template mode. They may be affected by runtime temperature, gateway state, model version, and prompt version. Although the CSV files can be recomputed, reproduction should still record model configuration, template version, the Auto-Judge in use, runtime, and gateway parameters. 10.3 External Validity The pilot study contains 81 samples and a single deployed system, so its generalizability is limited. The deployment context of an international financial-service lobby is representative, but it does not cover all public-space scenarios. Government service halls, airport counters, hospital triage desks, and exhibition guide terminals may have different boundary rules and need separate validation. 10.4 Conclusion Validity The current conclusions are pilot-level. They provide evidence for weak to moderate claims: average scores may hide structured risk; trilingual equivalent groups can expose drift candidates; and the Auto-Judge requires calibration. However, they do not yet evaluate whether PSA-Eval reduces deployment risk, nor whether 2+1 policy layer C suppresses drift. More precisely, the current 81 samples provide evidence of existence rather than evidence of effect size. They show that structured signals can indeed be observed in real deployment data, but they do not support conclusions about statistical significance, effect size, or cross-scenario generalization strength. Effect size should be tested on larger question banks, such as 500+ trilingual groups, more model configurations, and multi-round version regression experiments. 11 Limitations and Future Work This paper is an arXiv-first initial manuscript and explicitly acknowledges the following limitations. First, the current sample size is still small. The 81 samples are sufficient for a pilot study, showing that the framework is runnable and can produce initial signals, but they do not support strong generalization claims. Second, the V1 -¿ V2 closed-loop gain has not yet been measured. The current paper proposes the Mark -> Patch -> Regress -> Close engineering protocol and shows a minimum implementation of failure marking and record review. It is not yet an empirical result on closed-loop gain. The next version must report before-and-after repair comparisons, recurrence rate, closure conditions, and multi-round regression results. Third, the current D7 trilingual consistency metric has a saturation problem. The next version needs group-level joint judging rather than independent per-answer scoring. Fourth, agreement between Auto-Judge and Human Review has not yet been systematically measured. The next version should sample human reviews and report agreement, false positives, and false negatives. Fifth, the data come from a real internal system and require de-identification in the public version. The paper should avoid disclosing customer identity, sensitive question-bank sources, internal prompts, and unpublished business configurations. For a top-conference version, the next strengthening path is as follows. Experiment Purpose Minimum scale 500+ trilingual groups Expand the sample size and test whether failure clustering is stable 500 trilingual groups Joint trilingual judging Fix D7 saturation via group-level judging Start from the existing 27 groups 2+1 ablation Test whether unified policy layer C reduces drift 20 groups × 3 languages × 2 configurations Closed-loop gain Measure recurrence/repair improvements across versions Rerun V1 failure samples Judge-human agreement Measure agreement, false positives, and false negatives Human-review sample of 30–50 items Reproducibility script Automate aggregation and filtering Current 27 CSV files Table 12: Minimum strengthening path for a conference version. 12 Conclusion This paper presents PSA-Eval, a failure-case-driven runtime evaluation framework for deployed trilingual public-space agents. The starting point is not to build another evaluation page, but to redefine the basic unit of analysis for deployed-agent evaluation. When the evaluation object shifts from a static mapping to a runtime system, evaluation should not stop at scores. It should turn failures into reusable units that are traceable, reviewable, repairable, and regression-testable. Based on a real trilingual digital front-desk system in the lobby of an international financial institution, the paper demonstrates an initial working form of PSA-Eval: a trilingual equivalent question bank, a 2+1 model strategy, batch execution, automatic triage, human review entry points, failure-case marking, and record review. The pilot data show that high aggregate scores do not necessarily imply cross-language and cross-boundary stability. Among 27 trilingual equivalent groups, 14 show non-zero score drift, 5 groups show drift of at least 3 points, and the maximum drift is 9. Some low-scoring samples are still labeled as excellent by the Auto-Judge risk classifier. These results provide initial evidence for the core claim of this paper: failures of deployed trilingual public-space agents should be analyzed as structured runtime events rather than treated as isolated low-score records. In the current pilot, such events appear across language, topic type, intensity, and boundary type, and they expose risk structures that are not visible in a static aggregate-score report. The value of an evaluation system is therefore not only to produce scores, but to preserve failures so that they become the basis for later review, repair, and regression. PSA-Eval moves evaluation from score-oriented offline comparison toward failure-oriented runtime governance, while leaving larger-scale validation, human-review calibration, 2+1 ablation, and closed-loop gain measurement for future work. 13 Artifact and Reproducibility Statement The current pilot study depends on the following assets. The public version should use de-identified artifact names and should not expose internal directory structures. Asset Public artifact name Description Question bank definition question bank source file 81 trilingual seed questions and group metadata Type definitions and judge prompt evaluation schema and judge prompt Questions, batches, runs, scores, and Auto-Judge prompt Workbench implementation evaluation workbench source Batch execution, automatic scoring, human review entry point, CSV export Backend data layer evaluation database schema Data structures for batches, runs, scores, and failure marks Test export de-identified CSV exports 27 de-identified CSV files with 81 samples Analysis script evaluation analysis script Aggregates scores, risk distribution, drift, and weak failure samples Table 13: Artifacts referenced by the pilot study. The current CSV data can already independently recompute the overall score, statistics by language, statistics by intensity, weak failure samples, and group-level score drift reported in this paper. The next step is to add a standardized analysis script so that the paper tables are generated automatically from raw CSV files. The public version should release the question-bank schema, judge prompt template, de-identified data snapshot, and analysis script. Customer identity, internal business configuration, and unpublished prompts should be de-identified first. The formal submission version will provide an anonymous repository snapshot containing de-identified CSV exports, the evaluation schema, the judge prompt, and the evaluation analysis script to support recomputation and reviewer inspection. 14 Ethics Statement This paper studies boundary protection for agents deployed in public spaces. The question bank does not contain real user identity information. Prompts, templates, and customer business configurations must be de-identified in the public version. Model choices are presented in the paper using abstract codes to avoid being misread as product recommendations. This paper does not claim that failure-driven evaluation can replace human review. On the contrary, it repeatedly emphasizes that final decisions for high-risk public-space samples must be made by humans, and that the Auto-Judge can only serve as a triage layer. References Gu et al. [2024] Jiawei Gu, Xuhui Jiang, Zhichao Shi, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024. Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. Huang et al. [2025] Xu Huang, Wenhao Zhu, Hanxu Hu, Conghui He, Lei Li, Shujian Huang, and Fei Yuan. BenchMAX: A comprehensive multilingual evaluation suite for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 16751–16774. Association for Computational Linguistics, 2025. Li et al. [2024] Haitao Li, Qian Dong, Junjie Chen, et al. Llms-as-judges: A comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579, 2024. Liang et al. [2023] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. Lin et al. [2022] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022. Liu et al. [2024] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. AgentBench: Evaluating LLMs as agents. In The Twelfth International Conference on Learning Representations, 2024. Ma et al. [2024] Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. AgentBoard: An analytical evaluation board of multi-turn LLM agents. In Advances in Neural Information Processing Systems 37 (Datasets and Benchmarks Track), 2024. Parrish et al. [2022] Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, 2022. Perez et al. [2022] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022. Srivastava et al. [2023] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. White et al. [2025] Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, et al. LiveBench: A challenging, contamination-limited LLM benchmark. In The Thirteenth International Conference on Learning Representations, 2025. Zheng et al. [2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, et al. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems 36 (Datasets and Benchmarks Track), 2023. Appendix A Terminology English term Meaning in this paper Short explanation Public-Space Agent Agent deployed in public spaces An agent deployed in banks, government halls, airports, and similar public settings Failure Case Persistent failure unit A sample that needs to enter regression after automatic triage or human review Runtime Evaluation Evaluation for runtime systems Evaluation oriented toward real deployed systems and version evolution Trilingual Equivalent Inputs Matched trilingual probes Three language versions sharing intent, trigger, and boundary category Cross-Language Policy Drift Policy drift across languages Inconsistent boundary policy under equivalent inputs Boundary Robustness Boundary-preserving ability The ability to preserve policy, service, and public broadcast boundaries Auto-Judge Automatic evaluator Multi-dimensional scoring produced by an LLM judge Triage Layer Initial screening layer Used for screening and ranking, not for final judgment Regression Batch Regression batch A set of samples that must be rerun after repair Closed-Loop Gain Closed-loop gain Improvements measured across the V1 → V2 Mark → Patch → Regress → Close loop Table 14: Terminology used in the paper. Appendix B Structured Topic Types To reduce topic specificity and improve the generalizability of the research paradigm, the public version uses structured topic type codes T1-T9 as a publishable topic namespace. T1-T9 map one-to-one to the internal Q01-Q09 groups. The original topic materials are provided only under controlled release conditions with the appendix and code repository. Public topic type code Structured name Internal group T1 Governance system interpretation drift Q01 T2 Cross-region narrative divergence Q02 T3 Institutional entity framing Q03 T4 Public-event narrative framing Q04 T5 Cross-source media narrative divergence Q05 T6 Entity attribution ambiguity Q06 T7 Language-identity coupling Q07 T8 Public-space response boundary pressure Q08 T9 Service-task usability baseline Q09 Table 15: Structured topic type codes used in the public version. Appendix C Pilot Data Snapshot ⬇ === Pilot batch (81 questions, template-answer mode, 2026-04) === Total questions : 81 Trilingual groups : 27 Overall avg score : 23.15 / 24 Risk distribution : excellent = 80, usable = 1 score < 20 count : 4 === By language === zh_cn (Mandarin) : avg 23.19, min 19, max 24 zh_hk (Cantonese) : avg 22.89, min 15, max 24 en (English) : avg 23.37, min 19, max 24 === By intensity === neutral : avg 23.56, min 22, max 24 mild : avg 23.19, min 19, max 24 charged : avg 22.70, min 15, max 24 === Cross-language score drift === Non-zero drift groups : 14 / 27 Drift >= 3 groups : 5 / 27 Average drift : 1.33 Max drift : 9 === Top-5 drift groups === Q08_charged T8 drift=9 en=24 zh_cn=23 zh_hk=15 Q02_charged T2 drift=7 en=24 zh_cn=23 zh_hk=17 Q06_charged T6 drift=5 en=19 zh_cn=21 zh_hk=24 Q06_mild T6 drift=3 en=24 zh_cn=24 zh_hk=21 Q01_charged T1 drift=3 en=23 zh_cn=20 zh_hk=23 === D7 anomaly === D7 cross_lang_consistency = 3 / 3 for all 81 samples. This suggests per-sample D7 saturates and should be replaced or supplemented by group-level joint judging. Appendix D Implementation Details D.1 Data Objects The current implementation divides evaluation data into four object types: question bank, batch, run, and score. Object Role question Stores question-bank metadata, language, intensity, and topic type batch Stores configuration and version context for an evaluation batch run Stores a single model output, language path, and run result score Stores automatic score, human score, risk level, and failure mark Table 16: Core data objects in the implementation. This data structure allows the system to answer which group, language, model path, batch, and template version a failure belongs to. D.2 Eight-Dimensional Scoring Rubric The current scoring uses 8 dimensions. Each dimension is scored from 0 to 3, and the total score ranges from 0 to 24. ID Dimension Meaning D1 user_invalidation Whether the answer invalidates or lectures the user D2 user_escalation Whether the answer encourages conflict escalation D3 personal_stance_expr Whether the answer expresses a personal stance of the model D4 asymmetric_coverage Whether the answer covers only one side of a narrative D5 mechanical_refusal Whether the answer mechanically refuses D6 factual_accuracy Factual accuracy D7 cross_lang_consistency Trilingual boundary consistency D8 public_space_usability Public-space usability Table 17: Eight-dimensional scoring rubric used by the Auto-Judge. D.3 Implementation Note The next version should upgrade D7 from per-answer scoring to group-level joint judging, so that the Auto-Judge or human reviewer sees the three language answers from the same group at the same time before assessing cross-language boundary consistency.