Paper deep dive
When benchmark inferences do not compose: Projectibility in AI evaluation
Brett Reynolds
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/4/2026, 10:17:52 AM
Summary
The paper introduces the concept of 'projectibility' in AI evaluation, arguing that valid individual benchmark inferences do not automatically compose into valid chains of reasoning for broader claims. It proposes a non-composition principle where warrant does not transmit across links unless endpoints, assumptions, and uncertainties align. The paper outlines a 'projectibility audit' to diagnose unsupported joins in benchmark-to-use arguments, distinguishing this from construct validity or causal transportability.
Entities (8)
Relation Signals (6)
Non-composition principle â states â Warrant(C1) ⧠Warrant(C2) â Warrant(C2âC1)
confidence 98% ¡ The central claim of this paper is a non-composition principle: Warr(C1)â§Warr(C2)âWarr(C2âC1).
Projectibility audit â diagnoses â unsupported joins in benchmark-to-use arguments
confidence 96% ¡ The resulting projectibility audit diagnoses unsupported joins in benchmark-to-use arguments.
Projectibility â isdefinedby â bounded extension from observed to unobserved cases
confidence 95% ¡ Projectibility concerns whether a bounded extension from observed to unobserved cases is warranted.
Empirical Node â consistsof â object, population, conditions, outcome, period
confidence 93% ¡ For this audit, an empirical node has five fields: the object evaluated, the population of cases, the conditions under which the evaluation occurs, the outcome recorded, and the period to which the claim applies.
Goodman â supplies â problem of rival extensions
confidence 92% ¡ Goodman supplies the problem of rival extensions; argument-based validity supplies an architecture for testing them.
Kane â proposes â argument-based validity framework
confidence 90% ¡ Kane represents a proposed interpretation and use as a sequence of inferences and assumptions...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:An AI benchmark result rarely reaches a consequential claim in one step. Evaluators generalize it to further cases, interpret it as evidence of capability, extrapolate it to new tasks, transport it to another system or site, and combine it with assumptions about human review and downstream consequences. Validity-centred approaches require evidence for each claim. This paper identifies a further epistemic problem: warranted links don't automatically make a warranted chain. The target of one study may not be the source of the next; system, population, outcome, or conditions may change at the interface; and shared data or model lineage may make apparently independent support dependent. Projectibility concerns whether a bounded extension from observed to unobserved cases is warranted. Goodman supplies the problem of rival extensions; argument-based validity supplies an architecture for testing them. The paper's distinctive claim is a non-composition principle: support for adjacent projections warrants their composition only when endpoints and assumptions align and dependence and uncertainty are carried through. A legal-research case shows how benchmark evidence and a deployment study can each be sound while remaining parallel. A reanalysis and simulation show why aggregate stability can erase distinctions a later projection requires. The resulting projectibility audit diagnoses unsupported joins in benchmark-to-use arguments.
Tags
Links
- Source: https://arxiv.org/abs/2607.26159v1
- Canonical: https://arxiv.org/abs/2607.26159v1
Trouble viewing inline? Open PDF directly â
Full Text
120,644 characters extracted from source content.
Expand or collapse full text
[ Numbers=OldStyle, Ligatures=TeX, BoldFont=EBGaramond-Regular.ttf, ItalicFont=EBGaramond-Italic.ttf, BoldItalicFont=EBGaramond-Italic.ttf, ] [ BoldFont=CharisSIL-Bold.ttf, ItalicFont=CharisSIL-Italic.ttf, BoldItalicFont=CharisSIL-BoldItalic.ttf, ] [ BoldFont=HaranoAjiMincho-Bold.otf, ] [ BoldFont=Inconsolata-Regular.ttf, Scale=MatchLowercase, ] When Benchmark Inferences Do Not Compose: Projectibility in AI Evaluation Brett Reynolds Humber Polytechnic & University of Toronto brett.reynolds@humber.ca Abstract An AI benchmark result rarely reaches a consequential claim in one step. Evaluators generalize it to further cases, interpret it as evidence of capability, extrapolate it to new tasks, transport it to another system or site, and combine it with assumptions about human review and downstream consequences. Validity-centred approaches require evidence for each claim. This paper identifies a further epistemic problem: warranted links donât automatically make a warranted chain. The target of one study may not be the source of the next; system, population, outcome, or conditions may change at the interface; and shared data or model lineage may make apparently independent support dependent. Projectibility concerns whether a bounded extension from observed to unobserved cases is warranted. Goodman supplies the problem of rival extensions; argument-based validity supplies an architecture for testing them. The paperâs distinctive claim is a non-composition principle: support for adjacent projections warrants their composition only when endpoints and assumptions align and dependence and uncertainty are carried through. A legal-research case shows how benchmark evidence and a deployment study can each be sound while remaining parallel. A reanalysis and simulation show why aggregate stability can erase distinctions a later projection requires. The resulting projectibility audit diagnoses unsupported joins in benchmark-to-use arguments. Keywords: AI evaluation; benchmark validity; projectibility; construct validity; generalization; AGI AI use. The large language models ChatGPT 5; Claude Sonnet 4.5 and Opus 4.8; Gemini 3.1 Pro served as drafting and editing aids throughout the preparation of this paper. I am responsible for all theoretical claims, arguments, errors, and interpretive choices. 1 The Missing Problem: Composition A benchmark result rarely supports a consequential claim by itself. Consider a common chain of reasoning. A base model scores highly on a multidomain battery. The score is interpreted as evidence of general reasoning. A product built from that model is expected to perform a professional task. Human review is expected to improve the final work. That expectation, projected savings, and an error tolerance support authorization. The chain spans four empirical objects: benchmark responses, outputs from a tool-using application, work after human review, and consequences under a policy. A capability attribution concerns the base model and usually licenses the move from benchmark to application performance. The score records only the benchmark responses, not the other three objects. Psychological and educational measurement provide a disciplined response to this problem. On an argument-based view, validity belongs to a proposed interpretation and use of scores, not to a test considered apart from any claim [undefaaq, undefaai]. Recent work brings that view into AI evaluation. Claim-centred frameworks distinguish measurements, evaluations, and the assertions or decisions drawn from them; construct-validity studies ask whether benchmark content, structure, and relations support the capabilities named; and benchmark-epistemology accounts identify the assumptions needed to move from performance on a learning problem to a scientific conclusion [undefaav, undefap, undefaab]. These approaches substantially improve on treating a benchmark as valid or invalid without specifying what itâs supposed to warrant. They also make a further problem visible. Evaluation arguments are usually chains, but support for their links doesnât automatically compose. A benchmark may support a claim about new items generated under its rules. A firm may separately show that a particular humanâAI procedure works on a sample of its own cases. Both results can be warranted without the first supplying a premise for the second. The local study may bypass the benchmark rather than extend it. Conversely, two studies may appear to meet at a shared noun such as legal reasoning, draft quality, or human oversight while operationalizing different objects, populations, and outcomes. In either situation, adding the studies together doesnât produce the broad claim. These arenât only possibilities. Legal-research vendors advertised retrieval-grounded tools as hallucination-free on the strength of how those tools were built, and a preregistered evaluation later measured hallucination on more than 17% of queries [undefaan]. The problem isnât that machine outputs are fallible while human judgment isnât. Human judgments and institutions also fail. AI makes the interface problem especially pressing because one evaluated component can be replicated at high volume, modified into many applications, and inserted into workflows whose relevant evidence is distributed among different actors. An unsupported bridge can become a repeated, correlated failure while no single study observes the complete chain. LLMs can also make the remedy easier to apply by helping evaluators prepare typed source and target descriptions, trace assumptions and evidence, and flag possible mismatches for inspection. Those outputs donât certify the warrant they describe; they remain fallible contributions to a process answerable to independent evidence and accountable judgment. The central claim of this paper is a non-composition principle: WarrâĄ(C1)â§WarrâĄ(C2)â̸WarrâĄ(C2âC1).Warr(C_1)\ \ Warr(C_2)\ \ Warr(C_2\! C_1). (1) Here C1C_1 and C2C_2 are links presented as adjacent. If C1C_1âs target isnât C2C_2âs source, composition is undefined without a bridge; Equation 1 covers defined composition whose warrant fails to transmit. Equation 1 states a non-composition principle, not a theory of warrant. Writing WarrWarr as a predicate is a notational convenience: WarrâĄ(C)Warr(C) means warrant sufficient for the declared use. This threshold notation doesnât imply that support for a fixed claim is binary or reducible to one scale; warrant for the component links doesnât extend to their composition. Warranted composition requires alignment, or a separately warranted bridge, in object, population, conditions, outcome, and period; compatible assumptions and effect modifiers; and propagation of dependence and uncertainty. Projectibility concerns whether one bounded extension from observed to unobserved cases is warranted. The term comes from Goodmanâs treatment of induction, but the account developed here doesnât offer a general solution to induction and doesnât make entrenchment a sufficient condition. It gives a place within a validity argument for a narrower question: exactly what is being projected, across which boundary, under which assumptions, and with what evidence against the differences that could defeat the extension? A projectibility audit records the answer link by link and diagnoses unsupported joins rather than treating separately warranted links as automatically composable. This focus separates the paper from neighbouring proposals without displacing them. A nomological network, the set of relations a construct is predicted to bear to other constructs and observables, can make a capability claim more substantive [undefax, undefaaa]. An estimand can state precisely which quantity a study targets [undefaq]. A selection diagram can identify conditions for transporting a causal effect [undefaar]. Each may supply evidence or structure for a projection. None by itself guarantees that the target of one analysis is the source of the next. Projectibility concerns whether each boundary-specific extension is warranted; the audit also examines the interfaces among analyses. Section 2 locates projectibility within validity theory and distinguishes it from construct meaning, prediction, and decision. Section 3 states the interface requirements and separates composition from convergence and replacement. Section 4 works through a legal-research case whose hypothetical results support one projection, defeat another, and leave a third unresolved. Section 5 shows how an upstream mean can erase the item-level differences a downstream projection needs. Sections 6 and 7 apply the result to general-capability claims and divide the evidential work between developers and deployers. 2 Projectibility within a Validity Argument 2.1 Rival extensions from the same observations Goodmanâs new riddle of induction begins with observations that fit incompatible extensions equally well. Every emerald examined before time t is green. The observations fit the familiar hypothesis that emeralds are green. They also fit the rival hypothesis that emeralds are grue. Under this hypothesis, the observed emeralds qualify because theyâre green, but an emerald first examined after t would qualify only if it were blue. The hypotheses agree on every observed emerald and disagree on the next unexamined one. Fit to the source observations doesnât determine which predicate is fit for projection [undefaae, 74â80]. The benchmark analogue isnât that evaluators literally invent time-indexed predicates. Many classifications of unobserved cases agree on the scored items. Success can be grouped under general legal reasoning, under answering supplied-text legal questions, under recognizing patterns in questions drawn from familiar sources, or under still narrower descriptions. Those predicates may be extensionally equivalent on the test set and diverge on a research request that requires current authority, live retrieval, source comparison, and a cited memo. More observations sampled under the same narrow item-generating rule may distinguish none of them. Goodman appeals to entrenchment, the history of successful use of a predicate in past projections [undefaae, 84â98]. Other accounts place more weight on revisable background theory and causal structure [undefat, undefaak]. This account needs no single sufficient criterion. Its more modest lesson is that the rule of extension has to be exposed. A target declaration identifies the cases over which rival predicates disagree. Background knowledge identifies differences that could matter. Direct samples, controlled variations, and later observations test those differences. This doesnât remove induction; it makes a particular induction inspectable and revisable. 2.2 Definition and scope A projection extends an interpretation, prediction, or explanation from specified source observations to unobserved cases. Projectibility concerns whether that bounded extension is warranted relative to a declared target and use. Three restrictions follow. First, projectibility belongs to a bounded projective claim, not to a score, benchmark, or system considered in isolation. The same benchmark result may provide substantial warrant for performance on further items generated under a registered template and little warrant for performance on open-ended professional work. Calling the benchmark projectible without naming the target suppresses the point at issue. Second, projectibility isnât a scalar property. Support for a fixed projection may be stronger or weaker, but support, scope, and the kinds of evidence bearing on the claim shouldnât be collapsed onto one scale. A report should preserve heterogeneous results: a predictive estimate and interval, a defeated invariance assumption, an unresolved population gap, and a supported narrow use. Collapsing these into a projectibility score recreates the aggregation problem. The status should remain claim-specific: supported under a stated evidential standard, defeated by specified evidence, or unresolved because relevant evidence is absent or imprecise. Each status should name its limit, such as support for one request type or dependence on evidence that only another party can supply. Third, source fit is only part of the evidence. Warrant for a projection commonly comes from four places. Direct target sampling shows what happens in cases admitted by the target rule. Deliberate variation tests features suspected of changing the result while preserving what the claim treats as invariant. Background knowledge explains why some differences are plausible defeaters and others arenât. Replication across genuinely new levels (another template family, system lineage, site, or period) tests whether an earlier regularity survives the dimension being projected over. None can be replaced by a bare assertion that source and target are similar. This account belongs inside an argument-based approach to validity. Kane represents a proposed interpretation and use as a sequence of inferences and assumptions, each requiring support commensurate with the ambition of the claim [undefaai, 1â3, 22â25]. Messickâs unified account likewise treats validity as an evidential judgment about score meaning and use, drawing on content, response processes, internal structure, relations with external variables, and consequences [undefaaq, 741â749]. Projectibility concerns whether those parts of the argument that extend beyond cases observed under the source design are warranted. It doesnât subsume every validity question. Whether a rubric mis-scores the observed responses is a scoring problem before itâs a projection problem; whether a proposed capability has a coherent meaning is an interpretive problem that may condition a projection without being reducible to one. 2.3 What projectibility adds to neighbouring frameworks Neighbouring work can be sorted by how far along the inference it reaches. Some frameworks stop at what a benchmark collects; others follow a single claim from evidence to interpretation and use. This paper asks what warrants the join between two such claims. [undefaat] argue that broad benchmark claims routinely outrun the contextual tasks used to construct them, and [undefas] set out how construct validity fails in language-model evaluation. [undefaam] adapt evidence-centred design from educational assessment, formalizing benchmark construction into modules that require designers to describe, justify, and support each choice. Their subject is the evidence a benchmark collects about the capabilities it declares; movement beyond the benchmark isnât their topic. Recent validity-centred work on AI is closer still, so the difference has to be stated precisely. [undefaav] offer a claim-aware framework that maps measurements and evaluations to claims, differentiates forms of validity, and recognizes that measurement producers and downstream claimants may be different stakeholders. [undefaab] treat predictive benchmarks as measurement tools and specify internal, external, content, consequential, and auxiliary conditions for scientific inferences. [undefap] document construct-validity weaknesses across 445 language-model benchmarks and give practical recommendations for benchmark development. This paper accepts the shared premise: the inferential object is a particular claim, and stronger claims need more evidence. Its additional object is the interface between claims. A claim-centred framework can tell us which evidence would support a benchmark interpretation and which evidence would support a deployment criterion. A projectibility audit asks whether the conclusion of the first is actually a premise of the second. If the benchmark study concerns a base model answering supplied-text questions and the deployment study concerns final memos produced by a retrieval system and reviewed by lawyers, the studies donât meet merely because both are described as legal evaluation. Their evidence may converge on a narrow conclusion, one may replace the need for the other, or they may concern different objects altogether. Composition is a further relation. [undefaaa] argues that the inferential account provides insufficient resources for articulating the meaning of theoretical capability constructs, and that capability benchmarks should instead be embedded in nomological networks relating tasks, constructs, processes, and external criteria. That objection concerns a use of argument-based validity not made here. This paper neither proposes a meaning for reasoning nor infers the possession of reasoning from benchmark scores. It uses Kaneâs framework for the task it directly addresses: representing a proposed interpretation or use as a sequence of inferences and assumptions whose evidential support can be assessed. A substantive construct attribution, where one is proposed, needs whatever nomological, process, or causal support that attribution requires; the projectibility audit doesnât supply it. Nor would successful nomological validation remove the composition problem. A nomological network might support an inference from task scores to a reasoning construct, or establish a relation between that construct and an external criterion in a declared population. It wouldnât establish that the relation survives replacement of the benchmarked model by a retrieval-augmented application, that lawyer review catches the applicationâs errors, or that reviewed outputs produce the consequences a deployment policy assumes. Those are further projections over different objects, observations, and conditions. The non-composition result holds whether the construct attribution is warranted, unwarranted, or omitted. It can be omitted because it isnât an empirical endpoint. The step it appears to license still needs criterion evidence the attribution doesnât by itself supply. An estimand framework answers another indispensable question: what quantity is the analysis trying to estimate, over which population, under which data-acquisition and aggregation rule? [undefaq] show how benchmark conclusions become confused when those elements are left implicit. A chain may require several estimands. One study may estimate an expected score over benchmark items, another a post-review defect probability over local requests, and a third the rate at which defects reach clients. Each can be well specified while the chain remains unsupported because the first quantity doesnât predict the second or the second wasnât sampled under the conditions of the third. Finally, causal transportability supplies stronger formal results for a narrower kind of projection. Selection diagrams encode differences between source and target domains and establish when a causal relation is identifiable in the target from source experiments and target observations [undefaar]. Projectibility is broader: it covers descriptive generalization, construct interpretation, prediction, explanation, and sociotechnical outcomes, and it claims no identification theorem. Where the target claim is causal, a projectibility audit should defer to the corresponding causal requirements rather than treating generic robustness evidence as enough. Table 1 summarizes the division of labour. Its final column identifies the question that motivates this paper. Nothing in the non-composition principle conflicts with Kaneâs logic, and the objection that it adds nothing deserves a direct answer. A completely specified interpretation-and-use argument would include the interface assumptions among its inferences, and support for the whole argument would require support for those assumptions. Kaneâs own summary of chain reasoning shows where the remaining gap lies: âA chain of reasoning is only as strong as its weakest link, and strong evidence for part of an argument does not compensate for weaknesses in other parts of the argumentâ [undefaai, 64]. That principle presupposes that the inferences form a chain. Whether they do is a prior question, and where the answer is no, the individually strong links donât warrant the endpoint by composition. Composition becomes a distinct problem because of how AI evidence is actually produced. It arrives from separate studies, run by different actors, assembled later as though favourable conclusions were automatically adjacent. Kaneâs framework makes room for assumptions at the interfaces among inferences, but it doesnât treat composition as a distinct object of validation. This paper isolates a recurrent failure mode in distributed AI evaluation: separate studies may warrant their own conclusions while differing in the object, population, outcome, conditions, or uncertainty those conclusions need in order to serve as adjacent premises. The projectibility audit makes the interfaces explicit and states necessary conditions that a proposed composition has to satisfy. Table 1: Neighbouring frameworks and the composition question. Framework Primary object Question answered Further composition question Argument-based validity Interpretation-and-use argument Which assumptions and evidence support a stated interpretation or use? Does one supported conclusion supply the premise required by the next link? Nomological network Construct and its theoretical and empirical relations What gives a capability construct content, and do its predicted relations hold? Do those relations survive a change of task, system, site, or time? Estimand specification Metric, population, data acquisition, and aggregation Which quantity does this analysis estimate? Are the estimands at adjacent links defined over aligned cases and outcomes? Causal transportability Causal relation across source and target domains Under which assumptions is a target causal effect identifiable? Are later interpretive, workflow, or decision links also supported? Projectibility audit Specified extension and its interfaces What warrants this sourceâtarget extension, and can it be joined to adjacent extensions? The question is explicit rather than presumed. 3 Why Warranted Links Donât Automatically Compose 3.1 Typed nodes and projection edges The easiest way to conceal an inferential gap is to give both sides the same broad noun. A benchmark contains legal tasks, and a firm performs legal tasks; the shared label is taken to show that the score transfers. A developer tests the model, and a deployer uses the model; the shared label is taken to show continuity between the tested and deployed objects. I use type here in the type-theoretic sense, specifically as a record type: roughly, a schema whose instances assign values to labelled fields [undefaas, sec. 11.8]. For this audit, an empirical node has five fields: the object evaluated, the population of cases, the conditions under which the evaluation occurs, the outcome recorded, and the period to which the claim applies. These types are revisable representations for an audit, not claims that the objects share an essence or form a natural kind. A particular benchmark or deployment is represented as a node by filling those fields with values. Here, object names a field, whereas a named model build is a possible value of that field. Likewise, legal tasks may be intended as a value for the population field, but it supplies no case-generating or inclusion rule by which the benchmark and deployment populations could be shown to match. Write an empirical node as =â¨object,population,conditions,outcome,periodâŠ.N= ,\ population,\ conditions,\ outcome,\ period . (2) The population is a case-generating or inclusion rule, not just a list of cases, and the period covers the relevant system version as well as the span of time. In a benchmark node, the object might be a named model build; the population, held-out items produced under specified templates; the conditions, the prompt, decoding procedure, tool access, and scorer; the outcome, item correctness; and the period, the evaluation date. In a deployment node, the object might instead be a model, retrieval index, prompt stack, and lawyer-review procedure; the population, a rule admitting two kinds of logged research request; the conditions, the firmâs database and review instructions; the outcome, a substantive defect remaining in the final memo; and the period, the next six months. Typing doesnât guarantee that two nodes match. It makes the proposed match inspectable. The benchmark and deployment above might both be described as involving legal tasks, but their corresponding fields differ throughout. Repetition of the broad label supplies no warrant for the projection. A capability attribution isnât an empirical node of the form in Equation 2. Itâs an interpretive claim about a bearer, supported where appropriate by content, process, causal, and nomological evidence. [undefax, 290] state the constraint on it exactly: a construct âis not âreducedâ to the observations, but only combined with other constructs in the net to make predictions about observablesâ. It may be the terminal conclusion of one argument or a substantive premise in another. But it doesnât by itself specify the downstream population of cases, the criterion outcome, or the conditions under which the attributed capability is expected to predict that outcome. Where a capability attribution is invoked to license a step between empirical nodes, the audit asks which observable relation is required and whether that relation has been tested in the declared population. An empirical projection edge includes its source and target nodes, projective claim CeC_e, assumptions AeA_e, and evidence EeE_e bearing on the claim and assumptions. Write e=â¨s,t,Ce,Ae,EeâŠe= _s,N_t,C_e,A_e,E_e , also e:sâte:N_s _t. Generalizing from observed benchmark items to further items under the same generating rule is one empirical edge. Extrapolating to a new task population, transporting a relation to a new model build, and predicting a post-review outcome are further edges. Interpretive inferences remain part of the broader validity argument, but they arenât themselves sample-bearing interfaces between empirical studies. The notation forces no commitment that every argument has the same sequence. It prevents an argument from treating a sequence as one undifferentiated leap. Figure 1 shows a common evaluation chain. The projection appears above each arrow and evidence that can answer it below. No amount of evidence under one arrow automatically fills another. A. Benchmark-to-use architecture Observed benchmark responses Further test-rule performance Application outputs on eligible target tasks Final work after registered review Consequences under authorized policy Capability attribution backed by content, process, causal, and nomological evidence generalization template-family holdout task and system projection target-task and configuration sample review projection paired draft and final scoring outcome projection exposure and outcome surveillance interpretation criterion relation required B. Interface audit Endpoint alignment e1:0â1e2:1â˛â2e_1:N_0 _1 e_2:N _1 _2 1=?1â˛N_1 ?=N _1 object â ¡ population â ¡ conditions â ¡ outcome â ¡ period 1â 1â˛âN_1 _1 bridge owed; composition undefined Warrant transmission If the endpoints align, WarrâĄ(e1)â§WarrâĄ(e2)â̸WarrâĄ(e2âe1)Warr(e_1) (e_2) (e_2\! e_1) compatible assumptions and effect modifiers dependence and uncertainty carried through Figure 1: Architecture and interface failures in a benchmark-to-use argument. Panel A matches evidence to projections between empirical endpoints. A capability attribution can be supported while leaving the target cases and criterion relation needed for application performance untested. Panel B distinguishes endpoint mismatch, which makes composition undefined without a bridge, from failure of warrant transmission across aligned endpoints. 3.2 Endpoint alignment and warrant transmission Suppose e1:0â1e_1:N_0 _1 and e2:1â˛â2e_2:N_1 _2 are each supported. Two questions arise in order. First, do the links meet? This is an endpoint-alignment question about whether the proposed composition is well formed. Alignment requires continuity in object, population, conditions, outcome, and period, or a separately warranted bridge across the difference. Second, if they meet, does warrant transmit across the join? That further requires compatibility of assumptions and effect modifiers, and propagation of evidential dependence and uncertainty. The first failure makes the proposed composition undefined; the second leaves a defined composition unwarranted. The five alignment requirements read off Equation 2; the two transmission requirements read off the assumptions AeA_e and the evidence EeE_e carried by the edges. All seven presuppose empirical nodes at both endpoints, which the previous subsection secured. Endpoint alignment has five requirements. Where one fails, the mismatched field identifies the bridge thatâs owed. A result about supplied-text questions doesnât arrive at a population of live-retrieval requests. 1. Object continuity requires a declared system or workflow projection when the object changes. A base model, a model connected to a database, a complete application with citation validation, and a memo after lawyer review are different objects. Product names and version families donât establish continuity. Shared training, distillation, retrieval components, or evaluation data may create dependence without preserving behaviour. 2. Population alignment requires the cases reached by the upstream projection to have the distribution, support, and grouping assumed downstream. A study on short, self-contained questions doesnât supply cases for a study whose source consists of open-ended requests with disputed authorities. Even within one task label, a downstream study may sample routine files while the upstream claim includes novel, urgent, or multilingual work. The inclusion rule, not the label, determines alignment. 3. Condition alignment requires the operating conditions of the shared endpoint to match on both sides, or their difference to be shown harmless for the target claim. Two studies conducted under incompatible prompts, decoding settings, databases, retrieval configurations, or reviewer instructions donât compose merely because both are described as evaluating the same system. 4. Outcome and scale continuity requires the first edge to deliver the quantity the next uses. Benchmark accuracy, a factor score, the probability of a citation defect in a draft, a defect after review, and a client loss arenât interchangeable outcomes. Putting each measure on a zero-to-one scale changes its numerical range, not what a difference means. A change of .10 in benchmark accuracy isnât equivalent to a .10 change in defect probability or client loss. If the outcome changes, the relation between the measures is itself an inferential link. 5. Temporal alignment treats the period as a field like any other. A result established on one model build, database index, and work period doesnât reach a later one because the product name is unchanged. Where the downstream claim covers a span the upstream study didnât sample, the difference is a further projection, answered by a chronological holdout or a declared retest trigger rather than by an assumption of continuity. Warrant transmission has two further requirements once the endpoints align. 1. Assumptions and effect modifiers have to remain compatible across the join. Assumptions that support one edge can fail under the conditions of the next. Review may reduce one error class while increasing delay or inducing overreliance; retrieval may improve source coverage while introducing irrelevant but persuasive cases. Relevant effect modifiers have to be carried to the interface rather than averaged away. An effect modifier dropped at the join can leave both component estimates correct and the composed prediction wrong. 2. Dependence and uncertainty have to be propagated rather than reset at the next edge. A chain can look better supported than it is when its links reuse the same items, scorer, model lineage, or development decisions. Twenty outputs from one build arenât twenty systems, and thirty-two comparisons among related models and shared benchmarks arenât thirty-two independent replications. Where estimates are composed, their covariance, selection, and uncertainty should be represented. Where a qualitative assumption is uncertain, the chain inherits that uncertainty. Suppose one study finds that systems with higher benchmark scores produce fewer defective drafts, and another finds that lawyer review removes a certain proportion of draft defects. Itâs tempting to combine the results to predict the quality of final work. But that combination makes two claims that neither study establishes alone: that both results apply to the same target population, and that review works in the same way across systems once draft quality is held fixed. Let B be a benchmark result, D a draft-level outcome, and Z a final outcome after review. In a target population T, PTâ(ZâŁB)=âdPTâ(ZâŁD=d,B)âPTâ(D=dâŁB).P_T(Z B)= _dP_T(Z D=d,B)\,P_T(D=d B). (3) In words, consider each possible draft outcome. Multiply its probability at a given benchmark result by the probability of the final outcome after review for that draft and benchmark result, then add across the possible draft outcomes. The first study may estimate PTâ(DâŁB)P_T(D B), while the second estimates PTâ(ZâŁD)P_T(Z D). Equation 3, though, requires PTâ(ZâŁD,B)P_T(Z D,B). Substituting the second quantity is warranted only if the benchmark result adds no information about review success once the draft outcome is known, ZâBâŁDZ B D. That condition could fail if, for example, high-scoring systems produce more fluent errors that reviewers are likelier to trust. And if either study concerns a different population, it doesnât supply the corresponding relation in T. Perfectly aligned endpoint descriptions still donât ensure that warrant passes across the join. Formal causal claims require more, but the problem arises even for descriptive prediction. 3.3 Composition, convergence, and replacement Three relations among studies are easily conflated. In composition, the conclusion of one study supplies a premise or population for the next. A benchmark score predicts draft defects across systems; draft defects predict post-review defects under a specified procedure; those links may be composed if their systems, cases, outcomes, and conditions align. In convergence, distinct evidence bears on the same claim without forming a chain. A benchmark, expert assessment, and process intervention might each support a reasoning attribution. Their agreement can strengthen the claim, but no result is passed as an input from one study to another. Dependence still matters: three tests built from the same source materials donât provide three independent routes. In replacement, direct target evidence makes an ambitious upstream projection unnecessary for a narrower purpose. A firm that samples its own research requests and measures final reviewed defects can assess that workflow without first proving that the base model possesses general legal reasoning. The local study doesnât validate the generality of the benchmark. It starts from new source observations closer to the target. This is often epistemically preferable: direct target evidence can support a bounded policy while the capability question remains open. The distinctions matter because the phrase benchmark plus local validation can describe all three. If the local study compares several systems and tests whether their benchmark profiles predict their defect rates on the same requests, it tests a composable predictive edge. If it evaluates only one frozen application, the benchmark profile is constant across its requests and canât explain which request fails. The local study then replaces, rather than confirms, the benchmark-to-use projection. If both studies are cited merely because they sound favourable, they provide neither composition nor well-characterized convergence. 3.4 Two counterexamples The two failures separated above look alike on the page and come apart under inspection. The first is spurious adjacency, where the links never meet. Consider two individually true claims. First, a model answers 90% of held-out supplied-text legal questions correctly under the benchmark protocol. Second, lawyers following a written review instruction catch 99% of fabricated citations in drafts generated for a set of routine research requests. Neither result is defective on its own. Their conjunction still doesnât warrant that reviewed memos are substantively correct. The retrieval system may omit controlling authority without fabricating any citation. The benchmark never tests retrieval, and the review study counts only fabricated citations. Suppose the application omits a controlling case on 20% of requests, the omission isnât visible from the citations that remain, and the lawyersâ procedure doesnât require an independent update search. Both premises remain true while one fifth of final memos contain a substantive defect. The chain fails at two interfaces: the outcome changes from supplied-text correctness to citation fabrication, and the review evidence doesnât cover omitted authority. Multiplying the two percentages obscures rather than repairs the gap. The counterexample also shows what positive evidence would look like. The firm could prepare an authority list independently of the system, score omitted controlling authority as well as citation fabrication, and require reviewers to compare every final memo with that list or conduct a defined update search. The revised study wouldnât make the benchmark general. It would supply observations for the missing workflow link. Something close to this has already happened. Legal-research vendors advertised retrieval-grounded products as delivering â100% hallucination-free linked legal citationsâ or as avoiding hallucination by relying on trusted content, and [undefaan] found that no empirical evidence accompanied those claims. Their preregistered evaluation reports hallucination on more than 17% of queries for both the LexisNexis and Thomson Reuters tools, and incomplete answers on more than 60% for Thomson Reuters. The advertised inference runs from a component property, retrieval over an authoritative database, to what the product asserts, with no observations at the join. Incompleteness is the omission failure isolated above rather than the fabrication failure the marketing addressed. The second failure is harder to see, because the links do meet. Suppose several systems are evaluated on the same held-out requests, and a benchmark score predicts the frequency of draft defects across them. Suppose reviewers working from one instruction catch 90% of draft defects on those same requests. The draft-defect variable is now the same variable in both studies, over the same systems and the same requests, so every field of the shared endpoint matches and no bridge is owed. The composed prediction can still be wrong. If high-scoring systems fail mostly by omitting an authority that no citation flags, while low-scoring systems fail mostly by inventing a citation that any reviewer catches, then review effectiveness depends on which system produced the draft. In the terms of Equation 3, PTâ(ZâŁD,B)â PTâ(ZâŁD)P_T(Z D,B)â P_T(Z D), and a catch rate averaged over the whole set overstates what review achieves for exactly the systems the benchmark ranks highest. Both component relations are well estimated, the interface is aligned, and the chain still fails. Nothing here is repaired by checking that the two studies denote the same things; the defect is an effect modifier dropped at the join. 4 A Worked Projectibility Audit 4.1 The source result and the proposed use Suppose a developer reports a high legal-reasoning score for a base model. The battery presents each question with a fixed set of materials and marks the answer against a reference response. Its held-out design may support generalization to further questions produced under the same item rules. The score doesnât observe whether the model can find current law, distinguish controlling from merely similar authority, produce a linked citation, or work within a lawyerâs review procedure. An Ontario employment-law group is considering a product built from the model. The application receives a lawyerâs research request, searches the firmâs licensed legal database, and drafts a one-page memo. The firm proposes to authorize it only for two kinds of request: whether a termination clause is enforceable and what period of reasonable notice is likely. An eligible request is written in English, concerns Ontario law, identifies the relevant dates and employment facts, and asks one of those two questions. Requests concerning another jurisdiction, a different subject, missing facts that prevent research, or a novel constitutional issue are outside the target. Each memo must identify the issue, state the governing propositions, cite the current controlling authorities, link each proposition to a passage that supports it, and undergo line-by-line lawyer review before it leaves the firm. The object under consideration isnât the benchmarked base model. Itâs a frozen configuration consisting of the model build, legal-database index, retrieval settings, system prompt, drafting template, citation-validation component, and review instruction. The final outcome belongs to the complete workflow, not to the model alone. That distinction is familiar in sociotechnical analyses of AI: component accuracy doesnât by itself determine the quality of decisions or work produced after human interaction [undefay, undefav, undefaal]. Those measured rates donât estimate the defect rate of this application. They do identify a plausible defeater and an outcome the firm should record. The firm also needs to count unsupported propositions and omitted controlling authority. A citation can resolve and accurately quote a real case while the memo remains wrong because the decisive case never appeared. 4.2 The declared chain Table 2 states the proposed links before any local result is known. The source and target columns are deliberately repetitive: they show where an apparently continuous claim changes object, population, or outcome. Table 2: The legal-assistant argument decomposed into projection links. Source Target Evidence that bears on the link 1 Scored supplied-text questions Further questions under the registered benchmark rules Hold out whole templates or item families; document contamination checks and scorer reliability 2 Performance under the benchmark rules Draft memos for eligible local requests Sample logged requests by the inclusion rule; prepare authority lists independently; score draft defects 3 Frozen application drafts Final memos after the written review procedure Score the same outputs before and after review; record who reviewed them, what was corrected, and how long review took 4 Ordinary retrieval condition Retrieval with topically similar but legally irrelevant cases Use an answer-preserving alteration not seen during prompt or rubric development; keep it out of tuning 5 Termination-clause requests Reasonable-notice requests Sample and report each request type separately; donât infer one from the other by the label employment law 6 Current build and database index Later build, index, or six-month work period Use a chronological holdout and retest after a material component change 7 Estimated defects and review costs Authorization of a policy Compare no assistant, search suggestions, mandatory-review drafting, and broader use under stated values and constraints Link 2 isnât established by showing that the battery contains several legal domains. It requires observations from the firmâs task. Link 3 isnât established by assigning a lawyer to each output; review effectiveness is itself an empirical relation. Link 5 isnât established by averaging both request types. Link 7 isnât a projection from cases alone. It combines empirical premises with values, feasible alternatives, professional obligations, and institutional authority. 4.3 A confirmatory local design The firm develops its prompt, inclusion rule, reference-answer procedure, and scoring rubric on older closed files. Every request from one client file remains in the same split. Confirmation uses a later untouched block containing 200 eligible termination-clause requests and 200 eligible reasonable-notice requests, with one request selected from each client file. The counts are illustrative, but each number in the design has a function: 200 observations per declared stratum allow the firm to assess its chosen defect tolerance separately rather than letting the more frequent or easier request type dominate a pooled estimate. Before any output is generated, two lawyers who supervise these file types independently prepare, for each request, a list of controlling authorities and the legal propositions for which each authority is required. They resolve disagreements before seeing the applicationâs memo. The procedure avoids defining the reference standard around whatever the system happened to retrieve. For each request, the application produces one draft under the ordinary retrieval setting. It also produces a second draft after topically similar but legally irrelevant cases are inserted near the top of the retrieved set. The alteration leaves the correct legal answer unchanged and tests whether retrieval rank rather than legal relevance controls the memo. The two drafts from one request are assigned to different reviewing lawyers, and no lawyer sees both versions. Order is counterbalanced. Blind scorers assess each draft, the assigned lawyers conduct the registered review, and different blind scorers assess the final memos. A substantive defect is present when at least one citation doesnât resolve, a cited passage doesnât support the proposition attached to it, or the memo omits an authority on the independently prepared controlling-authority list. The three defect types are also reported separately. An unresolvable citation remaining in a final memo is a red-line outcome; it canât be offset by correct propositions elsewhere. Review time is recorded in minutes, but it isnât folded into the defect indicator. It enters the later policy comparison in its natural unit. The firm sets the confirmatory rule before opening the later block. For each request type and condition, support for mandatory-review use requires both (i) a one-sided 95% exact upper confidence bound below a post-review substantive-defect tolerance of 3% and (i) no unresolvable citation in a final memo. The claim is defeated if the corresponding lower bound exceeds 3% or if a red-line citation remains. Other results are unresolved. The 3% tolerance isnât supplied by the benchmark or proposed as a general legal standard. In the example, the firm would have to justify it by comparing the consequences and costs of the available policies. Fixing the rule in advance narrows analyst discretion without eliminating it. Adjudicating a borderline citation, pooling the two request types or reporting them apart, and handling a request that yields two outputs all remain open, and each can move the estimate. The audit records those choices and treats unplanned alternatives as exploratory rather than as independent confirmations [undefaad]. The exact bounds keep the constructed example transparent by treating the file-level outcomes as exchangeable Bernoulli draws. An actual study in which the same lawyers review many files should register a clustered or hierarchical uncertainty analysis and apply its support criterion to that estimate. The inferential point doesnât depend on the simple bound. This design supplies direct evidence for links 2â5. It doesnât test link 6 unless the later period or changed component is sampled, and it doesnât settle link 7. It also doesnât test whether the developerâs benchmark score predicts local defects. With one frozen application, the benchmark profile is constant across the 400 requests. A request-level prediction requires variables that vary by request, such as request type, the number and age of retrieved authorities, conflicting appellate treatment, or exposure to the registered alteration. A benchmark-to-local predictive study would instead need several genuinely distinct systems evaluated on the same held-out requests. 4.4 Hypothetical results and their inferential status Table 3 gives constructed results. They arenât observations about any commercial system. Their purpose is to show that the same audit can support, defeat, and suspend judgment about different projections without converting those statuses into one overall verdict. Table 3: Constructed confirmatory results for the worked example. Exact bounds are one-sided 95% bounds for the post-review defect probability. Request type and condition Draft defects Final defects Relevant exact bound Final red-line citations Status Termination clause, ordinary retrieval 38/200 1/200 upper 2.35% 0 Supported Termination clause, irrelevant-case alteration 46/200 2/200 upper 3.11% 0 Unresolved Reasonable notice, ordinary retrieval 52/200 18/200 lower 5.90% 2 Defeated Later database index or model build â â no estimate not observed Unresolved The ordinary termination-clause claim is supported under the registered rule. The upper bound is below the firmâs tolerance and no red-line citation survives. The result warrants a narrow projection from the sampled later files to other requests admitted by the same rule under the same frozen configuration and review procedure. It doesnât warrant unreviewed drafting, another office, another request type, or a later build. The alteration result is unresolved, not a failure disguised as success and not evidence of robustness. Its point estimate is low, but the registered upper bound remains above 3%. More importantly, the unresolved status identifies the exact missing precision. The firm can collect new alteration cases if the prospective value of that evidence justifies the cost; it canât pool the altered and ordinary outputs to make the interval narrower for a claim about the altered condition. The reasonable-notice claim is defeated. Its lower bound exceeds the tolerance, and two unresolvable citations remain after review. Success on termination-clause requests canât compensate for that result. The failed claim can lead to a prespecified narrower policy (for example, search suggestions only on reasonable-notice files) but not to a post hoc target defined around the requests that happened to pass. The future-build claim remains unresolved because no future build or database index appears in the observations. A chronological holdout from the current configuration doesnât become evidence about a materially changed configuration merely because the product name remains the same. Authorization can be conditioned on freezing the components or on retesting after a declared update trigger. Table 4 returns these results to the complete chain. The crucial row is the benchmark-to-local-defects projection, link 2 in Table 2. The local draft study estimates the frozen applicationâs performance on local requests, but it doesnât establish that the developerâs benchmark score predicted that performance. For the firmâs bounded decision, the direct local evidence replaces the benchmark-to-use projection. The benchmark remains useful as a description of the source evaluation and perhaps as a screening instrument for choosing systems to test. It doesnât acquire external reach from the local result. Table 4: Warrant status after the constructed local study. Projection Status Reason Observed benchmark items to further items under the benchmark rules Conditional Requires the developerâs template-level holdout, contamination checks, and scorer evidence; the local study adds nothing to this link Benchmark score to defects on local research drafts Unresolved One system supplies no across-system benchmarkâcriterion relation; the local request sample directly estimates draft defects instead Frozen application to reviewed termination-clause memos under ordinary retrieval Supported narrowly The direct target sample meets the registered defect and red-line criteria under the specified review procedure Ordinary retrieval to the irrelevant-case condition Unresolved The altered-condition upper bound misses the registered tolerance Termination-clause to reasonable-notice use Defeated The second request type has a separately observed post-review defect rate above tolerance and surviving red-line errors Current configuration to a later index or build Unresolved The changed object and period werenât sampled Empirical results to authorization Not settled by projectibility alone Authorization requires comparison of feasible policies, review time, confidentiality, professional obligations, and consequences The example gives projectibility a positive role rather than using it as a label for skepticism. One declared projection survives a test, one is contradicted, and one lacks enough evidence. The statuses arise from observations tied to the exact boundary, not from the breadth of the nouns used in the claim. The firm can authorize a specific mandatory-review workflow for termination-clause requests while declining or further testing the others. It neednât decide whether the base model possesses legal reasoning in general. 4.5 From empirical warrant to policy Even the supported termination-clause result is only one premise in a decision. The firm has at least four feasible policies: no assistant, assistant-generated search suggestions, drafting with the registered line-by-line review, and drafting with a lighter check. It can compare them on final defects, unresolvable citations, omitted authorities, lawyer minutes, confidentiality exposure, and whether an error reaches a client-facing memo or filed document. A policy that reduces draft time while increasing final defects may be inferior; a policy with slightly more review time may be preferable if it eliminates red-line outcomes. This separation matters because human review isnât a fixed safety multiplier. Studies of human oversight show that detection depends on base rates, signal quality, reviewer incentives, workload, and the information supplied with an output [undefaal]. More accurate component systems can also produce worse joint performance when people under- or over-rely on them [undefav]. The legal study records the workflowâs final result rather than multiplying a model accuracy by an assumed review rate. Values enter before the final authorization as well as at it. Treating an unresolvable citation as a noncompensatory red line is a choice about what errors may trade off against other gains. Choosing 3% as a tolerance and deciding whether lawyer minutes justify a narrower use are further choices. Validity evidence can show which empirical premises are supported; it canât choose the firmâs loss function or discharge its professional duties [undefaaq, 747â749]. 5 Aggregation Can Destroy Interface Evidence The non-composition problem isnât caused only by changes of system or site. It can arise because the source report discards distinctions that a later projection needs. A total score may answer the developerâs descriptive question while making the deployerâs target question unanswerable. This section isolates that information loss and gives a compact empirical illustration. The statistical quantities arenât proposed as a new universal scorecard. They show why the target claim has to determine what the source report preserves. 5.1 One stable mean, several incompatible states Consider the same N items scored under a baseline condition 0 and an altered condition p. Let siâcâ[0,1]s_icâ[0,1] be the score for item i under condition c, and let δi=siâpâsiâ0 _i=s_ip-s_i0. Define the signed level change L as the mean item change: L=1Nââi=1Nδi.L= 1N _i=1^N _i. (4) A value near zero says that positive and negative changes balance on average. It doesnât say that items were stable. To distinguish average stability from item movement and concentration, use the item instability (INSINS) and worst-tail degradation (WTDWTD) summaries introduced by [undefaay]. Item instability is the mean absolute paired change: INS=1Nââi=1N|δi|,INS= 1N _i=1^N| _i|, (5) and define its positive and negative directional components, F+F^+ and FâF^-, as the mean improvement and deterioration: F+=1Nââi(δi)+,Fâ=1Nââi(âδi)+.F^+= 1N _i( _i)_+, F^-= 1N _i(- _i)_+. (6) The two pairs are related by L=F+âFâL=F^+-F^- and INS=F++FâINS=F^++F^-. Reporting both makes cancellation observable rather than hidden. A mean can also hide concentration. For a prespecified tail fraction q, order the paired changes from smallest to largest, let m=âqâNâm= qN , and define worst-tail degradation as WTDq=â1mââj=1mδ(j).WTD_q=- 1m _j=1^m _(j). (7) A positive value records average deterioration in the worst-changing fraction. It still measures change. A serious error repeated in both conditions has δi=0 _i=0 and disappears from L, INSINS, and WTDqWTD_q. The distinction is visible without simulation. Suppose 100 items have a stable mean (Figure 2). â In state A, every item has δi=0 _i=0. Then L=0L=0, INS=0INS=0, and WTD.1=0WTD_.1=0. â In state B, 50 items improve by .20 and 50 deteriorate by .20. Then L=0L=0, but INS=.20INS=.20 and WTD.1=.20WTD_.1=.20. â In state C, every item is unchanged across conditions, but the same ten items carry an absolute substantive-loss probability of .80 in both. Then all three change quantities are zero while the worst-decile case-risk tail is .80. The states license different claims. State A supports stability on the observed items. State B defeats item stability despite the stable mean. State C shows why stability itself isnât safety: an unchanged serious failure remains serious. No estimator can recover which state obtained from L alone. The target claim determines which distinctions are required. A claim about answer-preserving context variation needs paired changes and their concentration; a deployment claim about final defects needs an absolute loss after the relevant review procedure. â.20-.200+.20+.20State A L=0L=0, INS=0INS=0, WTD.1=0WTD_.1=0 absolute risk unspecified â.20-.200+.20+.20State B L=0L=0, INS=.20INS=.20, WTD.1=.20WTD_.1=.20 absolute risk unspecified â.20-.200+.20+.20State C L=0L=0, INS=0INS=0, WTD.1=0WTD_.1=0 case-risk tail =.80=.80 Figure 2: Three item-level states behind one stable mean, constructed for illustration rather than measured. Each panel gives the distribution of per-item change δi _i; all three have L=0L=0. States A and C are indistinguishable in every change statistic, yet the ten worst items in C carry an absolute substantive-loss probability of .80 in both conditions. A change statistic canât stand in for an absolute risk measure, and a source report that publishes only L leaves a downstream claim about final defects unanswerable. For an absolute target loss, let âiâR(T)â[0,1] ^(T)_iRâ[0,1] be the realized loss for item i under residual response, scoring, and outcome variation R, and let Îźi(T)=Râ(âiâR(T)âŁi)Îź^(T)_i=E_R( ^(T)_iR i). Ordering Îźi(T)Îź^(T)_i from largest to smallest gives a case-risk tail; ordering realized draws gives a different tail. The first identifies cases expected to be risky, while the second includes residual bad luck within otherwise moderate-risk cases. A projectibility declaration has to say which object the decision concerns. Neither tail should be smuggled into a behavioural-change statistic. 5.2 Released-output illustration The logical possibilities occur in released model evaluations. [undefaay] add task-irrelevant context to benchmark items and report paired baseline and altered responses. Their released design crosses four benchmarks with eight models, yielding 32 modelâbenchmark comparisons with 20 baseline and 20 context trials per item. The empirical companion to this paper reanalyses those responses rather than regenerating proprietary-model outputs, pins the source version and data hashes, and provides the estimator code at https://github.com/BrettRey/benchmark-inference-composition. One comparison shows both cancellation and the need to separate tail selection from tail estimation. It pairs gpt-5.4 with MMLU-Pro, a more demanding version of the Massive Multitask Language Understanding benchmark [undefaax]. Baseline accuracy is .8119 and altered-condition accuracy .7904, a signed decline of .0215. Its directional components are F+=.0229F^+=.0229 and Fâ=.0444F^-=.0444, so the mean hides .0673 of two-sided item movement. Raw worst-decile degradation is .3680. Reusing the same finite response trials to select and estimate the tail can inflate its apparent magnitude. A response-half procedure instead selects items on one half and estimates their changes on the other; the resulting estimate is .2911. The result isnât that 29% is a deployment risk. A 2.15-percentage-point mean decline coexists with much larger deterioration among a selected group of items. Across the 32 fixed comparisons, both directional components exceed twice the absolute signed change in 22 cells. Raw worst-decile degradation exceeds the response-half estimate by .0906 on average, ranging from .0330 to .1696. Those summaries describe one crossed dataset with shared items and related model lineages, not 32 independent replications. They also show why a tail selected on noisy observed changes needs either a model of latent item effects or disjoint information for selection and estimation. A known-truth simulation makes the separation from absolute loss explicit. In the stable-poor scenario, the same low-performing items have unchanged response probabilities under baseline and altered conditions. Latent worst-tail change is zero, while their worst-decile conditional expected loss is .80. The response-half estimate of the change tail is .0013; the cross-fitted case-risk tail is .7911. The disagreement persists without ambiguity about the truth; it follows from the estimands. These results verify that source aggregates can erase cancellation, concentration, and persistent loss. They donât show that any benchmark quantity predicts legal research, another system, or a future period. Indeed, the released data contain no research requests, database searches, citations, lawyer reviews, or client outcomes. Their relevance is architectural: if an upstream evaluator publishes only a total, a downstream evaluator canât inspect whether the source failures line up with the features that define its target. 5.3 Evidence preservation at the interface A projectibility audit imposes an information requirement on source reports. The developer neednât publish every conceivable statistic, but it should retain the resolution at which target-relevant distinctions can later be checked: item identifiers or auditable equivalents, item-level outputs and scores, template or source clusters, prompts and tool conditions, repeated-response structure, scorer decisions, model build and lineage, and registered alterations. A domain profile is better than a total when domains matter, but it can still hide item-level reversals and stable failures within each domain. The requirement isnât maximal disaggregation for its own sake. Raw data can be sensitive, proprietary, or too large to release. The relevant standard is whether a declared downstream claim can be audited. Access controls, secure evaluation environments, independent auditors, or sufficiently detailed derived data may answer the need. What doesnât answer it is an aggregate from which several target-relevant states are observationally indistinguishable. The legal example illustrates the interface. A high legal-domain mean doesnât reveal whether errors concentrate on questions requiring current authority or on outputs with plausible but unsupported citations. A local evaluator needs those distinctions to decide what to sample and score. If the developer can report item changes under retrieval-like distractors, citation-related errors, and build provenance, that evidence may inform the firmâs design. It still doesnât replace the firmâs observations of its own requests and review procedure. 6 What the Non-Composition Principle Changes for AGI Evaluation Multidomain AGI evaluations make the composition problem unusually acute. The label general invites a conclusion that reaches beyond every fixed set of tasks, systems, and conditions used to construct the score. Adding domains increases coverage, but coverage and inferential reach arenât the same. Before a total or profile is read as evidence of general capability, three questions have to be separated: whether its components share a scale, whether the permitted tradeoffs are acceptable, and whether the resulting description projects to the target claim. 6.1 Commensurability, compensability, and projectibility Commensurability asks whether differences on component scores have a common interpretation that makes addition meaningful. Putting a legal score and a quantitative score on a common scale from zero to one doesnât show that a .05 increase represents the same amount of improvement in both. Equal weights donât solve the problem; components with different variances and covariances can make unequal statistical contributions even under equal nominal weights [undefau, 305â308]. Compensability asks whether gains on one component may offset failures on another for the interpretation or use at issue. A weighted sum permits such substitutions over the reported ranges. That may be acceptable for a descriptive index. Itâs unacceptable when the target imposes a bottleneck or red line. In the legal case, a high quantitative score doesnât compensate for an unresolvable citation, and success on routine termination-clause research doesnât compensate for failure on reasonable-notice requests if both uses are to be authorized. Value models can represent accepted tradeoffs, but the weights are range-dependent preferences rather than context-free measures of domain importance [undefaaj]. Projectibility asks whether the aggregate supports a specified claim beyond the cases and conditions used to construct it. Even a commensurable, defensibly weighted index may provide little warrant for a projection to a new task, system, or period. Conversely, a score can predict a bounded criterion without being an interpretable measure of a single latent capability. The measurement, prediction, and decision roles need different evidence. This tripartite distinction prevents one objection from doing the work of another. Showing that a total predicts an external outcome doesnât establish that its components measure one construct. Showing a coherent factor structure doesnât establish that its compensatory total is suitable for a decision. Showing that a battery samples ten domains doesnât establish that its score projects to open-ended general capability. 6.2 Four roles for a cognitive taxonomy The CHC framework used in some AGI proposals can enter an evaluation in at least four roles [undefaw, undefaao, undefaaw, undefaaf]. First, it can organize benchmark content: evaluators select tasks under labels inherited from a theory of human cognitive abilities. Second, correlations among task scores in a declared population of artificial systems can be summarized by a factor model. Third, the resulting scores can be interpreted as evidence of capacities with a nomological or process-based meaning. Fourth, the score or profile can be used to predict an external criterion or guide a decision. Evidence for one role doesnât establish the next. A content taxonomy can ensure that a battery doesnât consist entirely of one familiar task format. It doesnât establish that artificial systems reproduce the human covariance structure that motivated the taxonomy. A positive manifold among model scores can support a statistical general factor in the sampled modelâtask matrix without identifying the factor as human-like intelligence or a causally efficacious internal capacity. A factor interpretation can become more substantive through a nomological network of expected task, process, and external relations, but those relations still need to be tested in the artificial-system population [undefax, undefaaa]. [undefaz] distinguishes construct representation, an account of the processes, strategies, and knowledge involved in item responses, from nomothetic span, the network of relations between scores and external measures, groups, and tasks. The two rest on different units of variation. Construct representation âis concerned with task variability rather than subject variabilityâ, while nomothetic span âis assessed by individual differences dataâ, making it âpossible to obtain strong support for one, but not for the otherâ [undefaz, 180]. Process evidence can support an account of how a system produced benchmark responses without showing that the score predicts another task. External prediction can be strong while the process interpretation remains unsettled. Treating both as one achievement under the label validity obscures which conclusion the evidence supports. Empirical work on artificial-system psychometrics illustrates the gap. Across 591 nominally distinct models and 12 tests, [undefaag, 4â5] report a positive manifold and a strong general factor. Their proposed four-factor hierarchy beneath it wasnât interpretable, and the task set was largely verbal; dependence among nominally distinct models also limits the system population represented [undefaag, 7â9]. The study supports a factor model for that sampled matrix. It doesnât establish the CHC decomposition for a retrieval-augmented system, a humanâAI workflow, or future systems. Comparability across systems is another projection. Interpreting score differences as differences in the same abilities requires evidence that items and score relations retain their meaning across model families. Depending on the design, that can involve rubric checks, family-specific item analyses, differential item functioning, and invariance models [undefaap]. [undefaah] give a direct warning: across 17 language models, human psychometric instruments showed moderate reliability across item and prompt variations while their scores failed to align with, and sometimes ran opposite to, model behaviour on downstream tasks. Reliability of the instrument under selected variations didnât supply criterion validity. Philosophically informed benchmark construction can strengthen the content and interpretation links. For example, [undefao] develop a behavioural benchmark for scientific understanding around retrieval, explanation, and counterfactual performance. Such work makes the target construct more explicit than a battery assembled by domain coverage alone. It still leaves empirical questions about which artificial systems satisfy the proposed relations and whether benchmark performance reaches a different task or deployment setting. The non-composition principle isnât an objection to theorized benchmarks. It specifies the additional work their results require downstream. 6.3 Why accumulated coverage doesnât entail unrestricted generality A finite battery can define and measure an index over its construction sample. It can also support bounded generalizations when its item-generating process, sampling assumptions, and uncertainty are defensible. The stronger claim begins when the index is read as evidence of capability across tasks and conditions not represented by that process. More domains may make the source description broader, but a run of successful source descriptions doesnât by itself identify the rule by which they extend. This is the Goodman point in practical form. Passing tasks in law, mathematics, coding, and spatial reasoning fits a general-capability hypothesis. It can also fit a collection of narrower hypotheses under which performance depends on familiar interfaces, training overlap, static inputs, short horizons, or the absence of tool and user interactions. The hypotheses agree on the battery and diverge elsewhere. Domain count alone doesnât select among them. Evidence has to vary the suspected conditions or sample the target cases. Cross-domain covariation is stronger evidence than a domain count. Across 591 models and 12 tests, every score pair correlated positively, and a mathematics composite loaded most strongly on the general factor [undefaag]. The pattern makes one tested domain informative about others in that sampled matrix but doesnât establish unrestricted reach. Separate studies might support transfer from benchmark mathematics to unseen textbook problems, from coding exercises to repository-level bug fixes, and from supplied-text law questions to bounded research requests. Their union says more than any one, but without evidence relating those targets it doesnât establish arbitrary tasks, domain interactions, later systems, or novel tool environments. Bounded evidence can accumulate legitimately. If the cases reached by one projection are sampled by the next, interface assumptions are tested, and dependence and uncertainty are preserved, the chain can extend. A systemâs benchmark profile might predict target-task outcomes across independent builds; target-task outcomes might predict final workflow results across sites; those results might remain stable in chronological holdouts. The conclusion is as broad as the composed path and no broader. The non-composition principle blocks an automatic inference, not empirical progress. 6.4 Aggregates as descriptions, measures, predictors, and decision inputs Many disputes about AGI scores are disputes about role. A total can be a transparent descriptive index even when it isnât a measure of one attribute. It can be a useful predictor even when its construct interpretation is contested. It can enter a decision model without deciding the weights, constraints, or alternatives. Each role should be stated rather than allowed to slide into the next. As a description, the score reports a declared operation on the construction sample. Accuracy here is largely a matter of calculation, sampling, and scoring. As a measure, it represents a capability or attribute and requires content, structural, process, and external evidence appropriate to that interpretation [undefaaq, undefar]. As a predictor, it requires target-matched out-of-sample performance at the independent unit named by the claim. As a decision input, it also requires a defensible relation to consequences, feasible policies, and the people who bear their effects. A common error is to validate one role and write as though the others follow. A high testâretest reliability supports score stability under the repeated protocol, not a capability ontology. A factor loading records association in a fitted model, not an internal mechanism. A predictive relation across systems doesnât show that the predictor measures a causally efficacious AGI attribute. A policy that uses the predictor successfully in one organization doesnât show that the same rule is suitable elsewhere. The role distinctions also sharpen reporting. An AGI index should be named as an index when itâs defined by its scoring rule. A capability interpretation should state the nomological and process commitments that make the term more than a heading. A predictive claim should name the target cases, systems, and loss. A release decision should show why its tradeoffs and hard constraints are appropriate. Projectibility is then assessed for each move rather than attributed to the total as a prestige property. 7 Procedure and Evidential Responsibility A projectibility audit is useful only if it changes what evaluators record and test. Its procedure begins before calculation, with a sentence that can be contradicted by observations and a typed description of the source and target. It ends neither with a benchmark report nor with authorization. Later outcomes and material system changes reopen the links they bear on. 7.1 Who can supply the evidence The evidential burden is distributed because no single actor observes the whole chain. A developer can describe the source evaluation in detail. It can identify the exact model build and declared lineage; publish or provide controlled access to items, prompts, tool settings, scoring rules, item-level outputs, and repeated-response structure; document which templates or sources were held out; report registered alterations and known failures; and state which product changes invalidate the result. Psychometric analyses can improve the quality and interpretation of the instrument itself, as work on item-response methods for NLP benchmarks illustrates [undefan]. Those analyses remain conditional on the systems, items, and score interpretations studied. A deployer observes different facts. Only the law firm in the running example can define which requests it receives, sample them from its logs, prepare local authority lists, observe its lawyersâ review, compare the policies it can adopt, and record whether a defect remains internal or reaches a client or court. A hospital, school, or public agency would have corresponding task, workflow, exposure, and outcome records. The inability of a developer to foresee those details narrows the developerâs warranted claim. It doesnât turn a generic benchmark into evidence about every use. Some evidence belongs at the interface and requires cooperation. The deployer needs notice of model, safety-layer, retrieval, or API changes; enough configuration information to freeze the tested object; and access to outputs at the resolution required for local scoring. The developer needs feedback about failures that source evaluation didnât represent. Independent evaluators may test both sides, but they canât infer proprietary lineage or local workflow conditions from a public product label. Where information remains unavailable, the corresponding projection remains conditional or unsupported rather than being filled by an assumption of continuity. Responsibility is divided but not dissolved. Developers are responsible for the interpretations and uses they advertise and for foreseeable interface failures; deployers remain responsible for the decisions they make and the evidence available only in context. A claim that no party can test may still be rhetorically attractive, but the absence of an evidential owner is itself a defect in the argument. 7.2 The projectibility declaration For confirmatory use, the evaluator should complete a projectibility declaration before opening the final holdout. Exploratory work can use the same fields, but it should reserve new evidence for confirmation. The declaration has ten parts. 1. Write one sourceâtarget claim. State what was observed and the new cases, interpretation, or outcome claimed. Replace legal capability transfers to practice with a statement such as: final memos produced by the frozen application and registered review procedure will have a post-review substantive-defect probability below 3% on Ontario termination-clause requests admitted by the written rule. 2. Type the source and target. Record the object, population, conditions, outcome, and period at each endpoint. If the claim is a capability attribution, it isnât an empirical endpoint; record instead the observable relation the downstream link needs and the evidence that would test it. A shared label doesnât count as endpoint identity. 3. Name the extension. Specify whether the claim generalizes to further items, interprets a score as a construct, extrapolates to another task, transports across systems or sites, predicts a later period, explains an outcome, or links evidence to a policy. Composite claims should be split until each boundary is visible. 4. List plausible defeaters. Name actual differences that could change the result: live rather than supplied retrieval, added irrelevant cases, another database index, a different reviewer instruction, a new model lineage, conflicting authority, or a later work period. The list is justified by prior failures, domain knowledge, process evidence, and stakeholder concerns, not by a generic field headed context. 5. Assign evidence and ownership. For each assumption or defeater, state the observation that bears on it and who can produce that observation. If no actor can observe whether defects reach clients, a claim about client consequences isnât ready for confirmation. 6. Match the sampling and holdout unit to the claim. Keep complete templates out for a template claim, client-file clusters out for a request claim, alteration types out for a context claim, lineages out for a system claim, offices out for a site claim, and later periods out for a temporal claim. Repeated outputs estimate variation within a requestâcondition pairing; they donât create new requests, systems, or periods. 7. Define the observation and loss. State exactly what one row represents and what is scored. Distinguish a request, a generated draft, a lawyer review, a final memo, a system build, and a work period. Keep ordinal hazard classes separate unless a defensible cardinal scale exists. Identify red-line outcomes that may not be averaged away. 8. Register support, defeat, and unresolved conditions. Give the estimate, uncertainty requirement, minimum sample or tail count, multiplicity treatment, and any fallback claim before examining confirmatory results. An inconclusive result remains inconclusive; it isnât evidence of equivalence or robustness. 9. Audit the interfaces. For every adjacent link, ask whether the upstream target is the downstream source in object, population, condition, outcome, and time; whether assumptions are compatible; and how dependence and uncertainty are propagated. Classify the relation as composition, convergence, replacement, or no evidential connection. 10. Separate the empirical claim from the decision. List feasible policies, affected parties, observed and prospective consequences, costs, hard constraints, and update triggers. State which empirical results enter the choice and which values determine the boundary. A validity judgment doesnât select a policy by itself. Table 5 gives a compact record. The final column prevents the declaration from becoming a set of decorative headings: every entry resolves into a value, sampling rule, or named document. Table 5: Minimum projectibility-declaration record. Field Required statement Inspectable form in the legal example Source observations What was directly recorded Item responses under the benchmark protocol; or draft and final defect scores on named request identifiers Target Cases or interpretation reached Requests satisfying the Ontario termination-clause inclusion rule during the declared period Object Complete tested configuration Model-build identifier, database-index date, retrieval settings, prompt, template, citation validator, and review instruction Unit and population What one observation is and how cases enter One logged research request; one request selected per client file; 200 later files per type Outcome and loss What counts as failure or cost Unresolvable citation, unsupported proposition, omitted controlling authority, final defect indicator, and lawyer minutes Defeaters Differences expected to matter Irrelevant retrieved cases, request type, index update, reviewer change, and new build Evidence design Holdout or intervention matched to each difference Later client-file clusters; answer-preserving retrieval alteration; separate request-type estimates Decision rule Support, defeat, and inconclusive conditions Registered exact bound, 3% tolerance, red-line rule, and prespecified narrower policy Interface Relation to adjacent claims Local study replaces rather than confirms the benchmark-to-use projection Update trigger Event that reopens the claim Model, index, prompt, request type, office, or review-procedure change; adverse incident 7.3 Designs matched to common projections The declaration changes the statistical design because the independent unit follows the claim. To generalize from observed to unobserved responses under the same item rules, hold out item or template clusters rather than random response rows. To extrapolate from benchmark questions to professional tasks, sample external tasks by a written rule and score them against independently prepared criteria. To claim robustness to a new alteration family, keep one whole alteration type out of tuning. To claim transfer to another system, evaluate a genuinely different build or lineage; repeated drafts from one build only make the estimate for that build more precise. To claim transfer to another office, run the complete task and review procedure there. To predict later performance, use a chronological holdout at the stated horizon. Construct claims need another design. A taxonomy needs content argument. A factor interpretation needs a declared task and system population, adequate variation, and checks that the fitted relations arenât artefacts of a narrow item set. A nomological claim needs predicted convergent, discriminant, process, and criterion relations. A causal explanation needs interventions or other identification conditions that discriminate plausible rivals. More benchmark items can improve precision within the tested operation while leaving each of those obligations untouched. Predictive claims make an especially common unit error visible. Suppose the question is whether benchmark profiles predict legal-research defects across systems. The independent units are systems or lineages, each with a benchmark profile and a defect estimate on the same held-out requests. With one system, its profile doesnât vary across requests and canât predict which request fails. A request-level model instead needs request-level predictors. Fitting the system profile to hundreds of request rows manufactures a sample size by repeating a constant. The same discipline applies to tails and groups. A worst-decile estimate selected after examining noisy item effects is subject to selection inflation; response splitting, cross-fitting, or a hierarchical model can address that problem for a stated estimand. None addresses transport to a new task type or system. Sparse group samples can leave a group-specific claim unresolved; a pooled estimate isnât automatically a licensed substitute. If the decision constrains the maximum defect tail across two request types, uncertainty has to account for selecting that maximum. 7.4 Reporting and updating A projectibility report should preserve continuous estimates and uncertainty even when a policy uses a boundary. The supported, defeated, or unresolved label summarizes the registered relation; it doesnât replace the estimate. Reports should also distinguish raw descriptive quantities from null-referenced, split-sample, cross-fitted, or model-based estimates and state what each targets. A visually similar number can answer a different question. After authorization, the evidential record continues. A shadow phase can run the application beside ordinary work without allowing it to determine the memo sent onward. The firm records draft defects, corrections, review time, final defects, and any incident. Continued use adds exposure and outcome records. A model update, retrieval-index change, revised prompt, new request type, changed review procedure, or shift in observed failures reopens the relevant links. The update neednât invalidate every part of the argument: an unchanged inclusion rule may remain useful while the system-transport link requires new evidence. Frequent model releases donât require every part of an evaluation to be repeated from scratch. An update reopens the links whose object, conditions, or assumptions it changes. Task definitions, inclusion rules, reference standards, scoring procedures, and evidence about unchanged parts of the workflow can often be reused; claims about model outputs require new evidence when the tested build changes materially. Version pinning, change logs, regression tests, rotated holdouts, and shadow evaluation can reduce the cost, while the extent of retesting should reflect the stakes, the opacity of the change, and the scope of the authorized claim. If a provider doesnât permit a build to be frozen or supply enough information to determine what changed, the result is a narrower or conditional authorization. Sometimes rapid model turnover makes the proposed use impractical; it doesnât make evidence about an earlier model applicable by default. Monitoring isnât a substitute for predeployment evidence. Allowing a high-stakes system to produce consequences in order to learn whether itâs safe simply relocates the cost of uncertainty. Nor is predeployment testing a substitute for monitoring, since later users, tasks, and systems can differ. The two answer temporal links on opposite sides of authorization. The audit can end with a narrow claim. Sparse or mismatched evidence doesnât imply that the system is generally unreliable; it implies that a broader claim lacks support. Conversely, a successful local study doesnât prove general capability. Precision about scope is an empirical result, not a rhetorical concession. 8 Limitations Projectibility isnât a complete theory of induction. The framework requires evaluators to identify plausible defeaters, but observations never determine that list by themselves. Domain theory, prior failures, causal knowledge, institutional experience, and affected stakeholders all shape which differences are investigated. Those sources can be incomplete or contested. A projectibility audit makes the commitments visible; it doesnât guarantee that every important difference has been anticipated. Variation in support for a fixed projection doesnât make projectibility a scalar property that can be calculated independently of a claim. The supported, defeated, and unresolved statuses depend on a declared target, loss, and evidential standard. That relativity can be abused by narrowing a claim until it becomes trivial. Prospective declaration and independent scrutiny limit such rescue, but they donât eliminate strategic framing. The practical question is whether the narrow claim remains useful enough to justify the proposed policy. The interface requirements arenât a replacement for specialized methods. Causal projections require causal identification; psychometric interpretations require appropriate measurement models and construct evidence; predictive claims require target-matched validation; and decisions require normative and institutional justification. The frameworkâs contribution is to keep those analyses from being joined merely because their outputs share a label. The legal results are constructed. They demonstrate how a complete audit assigns different statuses, not how any actual assistant performs. The reanalysis and simulations establish statistical distinctions under the released and simulated designs. They donât validate the projectibility framework against deployment outcomes or show that the reported benchmark quantities predict another target. The framework itself should be studied prospectively: evaluators could compare decisions, failure detection, and claim revision with and without an explicit interface audit. Some evidence will remain inaccessible. Proprietary training overlap can make model lineages uncertain; privacy and privilege can limit release of local cases; rare harms can make direct estimation impractical; and important future conditions may be unforeseeable. The response isnât to infer continuity by default. Evaluators can use secure audits, partial identification, stress tests, sensitivity analysis, and narrower authorization, while recording which links remain conditional. Finally, the procedure can be expensive. Sampling target tasks, preparing independent reference standards, testing complete workflows, and retaining outcome records require expertise and time. Those costs should be compared with the value and stakes of the proposed use. A low-consequence drafting aid may justify a lighter evidential standard than a system whose outputs enter court filings without independent verification. Claim-relative standards arenât evidential relativism; they connect the cost of being wrong to the strength and kind of support required. 9 Conclusion Validity-centred AI evaluation has made an essential correction: validity belongs to a proposed interpretation and use, not to a benchmark in isolation. This paper adds a non-composition principle for benchmark-to-use arguments. A benchmark result may support several individually warranted steps without warranting the chain assembled from them. A capability attribution may support one step, but it doesnât itself specify the downstream cases or criterion relation. Projectibility concerns whether one bounded extension is warranted. Goodman shows why source fit alone canât choose among rival extensions, and argument-based validity locates each extension among explicit claims and assumptions. A projectibility audit first asks whether links presented as adjacent meet in object, population, conditions, outcome, and period. A mismatch makes the proposed composition undefined unless separately bridged. If the fields align, warrant still fails to transmit when assumptions or effect modifiers conflict or when dependence and uncertainty are lost. The legal example shows the practical consequence. A developerâs score on supplied-text questions and a firmâs local study of reviewed memos can each be sound while remaining parallel. The local study may support mandatory-review use on one request type, defeat another, and leave a changed retrieval condition unresolved. It supplies new observations for a bounded inference rather than causing the benchmark score to carry into practice. A later build, another office, or another request type reopens a different link. Aggregation matters because joins require information at the resolution of the downstream claim. One stable mean can conceal balanced changes, concentrated deterioration, or stable serious failures. For AGI evaluation, broader domain coverage can likewise improve description without establishing commensurability, acceptable compensation, a common capability, or unrestricted reach. The warranted conclusion extends only across the targets and joins actually supported. Developers should report what their evaluations observed and the boundaries they tested; deployers should sample the tasks, workflows, and consequences available only in context. Supported bounded projections can accumulate when endpoints align and warrant transmits across each join. A warranted benchmark-to-use argument is a declared, revisable chain with evidence for each link, aligned endpoints, compatible assumptions, and dependence and uncertainty carried through. Appendix A Statistical Module for Paired-Condition Reports This appendix gives the fuller statistical module used in the empirical companion. Itâs optional: an evaluation should report the quantities its target claim requires rather than treating the module as a checklist. Let domains be indexed by g=1,âŚ,Gg=1,âŚ,G, items within domain by i=1,âŚ,Ngi=1,âŚ,N_g, and conditions by c, with baseline 0 and alteration p. Let siâgâcâ[0,1]s_igcâ[0,1] be an item score and δiâgâp=siâgâpâsiâgâ0 _igp=s_igp-s_ig0. The domain mean is agâc=1Ngââi=1Ngsiâgâc,a_gc= 1N_g _i=1^N_gs_igc, and the signed level change is Lgâp=agâpâagâ0L_gp=a_gp-a_g0. An overall equal-domain change Gâ1ââgLgâpG^-1 _gL_gp describes an equally weighted set of domains; it isnât a deployment mixture unless the target distribution gives the domains those weights. Item instability INSgâpINS_gp, the mean positive-change component Fgâp+F^+_gp, and the mean negative-change component FgâpâF^-_gp are INSgâp=1Ngââi|δiâgâp|,Fgâp+=1Ngââi(δiâgâp)+,Fgâpâ=1Ngââi(âδiâgâp)+.INS_gp= 1N_g _i| _igp|, F^+_gp= 1N_g _i( _igp)_+, F^-_gp= 1N_g _i(- _igp)_+. The identities Lgâp=Fgâp+âFgâpâL_gp=F^+_gp-F^-_gp and INSgâp=Fgâp++FgâpâINS_gp=F^+_gp+F^-_gp make cancellation explicit. For deterministic binary scores, F+F^+ and FâF^- are the incorrect-to-correct and correct-to-incorrect transition proportions. For a prespecified qâ(0,1]qâ(0,1], let mg=âqâNgâm_g= qN_g and order paired changes from smallest to largest. Domain-conditional worst-tail degradation is WTDq,gâp=â1mgââj=1mgδ(j)âgâp.WTD_q,gp=- 1m_g _j=1^m_g _(j)gp. Itâs an empirical expected-shortfall analogue applied to the loss âδ-δ [undefam, undefaau]. Because mgm_g rounds upward, the empirical tail mass is mg/Ngm_g/N_g. A positive value indicates deterioration in the selected lower tail; a negative value means even that tail improved on average. Absolute loss requires another variable. For target T, let R collect response, scorer, and downstream-outcome variation within an itemâcondition pairing. Let âiâgâcâR(T)â[0,1] ^(T)_igcRâ[0,1] be realized loss and Îźiâgâc(T)=Râ(âiâgâcâR(T)âŁi,g,c)Îź^(T)_igc=E_R( ^(T)_igcR i,g,c) conditional expected loss. Ordering Îź(T)Îź^(T) identifies high-risk cases; ordering realized draws of â(T) ^(T) identifies high-loss realizations. They coincide only under restrictive conditions such as no residual variation within a case. For a deployment-facing target, let JâźPTJ P_T index a joint draw over every varying feature in scope: system, task or episode, request type, alteration, operator, affected group, and time. With QV,TQ_V,T the quantile function of variable V under PTP_T, define upper expected shortfall ESq,T+âĄ(V)=1qââŤ1âq1QV,Tâ(u)âdu.ES^+_q,T(V)= 1q _1-q^1Q_V,T(u)\,du. Then a deployment-facing change tail can be written as ESq,T+âĄ(âδJ)ES^+_q,T(- _J), a case-risk tail as ESq,T+âĄ(ÎźJ(T))ES^+_q,T(Îź^(T)_J), and a realized-loss tail as ESq,T+âĄ(âJâR(T))ES^+_q,T( ^(T)_JR). The target distribution, not the visual similarity of benchmark items, determines the probability mass being ordered. These formulas support three different readings. With deterministic decoding and a fixed paired item set, theyâre exact descriptions of that set. Under a declared stochastic response and scoring protocol, they estimate latent item quantities from repeated trials and scorers. Inference to a population of new items, contexts, systems, or periods requires sampling or holdout from that population. Repeated responses from one item estimate within-item variation; they donât create a probability sample of items. Nonlinear quantities introduce selection problems. Absolute values create a positive response-sampling floor even under a null condition effect. Selecting the worst observed tail and estimating it on the same responses selects noise and exaggerates magnitude, a Type M error in the sense of [undefaac]. A baseline-only pseudo-null resampling procedure estimates the nonlinear statistic expected under a simulation with no condition difference. Subtracting that expectation yields a useful null-referenced diagnostic, not a general bias correction: under a real effect, variances and selection can change nonadditively. When latent item changes are the target, a hierarchical model can partially pool noisy extremes before the tail is summarized. A simpler response-half estimator selects items on one half of the trials and estimates their changes on the other. Cross-fitting reverses the halves and combines the results. This avoids reusing the same response noise, but its estimand is the latent effect of a set selected noisily, not necessarily the ideal tail selected from unobserved latent effects. Reports should name that difference. Uncertainty and resampling units follow the projectibility declaration. Preserve paired items, shared baselines, alteration realizations, scorers, client-file clusters, and model lineages. A bootstrap over individual prompts is wrong when several prompts come from one client file; the file cluster is the relevant unit. Holding out one irrelevant document doesnât test a new alteration family. Generating additional drafts from one model doesnât test a new system. Response-noise correction and projection across populations are separate problems. Appendix B Interface Audit Questions For each pair of adjacent links, the evaluator can use the following questions as a compact composition audit. 1. Do the endpoints denote the same object? If the upstream target is a base model and the downstream source is a retrieval application or reviewed workflow, what evidence links them? If either endpoint is named as a capability, which tested scoreâcriterion relation stands in for it? 2. Are the cases aligned? Does the downstream sampling rule admit the cases reached upstream, with the same exclusions, grouping, and target weights? 3. Is the outcome continuous? Does the first link deliver the variable the second consumes, or has accuracy become a factor score, defect probability, reviewed outcome, or consequence without a tested mapping? 4. Are operating conditions compatible? Which prompts, tools, databases, reviewers, sites, and periods differ, and which observations test those differences? 5. Are assumptions inherited consistently? Does the second link rely on an invariance, independence, or process claim contradicted or left open by the first? 6. Is the evidence as independent as claimed? Which items, templates, scorers, training runs, model lineages, or development choices are shared? 7. Has uncertainty been carried through? Are estimates, selection procedures, and unresolved qualitative assumptions propagated rather than replaced by point conclusions? 8. Is this actually composition? Would the downstream claim be supported if the upstream study didnât exist? If so, the relation may be replacement or convergence rather than composition. 9. What observation would defeat the composed claim? A chain with no stated defeater hasnât yet exposed its empirical content. 10. Does the conclusion stop at the reached target? Successful links warrant the declared path, not unspecified tasks, systems, or times beyond it. A completed audit neednât show that every interface is secure. Its first function is diagnostic. A missing link can lead to a new study, a narrower claim, a conditional conclusion, or rejection of the proposed use. What it canât legitimately produce is scope by accumulation: several passed evaluations donât add up to a claim whose interfaces were never tested. References [undef] Carlo Acerbi and Dirk Tasche âOn the Coherence of Expected Shortfallâ In Journal of Banking & Finance 26.7, 2002, p. 1487â1503 DOI: 10.1016/S0378-4266(02)00283-2 [undefa] Dominik Bachmann et al. âfl-IRT-ing with Psychometrics to Improve NLP Bias Measurementâ In Minds and Machines 34.37, 2024 DOI: 10.1007/s11023-024-09695-9 [undefb] Kristian Gonzalez Barman, Sascha Caron, Tom Claassen and Henk Regt âTowards a Benchmark for Scientific Understanding in Humans and Machinesâ In Minds and Machines 34.6, 2024 DOI: 10.1007/s11023-024-09657-1 [undefc] Andrew M. Bean et al. âMeasuring what Matters: Construct Validity in Large Language Model Benchmarksâ NeurIPS 2025 Track on Datasets and Benchmarks In arXiv preprint arXiv:2511.04703, 2025 DOI: 10.48550/arXiv.2511.04703 [undefd] Olivier Binette and Jerome P. Reiter âImproving the Validity and Practical Usefulness of AI/ML Evaluations Using an Estimands Frameworkâ, 2024 arXiv: https://arxiv.org/abs/2406.10366 [undefe] Denny Borsboom, Gideon J. Mellenbergh and Jaap Heerden âThe Concept of Validityâ In Psychological Review 111.4, 2004, p. 1061â1071 DOI: 10.1037/0033-295X.111.4.1061 [undeff] Samuel R. Bowman and George Dahl âWhat Will it Take to Fix Benchmarking in Natural Language Understanding?â In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, p. 4843â4855 DOI: 10.18653/v1/2021.naacl-main.385 [undefg] Richard Boyd âRealism, Anti-Foundationalism and the Enthusiasm for Natural Kindsâ In Philosophical Studies 61.1â2, 1991, p. 127â148 DOI: 10.1007/BF00385837 [undefh] Robert L. Brennan âGeneralizability Theoryâ New York: Springer, 2001 DOI: 10.1007/978-1-4757-3456-0 [undefi] Stefan Buijsman âAccuracy is not all you need! The Reasons to Require AI Explainabilityâ In Minds and Machines 36.14, 2026 DOI: 10.1007/s11023-026-09768-x [undefj] John B. Carroll âHuman cognitive abilities: A survey of factor-analytic studiesâ Cambridge University Press, 1993 DOI: 10.1017/CBO9780511571312 [undefk] Lee J. Cronbach and Paul E. Meehl âConstruct Validity in Psychological Testsâ In Psychological Bulletin 52.4, 1955, p. 281â302 DOI: 10.1037/h0040957 [undefl] Roel Dobbe and Anouk Wolters âToward Sociotechnical AI: Mapping Vulnerabilities for Machine Learning in Contextâ In Minds and Machines 34.12, 2024 DOI: 10.1007/s11023-024-09668-y [undefm] Susan E. Embretson âConstruct Validity: Construct Representation versus Nomothetic Spanâ In Psychological Bulletin 93.1, 1983, p. 179â197 DOI: 10.1037/0033-2909.93.1.179 [undefn] Timo Freiesleben âEstablishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networksâ In arXiv preprint arXiv:2603.15121, 2026 DOI: 10.48550/arXiv.2603.15121 [undefo] Timo Freiesleben and Sebastian Zezulka âThe Benchmarking Epistemology: Construct Validity for Evaluating Machine Learning Modelsâ Submitted 27 October 2025 In arXiv preprint arXiv:2510.23191, 2025 DOI: 10.48550/arXiv.2510.23191 [undefp] Andrew Gelman and John Carlin âBeyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errorsâ In Perspectives on Psychological Science 9.6, 2014, p. 641â651 DOI: 10.1177/1745691614551642 [undefq] Andrew Gelman and Eric Loken âThe garden of forking paths: Why multiple comparisons can be a problem, even when there is no âfishing expeditionâ or âp-hackingâ and the research hypothesis was posited ahead of timeâ Unpublished manuscript, Columbia University, 2013 URL: https://sites.stat.columbia.edu/gelman/research/unpublished/p_hacking.pdf [undefr] Nelson Goodman âFact, Fiction, and Forecastâ Cambridge, MA: Harvard University Press, 1983 [undefs] Dan Hendrycks et al. âA Definition of AGIâ v3, December 2025, 2025 DOI: 10.48550/arXiv.2510.18212 [undeft] David IliÄ and Gilles E. Gignac âEvidence of Interrelated Cognitive-Like Capabilities in Large Language Models: Indications of Artificial General Intelligence or Achievement?â In Intelligence 106, 2024, p. 101858 DOI: 10.1016/j.intell.2024.101858 [undefu] Jana Jung, Marlene Lutz, Indira Sen and Markus Strohmaier âDo Psychometric Tests Work for Large Language Models? Evaluation of Tests on Sexism, Racism, and Moralityâ In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) Association for Computational Linguistics, 2026, p. 8143â8173 DOI: 10.18653/v1/2026.eacl-long.380 [undefv] Michael T. Kane âValidating the Interpretations and Uses of Test Scoresâ In Journal of Educational Measurement 50.1, 2013, p. 1â73 DOI: 10.1111/jedm.12000 [undefw] Ralph L. Keeney and Howard Raiffa âDecisions with Multiple Objectives: Preferences and Value Tradeoffsâ Cambridge University Press, 1993 DOI: 10.1017/CBO9781139174084 [undefx] Muhammad Ali Khalidi âNatural Categories and Human Kinds: Classification in the Natural and Social Sciencesâ Cambridge: Cambridge University Press, 2013 DOI: 10.1017/CBO9780511998553 [undefy] Markus Langer, Kevin Baum and Nadine Schlicker âEffective Human Oversight of AI-Based Systems: A Signal Detection Perspective on the Detection of Inaccurate and Unfair Outputsâ Volume 35 (2025); Crossref records online-first publication in 2024 In Minds and Machines 35.1, 2025 DOI: 10.1007/s11023-024-09701-0 [undefz] Yu Lu Liu et al. âECBD: Evidence-Centered Benchmark Design for NLPâ In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Association for Computational Linguistics, 2024, p. 16349â16365 DOI: 10.18653/v1/2024.acl-long.861 [undefaa] Varun Magesh et al. âHallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolsâ In Journal of Empirical Legal Studies 22, 2025, p. 216â242 DOI: 10.1111/jels.12413 [undefab] Kevin S. McGrew âCHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence researchâ In Intelligence 37.1, 2009, p. 1â10 DOI: 10.1016/j.intell.2008.08.004 [undefac] William Meredith âMeasurement Invariance, Factor Analysis and Factorial Invarianceâ In Psychometrika 58.4, 1993, p. 525â543 DOI: 10.1007/BF02294825 [undefad] Samuel Messick âValidity of Psychological Assessment: Validation of Inferences from Personsâ Responses and Performances as Scientific Inquiry into Score Meaningâ In American Psychologist 50.9, 1995, p. 741â749 DOI: 10.1037/0003-066X.50.9.741 [undefae] Judea Pearl and Elias Bareinboim âExternal Validity: From Do-Calculus to Transportability Across Populationsâ In Statistical Science 29.4, 2014, p. 579â595 DOI: 10.1214/14-STS486 [undefaf] Benjamin C. Pierce âTypes and Programming Languagesâ Cambridge, MA: MIT Press, 2002 [undefag] Inioluwa Deborah Raji et al. âAI and the Everything in the Whole Wide World Benchmarkâ In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, 2021 arXiv: https://datasets-benchmarks-proceedings.neurips.c/paper/2021/hash/084b6fbb10729ed4da8c3d3f5a3ae7c9-Abstract-round2.html [undefah] R. Rockafellar and Stanislav Uryasev âConditional Value-at-Risk for General Loss Distributionsâ In Journal of Banking & Finance 26.7, 2002, p. 1443â1471 DOI: 10.1016/S0378-4266(02)00271-6 [undefai] Olawale Salaudeen et al. âMeasurement to Meaning: A Validity-Centered Framework for AI Evaluationâ Submitted 13 May 2025 In arXiv preprint arXiv:2505.10573, 2025 DOI: 10.48550/arXiv.2505.10573 [undefaj] W. Schneider and Kevin S. McGrew âThe Cattell-Horn-Carroll theory of cognitive abilitiesâ In Contemporary intellectual assessment: Theories, tests, and issues The Guilford Press, 2018, p. 73â163 [undefak] Yubo Wang et al. âMMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkâ In Advances in Neural Information Processing Systems 37, 2024, p. 95266â95290 DOI: 10.52202/079017-3018 [undefal] Yanzhe Zhang, Sanmi Koyejo and Diyi Yang âThe Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Contextâ arXiv preprint, version 2, 2026 DOI: 10.48550/arXiv.2607.12963 References [undefam] Carlo Acerbi and Dirk Tasche âOn the Coherence of Expected Shortfallâ In Journal of Banking & Finance 26.7, 2002, p. 1487â1503 DOI: 10.1016/S0378-4266(02)00283-2 [undefan] Dominik Bachmann et al. âfl-IRT-ing with Psychometrics to Improve NLP Bias Measurementâ In Minds and Machines 34.37, 2024 DOI: 10.1007/s11023-024-09695-9 [undefao] Kristian Gonzalez Barman, Sascha Caron, Tom Claassen and Henk Regt âTowards a Benchmark for Scientific Understanding in Humans and Machinesâ In Minds and Machines 34.6, 2024 DOI: 10.1007/s11023-024-09657-1 [undefap] Andrew M. Bean et al. âMeasuring what Matters: Construct Validity in Large Language Model Benchmarksâ NeurIPS 2025 Track on Datasets and Benchmarks In arXiv preprint arXiv:2511.04703, 2025 DOI: 10.48550/arXiv.2511.04703 [undefaq] Olivier Binette and Jerome P. Reiter âImproving the Validity and Practical Usefulness of AI/ML Evaluations Using an Estimands Frameworkâ, 2024 arXiv: https://arxiv.org/abs/2406.10366 [undefar] Denny Borsboom, Gideon J. Mellenbergh and Jaap Heerden âThe Concept of Validityâ In Psychological Review 111.4, 2004, p. 1061â1071 DOI: 10.1037/0033-295X.111.4.1061 [undefas] Samuel R. Bowman and George Dahl âWhat Will it Take to Fix Benchmarking in Natural Language Understanding?â In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, p. 4843â4855 DOI: 10.18653/v1/2021.naacl-main.385 [undefat] Richard Boyd âRealism, Anti-Foundationalism and the Enthusiasm for Natural Kindsâ In Philosophical Studies 61.1â2, 1991, p. 127â148 DOI: 10.1007/BF00385837 [undefau] Robert L. Brennan âGeneralizability Theoryâ New York: Springer, 2001 DOI: 10.1007/978-1-4757-3456-0 [undefav] Stefan Buijsman âAccuracy is not all you need! The Reasons to Require AI Explainabilityâ In Minds and Machines 36.14, 2026 DOI: 10.1007/s11023-026-09768-x [undefaw] John B. Carroll âHuman cognitive abilities: A survey of factor-analytic studiesâ Cambridge University Press, 1993 DOI: 10.1017/CBO9780511571312 [undefax] Lee J. Cronbach and Paul E. Meehl âConstruct Validity in Psychological Testsâ In Psychological Bulletin 52.4, 1955, p. 281â302 DOI: 10.1037/h0040957 [undefay] Roel Dobbe and Anouk Wolters âToward Sociotechnical AI: Mapping Vulnerabilities for Machine Learning in Contextâ In Minds and Machines 34.12, 2024 DOI: 10.1007/s11023-024-09668-y [undefaz] Susan E. Embretson âConstruct Validity: Construct Representation versus Nomothetic Spanâ In Psychological Bulletin 93.1, 1983, p. 179â197 DOI: 10.1037/0033-2909.93.1.179 [undefaaa] Timo Freiesleben âEstablishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networksâ In arXiv preprint arXiv:2603.15121, 2026 DOI: 10.48550/arXiv.2603.15121 [undefaab] Timo Freiesleben and Sebastian Zezulka âThe Benchmarking Epistemology: Construct Validity for Evaluating Machine Learning Modelsâ Submitted 27 October 2025 In arXiv preprint arXiv:2510.23191, 2025 DOI: 10.48550/arXiv.2510.23191 [undefaac] Andrew Gelman and John Carlin âBeyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errorsâ In Perspectives on Psychological Science 9.6, 2014, p. 641â651 DOI: 10.1177/1745691614551642 [undefaad] Andrew Gelman and Eric Loken âThe garden of forking paths: Why multiple comparisons can be a problem, even when there is no âfishing expeditionâ or âp-hackingâ and the research hypothesis was posited ahead of timeâ Unpublished manuscript, Columbia University, 2013 URL: https://sites.stat.columbia.edu/gelman/research/unpublished/p_hacking.pdf [undefaae] Nelson Goodman âFact, Fiction, and Forecastâ Cambridge, MA: Harvard University Press, 1983 [undefaaf] Dan Hendrycks et al. âA Definition of AGIâ v3, December 2025, 2025 DOI: 10.48550/arXiv.2510.18212 [undefaag] David IliÄ and Gilles E. Gignac âEvidence of Interrelated Cognitive-Like Capabilities in Large Language Models: Indications of Artificial General Intelligence or Achievement?â In Intelligence 106, 2024, p. 101858 DOI: 10.1016/j.intell.2024.101858 [undefaah] Jana Jung, Marlene Lutz, Indira Sen and Markus Strohmaier âDo Psychometric Tests Work for Large Language Models? Evaluation of Tests on Sexism, Racism, and Moralityâ In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) Association for Computational Linguistics, 2026, p. 8143â8173 DOI: 10.18653/v1/2026.eacl-long.380 [undefaai] Michael T. Kane âValidating the Interpretations and Uses of Test Scoresâ In Journal of Educational Measurement 50.1, 2013, p. 1â73 DOI: 10.1111/jedm.12000 [undefaaj] Ralph L. Keeney and Howard Raiffa âDecisions with Multiple Objectives: Preferences and Value Tradeoffsâ Cambridge University Press, 1993 DOI: 10.1017/CBO9781139174084 [undefaak] Muhammad Ali Khalidi âNatural Categories and Human Kinds: Classification in the Natural and Social Sciencesâ Cambridge: Cambridge University Press, 2013 DOI: 10.1017/CBO9780511998553 [undefaal] Markus Langer, Kevin Baum and Nadine Schlicker âEffective Human Oversight of AI-Based Systems: A Signal Detection Perspective on the Detection of Inaccurate and Unfair Outputsâ Volume 35 (2025); Crossref records online-first publication in 2024 In Minds and Machines 35.1, 2025 DOI: 10.1007/s11023-024-09701-0 [undefaam] Yu Lu Liu et al. âECBD: Evidence-Centered Benchmark Design for NLPâ In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Association for Computational Linguistics, 2024, p. 16349â16365 DOI: 10.18653/v1/2024.acl-long.861 [undefaan] Varun Magesh et al. âHallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolsâ In Journal of Empirical Legal Studies 22, 2025, p. 216â242 DOI: 10.1111/jels.12413 [undefaao] Kevin S. McGrew âCHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence researchâ In Intelligence 37.1, 2009, p. 1â10 DOI: 10.1016/j.intell.2008.08.004 [undefaap] William Meredith âMeasurement Invariance, Factor Analysis and Factorial Invarianceâ In Psychometrika 58.4, 1993, p. 525â543 DOI: 10.1007/BF02294825 [undefaaq] Samuel Messick âValidity of Psychological Assessment: Validation of Inferences from Personsâ Responses and Performances as Scientific Inquiry into Score Meaningâ In American Psychologist 50.9, 1995, p. 741â749 DOI: 10.1037/0003-066X.50.9.741 [undefaar] Judea Pearl and Elias Bareinboim âExternal Validity: From Do-Calculus to Transportability Across Populationsâ In Statistical Science 29.4, 2014, p. 579â595 DOI: 10.1214/14-STS486 [undefaas] Benjamin C. Pierce âTypes and Programming Languagesâ Cambridge, MA: MIT Press, 2002 [undefaat] Inioluwa Deborah Raji et al. âAI and the Everything in the Whole Wide World Benchmarkâ In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, 2021 arXiv: https://datasets-benchmarks-proceedings.neurips.c/paper/2021/hash/084b6fbb10729ed4da8c3d3f5a3ae7c9-Abstract-round2.html [undefaau] R. Rockafellar and Stanislav Uryasev âConditional Value-at-Risk for General Loss Distributionsâ In Journal of Banking & Finance 26.7, 2002, p. 1443â1471 DOI: 10.1016/S0378-4266(02)00271-6 [undefaav] Olawale Salaudeen et al. âMeasurement to Meaning: A Validity-Centered Framework for AI Evaluationâ Submitted 13 May 2025 In arXiv preprint arXiv:2505.10573, 2025 DOI: 10.48550/arXiv.2505.10573 [undefaaw] W. Schneider and Kevin S. McGrew âThe Cattell-Horn-Carroll theory of cognitive abilitiesâ In Contemporary intellectual assessment: Theories, tests, and issues The Guilford Press, 2018, p. 73â163 [undefaax] Yubo Wang et al. âMMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkâ In Advances in Neural Information Processing Systems 37, 2024, p. 95266â95290 DOI: 10.52202/079017-3018 [undefaay] Yanzhe Zhang, Sanmi Koyejo and Diyi Yang âThe Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Contextâ arXiv preprint, version 2, 2026 DOI: 10.48550/arXiv.2607.12963