Paper deep dive
Efficient Detection of Bad Benchmark Items with Novel Scalability Coefficients
Michael Hardy, Joshua Gilbert, Benjamin Domingue
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/31/2026, 1:31:46 AM
Summary
The paper introduces a novel family of nonparametric scalability coefficients, specifically the 'signed isotonic R^2', for the efficient detection of globally problematic items in large-scale AI benchmarks and human assessments. By leveraging interitem isotonic regression, the method captures maximal monotone signal between items without assuming parametric forms, outperforming traditional psychometric diagnostics in ranking bad items for human review.
Entities (7)
Relation Signals (3)
Michael Hardy → authored → Efficient Detection of Bad Benchmark Items with Novel Scalability Coefficients
confidence 100% · Efficient Detection of Bad Benchmark Items with Novel Scalability Coefficients Michael Hardy * 1
signed isotonic R^2 → appliedto → MMLU
confidence 95% · Across three AI benchmark datasets (HS Math, GSM8K, MMLU)... the signed isotonic R^2 consistently achieves top-tier AUC
signed isotonic R^2 → detects → bad items
confidence 95% · The central contribution is the signed isotonic R^2, which measures the maximal proportion of variance in one item explainable by a monotone function of another... for efficiently detecting globally bad items
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The validity of assessments, from large-scale AI benchmarks to human classrooms, depends on the quality of individual items, yet modern evaluation instruments often contain thousands of items with minimal psychometric vetting. We introduce a new family of nonparametric scalability coefficients based on interitem isotonic regression for efficiently detecting globally bad items (e.g., miskeyed, ambiguously worded, or construct-misaligned). The central contribution is the signed isotonic $R^2$, which measures the maximal proportion of variance in one item explainable by a monotone function of another while preserving the direction of association via Kendall's $\tau$. Aggregating these pairwise coefficients yields item-level scores that sharply separate problematic items from acceptable ones without assuming linearity or committing to a parametric item response model. We show that the signed isotonic $R^2$ is extremal among monotone predictors (it extracts the strongest possible monotone signal between any two items) and show that this optimality property translates directly into practical screening power. Across three AI benchmark datasets (HS Math, GSM8K, MMLU) and two human assessment datasets, the signed isotonic $R^2$ consistently achieves top-tier AUC for ranking bad items above good ones, outperforming or matching a comprehensive battery of classical test theory, item response theory, and dimensionality-based diagnostics. Crucially, the method remains robust under the small-n/large-p conditions typical of AI evaluation, requires only bivariate monotone fits computable in seconds, and handles mixed item types (binary, ordinal, continuous) without modification. It is a lightweight, model-agnostic filter that can materially reduce the reviewer effort needed to find flawed items in modern large-scale evaluation regimes.
Tags
Links
- Source: https://arxiv.org/abs/2603.24999v2
- Canonical: https://arxiv.org/abs/2603.24999v2
Trouble viewing inline? Open PDF directly →
Full Text
74,130 characters extracted from source content.
Expand or collapse full text
Efficient Detection of Bad Benchmark Items with Novel Scalability Coefficients Michael Hardy * 1 Joshua Gilbert 2 Benjamin Domingue 1 Abstract The validity of assessments, from large-scale AI benchmarks to human classrooms, depends on the quality of individual items, yet modern eval- uation instruments often contain thousands of items with minimal psychometric vetting. We introduce a new family of nonparametric scal- ability coefficients based on interitem isotonic regression for efficiently detecting globally bad items (e.g., miskeyed, ambiguously worded, or construct-misaligned). The central contribution is the signed isotonic R 2 , which measures the max- imal proportion of variance in one item explain- able by a monotone function of another while pre- serving the direction of association via Kendall’s τ. Aggregating these pairwise coefficients yields item-level scores that sharply separate problem- atic items from acceptable ones without assum- ing linearity or committing to a parametric item response model. We prove that the signed iso- tonicR 2 is extremal among monotone predictors (it extracts the strongest possible monotone sig- nal between any two items) and show that this optimality property translates directly into practi- cal screening power. Across three AI benchmark datasets (HS Math, GSM8K, MMLU) and two human assessment datasets, the signed isotonic R 2 consistently achieves top-tier AUC for rank- ing bad items above good ones, outperforming or matching a comprehensive battery of classical test theory, item response theory, and dimensionality- based diagnostics. Crucially, the method remains robust under the small-n/large-p conditions typi- cal of AI evaluation, requires only bivariate mono- tone fits computable in seconds, and handles mixed item types (binary, ordinal, continuous) without modification. It is a lightweight, model- agnostic filter that can materially reduce the re- viewer effort needed to find flawed items in mod- ern large-scale evaluation regimes. 1 Stanford University, CA, United States 2 Harvard Univer- sity, MA, United States. Correspondence to: Michael Hardy <hardym[ατ ]stanford[·]edu>. Preprint. March 30, 2026. 1. Introduction The validity of any assessment–from classroom exams to large-scale AI benchmarks–depends on the quality of its individual items. Even a small number of flawed items (e.g., incorrect answer keys, scoring bugs, ambiguity, construct drift) can distort scores, undermine rankings, and lead to in- valid conclusions (Casabianca, 2025; Zhang et al., 2025). In high-stakes human testing, item development is typically ac- companied by extensive qualitative review and quantitative pilot analyses, and problematic items are usually removed before operational deployment. By contrast, many modern AI benchmarks are assembled at scale (often synthetically), frequently rely on automated grading for unstructured re- sponses, and may contain thousands of items with minimal psychometric vetting. As a result, benchmark scores can be sensitive to a small set of bad items, and the burden of validation is shifted to downstream users (Casabianca, 2025; Zhang et al., 2025; Truong et al., 2025; Reuel et al., 2024; Salaudeen et al., 2025). This paper focuses on global bad-item detection: efficiently prioritizing items for human review when some items are flawed in ways that degrade measurement overall (as distinct from differential item functioning, which concerns group- specific item behavior). The core problem is practical: given a response matrix with many items, how can we rank items so that the worst ones are found quickly? We propose a new family of nonparametric scalability co- efficients based on interitem isotonic regression. The main contribution is a signed coefficient derived from the isotonic- regression coefficient of determination, which we call the signed isotonicR 2 . For a response matrixY ∈ R n×p and for each ordered item pair(i,j), we fit the best monotone relationship between itemiand itemjand measure the fraction of variance inY j explained by a monotone function ofY i . Aggregating these pairwise associations yields item- level scores that sharply separate globally bad items from acceptable ones. Empirically, these coefficients improve practical detection efficiency (measured via AUC for rank- ing bad items above good ones) and remain computationally scalable for large benchmark regimes. 1 arXiv:2603.24999v2 [stat.AP] 27 Mar 2026 Efficient Detection of Bad Items 2. Background and Motivation 2.1. What counts as a “bad item”? An item is “bad” when it systematically breaks the intended measurement logic of the instrument. Across human assess- ments and AI benchmarks, we consider four common global failure modes: 1. Bad key: the labeled correct answer is wrong. 2. Bad grading: the scoring procedure is incorrect or inconsistently applied. 3. Ambiguity: multiple defensible answers or unclear prompt/specification. 4.Construct misalignment: the item elicits skills out- side the intended construct (e.g., format quirks, spuri- ous cues, irrelevant knowledge). These failure modes typically reduce an item’s coherence with the rest of the test, even when the overall instrument is intended to measure a dominant latent trait. Our aim is to detect such globally problematic items; we do not address fairness questions or group-conditional anomalies (e.g., DIF 1 ) in this work. 2.2. Existing tools are often inefficient in large regimes A wide range of item-quality indices exist across CTT, IRT, nonparametric scaling (Zijlmans et al., 2018a;b). These indices differ along three practical axes: Dynamic: “bad relative to what?” Common compar- isons include • Interitem (item vs. item): correlations, agreement, mutual information, pairwise scalability. •Item–rest (item vs. total/others): corrected item–total correlation, item-rest regression. •Item-drop (change in test statistic when removed): ∆α, ∆ reliability. •Parameter-based (item as a member of a model): IRT discrimination/fit, factor loadings. Baseline: “bad with respect to what property?” Dif- ferent statistics implicitly target different notions of misfit: linear association, mean differences, variance explained, local independence violations, or parameter inconsistency. 1 Items that demonstrate DIF may also be flagged by globally problematic items. Instead of a group differential item functioning, for this study, the reference group is the entire population. Assumptions: “bad under what model?”Many indices are efficient only when their assumptions hold (e.g., linear- ity, parametric ICC forms, well-behaved latent distributions, sufficient sample size). In AI benchmarks, these assump- tions are often strained: item types are mixed (binary/or- dinal/continuous scores), grading noise may be structured, and data may be sparse or highly imbalanced. Moreover, some widely used indices are direction-blind (they empha- size magnitude but not sign) or collapse nonlinear monotone effects into weaker linear proxies. 2.3. Item “behavior” as interitem social compatibility A useful intuition is to treat items as a team: good items “get along” with other items that measure the same construct. If a test is approximately unidimensional responses to any two items should be positively and monotonically related (up to noise and local dependence; (Sijtsma, 2009; Revelle & Condon, 2025; Mair & Leeuw, 2015; Ten Berge & So ˇ can, 2004)). Bad items often exhibit one or more of: (i) weak association with most other items, (i) non-monotone behav- ior (e.g., middle-ability respondents outperform high-ability respondents due to ambiguity or grading), (i) inversions (negative association) consistent with miskeying or system- atic scoring reversal. This motivates interitem measures of fit: quantify how well each item participates in the network of expected monotone dependencies. 2.4. Desiderata for scalable bad-item detection We seek item-level indices that are: 1.Practically efficient: prioritize bad items early in a review queue. 2.Computationally efficient: feasible for largen(re- sponses) and very large p (items). 3. Model-agnostic: avoid reliance on strict parametric IRT forms. 4.Monotonicity-aware: exploit the key qualitative con- straint of unidimensional measurement. 5.Type-flexible: handle mixed outcome types and asym- metric relationships. 2.5. Why isotonic regression? Isotonic regression provides a principled way to extract max- imal monotone signal between variables without committing to a parametric functional form. For item analysis, this is attractive because the key expectation under a dominant latent trait is monotonic dependence, not necessarily linear 2 Efficient Detection of Bad Items dependence. By measuring how much of an item’s variabil- ity can be explained by a monotone function of another item, we obtain a natural nonparametric analogue of “scalability” that (i) directly targets the monotonicity assumption and (i) can preserve directionality (through a signed association), enabling sharper detection of inversions such as miskeys. If a set of items measures a single latent trait, then perfor- mance on any two items should be positively and monotoni- cally related. The strength of this monotonic relationship, aggregated across all item pairs, becomes a powerful indi- cator of item fit. The primary contribution is a new family of nonparametric, nonlinear scalability coefficients built by maximizing the information gained by assuming monotonic- ity. Taking inspiration from Loevinger’sH(loe, 1947), 2 Mokken scaling (Wind, 2017; Mokken, 2011; Sijtsma & Molenaar, 2002; van der Ark, 2007), and advances in iso- tonic regression in IRT (Lee, 2002; 2007; Lee et al., 2009; Luzardo & Rodr ́ ıguez, 2015; Yu, 2022), this approach re- frames item fit as a function of its consistent, monotonic behavior with all other items on the scale. 3. Methods 3.1. Notation and setup LetY ∈ R n×p be the response matrix with respondentsr ∈ 1,...,nand itemsi ∈ 1,...,p. The column vector for itemiisy i ∈ R n . Responses may be binary, ordinal, or continuous; we assume higher values reflect greater success on the construct (after any required recoding). Our goal is to compute, for each itemi, a scalar badness score (or conversely a fit/scalability score) used to rank items for review. 3.2. Interitem isotonic regression Fix an ordered pair of distinct items (i,j). We model y j as a monotone function of y i : y rj ≈ f i→j (y ri ), f i→j ∈F ↑ ,(1) whereF ↑ is the set of non-decreasing functions on the ob- served support of y i . The isotonic regression estimator is ˆ f i→j ∈ arg min f∈F ↑ n X r=1 (y rj − f (y ri )) 2 ,(2) computed efficiently using the Pool Adjacent Violators Al- gorithm (PAVA; (Busing, 2022)) after sorting observations byy i (with standard handling of ties). This yields fitted values ˆy (i→j) rj = ˆ f i→j (y ri ). Asymmetry and mixed item types. Because the regres- sion is directional, the strength ofi → jneed not equal 2 The initial inspiration for these proposed solutions. j → i, which is useful when item types differ (e.g., a ordi- nal partial-credit item may monotonically explain a binary item differently than vice versa). This asymmetry is a fea- ture: it permits detection based on predictable directional structure rather than forcing symmetry. 3.3. Signed isotonic R 2 as a pairwise scalability coefficient We quantify the monotone explanatory power of itemi for itemjusing an isotonic analogue of the coefficient of determination: R 2 i→j = 1− P n r=1 y rj − ˆy (i→j) rj 2 P n r=1 (y rj − ̄y j ) 2 , ̄y j = 1 n n X r=1 y rj . (3) To preserve directionality (inversions) we attach a sign based on the global direction of association betweeny i andy j based on Kendall’s τ . Let s ij = sign(τ (y i ,y j )),(4) with the convention thats ij = 0if the correlation is nu- merically0or undefined (e.g., zero variance). The signed isotonic coefficient is M i→j = s ij R 2 i→j .(5) Intuitively,M i→j estimates the proportion of monotone variance explained, while preserving whether the relation- ship aligns with the expected positive direction. Bad keys and systematic grading reversals tend to induce negative or unusually small signed values across many pairs. 3.4. A formal interpretation: optimality among monotone predictors The empirical advantage of isotonicR 2 is explained by an optimization property: it measures the maximal proportion of variance explainable by any monotone transformation. Proposition 3.1 (Maximal monotone explained variance). Fix item pair(i,j)and consider predictors ofY j of the form f (Y i )wherefis non-decreasing. Let ˆ f be the isotonic regression solution as defined in Eq. 2. Then for any non- decreasing g, n X r=1 (y rj − ˆ f (y ri )) 2 ≤ n X r=1 (y rj − g(y ri )) 2 , and thereforeR 2 i→j computed from ˆ fis the largest achiev- able R 2 among monotone predictors. 3 Efficient Detection of Bad Items 3.5. From pairwise coefficients to item-level badness scores For each focal itemi, we aggregate its pairwise signed isotonic relationships with all other items: Fit(i) = 1 p− 1 X j̸=i M i→j .(6) We then rank items by increasingFit(i)(lower implies more suspicious). Variants we consider in ablations (not required for using the method) include: (i) symmetrized aggregation M iso = 1 2 (M i→j +M j→i ) , (i) robust aggregation using trimmed means/medians to reduce sensitivity to local depen- dence clusters, (i) nonnegative aggregation using|M i→j | when direction is known to be unreliable. 3.6. Computational considerations For each ordered pair(i,j), isotonic regression reduces to sorting byy i and a linear-time PAVA pass. In practice, when many items are binary or low-cardinality, sorting can be implemented via counting/bucketing, making pairwise fits fast. The full pairwise matrix isO(p 2 )fits; we therefore use two scalable strategies depending on regime: •All-pairs for moderatep(typical in human assess- ments). •Subsampled neighbors for very largep(typical in AI benchmarks): computeFit(i)using a fixed-size set of comparison items peri(random, stratified by difficulty, or chosen via a computationally cheap pre-screen such as correlation). This preserves ranking quality while reducing compute to O(pK) fits for K ≪ p. 3.7. Evaluation protocol: efficiency as ranking performance We evaluate item-detection efficiency by treating each met- ric as a scoring function that ranks items from most to least suspicious, then computing the area under the ROC curve (AUC) for classifying known bad items. Formally, for item scoresS(i)where larger indicates “more bad” (we use S(i) =−Fit(i)), AUC equals Pr(S(i bad ) > S(i good )),(7) the probability that a randomly chosen bad item is ranked above a randomly chosen good item. This directly reflects expected reviewer time saved: higher AUC concentrates bad items earlier in the queue. Thus we are evaluating bad- item detection as a ranking problem: a statistic assigns each itemia scoreS(i), and we sort items from most suspicious to least suspicious. Ground-truth labelsL(i)∈ 0, 1 indicate whether an item is globally bad (Sec. 2; not DIF). Performance is measured by how effectively the ranking prioritizes bad items for review. 3.8. Baselines To contextualize gains, we compare signed isotonic R 2 against a broad suite of established CTT/IRT and association-based indices, covering: (i) interitem associ- ation (e.g., linear and information-theoretic dependence), (i) item-rest statistics (e.g., corrected item-total correlation and monotone dependence), (i) parameter-based proxies (e.g., discrimination or explained-variance loadings), (iv) item-drop deltas (e.g., reliability changes). All methods produce an item ranking, evaluated under the same AUC protocol. 3.9. Scope Our methods target global item misfit detectable via dis- rupted monotone coherence with the rest of a scale. We do not attempt to diagnose the causal source of misfit (key- ing vs. ambiguity vs. construct drift), nor do we conduct group-conditional DIF analyses; rather, we provide a compu- tationally efficient front-end filter that materially improves the rate at which human reviewers find problematic items in both traditional assessments and large AI benchmarks. 4. Empirical Approach 4.1. Datasets and labels We use both human assessment data and AI benchmark data. Human datasets reflect conventional test development pipelines where few bad items survive, but when they do, they are consequential. AI benchmark datasets represent large item banks assembled without comparable validation pipelines; these are the main motivation for scalable detec- tion tools. Each dataset includes an externally curated list of globally bad items (bad key, bad grading, ambiguity, construct mis- alignment). For experiments involving bootstraps, we en- force a minimum number of bad items in each resample so that AUC is well-defined and not dominated by degenerate cases. 4.1.1. APPLICATION DOMAIN: AI BENCHMARK ASSESSMENT The emergence of large language models (LLMs) has cre- ated unprecedented challenges for psychometric evaluation. AI benchmarks often lack human comparison data, hav- ing been designed specifically for machine evaluation. As model performance approaches saturation on existing bench- marks, the identification of problematic items becomes cru- cial—differences of one or two poor items can determine rankings among state-of-the-art models. 4 Efficient Detection of Bad Items alpha_drop cneg cmean g_15 Zi Hi a1_2 a1_3 z_outfit_3 adj_depth tau iso 0100200300400500 MMLU5 Bad Item Detection Metrics Data: MMLU5 Lite Figure 1. Examples of Differences in Item Detection Efficiencies based on Technique:Bad Item detection as ordered by each item-fit metric. iso =M iso ; tau =M τ ; adjdepth = Isolation Forest Tree Depth (anomaly detection), zoutfit3 = standardized absolute 3PL outfit statistic, a13 and a12 = discrimination parameter for 3PL and 2PL, respectively; Hi =H i ; Zi =Z i , g15 = loading on the general factor of a 15-factor estimation of McDonald’sω; cmean = Mean inter-item tetrachoric correlation; cneg = proportion of inter-item tetrachoric correlations ¡ 0; alphadrop = benchmark reliability (Cronbach’s α) with item removed 4.1.2. UNIDIMENSIONALITY ASSUMPTION IN AI ASSESSMENT While human cognitive assessment typically reveals multi- dimensional ability structures, AI models present a unique case. Contemporary LLMs share a fundamental training ob- jective: autoregression of Internet text. Despite subsequent modifications through instruction tuning and reinforcement learning from human feedback (RLHF), we hypothesize that this primary objective dominates performance across diverse benchmarks (McCoy et al., 2023). This suggests that for AI evaluation contexts, a unidimen- sional latent ability model may be appropriate, or alterna- tively, that the autoregressive ability component substan- tially outweighs other potential factors. We validate this assumption using the MMLU (Massive Multitask Language Understanding) dataset, which spans five distinct subject areas. 4.1.3. AI AND BENCHMARKS AND GROUND TRUTH We employed multiple datasets with independently validated item quality assessments, allowing for objective evaluation of metric performance. Each dataset contained items pre- viously identified as problematic through expert review or statistical flagging procedures. The responses come from Holistic Evaluations of Language Models (HELM) datasets (Liang et al., 2023). 3 From HELM, we utilize the model responses and combined bad item labels of the HS Math, GSM8K, and HELM-Lite 5-subject MMLU Benchmark from the (Truong et al., 2025; Vendrow et al., 2025) studies. 3 https://crfm.stanford.edu/helm/ 4.1.4. HUMAN DATASETS Publicly available datasets datasets with “bad” items are rare, as bad items are typically removed before public re- lease. We were fortunate enough to obtain access to two human data, in addition to the main AI benchmark datasets above. , we have two human datasets where “bad” items have been identified. One is publicly available and found within the Item Response Warehouse. It has 31 items and 7780 respondents. The second human dataset is a private dataset from an educational intervention pilot study in high school science. The flagged items were identified as either ambiguous or misaligned with the intended construct out of 20 items, each having 106 individual responses. We conjec- ture that pilot studies, which are rarely accessible publicly, would be a natural format for using these analyses. 4.2. Evaluation metric: AUC as reviewer-efficiency We operationalize efficiency as the area under the ROC curve (AUC) when metrics are used to rank-order items by suspected quality. AUC represents the probability that a classifier will rank a randomly chosen instance of a bad item the higher than a randomly chosen instance from items without issues. For any item scoring ruleS, we compute the area under the ROC curve (AUC) for classifying items using S(i): AUC(S) = Pr(S(i bad ) > S(i good ))(8) for i bad ∼ L = 1, i good ∼ L = 0. AUC has a direct operational interpretation: it is the proba- bility that a randomly chosen bad item is ranked ahead of a randomly chosen good item. Thus, higher AUC corresponds to less manual effort to discover flawed items. Specifically, 5 Efficient Detection of Bad Items we simulate the workflow of a human reviewer who exam- ines items in order of decreasing fit (i.e., starting with the most problematic items as identified by each metric). Thus, the efficiency score is alternatively defined as: Efficiency = AUC = Z 1 0 TPR(FPR)d(FPR) where TPR (True Positive Rate) represents the proportion of truly problematic items identified, and FPR (False Positive Rate) represents the proportion of acceptable items incor- rectly flagged. The estimation of the AUC was implemented with the pROC package (Robin et al., 2011). Sign convention. Many fit indices (including our signed isotonic coefficient) are “higher is better.” For AUC, we use a consistent convention that largerS(i)means “more suspicious” by negating fit measures when needed. 4.3. Competing methods We compare the signed isotonicR 2 coefficient against rep- resentative families of item-level diagnostics: •Interitem association (pairwise, aggregated to items): agreement-based statistics (e.g., SMC,κ), correlation- based statistics (e.g.,φ), and information-theoretic de- pendence (MI, symmetric uncertainty). •Item–rest statistics: correlation or monotone depen- dence between item i and the rest score X −i . •Model-based parameters (when feasible): IRT dis- crimination (a) and difficulty (d) parameters. Our method is computed from interitem isotonic regres- sions (Sec. 3); item-level scores are obtained by aggregating signed monotone R 2 i→j across j ̸= i. 5. Experiments 5.1. Experiment 1: full-dataset AUC across analysis levels The first experiment evaluates detection efficiency on the full datasets. For each method, we compute an item score on all five datasets and report the AUC (Table 1). Our first experiment answers: When the dataset is fixed, which statistics best prioritize globally bad items? 5.2. Experiment 2: subsampled benchmark stress test (AI) AI benchmark regimes often face small-n/large-pcondi- tions: relatively few “respondents” (models, prompts, or runs) and many items. To probe robustness in this regime, we runB = 20subsampling trials for each AI benchmark. In each trial, we sample without replacement p = 200 items and n = 50 respondents, compute a large battery of item statistics, and record AUC for each statistic on that subsample. We then aggregate performance across bootstraps by ranking statistics within each trial (by AUC) and averaging ranks; the comprehensive aggregated table is reported in Appendix Table A. This experiment evaluates the consistency of the many estimates in our statistical suite to generalize the statistical findings of the first experiment and reduce any sensitivity to particular combinations of items. Thus, our second experiment answers: Which methods re- main reliable when both items and respondents are not fixed 5.3. Experiment 3: scalability across n× p regimes (interitem-only) The third experiment isolates computationally light in- teritem methods to explore scaling behavior as the data aspect ratio varies. This setting is especially important for benchmarks that evolve over time (items added/removed) and for emerging evaluation settings where n is limited. We consider two AI benchmarks (GSM8K and MMLU) and generate 5600 resamples total: • p∈2 3 , 2 4 ,..., 2 9 =8, 16,..., 512, • nvaries over a grid of proportions of the original re- spondents, including deciles in[0.4, 1.0]and an extrap- olated setting at 1.1, •for each(n,p)configuration, we run100resamples with replacement (subject to a minimum number of bad items). For each resample, we compute AUC for each interitem statistic and then average AUC over the100resamples per(n,p). Finally, to summarize overall robustness across regimes, we apply a Borda count aggregation: each(n,p) configuration votes on a total ordering of methods by AUC, and we sum votes across configurations. The third experiment answers: Which interitem metrics remain strong across wide changes in sample size and test length? 5.4. Estimation details Detailed results are presented in the appendix, demonstrat- ing the practical advantages of these novel scalability formu- lations for modern psychometric applications. Experiment 6 Efficient Detection of Bad Items Table 1. Bad-item detection efficiency (AUC) for association-based baselines. Interitem methods score an item by its mean pairwise association with all other items. Item-rest methods score an item by its association with the rest-scoreX −i . Higher AUC indicates better prioritization of bad items for review. Abbrev.Statistic (pairwise)HS MathGSM8KMMLU-5Human Interitem comparisons (mean pairwise association for item i with other items j ̸= i) AccP(X i = 1,X j = 1) (probability both correct)0.5850.7810.7700.667 F1 2 P(1| X i = 1) P(1| X j = 1) P(1| X i = 1) + P(1| X j = 1) (cond. prob. both correct)0.6270.8210.7910.683 SMCP(X i = X j ) (probability of item agreement)0.8850.8410.8270.733* MIH(X i ) + H(X j )− H(X i ,X j ) (mutual dependence)0.8140.8630.7630.817* φCorr(X i ,X j ) (linear dependence)0.8890.8720.8650.917* ρ tet Corr(θ i ,θ j ) (linear dependence among latent θs)0.7650.8720.8660.896* κ P(X i = X j )− E[P(X i = X j )] 1− E[P(X i = X j )] (meas. error for same θ)0.8990.8620.8690.900* U sym ̄ U i = 2 n X j̸=i MI(X i ;X j ) H(X i ) + H(X j ) (symmetrized uncertainty)0.7800.8790.6810.883* M iso Signed isotonic R 2 (prop. interitem signal explained)0.908 0.8730.8610.983* Item-group comparisons (association of item i against all items/rest-score X −i ) MI X,X(−i) H(X i ) + H(X −i )− H(X i ,X −i ) (mutual information)0.5760.5530.5970.590 ρCorr(X i ,X −i ) (linear dependence)0.7890.8700.8710.896* z √ N − 1 P j̸=i Cov(X i ,X j ) q P j̸=i Var(X i ) Var(X j ) (monotone depen- dence) 0.7880.8710.8710.933* R 2 iso:X,i Signed isotonic R 2 (item signal explained by X )0.9040.8730.8300.932* Item-drop comparisons (difference in statistic upon removal of i) ∆αChange in Reliability; α X − α X−i 0.8050.8730.8500.922* ∆ ̄ρChange in Mean Correlation; ̄ρ X − ̄ρ X−i 0.7880.8700.8710.895* Item-as-parameter comparisons (Use of parameter value) d 2PL 2PL difficulty parameter; σ(a i (θ− d i ))0.6700.7460.7610.771 a 2PL 2PL discrimination parameter; σ(a i (θ− d i ))0.7110.8730.8370.917* Bold Underlineindicates the best AUC for a given dataset; Bold indicates the second-best; italics indicate others within0.01from best. ‘*’ signifies that for the first human dataset discussed in Sec. 4.1.4, the one “bad” item was correctly sorted first;H(·)denotes Shannon entropy;MI(·;·)mutual information.σ(·)denotes sigmoid. “tet.” denotes tetrachoric correlation (latentθcorrelation implied by dichotomous items). 1 compares all the different computation costs: all interitem, except tetrachoric, relationships and nonparametric rela- tionships were computed inR(Team) in 4.0, 8.1, and 22.4 seconds for HS math, MMLU, and GSM8K respectively; whereas tetrachoric alone took 8.2, 39.0, and 190.5 seconds usingpsychpackage (Revelle, 2024); fitting 2PL mod- els usingmirt(Chalmers, 2012) took 5.0, 24.5, and 15.4 seconds, respectively. Detailed results are presented in the appendix with all comparisons computed as measured under a bootstrapped Borda count regime for ranking across sub- sample sizes. AUC is computed usingpROC(Robin et al., 2011). We drop degenerate cases where a method yields constant scores or where a resample contains only one label class. Missingness rates were recorded per metric per resam- ple to distinguish statistical weakness from non-estimability. In the present study, missingness was negligible. 6. Results Results demonstrate substantial improvements in identifica- tion efficiency for both proposed metrics. Table A presents AUC values across multiple datasets, withM iso consistently ranking among the top-performing indices. 7 Efficient Detection of Bad Items 6.1. Experiment 1: signed isotonic R 2 is consistently top-tier Table 1 reports AUC for representative interitem, item–rest, and model-based baselines across datasets. Three patterns are consistent. (i) Interitem monotone variance explained is highly di- agnostic. The signed isotonicR 2 achieves the best or near-best AUC across datasets, including the strongest per- formance on the human dataset (AUC≈ 0.98). This indi- cates that globally bad items are precisely those that fail to participate in the monotone dependency structure shared by most items. (i) Nonlinear monotone signal matters beyond linear correlation. Linear association (φ) and agreement-based measures (SMC,κ) are competitive on some datasets, but the signed isotonicR 2 improves or matches them without assuming linearity and while preserving the direction of contribution. This is particularly important when misfit produces nonlinearity (e.g., ambiguity affecting mid-ability respondents disproportionately). (i) Item–rest summaries can underperform interitem structure. Item–rest mutual information is substantially weaker than the best interitem methods in these datasets. Aggregating through the rest scoreX −i can wash out pair- wise violations, especially when a small number of bad items is diluted by many good ones. 6.2. Experiment 2: robustness under subsampling and broad method comparison Appendix Table A reports results from a comprehensive bat- tery of indices (CTT, IRT, scalability, PCA/omega, anomaly metrics) underB = 20subsamples per AI benchmark. Signed isotonicR 2 ranks among the most efficient methods by average rank and percentile-AUC summaries, outper- forming many widely used diagnostics. Two observations are particularly practically relevant: Stability under benchmark-sized subsamples. When restricted to onlyn = 50respondents andp = 200items, many model-based procedures can become unstable, slow, or sensitive to estimation choices. The signed isotonicR 2 remains competitive because it depends only on bivariate monotone fits and aggregation. Comparable accuracy with substantially lighter com- putation. On the full datasets, computing the full set of interitem statistics (including isotonic regression) required seconds, whereas tetrachoric correlations and some IRT fits required substantially longer. This supports the use of signed isotonicR 2 as an early-stage screening tool: it produces high-quality review queues cheaply. 6.3. Experiment 3: best overall scaling robustness across n× p regimes Across5600resamples spanningpup to512and varying n, Borda aggregation yields a clear ordering: the signed isotonicR 2 is the most robust interitem method overall, fol- lowed by the signed adjusted isotonicR 2 , then correlation- based (φ/MCC) and classic agreement-based metrics. This result is consistent with the interpretation that signed isotonicR 2 captures a fundamental regularity of unidimen- sional measurement–monotone coherence–while remaining insensitive to functional-form misspecification and changes in aspect ratio. Proposition 3.1 shows that the signed isotonicR 2 is not merely another association coefficient: it is an extremal measure of monotone predictability. When most items share a latent ordering, a good item should be predictably mono- tone from many others; globally bad items systematically reduce this maximal monotone predictability. 7. Discussion 7.1. What the experiments collectively show Across full datasets (Exp. 1), stress-tested subsamples (Exp. 2), and wide aspect-ratio regimes (Exp. 3), signed isotonicR 2 is consistently among the best detectors of glob- ally bad items. These findings support two claims: 1.Measurement coherence is fundamentally mono- tone. The strongest signal distinguishing good from bad items is whether an item participates in the ex- pected monotone dependency structure induced by a dominant latent trait. 2. Maximal monotone explainability is a practical screening principle. Quantifying “how much of an item can be monotone-explained by other items” yields review queues that are efficient across diverse item pathologies. 7.2. Why signed isotonic R 2 detects global badness Bad items can fail in different ways (bad key, grading, am- biguity, construct drift), but they share a common footprint: they break monotone coherence with the rest of the instru- ment. Directionality matters. Miskeyed or systematically in- verted grading induces negative association with many items. A signed statistic can surface these inversions di- rectly, whereas unsigned dependence measures may treat 8 Efficient Detection of Bad Items inversions as “strong signal” and mis-rank the item. This is one reason the signed isotonic family tends to outperform the unsigned isotonicR 2 (which, for dichotomous items, collapses toward a squared correlation-style quantity). Nonlinearity matters.Ambiguity and construct drift can produce non-monotone or saturating effects: an item may be- have normally for low-ability respondents but become noisy for high-ability respondents (or vice versa). Linear correla- tion averages these regimes; isotonic regression explicitly extracts the strongest monotone component and penalizes deviations through reduced R 2 . 7.3. Implications for AI benchmarks AI benchmarks lack the institutionalized validation pipelines typical of human testing. Our results suggest a lightweight, model-agnostic workflow: 1. Compute signed isotonicR 2 item scores from an eval- uation matrix (models×items or runs×items). 2. Review the top-k most suspicious items. 3.Fix or remove problematic items; re-score models; repeat. Because the method is bivariate and nonparametric, it can be applied even when item formats vary (binary/ordinal/- continuous) and when the “respondents” are heterogeneous systems rather than humans. 7.4. How to read Appendix Table A The appendix table is intentionally comprehensive: it re- ports (i) a large suite of indices spanning interitem associ- ation, Mokken scalability, item-drop reliability, IRT item fit, and dimensionality proxies, and (i) aggregated perfor- mance summaries across subsamples. Rather than serving as a central narrative element, it functions as a robustness audit: the signed isotonicR 2 remains highly ranked even against dozens of alternative diagnostics and under repeated perturbations of the dataset. In the main text we therefore emphasize: (i) the head-to- head comparisons most interpretable to a broad audience (Table 1), and (i) the scaling regime where AI benchmark practice is most challenged (Exp. 2–3). 7.5. Conclusion Across both human assessments and AI benchmarks, signed isotonicR 2 provides a simple rule with a strong empirical and mathematical basis: Items that cannot be well explained by monotone functions of other items are the ones most likely to be globally bad. This principle yields review queues that are more efficient, more scalable, and less assumption-laden than many tradi- tional alternatives, making it well-suited for modern large- scale evaluation regimes. 7.6. Limitations and scope Global detection, not diagnosis. High suspicion scores indicate that an item fails to cohere with the rest of the instrument, but they do not identify the cause (miskey vs. ambiguity vs. construct drift). In practice, the score is a prioritization tool for human review. Not a DIF tool. Because we do not condition on group membership, this method does not detect group-conditional misfit. Combining signed isotonicR 2 with stratified analy- ses is a promising extension. Dependence on a dominant monotone structure.If the instrument is strongly multidimensional or intentionally non- monotone, interitem monotone coherence is not the correct baseline and may over-flag items. In such cases, applying the method within clusters (e.g., topics/subscales) or after dimensionality screening is recommended. Acknowledgments We’d like to thank Lijin Zhang, Sanmi Koyejo, Sang Truong, Yuheng Tu, Elizabeth Childs, Yunsung Kim, Anka Reuel, Hansol Lee, and Jason Cho for their time, input, and support. References A systematic approach to the construction and evaluation of tests of ability. Psychological Monographs, 61(4):i–49, 1947. ISSN 0096-9753. doi: 10.1037/h0093565. Place: US. Busing, F. M. T. A. Monotone Regression: A Simple and Fast O(n) PAVA Implementation. Journal of Statistical Software, 102:1–25, May 2022. ISSN 1548-7660. doi: 10.18637/jss.v102.c01. URLhttps://doi.org/10 .18637/jss.v102.c01. Casabianca, J. M. Psychometrics is all you need, November 2025. URLhttps://osf.io/preprints/eda rxiv/7w6pz_v1/. Chalmers, R. P. mirt: A Multidimensional Item Response Theory Package for the R Environment. Journal of Statistical Software, 48:1–29, May 2012. ISSN 1548- 7660. doi: 10.18637/jss.v048.i06. URLhttps: //doi.org/10.18637/jss.v048.i06. 9 Efficient Detection of Bad Items Lee, Y.-S. Applications of isotonic regression in item response theory. Ph.D., The University of Wisconsin - Madison, United States – Wisconsin, 2002.URL https://w.proquest.com/docview/305 527109/abstract/59C2F5794504049PQ/1. Lee, Y.-S. A Comparison of Methods for Nonparamet- ric Estimation of Item Characteristic Curves for Bi- nary Items. Applied Psychological Measurement, 31 (2):121–134, March 2007.ISSN 0146-6216.doi: 10.1177/0146621606290248. URLhttps://do i.org/10.1177/0146621606290248. Lee, Y.-S., Wollack, J. A., and Douglas, J. On the Use of Nonparametric Item Characteristic Curve Estimation Techniques for Checking Parametric Model Fit. Educa- tional and Psychological Measurement, 69(2):181–197, April 2009. ISSN 0013-1644. doi: 10.1177/00131644 08322026. URLhttps://doi.org/10.1177/00 13164408322026. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., R ́ e, C., Acosta-Navas, D., Hudson, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong, F., Ren, H., Yao, H., Wang, J., Santhanam, K., Orr, L., Zheng, L., Yuksekgonul, M., Suzgun, M., Kim, N., Guha, N., Chatterji, N., Khattab, O., Henderson, P., Huang, Q., Chi, R., Xie, S. M., Santurkar, S., Ganguli, S., Hashimoto, T., Icard, T., Zhang, T., Chaudhary, V., Wang, W., Li, X., Mai, Y., Zhang, Y., and Koreeda, Y. Holistic Evaluation of Language Models, October 2023. URLhttp:// arxiv.org/abs/2211.09110 . arXiv:2211.09110 [cs]. Luzardo, M. and Rodr ́ ıguez, P. A Nonparametric Estimator of a Monotone Item Characteristic Curve. In Quantitative Psychology Research, p. 99–108. Springer, Cham, 2015. ISBN 978-3-319-19977-1. doi: 10.1007/978-3-319-199 77-18. URLhttps://link.springer.com/ch apter/10.1007/978-3-319-19977-1_8. Mair, P. and Leeuw, J. D. Unidimensional Scaling. In Wiley StatsRef: Statistics Reference Online, p. 1–3. John Wiley & Sons, Ltd, 2015. ISBN 978-1-118-44511-2. doi: 10.1002/9781118445112.stat06462.pub2. URLhttps: //onlinelibrary.wiley.com/doi/abs/10 .1002/9781118445112.stat06462.pub2. McCoy, R. T., Yao, S., Friedman, D., Hardy, M., and Griffiths, T. L. Embers of Autoregression: Understand- ing Large Language Models Through the Problem They are Trained to Solve, September 2023. URLhttps: //arxiv.org/abs/2309.13638v1. Mokken, R. J. A Theory and Procedure of Scale Analy- sis: With Applications in Political Research. De Gruyter Mouton, July 2011. ISBN 978-3-11-081320-3. doi: 10.1515/9783110813203. URLhttps://w.degr uyterbrill.com/document/doi/10.1515/ 9783110813203/html. Reuel, A., Hardy, A., Smith, C., Lamparth, M., Hardy, M., and Kochenderfer, M. J. BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices, November 2024. URLhttp://arxiv.or g/abs/2411.12990. arXiv:2411.12990 [cs] version: 1. Revelle, W. psych: Procedures for Psychological, Psy- chometric, and Personality Research, June 2024. URL https://cran.r-project.org/web/packa ges/psych/index.html. Revelle, W. and Condon, D. Unidim: An index of scale homogeneity and unidimensionality. Psychological Meth- ods, 2025. ISSN 1939-1463. doi: 10.1037/met0000729. Place: US. Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., and M ̈ uller, M. pROC: an open- source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12(1):77, March 2011. ISSN 1471-2105. doi: 10.1186/1471-2105-12-77. URL https://doi.org/10.1186/1471-2105-1 2-77. Salaudeen, O., Reuel, A., Ahmed, A., Bedi, S., Robertson, Z., Sundar, S., Domingue, B., Wang, A., and Koyejo, S. Measurement to Meaning: A Validity-Centered Frame- work for AI Evaluation, June 2025. URLhttp://ar xiv.org/abs/2505.10573. arXiv:2505.10573 [cs]. Sijtsma, K. On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha. Psychometrika, 74(1): 107–120, March 2009. ISSN 0033-3123, 1860-0980. doi: 10.1007/s11336-008-9101-0. URLhttps://w.ca mbridge.org/core/journals/psychometr ika/article/on-the-use-the-misuse-and -the-very-limited-usefulness-of-cronb achs-alpha/72E9A648D5324412AF5506701 B6BE325. Sijtsma, K. and Molenaar, I. The Monotone Homogene- ity Model: Scalability Coefficients. In Introduction to Nonparametric Item Response Theory, p. 49–64. SAGE Publications, Inc., 2002. ISBN 978-1-4129-8467- 6. doi: 10.4135/9781412984676. URLhttps: //methods.sagepub.com/book/mono/intr oduction-to-nonparametric-item-respo 10 Efficient Detection of Bad Items nse-theory/chpt/monotone-homogeneity -model-scalability-coefficients. Team, R. C. R: A Language and Environment for Statistical Computing. URLhttps://w.r-project.org /. Ten Berge, J. M. F. and So ˇ can, G. The Greatest Lower Bound to the Reliability of a Test and the Hypothesis of Unidimensionality. Psychometrika, 69(4):613–625, December 2004. ISSN 0033-3123, 1860-0980. doi: 10.1 007/BF02289858. URLhttps://w.cambridge. org/core/product/identifier/S0033312 30002353X/type/journal_article. Truong, S., Tu, Y., Hardy, M., Reuel, A., Tang, Z., Bu- rapacheep, J., Perera, J., Uwakwe, C., Domingue, B., Haber, N., and Koyejo, S. Fantastic Bugs and Where to Find Them in AI Benchmarks, November 2025. URLhttp://arxiv.org/abs/2511.16842. arXiv:2511.16842 [cs]. van der Ark, L. A. Mokken Scale Analysis in R. Journal of Statistical Software, 20:1–19, February 2007. ISSN 1548-7660. doi: 10.18637/jss.v020.i11. URLhttps: //doi.org/10.18637/jss.v020.i11. Vendrow, J., Vendrow, E., Beery, S., and Madry, A. Do Large Language Model Benchmarks Test Reliability?, February 2025. URLhttp://arxiv.org/abs/25 02.03461. arXiv:2502.03461 [cs]. Wind, S. A.An Instructional Module on Mokken Scale Analysis.Educational Measurement:Is- sues and Practice, 36(2):50–66, 2017.ISSN 1745-3992.doi: 1 0 . 1 1 1 1 / e m i p . 1 2 1 53.URL https://onlinelibrary.wiley.co m/doi/abs/10.1111/emip.12153.eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/emip.12153. Yu, A. Learning models and the double monotone model. Thesis, University of Illinois at Urbana-Champaign, Oc- tober 2022. URLhttps://hdl.handle.net/2 142/117700. Zhang, G., Dorner, F. E., and Hardt, M. How Benchmark Prediction from Fewer Data Misses the Mark, June 2025. URLhttp://arxiv.org/abs/2506.07673. arXiv:2506.07673 [cs]. Zijlmans, E. A. O., Tijmstra, J., van der Ark, L. A., and Sijtsma, K. Item-Score Reliability in Empirical- Data Sets and Its Relationship With Other Item In- dices. Educational and Psychological Measurement, 78(6):998–1020, December 2018a. ISSN 0013-1644. doi: 10.1177/0013164417728358. URLhttps: //doi.org/10.1177/0013164417728358. Zijlmans, E. A. O., van der Ark, L. A., Tijmstra, J., and Sijtsma, K. Methods for Estimating Item-Score Reliability. Applied Psychological Measurement, 42 (7):553–570, October 2018b. ISSN 0146-6216. doi: 10.1177/0146621618758290. URLhttps://doi.or g/10.1177/0146621618758290. 11 Efficient Detection of Bad Items A. Appendix MetricMetric TypeAve. Rank R 1.0 MMLU5 R 1.0 GSM8K R 1.0 HSMath M iso Isotonic12.670.910.830.85 c75Inter-Item Correlations (tet), 75-quantile14.000.910.830.84 a 2PL IRT: Discrimination19.330.910.830.83 g (5) Omega: General Factor25.000.900.820.81 g (3) Omega: General Factor25.000.880.810.90 h 2 (3) Omega: Variance25.500.730.820.95 u 2 (3) PCA: Variance26.830.730.820.95 h 2 (5) Omega: Variance26.830.730.820.96 min(ρ i(tet) )Inter-Item Correlations (tet)27.330.900.810.86 u 2 (5) PCA: Variance28.500.720.820.96 ρ τ meanInter-Item Correlations (tetrachoric)28.830.910.830.77 M τ,j Kendall’s τ30.000.910.830.77 a 3PL IRT: Discrimination31.830.840.790.89 h 2 (15) Omega: Variance31.830.670.820.96 ∆αCTT: Reliability32.170.860.830.78 ZiScalability34.000.900.830.77 ∆rother34.000.900.830.77 u 2 (15) PCA: Variance34.330.670.810.96 cmedseInter-Item Correlations (tet)35.000.690.760.97 Hi (Loevinger)Scalability35.330.910.790.79 g (15) other36.830.910.830.73 SE(ρ)Inter-Item Correlations (tet)37.830.700.780.94 Var(ρ)Inter-Item Correlations (tet)37.830.700.780.94 median(ρ)Inter-Item Correlations (tet)41.170.900.820.72 dcor dropRest Distance Correlation42.170.770.820.77 p25Omega: Variance48.170.730.760.81 p23Omega: Variance48.670.620.690.97 Pmanifest50.670.910.770.66 cnegInter-Item Correlations (tet)50.670.890.800.67 corrdropRest Correlation50.670.860.800.70 c25Inter-Item Correlations (tet)51.330.890.820.60 com 15Omega: Variance51.830.580.720.96 tc5 15PCA: Variance54.000.630.650.92 p215Omega: Variance56.170.750.790.73 c5Inter-Item Correlations (tet)58.830.800.810.63 g22Omega: Variance59.000.570.680.95 ΣViol.Scalability (Mokken)59.500.830.810.61 f3 5Omega: Variance61.670.640.640.83 tc1315PCA: Variance61.670.640.610.91 c10Inter-Item Correlations (tet)62.330.860.820.51 tc1115PCA: Variance63.330.710.470.96 f1315Omega: Variance64.170.670.590.87 tc45PCA: Variance64.330.640.670.80 raw alphaItem-drop64.50NANA0.78 N viol. Scalability65.170.850.810.51 f1515Omega: Variance65.670.680.530.90 critInter-Item Correlations (tet)66.330.860.770.53 tc2 15PCA: Variance66.670.670.580.85 adjdepthAnomaly: Item Isolation66.670.460.820.81 com 3Omega: Variance67.330.600.680.82 f15Omega: Variance67.500.810.620.69 Max Viol.Scalability (Mokken)67.670.720.740.65 tc7 15PCA: Variance68.000.600.630.86 tc23PCA: Variance68.330.740.620.76 x22IRT: Item Fit68.500.540.650.88 f2 3Omega: Variance68.670.740.620.76 f1115Omega: Variance68.670.610.600.91 adjdensityAnomaly: Item Isolation69.000.470.820.79 sum numberacScalability (Mokken)69.330.820.750.54 f1215Omega: Variance69.330.670.540.87 12 Efficient Detection of Bad Items numberacScalability69.330.660.740.73 numbervinumberacScalability69.830.810.720.58 f915Omega: Variance69.830.550.600.95 f25Omega: Variance70.670.740.670.63 tc15PCA: Variance71.170.810.640.63 tc25PCA: Variance72.000.740.620.68 tc13PCA: Variance72.330.820.680.56 tc115PCA: Variance73.670.600.620.80 f13Omega: Variance75.000.820.640.56 tc55PCA: Variance75.330.620.600.81 f215Omega: Variance75.670.650.560.80 averagerother76.00NANA0.77 g6 smcCTT: Reliability76.00NANA0.77 rcorItem-total Correlation76.00NANA0.77 rdropalphaItem-drop76.00NANA0.77 rawrItem-drop76.00NANA0.77 SNRAlpha76.00NANA0.77 stdalphaItem-drop76.00NANA0.77 stdrItem-drop76.00NANA0.77 f515Omega: Variance76.670.630.640.74 rmseax22Fit Deviation76.830.500.610.88 f4 5Omega: Variance77.670.620.600.80 complexity5PCA: Variance77.830.620.510.87 f815Omega: Variance79.000.560.600.86 outfit3IRT: Item Fit80.330.560.740.73 f315Omega: Variance81.170.670.610.69 f115Omega: Variance82.170.630.590.78 rmseag22Fit Deviation82.330.570.530.87 tc35PCA: Variance82.330.490.620.83 c1Inter-Item Correlations (tet)83.330.610.800.57 tc10 15PCA: Variance83.670.620.530.80 tc1215PCA: Variance84.330.540.590.84 com 5Omega: Variance85.170.580.700.66 complexity 15PCA: Variance86.000.580.510.85 tc615PCA: Variance86.170.620.480.81 cminInter-Item Correlations (tet)86.670.540.500.90 f55Omega: Variance87.000.470.620.80 z infit3IRT: Item Fit87.670.780.540.60 f1415Omega: Variance88.330.660.640.56 tc1515PCA: Variance88.330.630.630.61 f33Omega: Variance88.500.470.620.79 rmseag23Fit Deviation88.670.580.550.79 g23Omega: Variance89.000.550.710.64 tc33PCA: Variance91.500.470.610.79 tc315PCA: Variance92.500.520.590.78 tc1415PCA: Variance94.000.690.550.59 z outfit3IRT: Item Fit94.170.590.620.64 f10 15Omega: Variance94.500.460.610.77 guess3IRT: Discrimination95.170.680.460.66 outfit 2IRT: Item Fit95.170.530.700.61 tc915PCA: Variance95.500.660.520.64 infit3IRT: Item Fit95.670.780.520.53 x2 3IRT: Item Fit95.670.500.690.63 tc815PCA: Variance99.670.630.600.54 rmseax23Fit Deviation100.330.490.640.63 complexity 3PCA: Variance100.830.630.580.58 zinfit2IRT: Item Fit103.000.640.500.61 f715Omega: Variance104.170.600.610.51 f4 15Omega: Variance106.000.510.520.75 infit 2IRT: Item Fit106.170.600.520.63 f615Omega: Variance107.500.630.460.63 z outfit2IRT: Item Fit110.670.550.590.56 numberzsigScalability111.330.620.560.42 tc415PCA: Variance111.330.560.560.56 13 Efficient Detection of Bad Items vandermaarelyule_q_gammayule_wyule_yzi sym_jaccardsym_slopessym_tarantulasym_uncertaintytarantulatarwidtaub2tetra_approxtetra_approx2thielUunsigned_iso shape_differencesize_differencesmsokal_sneath_1sokal_sneath_2sokal_sneath_3sokal_sneath_4sorgenfreistilesswjaccardsym_cole_phi pearson_heron_2peirce1peirce2peirce3phipmirandrogers_tanimotorogot_goldbergrussell_raoscott maxwell_pillinermcconnaugheymichaelmono_distmono_efficiencymountford ochiai2_sokal_sneath_5odds_ratiooverlapp4pattern_difference hellingerhiinnerproductiso_monojaccardjohnsonkappakendall_tau_akendall_tau_bkulczynski1kulczynski2 forbes_2fossum goodman_kruskal_lambdagoodman_kruskal1goodman_kruskal2gowerh_2h_symhamannhammingharris_lahey ct5dennisdiceidiceiidispersioneuclideaneyraudf1fager_mcgowanfaithforbes_1 cole_1acole_1bcole_2cole_3contingencycosine_ochiaicramer_vct1ct2ct3ct4 a1_2adj_contingencyanderberganderberg1ariaustin_colwell baroni_urbani_buser_1baroni_urbani_buser_2braun_blanquetcanberrachi_sq 0 25 50 750 25 50750 25 50 750 25 50 750 25 5075 0 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 5075 0 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 5075 0 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 5075 0 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 5075 0 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 5075 0 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 5075 0 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 5075 0 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 5075 0 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 50750 25 50 750 25 50 750 25 5075 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 n log(p) auc 0.6 0.7 0.8 0.9 Figure 2. Instrument Composition Permutation Bootstrap Results (n x p) cmaxInter-Item Correlations (tet)112.000.510.530.63 MIdropRest Mutual Information118.000.560.520.52 msa15PCA: Variance127.500.500.500.50 msa 3PCA: Variance127.500.500.500.50 msa5PCA: Variance127.500.500.500.50 Table 2. The first numeric column represents the average Borda count rank. The subsequent columns represent the average percentile rank for AUC. Estimation of the AUC was done with the pROC package (Robin et al., 2011). B. Full List of Binary Associations Table 3. Intersection and Agreement-Based Similarity Measures Measure (Alias)ExpressionDescription & RationaleRef. Group 1: Intersection-Based (Focus on Joint Presence, ‘a’) Jaccard Index a a + b + c Ratio of the intersection to the union of positive responses. Ignores joint absences (d), making it suitable when co-presence is the primary signal of similarity. () Dice-Sørensen Coefficient (F1 Score) 2a 2a + b + c The harmonic mean of precision and recall. It is monotonic with Jaccard but gives greater weight to the intersection term (a). () Continued on next page 14 Efficient Detection of Bad Items Table 3 – continued from previous page Measure (Alias)ExpressionDescription & RationaleRef. Overlap Coefficient (Szymkiewicz- Simpson) a min(a + b,a + c) An asymmetric measure of inclusion; represents the proportion of the smaller set that is contained within the larger set. Reaches 1 if one item’s endorsers are a subset of the other’s. () Cosine Similarity (Ochiai, Fowlkes-Mallows) a p (a + b)(a + c) Geometric mean normalization. Interprets response vectors in a high-dimensional space and measures the cosine of the angle between them. Insensitive to vector magnitude. () Kulczy ́ nski 1 a b + c Ratio of joint presences to disagreements. Highly sensitive to low frequencies of disagreement, diverging as b+c approaches zero. () Forbes I n· a (a + b)(a + c) Ratio of the observed frequency of joint presence (a) to its expected frequency under statistical independence. A value ¿ 1 indicates positive association. () Group 2: Agreement-Based (Focus on Agreement, ‘a+d’) Simple Matching (Sokal-Michener) a + d n The most straightforward measure of agreement; the proportion of total cases where both items yield the same outcome. Treats joint presence and absence as equally informative. () Rogers-Tanimoto a + d a + d + 2(b + c) A variant of Simple Matching that penalizes disagreements by giving them double weight in the denominator. This results in lower similarity values compared to Simple Matching. () Hamann (a + d)− (b + c) n Proportion of agreement minus the proportion of disagreement. Ranges from -1 (perfect disagreement) to +1 (perfect agreement). A signed measure sensitive to the balance of concordant and discordant pairs. () Sokal-Sneath 2 (Gower-Legendre) 2(a + d) 2(a + d) + b + c A variant of the Dice-Sørensen coefficient that is applied to both agreement states (a and d). () Continued on next page 15 Efficient Detection of Bad Items Table 3 – continued from previous page Measure (Alias)ExpressionDescription & RationaleRef. Faith a + 0.5d n Asymmetrically weights agreement, considering a joint presence (a) twice as informative as a joint absence (d). Useful when positive matches are considered stronger evidence of similarity. () Table 4. Covariance, Correlation, and Information-Theoretic Measures Measure (Alias)ExpressionDescription & RationaleRef. Group 3: Covariance and Correlation-Based (Focus on ‘ad-bc’) Phi (φ) Coefficient (Matthews Correlation) ad− bc p (a + b)(c + d)(a + c)(b + d) The Pearson product-moment correlation for two dichotomous variables. It is a chance-corrected measure of association, sensitive to both marginal distributions and the covariance term. () Yule’s Q (γ) ad− bc ad + bc A measure of ordinal association that is independent of the marginal distributions. It represents the probability of concordance minus the probability of discordance, given that no ties exist. Reaches±1if any cell is zero. () Yule’s W √ ad− √ bc √ ad + √ bc The ”coefficient of colligation.” It is more conservative than Yule’s Q and is not independent of the marginals. It relates to the average of conditional probabilities. () Odds Ratio (OR) ad bc The ratio of the odds of a positive outcome on Item j given a positive outcome on Item i, to the odds of a positive outcome on j given a negative outcome on i. Widely used in epidemiology; not bounded above. () Cohen’s Kappa (κ) p o − p e 1− p e where p o = a+d n and p e = (a+b)(a+c)+(c+d)(b+d) n 2 A measure of inter-rater reliability that corrects the observed proportion of agreement (p o ) for the agreement expected by chance (p e ). Robust against unequal marginal probabilities. () Dispersion ad− bc n 2 The covariance between the two binary items. It is the fundamental building block of many correlation coefficients but is sensitive to the variance of the items. () Group 4: Information-Theoretic Continued on next page 16 Efficient Detection of Bad Items Table 4 – continued from previous page Measure (Alias)ExpressionDescription & RationaleRef. Pointwise Mutual Information (PMI) log 2 P (i,j) P (i)P (j) = log 2 a· n (a + c)(a + b) Quantifies the discrepancy between the probability of the items’ co-occurrence given their joint distribution versus their individual distributions (assuming independence). () Symmetric Uncertainty 2· I(i;j) H(i) + H(j) A normalized variant of mutual information,I(i;j). It corrects for the bias of mutual information toward variables with more states (though not an issue for binary data) and produces a value in [0, 1]. () Table 5. Psychometric and Monotonicity-Based Measures Measure (Alias)ExpressionDescription & RationaleRef. Loevinger’s H (Pairwise, H ij ) Cov(i,j) Cov max (i,j) For Cov > 0: ad− bc min [(a + c)(b + d), (a + b)(c + d)] A coefficient of homogeneity. It normalizes the covariance by the maximum possible covariance given the items’ marginal frequencies. This assesses how well the items conform to a deterministic Guttman-type relationship. A cornerstone of Mokken Scale Analysis. () Standardized Z (Z i )Conceptually: H i − E[H i ] p Var(H i ) A test statistic for the significance of an item’s scalability (H i , its average H-value with all other items). It assesses whether an item’s monotonicity with the underlying scale is statistically significant under the null model of non-monotonicity. Not directly computed from a single pair. () 2PL Discrimination (a i ) Estimated via Marginal Maximum Likelihood (MML) A parameter from a latent trait (Item Response Theory) model. It represents the slope of the Item Characteristic Curve at the point of inflection (θ = b i ). It is not expressible in contingency table terms, as it is estimated from the full response matrix under the assumption of a continuous latent trait. It is analogous to a factor loading. () Yule’s Q (γ) ad− bc ad + bc Included again for comparison. In psychometrics, its property of being independent of marginals makes it a pure measure of ordinal association, but this can be a drawback, as it may inflate association for items with extreme difficulties. () 17 Efficient Detection of Bad Items Table 5 – continued from previous page Measure (Alias)ExpressionDescription & RationaleRef. Phi (φ) Coefficient ad− bc p (a + b)(c + d)(a + c)(b + d) Included again for comparison. In contrast to Q, Phi is sensitive to item difficulty differences. Two items with high Phi must have similar difficulties, making it a measure of both association and difficulty-matching. Its square is the proportion of variance explained. () C. Formal Treatment of Isotonic Regression This appendix builds the mathematical relationships of isotonic regression in the context of scalability and bad item detection. C.1. The Probability Space We begin by defining the measure-theoretic foundation. Let (Ω,F,P ) be a probability space, where: • Ω is the sample space (a set of outcomes). • F is a σ-algebra of events (a set of subsets of Ω). • P is a probability measure onF . LetXandYbe two real-valued random variables, which are measurable functions from our probability space to the real numbers: • X : Ω→ R • Y : Ω→ R The expectation of a random variableZ(or any integrable functiong(X,Y )) is its Lebesgue integral with respect to the measure P : E[Z] := R Ω Z(ω)dP (ω) We assume thatXandYhave finite second moments, i.e.,E[X 2 ] <∞andE[Y 2 ] <∞. This ensures that their variances are well-defined and finite. • Var(X) = E[(X − E[X]) 2 ] = R Ω (X(ω)− E[X]) 2 dP (ω) • Var(Y ) = E[(Y − E[Y ]) 2 ] = R Ω (Y (ω)− E[Y ]) 2 dP (ω) C.2. Signed Isotonic (Monotonic) Regression Isotonic regression finds the best-fitting monotonic function. Unlike linear regression, we search over the entire space of monotonic functions to minimize the mean squared error (MSE). We define the two sets of monotonic, measurable functions: • The set of non-decreasing (isotonic) functions:M ↑ = f : R → R | f is measurable and x 1 ≤ x 2 =⇒ f (x 1 ) ≤ f (x 2 ) •The set of non-increasing (antitonic) functions:M ↓ = f : R → R | f is measurable and x 1 ≤ x 2 =⇒ f (x 1 ) ≥ f (x 2 ) 18 Efficient Detection of Bad Items Table 6. Binary Item Similarity Measures: Mathematical Formulations and Characteristics MeasureAliasesFormulaDescription & Key FeaturesRefs Intersection-Based Measures Jaccard IndexTanimoto a a+b+c Intersection over union; satisfies triangle inequality; standard for binary similarity Dice CoefficientSørensen, F1-score 2a 2a+b+c Harmonic mean of precision/recall; emphasizes joint occurrences Overlap CoefficientSimpson a min(a+b,a+c) Similarity relative to smaller set; asymmetric upper bound Cosine SimilarityOchiai, Driver-Kroeber a √ (a+b)(a+c) Geometric angle measure; normalized dot product for binary vectors Kulczy ́ nski-2 1 2 a a+b + a a+c Arithmetic mean of conditional probabilities; symmetric precision measure Agreement-Based Measures Simple MatchingSokal- Michener a+d n Total agreement proportion; treats positive and negative matches equally Rogers-Tanimoto a+d a+d+2(b+c) Penalized matching coefficient; reduces weight of disagreements Hamming Similarity 1− b+c n Complement of normalized edit distance; symmetric disagreement penalty Correlation and Covariance Measures Phi CoefficientMatthews C, Pearson ad−bc √ (a+b)(c+d)(a+c)(b+d) Tetrachoric correlation for binary data; ranges [-1,1]; measures linear association Yule’s QGoodman- Kruskal γ ad−bc ad+bc Ordinal association measure; emphasizes concordant vs. discordant pairs Kendall’s τ b ad−bc √ (n c −T x )(n c −T y ) Rank correlation with tie correction; robust to marginal distributions Cram ́ er’s V q χ 2 n Chi-square based association; standardized effect size measure Proportional Reduction in Error Cohen’s κ p o −p e 1−p e Chance-corrected agreement; accounts for marginal probability agreement Adjusted Rand Index ( n 2 ) (P +Q)−[(P +R)(P +S)+(S+T )(R+T )] ( n 2 ) 2 −[(P +R)(P +S)+(S+T )(R+T )] Clustering agreement corrected for chance; extended kappa for partitions Information-Theoretic Measures Mutual InformationI(X;Y ) = H(X) + H(Y )− H(X,Y ) Shared information content; measures statistical dependence reduction in uncertainty Normalized MISymmetric Uncertainty 2I(X;Y ) H(X)+H(Y ) Standardized mutual information; bounded [0,1]; symmetric measure Psychometric Scaling Measures Loevinger’s H i P ij −P i P j max(P ij )−P i P j Scalability coefficient; measures departure from independence relative to maximum possible Standardized Z i H i SE(H i ) Standardized scalability; provides significance testing for monotone homogeneity 2PL DiscriminationIRT a-parameter a i = 1.7φ ij √ 1−φ 2 ij Item discrimination parameter; relates tetrachoric correlation to IRT slope Note: Contingency table notation:a= both items positive,b= itemipositive/jnegative,c= iteminegative/jpositive,d= both items negative, n = total sample size. P i , P j denote marginal probabilities; P ij denotes joint probability. 19 Efficient Detection of Bad Items Table 7. Specialized Binary Similarity Measures for Item Response Applications MeasureAliasesFormulaDescription & Key FeaturesRefs Expectation-Adjusted Measures Forbes Coefficient na (a+b)(a+c) Ratio of observed to expected co-occurrence; sensitive to rare item pairs Tarwid Index na−(a+b)(a+c) na+(a+b)(a+c) Standardized deviation from independence; symmetric about zero McConnaughey a 2 −bc √ (a+b)(a+c) Squared overlap relative to geometric mean; emphasizes strong associations Monotonicity and Scalability Symmetric HMokken H ij 1− b+c 2P i (1−P j )+2P j (1−P i ) Symmetric scalability index; measures violation of monotone homogeneity Isotonic R 2 Unsigned Monotone φ 2 Squared correlation; measures strength ignoring direction of association Signed IsotonicMonotone R 2 sign(ad− bc)· φ 2 Directional monotone association; preserves sign of relationship Robust and Weighted Measures Baroni-Urbani- Buser √ ad+a √ ad+a+b+c Includes negative matches in geometric weighting; robust to marginal asymmetry Yule’s W √ ad− √ bc √ ad+ √ bc Square-root stabilized odds ratio; reduces influence of extreme cell counts Stiles Indexlog 10 n(|ad−bc|−n/2) 2 (a+b)(c+d)(a+c)(b+d) Yates-corrected chi-square; provides continuity correction for small samples Distance-Based Transformations Hellinger Similarity1− r 1 2 b a+b + c a+c Probabilistic distance measure; derived from Hellinger distance between distributions Austin-Colwell 2 π arcsin q a+d n Arcsine-transformed agreement; variance-stabilizing transformation Pattern Difference1− 4bc n 2 Shape-based similarity; measures configurational rather than marginal differences Note: These measures address specific psychometric challenges including marginal heterogeneity, sample size sensitivity, and monotonicity assumptions in item response modeling. Selection depends on theoretical assumptions about item relationships and desired mathematical properties. 20 Efficient Detection of Bad Items C.3. Signed R 2 for Y regressed on X First, we consider the regression ofYonX. We must find the best monotonic fit, which could be either non-decreasing or non-increasing. 1. Find the best non-decreasing fit (f ↑ Y|X ): This function minimizes the MSE over all functions inM ↑ .f ↑ Y|X := arg min f∈M ↑ E[(Y − f (X)) 2 ] = arg min f∈M ↑ R Ω (Y (ω) − f (X(ω))) 2 dP (ω) Let the resulting minimum MSE be MSE ↑ Y|X = E[(Y − f ↑ Y|X (X)) 2 ]. 2. Find the best non-increasing fit (f ↓ Y|X ): This function minimizes the MSE over all functions inM ↓ .f ↓ Y|X := arg min f∈M ↓ E[(Y − f (X)) 2 ] = arg min f∈M ↓ R Ω (Y (ω) − f (X(ω))) 2 dP (ω) Let the resulting minimum MSE be MSE ↓ Y|X = E[(Y − f ↓ Y|X (X)) 2 ]. 3. Determine the Direction and the Signed R²: The direction of the monotonicity is determined by which fit is better (i.e., has a lower MSE). We define a sign, σ Y|X , based on this comparison. σ Y|X := ( +1 if MSE ↑ Y|X ≤ MSE ↓ Y|X −1 if MSE ↑ Y|X > MSE ↓ Y|X The overall best monotonic MSE ismin(MSE ↑ Y|X , MSE ↓ Y|X ). The corresponding unsignedR 2 is:R 2 Y|X = 1 − min(MSE ↑ Y|X ,MSE ↓ Y|X ) Var(Y ) The signed R² for Y on X, which we denoteS 2 Y|X , is the product of the sign and the unsigned R²:S 2 Y|X := σ Y|X · 1− min(E[(Y−f ↑ Y|X (X)) 2 ],E[(Y−f ↓ Y|X (X)) 2 ]) Var(Y ) C.4. Signed R² for X regressed on Y The process is perfectly symmetric. We now regressXonY. We need a new set of functions, which we’l callg, to avoid confusion. 1. Find the best non-decreasing fit (g ↑ X|Y ): g ↑ X|Y := arg min g∈M ↑ E[(X − g(Y )) 2 ] MSE ↑ X|Y = E[(X − g ↑ X|Y (Y )) 2 ] 2. Find the best non-increasing fit (g ↓ X|Y ): g ↓ X|Y := arg min g∈M ↓ E[(X − g(Y )) 2 ] MSE ↓ X|Y = E[(X − g ↓ X|Y (Y )) 2 ] 3. Determine the Direction and the Signed R²: σ X|Y := ( +1 if MSE ↑ X|Y ≤ MSE ↓ X|Y −1 if MSE ↑ X|Y > MSE ↓ X|Y The signed R² for X on Y, denoted S 2 X|Y , is: S 2 X|Y := σ X|Y · 1− min(E[(X−g ↑ X|Y (Y )) 2 ],E[(X−g ↓ X|Y (Y )) 2 ]) Var(X) C.5. Final Notation for the Association Measure The final association measure is the mean of the two signed R² values. Let’s call this measureρ 2 m (X,Y ), where the subscript ’m’ stands for monotonic. The final mathematical notation is: ρ 2 m (X,Y ) = 1 2 S 2 Y|X + S 2 X|Y where: S 2 Y|X = sgn(MSE ↓ Y|X − MSE ↑ Y|X )· 1− min(MSE ↑ Y|X ,MSE ↓ Y|X ) Var(Y ) S 2 X|Y = sgn(MSE ↓ X|Y − MSE ↑ X|Y )· 1− min(MSE ↑ X|Y ,MSE ↓ X|Y ) Var(X) and the components are defined using Lebesgue integrals over the probability space (Ω,F,P ) as follows: • MSE ↑ Y|X = inf f∈M ↑ R Ω (Y (ω)− f (X(ω))) 2 dP (ω) 21 Efficient Detection of Bad Items • MSE ↓ Y|X = inf f∈M ↓ R Ω (Y (ω)− f (X(ω))) 2 dP (ω) • MSE ↑ X|Y = inf g∈M ↑ R Ω (X(ω)− g(Y (ω))) 2 dP (ω) • MSE ↓ X|Y = inf g∈M ↓ R Ω (X(ω)− g(Y (ω))) 2 dP (ω) • Var(Y ) = R Ω Y (ω)− R Ω Y (ω ′ )dP (ω ′ ) 2 dP (ω) • Var(X) = R Ω X(ω)− R Ω X(ω ′ )dP (ω ′ ) 2 dP (ω) ρ 2 miso (X,Y ) = sgn mono · 2− argmin g∈M ( R Ω (X(ω)−g(Y (ω))) 2 dP (ω)) R Ω ( Y (ω)− R Ω Y (ω ′ )dP (ω ′ ) ) 2 dP (ω) − argmin f∈M ( R Ω (Y (ω)−f (X(ω))) 2 dP (ω)) R Ω ( X(ω)− R Ω X(ω ′ )dP (ω ′ ) ) 2 dP (ω) ! M ↑ andM ↓ are the sets of non-decreasing and non-increasing measurable functions, respectively. The use of ‘inf‘ (infimum) is slightly more rigorous than ‘min‘ (minimum) as the minimum might not be achieved by a specific function within the set, though in the context of theseL 2 projections, it is. The ‘argmin‘ notation used earlier implies this existence. The sign function ‘sgn‘ is defined as sgn(z) = +1 for z ≥ 0 and sgn(z) =−1 for z < 0, which matches our definition of σ. 22 Efficient Detection of Bad Items NameAliasesFormulaSalient Features & Use Case I. Intersection-Based Measures (Joint Positive Occurrence) Dice PrecisionS P (Precision) a a+b Proportion ofX = 1alsoY = 1; insensitive to co-absences, directional. Dice RecallS R (Recall) a a+c Proportion ofY = 1alsoX = 1; equally direc- tional. Jaccard(Tanimoto) a a+b+c Fraction of total positives shared; ignores double- zeros; popular in ecology/text. Weighted Jaccard 3a 3a+b+c Increased weight on matches; strengthens inter- section focus. Russell-Rao a n Absolute fraction of joint positives; marginal cov- erage emphasized, sparse for rare items. Sokal-Sneath 1 a a+2(b+c) Penalizes discord more than Jaccard; stricter sim- ilarity. Braun-Blanquet a max(a+b,a+c) Ratio of joint positives to maximum positive marginal; overlap normalization. Mountford 2a a(b+c)+2bc Sensitive to shared presence among rare items. Sorgenfrei a 2 (a+b)(a+c) Squared joint positive, normalized for expected chance; rare-item association. Ochiai/ Cosine a √ (a+b)(a+c) Geometric mean scaling; adjusts for marginal frequencies; popular in text mining. I. Agreement (Matching) Measures Simple Matching (SMC) a+d n Overall agreement; considers both co-presence and co-absence. Sokal-Sneath 3 a+d b+c Ratio of total agreement to disagreement; un- bounded above. Gower a+d √ (a+b)(a+c)(b+d)(c+d) Adjusts for marginals; robust for unbalanced data. Rogot-Goldberg a 2a+b+c + d 2d+b+c Symmetrized precision foraandd; accentuates strong matches. I. Distance and Dissimilarity-Derived Measures Normalized Euclidean1− √ b+c √ n Transforms Hamming/L 2 distance to similarity; interpretable as spatial proximity. Canberra1− 1 2 b a+b + c a+c Emphasizes mismatches when marginals are small. Size Difference1− (b+c) 2 n 2 Quadratic disagreement penalty; accentuates out- lier differences. IV. Covariance, Correlation, and Dependency Measures Determinantad− bcRaw cross-patterning between pairs; basis for several associations below. Yule’s Q (Goodman-Kruskal γ) ad−bc ad+bc Scaling from perfect agreement (1) to inversion (−1); extreme-focus. Phi / Matthews Correlation ad−bc √ (a+b)(a+c)(b+d)(c+d) Pearson for binaries;[−1, 1]range; symmetric; interpretable as ”latent” correlation. Cohen’s Kappa P o −P e 1−P e Chance-corrected agreement; robust to marginal imbalance. Kendall’s Tau-a ad−bc n(n−1)/2 Concordance-discordance scaled for pairs; rank- based. Tetrachoric Approx.cos π 1+ √ OddsRatio Approximates correlation between dichotomized normals. V. Information-Theoretic Measures Mutual InformationMI(X;Y ) = H(X) + H(Y )− H(X,Y )Nonlinear association; accounts for total and joint uncertainty. Theil’s U (Uncertainty coefficient) H(Y )−H(Y|X) H(Y ) Proportion ofY’s entropy explained byX; highly directional, bounded [0, 1]. VI. Cluster and Proportional Reduction in Error Indices Rand Index A+ad n 2 Agreement over all item pairs; adjusts for cluster structure and chance. Goodman-Kruskal LambdaSee note a Proportion by which knowing one variable re- duces error in predicting the other; direction mat- ters. Notes. Notation for marginal and joint counts:a,b,c,das in the text;n = a + b + c + d. Where other symbols appear:P o is observed agreement, P e is expected agreement by chance, OddsRatio = ad bc , H(·) is (Shannon) entropy, and A is the number of pairs clustered together in both variables. a See Supplement for full Goodman-Kruskal formulae and context. 23