Paper deep dive
Validation-Frontier Representation Selection under Constrained Observation
Wesley Shu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/18/2026, 5:30:11 AM
Summary
This paper introduces a validation-frontier selector for state representation selection under constrained observation, where raw accuracy is not the sole criterion. The method combines balanced accuracy with penalties for feature cost, overfit gap, and validation-test instability. Evaluated on three scikit-learn datasets across five observation regimes, the adaptive selector improved the frontier score by 0.025801 and reduced mean feature count by 22.733 compared to full trace features, with no statistically significant difference in balanced accuracy. However, broader stress tests showed mixed results, indicating that adaptive selection improves robustness-efficiency frontiers in specific matched settings but does not universally dominate trace baselines.
Entities (13)
Relation Signals (8)
Validation-Frontier Selector → comparedagainst → Full Trace Features
confidence 95% · The central comparison is between the adaptive selector and full trace features
Validation-Frontier Selector → improves → Frontier Score
confidence 95% · the adaptive selector improves frontier score over full trace features by 0.025801
Validation-Frontier Selector → reduces → Mean Feature Count
confidence 95% · reducing mean feature count by 22.733
Strong Robustness v2 → uses → UCI Wine Recognition
confidence 95% · The Strong Robustness v2 benchmark uses three public scikit-learn bundled datasets: ... UCI wine recognition ...
Strong Robustness v2 → uses → Wisconsin Breast Cancer Diagnosis
confidence 95% · The Strong Robustness v2 benchmark uses three public scikit-learn bundled datasets: Wisconsin breast cancer diagnosis...
Strong Robustness v2 → uses → Fisher Iris Classification
confidence 95% · The Strong Robustness v2 benchmark uses three public scikit-learn bundled datasets: ... and Fisher iris classification.
Validation-Frontier Selector → doesnotuniversallydominates → Trace Baselines
confidence 90% · adaptive representation selection can improve a constrained-observation robustness-efficiency frontier in matched benchmark settings, but does not universally dominate trace baselines.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:AI systems deployed outside clean benchmark settings often rely on observations that are incomplete, unstable, costly, or degraded by monitoring failures. This paper studies representation selection under constrained observation: choosing a state representation when raw accuracy is not the only operational criterion. We propose a validation-frontier selector that combines balanced accuracy with penalties for feature cost, overfit gap, and validation-test instability. In a focused public-tabular benchmark using three scikit-learn datasets, five observation regimes, 45 matched task cells, 720 candidate actions, and 405 representation rows, the adaptive selector improves frontier score over full trace features by 0.025801 while reducing mean feature count by 22.733. Balanced-accuracy difference is small and not statistically significant. A broader offline stress test gives mixed results. The supported claim is therefore bounded: adaptive representation selection can improve a constrained-observation robustness-efficiency frontier in matched benchmark settings, but does not universally dominate trace baselines.
Tags
Links
- Source: https://arxiv.org/abs/2608.15095v1
- Canonical: https://arxiv.org/abs/2608.15095v1
Trouble viewing inline? Open PDF directly →
Full Text
46,674 characters extracted from source content.
Expand or collapse full text
Validation-Frontier Representation Selection under Constrained Observation Wesley Szu-Way Shu Affiliation: Independent Researcher Email: shuwesley@gmail.com Abstract AI systems deployed outside clean benchmark settings often rely on observations that are incomplete, unstable, costly, or degraded by monitoring failures. This paper studies representation selection under constrained observation: choosing a state representation when raw accuracy is not the only operational criterion. We propose a validation-frontier selector that combines balanced accuracy with penalties for feature cost, overfit gap, and validation-test instability. In a focused public-tabular benchmark using three scikit-learn datasets, five observation regimes, 45 matched task cells, 720 candidate actions, and 405 representation rows, the adaptive selector improves frontier score over full trace features by 0.025801 while reducing mean feature count by 22.733. Balanced-accuracy difference is small and not statistically significant. A broader offline stress test gives mixed results. The supported claim is therefore bounded: adaptive representation selection can improve a constrained-observation robustness-efficiency frontier in matched benchmark settings, but does not universally dominate trace baselines. Keywords: artificial intelligence, robust AI systems, representation selection, constrained observation, model selection, dataset shift, validation frontier 1 Introduction Modern AI systems do not act on unlimited, stable, and costless information. They often depend on telemetry, logs, sensor streams, partial traces, structured summaries, retrieved context, or monitoring channels that degrade under missingness, distribution shift, class imbalance, or instrumentation cost. A representation can be accurate when all variables are observable, but inefficient when every feature must be collected; compact but brittle when the environment shifts; or stable but too lossy for the control decision it supports. These are not merely engineering inconveniences. They determine whether an AI system can preserve useful behavior when its observation layer changes. This paper studies the problem as state representation selection under constrained observation. Given a family of candidate representations and probe models, the system must choose a representation using validation evidence before reporting held-out test performance. The selection rule should not treat all observed variables as free, nor should it assume that the representation with the highest validation accuracy is most reliable under operational constraints. The proposed solution is a validation-frontier criterion that combines balanced accuracy with penalties for feature cost, overfit gap, and validation-test instability. This study frames representation choice as an AI-system reliability problem. When observation channels are costly, incomplete, or unstable, the best state representation is not necessarily the one with the highest unconstrained validation accuracy, but the one that preserves the strongest accuracy-cost-stability frontier under matched deployment stressors. The contribution is not a new deep representation architecture, not a representation-learning theorem, and not a claim that a fixed state vector universally dominates trace features. It is a reproducible evaluation framework and bounded empirical result for selecting state representations under constrained observation. This reframing matters because unrestricted trace representations are often strong. If all variables are available, stable, and costless, full traces can be hard to beat. But deployed AI systems rarely live in that condition. The relevant question becomes whether a representation selector can preserve comparable accuracy while improving a frontier that includes observation cost and stability. The empirical answer here is conditional. The Strong Robustness v2 benchmark gives a positive frontier result against full trace features. The broader v5 stress test then shows that the advantage does not generalize into universal dominance against every trace baseline. This pattern is scientifically useful: it identifies where adaptive representation selection helps and where the evidence should not be overstated. Contributions. The paper makes four contributions. First, it formalizes constrained-observation state representation selection as a frontier-based AI reliability problem. Second, it defines reproducible public-label benchmark regimes that stress missingness, low-resource training, class imbalance, and covariate shift. Third, it evaluates weak summaries, raw traces, full trace features, PCA, random projections, fixed state representations, and an adaptive selector in matched task cells. Fourth, it preserves negative evidence, showing that fixed state representations do not dominate trace baselines and that broad stress testing supports conditional, not universal, claims. 2 Results 2.1 Constrained-observation representation selection Let X denote an observed row from a public dataset and Y its target label. A representation map ϕ∈Φφ∈ transforms the observation into a state vector Zϕ=ϕ(X)Z_φ=φ(X). A probe model h∈ℋh is trained on ZϕZ_φ and evaluated on held-out labels. Conventional model selection would choose ϕφ and h by validation accuracy. Under constrained observation, however, high validation accuracy can select representations that are costly to observe, unstable under train-test changes, or overfit to the validation condition. Each candidate representation-probe pair therefore produces four reported quantities: held-out balanced accuracy B, overfit gap O, validation-test stability gap S, and normalized feature cost C. The validation-frontier score is F(ϕ,h)=B−λOO−λCC−λSS,F(φ,h)=B- _OO- _CC- _SS, (1) where penalty weights are chosen before evaluation and raw components are still reported. The scalar score is not meant to hide trade-offs. It forces the selection process to make observation cost and instability visible instead of treating them as free side effects of predictive accuracy. Definition 1 (Constrained observation regime). A constrained observation regime is a reproducible transformation of the training, validation, or test condition that preserves the public target labels while changing the information conditions under which a representation is learned or evaluated. The regimes used here are natural splitting, low-resource training, missingness injection, class-imbalanced training, and covariate-shifted testing. Definition 2 (Frontier-based adaptive representation selection). Given a candidate representation family Φ , probe family ℋH, and validation-frontier score F, frontier-based adaptive representation selection chooses (ϕ^,h^)=argmaxϕ∈Φ,h∈ℋFval(ϕ,h),( φ, h)= _φ∈ ,h F_val(φ,h), and reports held-out test metrics for the selected pair. The method is intentionally modest. It does not assert that the selected representation is causally sufficient, fully interpretable, or optimal under all future shifts. It is a practical selection rule for AI systems that must choose what state to expose when observations have costs and stability risks. 2.2 Representation family The benchmark compares nine reported representation conditions. Weak and generic baselines. The weak narrative proxy exposes only row availability and magnitude summaries. The generic summary uses a small number of normalized state statistics. These conditions approximate settings in which an AI system receives broad summaries rather than full trace-level observations. Trace baselines. The raw trace uses original numerical features. The full trace representation augments raw features with missingness channels. These are strong baselines by design: if trace variables are complete and costless, they should often perform well. Compressed baselines. PCA-8 and random projection-8 provide standard compressed representation baselines. Their inclusion prevents the benchmark from treating compactness as a unique property of the proposed state vectors. Fixed state representations. The compressed state vector uses selected normalized variables, tail indicators, and a limited set of interaction channels. The robust state vector expands the state with additional missingness, tail, and interaction features. These fixed state vectors are not assumed to dominate full traces. Adaptive selector. The adaptive selector chooses among compact state, robust state, PCA, random projection, and trace representations using the validation-frontier score in Equation 1. The selector may choose a simple baseline when that baseline better preserves the frontier. This design is conservative: a positive result cannot come from forcing every task into a preferred representation. 2.3 Benchmark design The empirical evidence has two layers. 2.3.1 Strong Robustness v2 The main benchmark uses three public scikit-learn datasets, five observation regimes, 45 matched task cells, 720 candidate actions, and 405 representation rows. Each task cell compares representation families under identical data splits and regime conditions. The central comparison is between the adaptive selector and full trace features, because full trace features represent the natural alternative when all variables are exposed. This benchmark is intentionally presented as focused public-tabular evidence, not as a large-scale foundation-model or OpenML-wide claim. 2.3.2 BroadRealPublic v5 The broader stress test checks whether the positive v2 result is fragile. The completed offline v5 benchmark uses five public source datasets and fixed public-target tasks. It is reported as breadth and boundary evidence rather than as an independent OpenML battery. Its role is to prevent overclaiming: if adaptive selection fails against strong trace baselines in some broader cells, the paper should preserve that failure rather than hide it. 2.3.3 Evaluation principle The benchmark reports both scalar frontier scores and raw components. A frontier improvement is meaningful only if it is not purchased by unacceptable accuracy collapse or hidden instability. This is why balanced accuracy, feature count, overfit gap, and clean-efficiency score are reported beside the frontier score. 2.4 Strong Robustness v2 results Table 1 summarizes the Strong Robustness v2 representation results. Raw trace and PCA remain strong. The adaptive selector does not dominate raw accuracy. Its advantage appears on the robustness-efficiency frontier, where it combines competitive balanced accuracy with lower feature cost and lower overfit gap. Figure 1 visualizes this frontier trade-off by plotting frontier score against mean feature count. Table 1: Strong Robustness v2 representation summary over 45 matched task cells. Representation Bal. acc. Frontier Clean eff. Feat. Overfit baseline proxy 0.7717 0.7211 0.7544 5.0 0.1433 generic summary 0.8674 0.8323 0.8529 4.0 0.0598 raw trace 0.9447 0.8924 0.9170 15.7 0.0542 trace features 0.9357 0.8577 0.8917 31.3 0.0597 PCA-8 0.9294 0.8900 0.9106 6.7 0.0378 random proj. 0.8770 0.8349 0.8582 6.7 0.0558 Compressed-State 0.8990 0.8113 0.8506 27.0 0.0636 Robust-State 0.9266 0.8113 0.8588 56.3 0.0647 adaptive selector 0.9256 0.8835 0.9036 8.6 0.0239 0055101015152020252530303535404045455050555560600.70.70.750.750.80.80.850.850.90.9Mean feature countFrontier scorebaseline proxygeneric summaryraw tracetrace featuresPCA-8random projectionCompressed-StateRobust-Stateadaptive selector Figure 1: Strong Robustness v2 frontier score versus mean feature count. The adaptive selector occupies a high-frontier, low-feature region relative to full trace features, while raw trace and PCA remain strong competitors. The figure is generated directly from Table 1, keeping the arXiv source self-contained. Table 2 gives the key paired comparisons. Compared with full trace features, the adaptive selector improves frontier score by 0.025801 and clean-efficiency score by 0.011865 while reducing mean feature count by 22.733. The balanced-accuracy gap is −0.010050-0.010050 and is not significant. This supports a frontier claim, not an accuracy-dominance claim. Cell-wise support is also asymmetric in the expected direction: adaptive selection has higher frontier score than trace features in 33 of 45 matched cells, lower frontier score in 9 cells, and ties in 3 cells; the median paired frontier difference is 0.030956, with an exact two-sided sign-test value of 2.72×10−42.72× 10^-4. Table 3 reports the primary matched-cell statistical tests for the adaptive selector relative to trace features, including bootstrap intervals and Wilcoxon signed-rank tests. Table 2: Strong Robustness v2 headline paired comparisons over matched task cells. Positive values favor the first method. Comparison Acc. diff Frontier diff Eff. diff Feature diff pFp_F adaptive vs trace features -0.010050 0.025801 0.011865 -22.733 4.70×10−54.70× 10^-5 adaptive vs PCA-8 -0.003750 -0.006495 -0.006992 1.933 0.0868 Compressed-State vs trace -0.036679 -0.046366 -0.041079 -4.333 9.62×10−59.62× 10^-5 Robust-State vs trace -0.009081 -0.046363 -0.032903 25.000 1.18×10−61.18× 10^-6 Table 3: Primary matched-cell statistical report for adaptive selector versus trace features. Differences are adaptive minus trace features over 45 matched cells. Tests are two-sided Wilcoxon signed-rank tests; bootstrap intervals use 10,000 matched-cell resamples. Metric Mean diff. 95% bootstrap CI Wilcoxon p Claim use Balanced accuracy -0.010050 [-0.0374, 0.0092] 0.669 no accuracy-superiority claim Frontier score 0.025801 [-0.0054, 0.0480] 4.70×10−54.70× 10^-5 primary frontier claim Clean efficiency 0.011865 [-0.0163, 0.0319] 0.00298 secondary Feature count -22.733 [-29.067, -16.644] 8.05×10−88.05× 10^-8 secondary cost reduction Overfit gap -0.035855 [-0.0484, -0.0241] 4.99×10−64.99× 10^-6 secondary stability The negative comparisons are important. Fixed compressed and robust state vectors do not dominate trace features. Their frontier scores are worse than trace features in the verified v2 run. The result that survives is narrower: a validation-frontier selector can avoid a universal fixed-vector claim by selecting compact or robust representations only when they are useful. 2.5 BroadRealPublic v5 stress test and claim boundary The BroadRealPublic v5 result is mixed. It does not strengthen a universal dominance claim. In the visible locked comparison against raw trace, the adaptive selector has lower balanced accuracy by 0.021305 and lower frontier score by 0.010110 while using 5.941818 fewer features. Another visible comparison shows a small positive balanced-accuracy gap but a negative frontier gap. These results mean that the broad stress test should be read as a boundary: adaptive selection remains a plausible constrained-observation reliability method, but it does not dominate every strong trace baseline across expanded target tasks. This mixed outcome improves the manuscript if it is reported honestly. It prevents the paper from depending on a fragile universal-gain narrative. The final claim combines the two layers: v2 provides the main positive matched evidence for frontier improvement against full trace features; v5 provides breadth and shows that the advantage is conditional, baseline-sensitive, and not equivalent to raw accuracy dominance. Table 4 summarizes the resulting evidence hierarchy and claim boundary. Table 4: Evidence hierarchy after the broad v5 stress test. Layer What it supports What it does not support Strong Robustness v2 Main positive frontier result: adaptive selection improves frontier over full trace features with comparable balanced accuracy. Universal fixed-state dominance; broad independent OpenML coverage. BroadRealPublic v5 Broader offline real-public target-task stress test; conditionality and robustness of the claim boundary. Universal adaptive dominance over raw trace or all trace baselines. Earlier controlled scaffold Mechanism sanity check for structural probes. Real full benchmark evidence. 3 Discussion 3.1 Interpretation for robust AI systems The results identify three practical mechanisms in representation design. First, full trace access is valuable but not always efficient. Trace features and raw traces achieve strong raw accuracies in several regimes. If observation is free, trace baselines are difficult to beat. If observation is constrained, the relevant question becomes whether comparable accuracy can be preserved with fewer features and lower stability cost. Second, fixed state vectors are brittle as universal answers. The compressed state loses raw accuracy, and the robust state pays a high feature cost. A fixed representation can protect some control-relevant channels and still be inferior on a generic supervised-learning frontier. Third, adaptivity is the defensible object. The adaptive selector does not require one representation to be best everywhere. It selects according to a validation frontier. The v2 run shows a frontier improvement against full trace features; the v5 run shows that this does not transfer into universal dominance over every trace baseline. The method is therefore best understood as controlled representation selection under constraints. For AI reliability, this distinction matters. A system that always uses the largest trace may over-instrument its environment, increase monitoring burden, and become vulnerable to missingness or shift. A system that always uses a compact state may under-observe the task. Frontier-based selection provides a mechanism for choosing between these risks using validation evidence rather than architectural preference. 4 Relation to Existing Work The work sits at the intersection of representation learning, model selection, state abstraction, partial observability, feature selection, and robustness under distributional stress. Representation-learning surveys emphasize that useful features should expose factors of variation relevant to downstream prediction [2]. State abstraction and partially observable decision-process literatures show that the available state is often an approximation of a larger latent condition [3, 4]. Information bottleneck and compressed representation methods similarly treat predictive sufficiency and compression as coupled objectives [5, 6]. Classical cross-validation and bootstrap methods provide mechanisms for estimating out-of-sample behavior [7, 8, 9], while feature-selection and sparsity methods show that predictive quality and feature cost have long been treated as coupled objectives [10, 11]. Dataset-shift and domain-adaptation work show that train-test stability cannot be assumed [12, 13]. Recent robustness benchmarks and group-distributionally robust methods similarly emphasize that average accuracy can hide failure modes under shifted subpopulations [14, 15]. Shortcut-learning and safety literatures warn that systems can exploit available signals without preserving the intended objective [16, 17]. Robustness and deployment-shift evaluations further show that reported performance can change under natural distribution changes, corruptions, and domain shifts [18, 19, 20, 21]. Missing-data methodology also motivates treating observation failure as a first-class modeling condition rather than as a cosmetic preprocessing issue [22]. The present paper differs from this literature in its object of selection. It does not propose a new neural architecture, a new regularizer, or a new domain-adaptation bound. It proposes a benchmarked AI-system selection problem: given competing representations, choose the one that preserves a prediction-control frontier under constrained observation. The novelty is not that accuracy, cost, or stability are individually new quantities. The contribution is to bind them into a reproducible representation-selection task and to show, with positive and negative evidence, when an adaptive selector is more defensible than a fixed state vector or unrestricted trace representation. 5 Methods Datasets and target labels. The Strong Robustness v2 benchmark uses three public scikit-learn bundled datasets: Wisconsin breast cancer diagnosis, UCI wine recognition, and Fisher iris classification. The target labels are the original dataset targets supplied with each loader; the benchmark does not create synthetic target labels. The manifest records 250 breast-cancer rows with 30 original features and two classes, 178 wine rows with 13 original features and three classes, and 150 iris rows with four original features and three classes. The scikit-learn library is cited for dataset loading and baseline tooling.[1] Splitting and matched cells. Each dataset is evaluated under five observation regimes and three random seeds, producing 45 matched dataset-regime-seed task cells. Within each cell, all representation and probe candidates use the same train, validation, and test split. This matched-cell design ensures that paired comparisons attribute differences to the representation/probe condition rather than to different rows or target definitions. Observation regimes. The natural regime uses the standard split without additional degradation. The low-resource regime reduces training rows while preserving validation and test evaluation. The missingness regime injects 20% observation missingness into feature channels and exposes corresponding missingness effects to the representations. The class-imbalance regime changes the training distribution to stress minority-class preservation while preserving the public test labels. The covariate-shift regime evaluates on shifted regions of the public feature space. These regimes alter observation and sampling conditions; they do not replace public labels with generated labels. Representations. The benchmark evaluates nine representation conditions: a weak narrative proxy, a four-feature generic summary, raw trace features, trace features augmented with missingness channels, PCA-8, random projection-8, Compressed-State, Robust-State, and the adaptive frontier selector. Feature count is recorded for each representation and normalized within each task cell by the original feature count, making the cost term comparable across datasets with different dimensionalities. Probe models and candidate actions. Each representation is paired with logistic regression and decision-tree probes. A candidate action is a representation-probe pair evaluated within a matched task cell. Strong Robustness v2 evaluates 720 candidate action rows and 405 best-model-by-representation rows. The probe family is intentionally simple: the purpose is to test representation selection under observation constraints, not to introduce a new deep architecture. Metrics and frontier score. The primary predictive metric is held-out balanced accuracy, which prevents majority-class performance from dominating imbalanced regimes. Additional recorded quantities are train balanced accuracy, validation balanced accuracy, test macro-F1, overfit gap, validation-test stability gap, raw feature count, normalized feature cost, clean-efficiency score, and frontier score. The frontier score subtracts pre-specified penalties for feature cost, overfit gap, and instability from balanced accuracy. The scalar score is used for selection, while all raw components are reported to avoid hiding trade-offs. Adaptive selection protocol. For every task cell, each candidate action is trained on the training split and evaluated on the validation split. The adaptive selector chooses the representation-probe pair with the highest validation-frontier score, then reports held-out test performance only after selection. Baseline comparisons are made on matched cells against raw trace, trace features, PCA-8, and fixed state representations. Statistics and reproducibility. Paired headline comparisons are computed over the 45 matched cells. The prespecified primary comparison is adaptive selector versus trace features on test frontier score. The null hypothesis is zero median paired difference across matched cells. The reported nonparametric test is a two-sided Wilcoxon signed-rank test with significance level α=0.05α=0.05. No multiplicity-adjusted claim is made across all pairwise comparisons; secondary comparisons are descriptive and are used to define claim boundaries. For the primary comparison, the mean frontier-score difference is 0.025801, the median paired difference is 0.030956, and adaptive selection is higher in 33 cells, lower in 9 cells, and tied in 3 cells. The Wilcoxon result is p=4.70×10−5p=4.70× 10^-5; the exact two-sided sign-test value, excluding ties, is 2.72×10−42.72× 10^-4. The bootstrap 95% confidence interval for the mean paired frontier difference is [−0.0054,0.0480][-0.0054,0.0480] over matched cells. The Wilcoxon signed-rank test evaluates paired signed-rank evidence over matched cells, whereas the bootstrap interval summarizes uncertainty in the mean paired difference; the primary claim is therefore based on the pre-specified paired nonparametric test, with the bootstrap interval reported descriptively. The balanced-accuracy difference is −0.010050-0.010050 with Wilcoxon p=0.669p=0.669 and bootstrap 95% confidence interval [−0.0374,0.0092][-0.0374,0.0092], so the paper does not claim accuracy superiority. Feature-count difference is −22.733-22.733 features with Wilcoxon p=8.05×10−8p=8.05× 10^-8 and bootstrap 95% confidence interval [−29.067,−16.644][-29.067,-16.644]. Clean-efficiency difference is 0.011865 with Wilcoxon p=0.00298p=0.00298. Bootstrap intervals use 10,000 resamples with a fixed random seed and resample matched cells with replacement. BroadRealPublic v5 stress test. The v5 stress test expands public source-dataset and fixed target-task breadth. A prior OpenML-dependent route was not certified because OpenML access was unavailable; therefore the completed v5 run should not be described as an independent OpenML battery. The v5 result is used only as a claim-boundary stress test. It shows mixed adaptive-versus-trace outcomes and is not used to claim universal dominance. Execution environment and verification. The Strong Robustness v2 manifest records Python 3.13.5, scikit-learn 1.8.0, pandas 2.2.3, numpy 2.3.5, and Linux execution. The verification files state that synthetic-or-controlled labels, generated scenarios, and benchmark-created labels are false for the central v2 evidence. The BroadRealPublic v5 freeze records SHA256 prefix fba962c550419bfb. All methods needed to understand the study are reported here; the supplementary files provide code, outputs, and file inventory rather than additional Supplementary Methods. Use of AI tools. AI-assisted tools were used for drafting support, language refinement, mathematical exposition, and package preparation. The author reviewed, revised, and approved the final manuscript and takes full responsibility for all claims, analyses, and final text. 5.1 Limitations The strongest limitation is dataset independence. The broad v5 benchmark expands target-task breadth, but several tasks are deterministic transformations of the same source datasets. This is useful for stress testing, but weaker than a large independent OpenML battery. A stronger future version should add many independently sourced public datasets once network access is reliable. A second limitation is that the observation regimes are benchmark stressors, not live deployment logs. The benchmark changes sampling, missingness, imbalance, and shifted test regions using real rows and public labels; it does not observe actual institutional interventions or production AI traffic. A third limitation is that the frontier score contains design choices. The penalty weights are deliberately modest, but alternative deployments may value feature cost or stability differently. The package therefore reports raw metrics as well as frontier scores so reviewers can inspect the trade-off rather than accept a single scalar. A fourth limitation is scale. Three central datasets and 45 matched cells are sufficient to make a reproducible focused benchmark, but not sufficient to establish broad empirical coverage across modern AI application domains. The correct reading is therefore methodological and bounded: the paper demonstrates a falsifiable selection framework and a positive matched frontier result, while leaving larger multi-dataset validation to future work. 5.2 Conclusion The verified evidence supports a bounded AI reliability result: frontier-based adaptive state representation selection can improve the robustness-efficiency frontier relative to full trace features while preserving statistically comparable balanced accuracy under constrained observation regimes. The broader v5 stress test clarifies that the advantage is conditional and baseline-sensitive. The scientific claim is therefore not universal dominance, but a controlled selection principle: when observation is costly or unstable, representation choice should be evaluated by the joint frontier it preserves, not by raw accuracy alone. Data availability The data and reproducibility materials supporting this study are openly archived on Zenodo in the record Reproducibility Artifact for Validation-Frontier Representation Selection under Constrained Observation, DOI: 10.5281/zenodo.21944174. The benchmark uses public scikit-learn bundled datasets and fixed public-target transformations as documented in the archived dataset manifests and evidence-lock files. The archive includes the generated CSV outputs containing matched-cell results, representation summaries, pairwise comparisons, bootstrap summaries, and verification records needed to interpret and replicate the reported findings. Code availability The benchmark code, reproduction runner, validation scripts, generated CSV outputs, JSON manifests, evidence-lock files, and associated verification materials are openly archived in the same Zenodo reproducibility record, DOI: 10.5281/zenodo.21944174. The deposited archive is the public code-and-output artifact associated with this manuscript. Acknowledgements The author received no external funding for this work. Author contributions W.S. conceived the study, prepared the reproducibility package, conducted the analysis, wrote the manuscript, and approved the final version. Competing interests The author is developing the Energetic Paradigm framework and related applications. The author declares no other competing interests directly related to this manuscript. Use of AI tools AI-assisted tool use is documented in the Methods section. The author reviewed, revised, and approved the final manuscript and takes full responsibility for its content. Appendix A Mathematical Formulation of Frontier-Based State Representation Selection This appendix gives the formal layer underlying the selection rule used in the manuscript. The purpose is not to claim a new universal representation theorem, but to make explicit the quantities optimized by the validation-frontier selector and the conditions under which a selected representation can improve the reliability frontier without necessarily improving raw predictive accuracy. A.1 Observation model and representation maps Let (Ω,ℱ,ℙ)( ,F,P) be a probability space and let (X,Y)∼P(X,Y) P denote an observation-label pair with X∈X and Y∈1,…,KY∈\1,…,K\. A constrained observation regime is represented by an observation operator Or:→r,O_r:X _r, (2) where r∈ℛr indexes missingness, low-resource sampling, imbalance, covariate shift, or other operational degradation. A candidate state representation is a measurable map ϕj:r→ℝdj,j∈1,…,m. _j:X_r ^d_j, j∈\1,…,m\. (3) The induced state is Zj=ϕj(Or(X))Z_j= _j(O_r(X)). The dimension djd_j is not treated as a harmless formatting detail: it is an observable proxy for instrumentation burden, monitoring cost, and exposure to missing or unstable channels. For a learning algorithm AℓA_ and training sample Sn=(Xi,Yi)i=1nS_n=\(X_i,Y_i)\_i=1^n, the trained predictor associated with representation j and learner ℓ is hjℓ=Aℓ((ϕj(Or(Xi)),Yi)i=1n).h_j =A_ (\( _j(O_r(X_i)),Y_i)\_i=1^n ). (4) The candidate action set is therefore =a=(j,ℓ):j∈1,…,m,ℓ∈1,…,L.A=\a=(j, ):j∈\1,…,m\,\ ∈\1,…,L\\. (5) In the reported benchmark, rows in A correspond to representation-probe combinations evaluated on matched dataset-regime cells. A.2 Balanced accuracy and operational cost For a classifier h, define class-conditional recall Reck(h,P)=ℙh(Z)=k∣Y=k,Rec_k(h;P)=P\h(Z)=k Y=k\, (6) with Z=ϕj(Or(X))Z= _j(O_r(X)) for the representation associated with h. The population balanced accuracy is B(h,P)=1K∑k=1KReck(h,P).B(h;P)= 1K _k=1^KRec_k(h;P). (7) This objective is used instead of raw accuracy because several regimes include imbalance or shifted label distributions. Let cj≥0c_j≥ 0 denote the normalized feature cost of representation j. In the simplest implementation, cj=djdmax,dmax=max1≤q≤mdq.c_j= d_jd_ , d_ = _1≤ q≤ md_q. (8) Other cost functions may be substituted when features have unequal collection costs. For example, if feature channel q has cost κq≥0 _q≥ 0 and representation j uses channel set IjI_j, then cj=∑q∈Ijκqmax∑q∈Issκq.c_j= _q∈ I_j _q _s _q∈ I_s _q. (9) The empirical experiments use feature count as a transparent and reproducible cost proxy. A.3 Validation frontier objective Let B^tr(a) B_tr(a), B^val(a) B_val(a), and B^te(a) B_te(a) be the training, validation, and held-out test balanced accuracies for action a=(j,ℓ)a=(j, ). Define the empirical overfit gap and validation-test instability as G(a) G(a) =[B^tr(a)−B^val(a)]+, = [ B_tr(a)- B_val(a) ]_+, (10) I(a) I(a) =|B^val(a)−B^te(a)|, = | B_val(a)- B_te(a) |, (11) where [u]+=maxu,0[u]_+= \u,0\. The validation frontier score is Fval(a)=B^val(a)−λcj−μG(a),F_val(a)= B_val(a)-λ c_j-μ G(a), (12) with λ,μ≥0λ,μ≥ 0. The held-out reporting score is Fte(a)=B^te(a)−λcj−μG(a)−νI(a),F_te(a)= B_te(a)-λ c_j-μ G(a)-ν I(a), (13) with ν≥0ν≥ 0. Equation (12) is used for selection; Equation (13) is used for reporting because the test split must not determine the selected representation. The adaptive selector is a^∗∈argmaxa∈Fval(a),ϕ^∗=ϕj^for a^∗=(j^,ℓ^). a^*∈ _a F_val(a), φ^*= _ j a^*=( j, ). (14) Ties are resolved by smaller feature cost and then by higher validation balanced accuracy. This tie rule makes the selector conservative: it does not choose a larger observation surface unless validation evidence justifies it. A.4 Frontier dominance and why accuracy dominance is not required For two actions a=(j,ℓ)a=(j, ) and b=(q,s)b=(q,s), define the test-frontier difference ΔF(a,b)=Fte(a)−Fte(b). _F(a,b)=F_te(a)-F_te(b). (15) Expanding Equation (13) gives ΔF(a,b)=[B^te(a)−B^te(b)]−λ(cj−cq)−μG(a)−G(b)−νI(a)−I(b). split _F(a,b)&= [ B_te(a)- B_te(b) ]-λ(c_j-c_q)\\ & -μ\G(a)-G(b)\-ν\I(a)-I(b)\. split (16) Thus a representation can have lower test balanced accuracy and still have a higher frontier score if its cost, overfit, or instability reductions are large enough. Proposition 1 (Frontier gain without accuracy gain). Let a and b be two candidate actions. Suppose B^te(a)≤B^te(b) B_te(a)≤ B_te(b). Then a has a higher reported frontier score than b if and only if λ(cb−ca)+μG(b)−G(a)+νI(b)−I(a)>B^te(b)−B^te(a).λ(c_b-c_a)+μ\G(b)-G(a)\+ν\I(b)-I(a)\> B_te(b)- B_te(a). (17) Proof. Move terms in Equation (16). The condition ΔF(a,b)>0 _F(a,b)>0 is equivalent to Equation (17). No distributional assumption is required because the statement is algebraic for the empirical reporting score. ∎ This proposition is the mathematical reason the empirical claim is phrased as a frontier claim rather than as raw accuracy dominance. In the Strong Robustness v2 comparison against full trace features, the adaptive selector has a small negative balanced-accuracy difference but a positive frontier-score difference because it reduces feature count and overfit gap. A.5 Sufficiency, redundancy, and constrained observation A representation ϕj _j is label-sufficient under regime r if Y⟂Or(X)|ϕj(Or(X)).Y O_r(X) _j(O_r(X)). (18) Exact sufficiency is rarely verifiable in finite benchmark settings. The paper therefore uses an empirical relaxation. For tolerance ϵ≥0ε≥ 0, representation j is ϵε-competitive with representation q on a matched cell if B^te(j)≥B^te(q)−ϵ. B_te(j)≥ B_te(q)-ε. (19) It is ϵε-frontier-superior if additionally Fte(j)>Fte(q).F_te(j)>F_te(q). (20) The reported positive v2 result should be read in this relaxed sense: the adaptive selector is approximately competitive in balanced accuracy and superior on the frontier against full trace features in the matched benchmark cells. Redundant observation channels can be expressed by decomposing a trace representation as T=(S,R),T=(S,R), (21) where S contains control-relevant channels and R contains channels that are predictive only under some regimes or unstable under shift. If R increases cjc_j, G(a)G(a), or I(a)I(a) without increasing held-out balanced accuracy enough, Equation (17) predicts a frontier loss for unrestricted trace features. Conversely, if R carries stable predictive signal, raw trace can remain superior. This is why the manuscript reports both positive and negative evidence. A.6 Cell-wise aggregation Let C be the set of matched benchmark cells, where a cell is a dataset-regime-target split with a fixed training/validation/test partition. For method M and baseline Q, define the cell-wise frontier difference Dc(M,Q)=Fte,c(M)−Fte,c(Q),c∈.D_c(M,Q)=F_te,c(M)-F_te,c(Q), c . (22) The reported mean frontier difference is D¯(M,Q)=1||∑c∈Dc(M,Q). D(M,Q)= 1|C| _c D_c(M,Q). (23) The matched-cell design is important because it blocks a common reporting error: comparing representation averages computed over different tasks or different splits. The paired nonparametric test reported in the manuscript is applied to the matched sequence Dc(M,Q):c∈\D_c(M,Q):c \ rather than to unpaired summary rows. A.7 Selection risk and validation uncertainty Let a∗=argmaxa∈FP(a)a^*= _a F_P(a) be the population-best action under the population frontier score FPF_P, and let a^∗ a^* be the validation-selected action in Equation (14). The excess frontier risk is ℰ(a^∗)=FP(a∗)−FP(a^∗).E( a^*)=F_P(a^*)-F_P( a^*). (24) The benchmark does not claim a new finite-sample bound, but the usual uniform-deviation argument clarifies what the validation-frontier procedure requires. If supa∈|F^val(a)−FP(a)|≤η, _a | F_val(a)-F_P(a) |≤η, (25) then ℰ(a^∗)≤2η.E( a^*)≤ 2η. (26) Indeed, FP(a∗)≤F^val(a∗)+η≤F^val(a^∗)+η≤FP(a^∗)+2η. splitF_P(a^*)&≤ F_val(a^*)+η\\ &≤ F_val( a^*)+η\\ &≤ F_P( a^*)+2η. split (27) This standard inequality motivates the use of matched validation evidence and the reporting of overfit and validation-test instability. When validation estimates are noisy or shifted, the selector can fail. The BroadRealPublic v5 stress test is therefore interpreted as a claim boundary, not as a universal dominance result. A.8 Decision rule summary For clarity, the full selection-and-reporting protocol can be written as: Train each a∈ on the training split, each a on the training split, (28) compute B^tr(a),B^val(a),cj,G(a), B_tr(a), B_val(a),c_j,G(a), (29) a^∗∈argmaxa∈B^val(a)−λcj−μG(a), a^*∈ _a \ B_val(a)-λ c_j-μ G(a) \, (30) report B^te(a^∗),Fte(a^∗),cj^,G(a^∗),I(a^∗), B_te( a^*),F_te( a^*),c_ j,G( a^*),I( a^*), (31) compare against baselines only on matched cells. (32) This protocol is the mathematical core of the manuscript. It makes the claim falsifiable: a trace representation wins when its accuracy advantage exceeds its cost and stability penalties, while an adaptive representation wins only when validation evidence identifies a better operating point on the constrained-observation frontier. Appendix B Evidence Boundary The controlled structural benchmark from earlier package versions is retained only as historical scaffold evidence. It used generated structural scenarios and should not be described as real full benchmark evidence. The present manuscript uses the Strong real-label robustness benchmark as its central positive evidence and BroadRealPublic v5 as a broader offline stress test. Appendix C M4P Evidence Locks The Strong Robustness v2 M4P execution log reports: benchmark level Strong Robustness v2; datasets 3; regimes 5; task cells 45; candidate action rows 720; best-model representation rows 405; and validation status PASS. The same log reports the adaptive-selector comparison against trace features as balanced-accuracy difference −0.010050-0.010050, frontier-score difference 0.0258010.025801, clean-efficiency difference 0.0118650.011865, feature-count difference −22.733333-22.733333, and frontier Wilcoxon p=4.70007×10−5p=4.70007× 10^-5. The BroadRealPublic v5 M4P execution completed and froze. The completed package is the BroadRealPublic v5 freeze; the SHA256 prefix is fba962c550419bfb. The visible verification states that v5 should be described as source-dataset plus target-task breadth, not independent OpenML breadth. The visible adaptive-versus-raw-trace comparison reports balanced-accuracy difference −0.021305-0.021305, frontier-score difference −0.010110-0.010110, clean-efficiency difference −0.009813-0.009813, and feature-count difference −5.941818-5.941818. References [1] Pedregosa, F. et al. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 12, 2825-2830 (2011). [2] Bengio, Y., Courville, A. & Vincent, P. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798-1828 (2013). [3] Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction. 2nd edn (MIT Press, 2018). [4] Kaelbling, L. P., Littman, M. L. & Cassandra, A. R. Planning and acting in partially observable stochastic domains. Artif. Intell. 101, 99-134 (1998). [5] Tishby, N., Pereira, F. C. & Bialek, W. The information bottleneck method. In Proc. 37th Annu. Allerton Conf. Commun. Control Comput. 368-377 (2000). [6] Alemi, A. A., Fischer, I., Dillon, J. V. & Murphy, K. Deep variational information bottleneck. In International Conference on Learning Representations (2017). [7] Stone, M. Cross-validatory choice and assessment of statistical predictions. J. R. Stat. Soc. Ser. B 36, 111-133 (1974). [8] Kohavi, R. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proc. IJCAI 1137-1145 (1995). [9] Arlot, S. & Celisse, A. A survey of cross-validation procedures for model selection. Stat. Surv. 4, 40-79 (2010). [10] Guyon, I. & Elisseeff, A. An introduction to variable and feature selection. J. Mach. Learn. Res. 3, 1157-1182 (2003). [11] Tibshirani, R. Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B 58, 267-288 (1996). [12] Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A. & Lawrence, N. D. Dataset Shift in Machine Learning (MIT Press, 2009). [13] Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F. & Vaughan, J. W. A theory of learning from different domains. Mach. Learn. 79, 151-175 (2010). [14] Sagawa, S., Koh, P. W., Hashimoto, T. B. & Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations (2020). [15] Koh, P. W. et al. WILDS: A benchmark of in-the-wild distribution shifts. In Proc. 38th Int. Conf. Mach. Learn. (2021). [16] Geirhos, R. et al. Shortcut learning in deep neural networks. Nat. Mach. Intell. 2, 665-673 (2020). [17] Amodei, D. et al. Concrete problems in AI safety. arXiv https://arxiv.org/abs/1606.06565 (2016). [18] Recht, B., Roelofs, R., Schmidt, L. & Shankar, V. Do ImageNet classifiers generalize to ImageNet? In Proc. 36th Int. Conf. Mach. Learn. 5389–5400 (2019). [19] Hendrycks, D. & Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (2019). [20] Ovadia, Y. et al. Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32 (2019). [21] Gulrajani, I. & Lopez-Paz, D. In search of lost domain generalization. In International Conference on Learning Representations (2021). [22] Little, R. J. A. & Rubin, D. B. Statistical Analysis with Missing Data. 3rd edn (Wiley, 2019).