Paper deep dive
Beyond Static Snapshots: A Grounded Evaluation Framework for Language Models at the Agentic Frontier
Jazmia Henry
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 4/27/2026, 8:20:32 AM
Summary
The paper identifies four systematic failures in current LLM evaluation frameworks (distributional, temporal, scope, and process invalidity) that make them inadequate for agentic systems and RLHF. It proposes the Grounded Continuous Evaluation (GCE) framework and introduces ISOPro, a simulation-based fine-tuning system. ISOPro replaces learned reward models with deterministic verifiers to eliminate reward hacking and uses LoRA adapter updates on CPU to reduce hardware barriers. Validation on a resource-constrained scheduling domain shows that ISOPro achieves a 3x accuracy improvement over zero-shot baselines and demonstrates capability emergence through an implicit curriculum.
Entities (12)
Relation Signals (7)
Grounded Continuous Evaluation â proposes â ISOPro
confidence 100% · We propose the Grounded Continuous Evaluation (GCE) framework and present ISOPro
Grounded Continuous Evaluation (GCE) â proposes â ISOPro
confidence 100% · We propose the Grounded Continuous Evaluation (GCE) framework and present ISOPro
ISOPro â replaces â Learned Reward Model
confidence 100% · ISOPro replaces the learned reward model with a deterministic ground-truth verifier
DeepSeek-R1 â uses â GRPO
confidence 100% · DeepSeek-R1âs GRPO [8]
ISOPro â uses â LoRA
confidence 100% · operates on LoRA adapter weights updatable on CPU
ISOPro â uses â LoRA
confidence 100% · operates on LoRA adapter weights updatable on CPU
ISOPro â validatedon â Resource-Constrained Project Scheduling (RCPSP)
confidence 100% · We validate ISOPro on a resource-constrained scheduling domain
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We argue that current evaluation frameworks for large language models (LLMs) suffer from four systematic failures that make them structurally inadequate for assessing deployed, agentic systems: distributional invalidity (evaluation inputs do not reflect real interaction distributions), temporal invalidity (evaluations are post-hoc rather than training-integrated), scope invalidity (evaluations measure single-turn outputs rather than long-horizon trajectories), and process invalidity (evaluations assess outputs rather than reasoning). These failures compound critically in RLHF, where reward models are evaluated under conditions that do not hold during RL training, making reward hacking a predictable consequence of evaluation design rather than a training pathology. We propose the Grounded Continuous Evaluation (GCE) framework and present ISOPro, a simulation-based fine-tuning and evaluation system. ISOPro replaces the learned reward model with a deterministic ground-truth verifier, eliminating reward hacking by construction in verifiable-reward domains, and operates on LoRA adapter weights updatable on CPU, reducing the hardware barrier by an order of magnitude. We validate ISOPro on a resource-constrained scheduling domain with six difficulty tiers, demonstrating capability emergence visible only through continuous evaluation, an implicit curriculum that forms without researcher curation, and a 3x accuracy improvement over zero-shot baselines, all on consumer hardware with 0.216% trainable parameters.
Tags
Links
- Source: https://arxiv.org/abs/2604.17573v1
- Canonical: https://arxiv.org/abs/2604.17573v1
Trouble viewing inline? Open PDF directly â
Full Text
28,806 characters extracted from source content.
Expand or collapse full text
Beyond Static Snapshots: A Grounded Evaluation Framework for Language Models at the Agentic Frontier Jazmia Henry University of Oxford Stanford HAI Collide jazmia.henry@gmail.com github.com/iso-ai/isopro Abstract We argue that current evaluation frameworks for large language models (LLMs) suffer from four systematic failures that make them structurally inadequate for assessing deployed, agentic systems: distributional invalidity (evaluation inputs do not reflect real interaction distributions), temporal invalidity (evaluations are post-hoc rather than training-integrated), scope invalidity (evaluations measure single-turn outputs rather than long-horizon trajectories), and process invalidity (evaluations assess outputs rather than reasoning). These failures compound critically in RLHF, where reward models are evaluated under conditions that do not hold during RL trainingâmaking reward hacking a predictable consequence of evaluation design rather than an unpredictable training pathology. We further argue that RLHFâs dual-model architecture imposes a structural compute and memory barrier that limits evaluation accessibility and reproducibility. We propose the Grounded Continuous Evaluation (GCE) framework and present ISOPro, a simulation-based fine-tuning and evaluation system, as a reference implementation. ISOPro replaces the learned reward model with a deterministic ground-truth verifierâeliminating reward hacking by construction in verifiable-reward domainsâand operates on LoRA adapter weights updatable on CPU, reducing the hardware barrier by an order of magnitude. We validate ISOPro on a resource-constrained scheduling domain with six difficulty tiers, demonstrating capability emergence visible only through continuous evaluation, an implicit curriculum that forms without researcher curation, and a 3.0Ă accuracy improvement over zero-shot baselinesâall on consumer hardware with 0.216% trainable parameters. We situate our work alongside DeepSeek-R1âs GRPO [8], which independently arrived at the same architectural conclusion at scale: for verifiable-reward domains, the verifier is the reward signal. 1 Introduction The dominant paradigm for evaluating large language models rests on a deceptively simple assumption: that performance on curated benchmark tasks predicts capability in deployment. For single-turn tasks evaluated against a reference answer, this assumption is defensible. For agentic systems that plan across long horizons, use tools, and adapt to dynamic environments, it is not. Current evaluation practice fails on four dimensions that compound as deployment complexity increases. Benchmark inputs are drawn from researcher-constructed distributions that do not reflect actual interactions. Evaluations are conducted at discrete checkpoints, making training dynamics invisible. Single-turn output evaluation cannot assess multi-step trajectory quality. And evaluating outputs rather than reasoning rewards correct-seeming outputs produced by gameable reasoning. These failures have an institutional dimension: RLHF [4]âthe dominant remedyâintroduces architectural requirements that create a substantial hardware barrier. Enforcing the KL penalty requires loading two full model copies simultaneously. For a 7B model in half-precision, this requires âŒ28 28GB of VRAM before accounting for optimizer states. This structural constraint limits RLHF-based evaluation research to organizations with data center GPU access, reducing the diversity of approaches and slowing progress on evaluation methodology. We make four contributions. First, we formalize a taxonomy of evaluation failure modes organized around measurement validity theory [11]. Second, we show these failures compound in RLHF, where reward model invalidity makes reward hacking structurally predictable. Third, we argue that RLHFâs architecture imposes a reproducibility barrier that simulation-based alternatives eliminate. Fourth, we propose GCE and present ISOPro as a reference implementation, validated on a constraint-satisfaction scheduling domain that demonstrates capability emergence, implicit curriculum formation, and evaluation observabilityâall on a consumer laptop. 2 Background and Related Work The Benchmark Paradigm. HELM [1] broadens evaluation coverage across scenarios and models. BIG-Bench [2] crowdsources tasks probing emergent abilities. Both inherit structural assumptions: researcher-curated inputs, post-hoc timing, and output-level assessment. Human Preference Evaluation. Chatbot Arena [3] grounds evaluation in real user interactions, meaningfully addressing distributional invalidity at the input level. However, human raters assess final responses, not the reasoning that produced them. As reasoning becomes less transparent, output-grounded preference evaluation becomes less reliable as a capability signal. Process Reward Models. PRMs [7] address process invalidity by scoring intermediate reasoning steps. However, PRMs do not address distributional invalidity (step labels are collected under annotation conditions), temporal invalidity (RM evaluation precedes RL training), or the hardware barrierâtraining a PRM requires the full RLHF compute stack. GRPO and Verifiable-Reward Training. DeepSeek-R1 [8] demonstrated at frontier scale what ISOPro implements at accessible scale: for verifiable-reward domains, the reward model can be replaced entirely by a ground-truth verifier. GRPO generates multiple rollouts per prompt, scores them against a verifier, and uses relative scores as the reward signalâwithout a separate reward model or frozen reference model. Chain-of-thought reasoning emerges as a byproduct. The RLHF Evaluation Gap. Standard practice evaluates the RM on held-out preference pairs from the same annotation processâwith three critical limitations: annotation-condition inputs do not reflect policy-generated inputs during RL; the RM is evaluated before RL begins and not monitored thereafter; and pairwise preference annotation cannot assess RM reliability across long-horizon interactions. The consequenceâreward hackingâis treated as a training instability. We argue it is more accurately characterized as an evaluation design failure. 3 A Taxonomy of Evaluation Failure Modes We organize evaluation failures around measurement validity theory [11]. An evaluation is valid to the extent that it measures what it purports to measure. We identify four distinct validity failures. Distributional Invalidity. An evaluation exhibits distributional invalidity when the distribution of evaluation inputs does not reflect deployment inputs. In RLHF, this manifests at the reward model level: preference annotations are collected under annotation conditions, not deployment conditions. The RM is optimized for annotation-condition inputs; it is deployed against policy-generated inputs that increasingly diverge as RL progresses. Temporal Invalidity. An evaluation exhibits temporal invalidity when conducted at discrete time points rather than continuously, making it unable to capture capability dynamics. In RLHF, the RM is evaluated before RL begins, under conditions that assume the policy remains close to the SFT baseline. As the policy drifts, RM reliability degrades on out-of-distribution inputsâinvisible to pre-training evaluation. Scope Invalidity. An evaluation exhibits scope invalidity when the unit of assessment (individual output) does not match the unit of deployment (multi-step trajectory). Pairwise preference annotation cannot assess RM coherence across long-horizon trajectories, enabling RM inconsistency undetectable by single-turn evaluation. Process Invalidity. An evaluation exhibits process invalidity when it assesses only final outputs, unable to distinguish correct reasoning from correct-looking outputs produced by gameable reasoning. In RLHF, this is the direct mechanism of sycophancy: reward models learn that agreement is preferred regardless of correctness. Table 1: Taxonomy of evaluation failure modes, their definitions, and manifestations in base model and RLHF evaluation. Failure Mode Definition Base Model Eval RLHF / RM Eval Distributional Eval inputs â deployment distribution Researcher-curated â real interaction Annotation pairs â RL-training inputs Temporal Point-in-time misses dynamics Training dynamics invisible Pre-RL eval misses RM degradation Scope Single-turn â trajectory Output eval misses multi-step Pairwise â long-horizon coherence Process Output eval misses reasoning Correct outputs mask flawed reasoning RM rewards preferred outputs 4 The Grounded Continuous Evaluation (GCE) Framework GCE is organized around three principles, each addressing a subset of the identified failure modes. Principle 1: Interaction-Grounded Prompt Sampling. Addressing distributional invalidity requires evaluation inputs sampled from distributions reflecting real interactionsâcharacterizing deployment distributions empirically, constructing evaluation datasets whose marginals match, and validating through coverage metrics. Principle 2: Training-Integrated Continuous Evaluation. Addressing temporal invalidity requires evaluation integrated into training: lightweight probes at training frequency, metrics detecting capability dynamics rather than absolute levels, and instrumentation for trajectory analysis. Principle 3: Simulation-Based Agentic Assessment. Addressing scope and process invalidity requires evaluating behavior across trajectories in structured environments, enabling trajectory-level assessment, failure analysis, recovery behavior evaluation, intermediate checkpoint scoring, and counterfactual probing. 5 ISOPro: A Reference Implementation of GCE ISOPro is a simulation-based fine-tuning and evaluation framework demonstrating that GCE is practically implementable. Each mechanism addresses one or more failure modes from our taxonomy. 5.1 Architecture Overview ISOPro consists of three components: a simulation environment layer providing structured task contexts with deterministic verifiers, an AI agent layer (LLM) generating responses, and a communication wrapper managing state, evaluation, and feedback loops. The framework supports parallel and sequential execution modes and multiple simulation environments, including mathematical reasoning, domain-specific task environments (scheduling, engineering), and multi-agent orchestration. 5.2 Mechanism 1: Gradient Descent on Correct Reasoning Traces When the model produces a correct answer, ISOPro runs a forward pass with prompt tokens masked (labels set to â100-100). The loss is computed only on generated tokensâthe gradient signal is the reasoning sequence that produced correctness. LoRA adapter weights shift toward that pattern. This is process-level supervision: the model is trained on the reasoning trajectory, not correctness as a label. 5.3 Mechanism 2: Rejection Sampling as Continuous Self-Filter The model generates responses at high temperature (T=0.8T=0.8) and a ground-truth verifier determines correctness. Only verified correct responses enter the replay buffer. This creates a continuous evaluation regime: at every iteration, the modelâs capability is evaluated against ground truth. Capability dynamics are visible in the buffer compositionâgranularity that checkpoint evaluation cannot provide. 5.4 Mechanism 3: Implicit-Curriculum Replay Buffer Correct rollouts accumulate across iterations, creating a training distribution anchored to actual capability. Easy wins dominate early; harder problems enter as capability develops. The curriculum emerges from the modelâs trajectory rather than researcher curation. By iteration N, the model trains on everything solved correctly in iterations 1 through Nâcompounding correct signal without compounding error. 5.5 Mechanism 4: Activation-Guided LoRA Targeting ISOPro identifies the most active layers on the target domain through activation probing and concentrates LoRA updates there. In our scheduling experiments, query and value projections in layers 28â35 were identified as the primary locus of constraint reasoning. LoRA updates target 6.6M parameters (0.216% of 3.1B total). Critically, LoRA weights are updated on CPUâthe base model stays frozen in quantized form, enabling consumer-hardware operation. 5.6 Experimental Validation: Scheduling Domain We validate ISOPro on resource-constrained project scheduling (RCPSP)âa verifiable-reward problem requiring constraint satisfaction across precedence dependencies, resource capacity limits, and deadlines. Problems are generated programmatically at six difficulty tiers, solved by an OR-Tools CP-SAT solver for ground truth, and rendered as natural language prompts. Tier design. T0 (4-job warmup, dependencies only) through T5 (10-job full composition with all constraints). Tiers 0â4 appear in training; Tier 5 is held out for compositional generalization. The key compositional structure: T1 tests sequencing, T2 tests resource allocation, T3 tests deadline satisfaction, T4 combines pairs of constraints, and T5 combines all threeânever seen during training. Evaluation conditions. Five conditions: (1) zero-shot prompting, (2) 3-shot in-context learning, (3) IsoZero simulation-based multi-step reasoning (no training), (4) multi-turn revision with verifier feedback (no training), and (5) ISOPro rejection-sampling fine-tuning with LoRA. Hardware. All experiments run on an Apple M1 with 32GB unified memory using Qwen 2.5 3B Instruct via MLX [17]. The full pipelineâ6 training iterations, 504 rollouts, 119 training traces, and per-tier evaluationâcompletes in approximately 90 minutes. Peak memory: under 8GB. No GPU required. Figure 1: Per-tier accuracy across all evaluation conditions. ISOPro + LoRA (red) achieves the highest accuracy on T0 (100%), T1 (66.7%), and T3 (66.7%). T2, T4, and T5 remain at 0% across all methods, indicating capability boundaries visible through the GCE frameworkâs per-tier continuous evaluation. 5.7 Results Table 2 presents the full results. ISOPro achieves 39.8%±3.539.8\%± 3.5 mean accuracy across three seedsâa 3.0Ă3.0Ă improvement over zero-shot (13.3%)âwhile training on only self-generated correct reasoning traces with no oracle solutions, no reward model, and no KL penalty. Table 2: Full results across all methods on Qwen 2.5 3B Instruct in the scheduling domain. Method T0 T1 T2 T3 T4 T5 Mean Trains? Zero-shot 80% 0% 0% 0% 0% 0% 13.3% No 3-shot 20% 0% 0% 20% 0% 0% 6.7% No Multi-turn (Ă3) 100% 0% 0% 0% 0% 0% 16.7% No IsoZero (sim.) 100% 60% 20% 20% 0% 0% 33.3% No ISOPro + LoRA 100% 66.7% 0% 66.7% 0% 0% 39.8% Yes Eval: 3â5 problems per tier. ISOPro: 6 iterations, 504 rollouts, 119 correct traces, mean across 3 seeds. Hardware: Apple M1, 32GB. Capability emergence (T1: 0%â 66.7%). The model learns sequencing entirely through self-generated correct tracesâa capability inaccessible through any form of prompting. This demonstrates that the rejection sampling loop bootstraps genuine capability acquisition, not mere prompt reformulation. 3-shot degradation (13.3%â 6.7%). Few-shot in-context learning degrades performance on constraint satisfaction. The long prefix causes format drift on harder tiers, consistent with recent work on prompt sensitivity in structured output domains. This finding supports the distributional invalidity argument: researcher-curated examples can actively mislead. Multi-turn failure pattern. Multi-turn revision achieves 100% on T0 but 0% recovery on T1âT5. The model oscillates between fixing one constraint and breaking anotherâa failure pattern visible only through trajectory-level assessment, validating the scope validity argument (Section 3). Training inflection. The rollout hit rate doubles from 15% to 30% between iterations 2 and 3 (Figure 3), coinciding with the sharpest loss decrease (0.351â 0.232). This inflectionâvisible only through continuous evaluationâvalidates the temporal validity claim. Figure 2: Training dynamics: rollout hit rate (blue) and cross-entropy loss (red). The inflection at iteration 3 marks when the modelâs correct rollout rate doubledâvisible only through continuous evaluation. Figure 3: Implicit curriculum: replay buffer composition over training iterations. T0 traces dominate early; T1, T3, and T4 enter as capability developsâa training distribution anchored to actual capability. Implicit curriculum dynamics. Figure 3 reveals the implicit curriculum. T0 warmup traces dominate early iterations. T3 deadline traces grow steadily. T1 sequencing enters at iteration 2. T4 pairwise (8-job, dual-constraint problems) first appears at iteration 3 with 2 traces. By iteration 6, the buffer contains 60 T0, 39 T3, 18 T1, and 2 T4 tracesâa training distribution that emerged from the modelâs capability trajectory without researcher curation. Capability emergence heatmap. Figure 4 captures the exact iteration when each tier enters the buffer: T0 and T3 from iteration 1 (base model capability), T1 at iteration 2 (LoRA unlocked sequencing), T4 at iteration 3 (compositional reasoning emerging). T2 and T5 remain unreachedâhonest limitations suggesting resource reasoning requires either more iterations or architectural changes. Figure 4: Capability emergence heatmap. Each cell shows when a tier first produces correct traces in the buffer. T2 Resource and T5 Full remain unreached after 6 iterationsâthe framework honestly reports capability boundaries. 5.8 Ablation Study We ablate two of ISOProâs mechanisms against the full configuration on the scheduling domain, averaging across three seeds (42, 123, 456) with six training iterations each (Table 3). Table 3: Ablation results (mean ± std across 3 seeds). Both chain-of-thought and buffer accumulation contribute meaningfully; buffer removal additionally destabilizes training. Condition Mean ± Std Î vs Full Note Full ISOPro 39.8±3.5%39.8± 3.5\% â baseline No chain-of-thought 31.5±2.6%31.5± 2.6\% â8.3-8.3p largest loss on T3 No buffer accumulation 27.8±13.6%27.8± 13.6\% â12.0-12.0p 4Ă4Ă variance Random LoRA layers 40.7±2.6%40.7± 2.6\% +0.9+0.9p (ns) see Appendix A Removing chain-of-thought prompting from training and evaluation drops accuracy to 31.5%±2.631.5\%± 2.6 (â8.3-8.3p), with the largest losses on T3 Deadline Pressure (67%â34%67\%â 34\%)âstructured reasoning is a substantial contributor on deadline-constrained tiers. Removing buffer accumulation, so that each iteration trains only on current-iteration correct traces rather than the accumulated replay buffer, drops accuracy to 27.8%±13.627.8\%± 13.6 (â12.0-12.0p) and increases seed-to-seed variance nearly 4Ă4Ă. The implicit-curriculum buffer therefore serves two distinct functions: raising mean accuracy and stabilizing training across seeds (Figure 5). Both ablations leave T2 Resource and T5 Full Composition at 0%, consistent with the main resultsâthese tiers mark capability boundaries not crossed by any configuration in this compute budget. Figure 5: Ablation results (mean ± std, n=3n=3 seeds). Left: per-tier accuracy by condition. The No Buffer condition shows high variance and a spurious T3 advantage driven by seed instability. Right: overall mean accuracy. Buffer accumulation (â12.0-12.0p, 4Ă4Ă variance) and chain-of-thought (â8.3-8.3p) are both load-bearing mechanisms. 6 A Principled Comparison: ISOPro vs. RLHF 6.1 The Reward Model as Evaluation Failure Point The PPO objective maximizes â[râ(x)]âÎČâ KLâ(Ïâ„Ïref)E[r(x)]-ÎČ·KL(Ï\| _ref). Because râ(x)r(x) is a learned proxy, the RL policy finds the reward modelâs blind spotsâproducing fluent, confident, wrong outputs that score well. ISOPro replaces the RM with a deterministic verifier, perfectly calibrated by construction. There is no score to game. Reward hacking is eliminated architecturally. 6.2 The Dual-Model VRAM Problem RLHFâs KL penalty requires both ÏΞ _Ξ and Ïref _ref in GPU memory simultaneouslyââŒ4âP 4P bytes for P parameters in half-precision, before optimizer states. At 7B: âŒ28 28GB; at 70B: âŒ280 280GB. ISOPro requires no reference model. LoRA adapters (6.6M parameters) update on CPU; our pipeline runs with under 8GB peak memory. 6.3 Honest Tradeoffs RLHF retains genuine advantages: cold start (SFT provides signal from step 0; ISOPro needs independent correct generation), partial credit (graded vs. binary scores), reasoning diversity (ISOPro is bounded by self-generated sequences), and non-verifiable domains (no verifier exists for safety or style judgments). GCE extends to such domains through rubric-based trajectory assessment, but reward hacking elimination holds only for verifiable rewards. 7 Convergence with DeepSeek-R1 DeepSeek-R1 [8] demonstrated at 671B parameters what ISOPro implements on a 3B model on a laptop: for verifiable-reward domains, the learned reward model is an unnecessary intermediary. This convergence reflects a structural insight: the learned RM introduces all four validity failures from our taxonomyâdistributional, temporal, scope, and processâwhich the deterministic verifier avoids by construction. Table 4 summarizes the architectural comparison. Table 4: Architectural comparison between ISOPro, standard RLHF, and DeepSeek-R1 GRPO. ISOPro (Ours) RLHF (Standard) GRPO Reward Signal Deterministic verifier Learned RM Deterministic verifier Stability Rejection sampling KL penalty (dual model) Group-relative advantages Models in Memory 1 2+ 1 Trainable Params 0.216% (6.6M) 100% (full) 100% (full) Min. Memory ⌠6 GB (3B) ⌠28 GB (7BĂ2) ⌠280 GB (70B) Hardware Consumer laptop Data center GPU GPU cluster Reward Hacking Impossible Predictable Impossible 8 Discussion 8.1 Limitations and Scope GCE is a framework requiring empirical validation per deployment context. Our results demonstrate implementability on one domain; broader validation across tasks, model families, and scales is necessary. Per-tier sample sizes (3â5 problems) mean individual tier accuracies are directional estimates, though multi-seed averaging (n=3n=3) and ablation analysis (Table 3) confirm the main findings are robust. T2 and T5 at 0% across all configurations suggest resource reasoning requires more iterations, scaffolding, or stronger base models. For non-verifiable domains, the deterministic verifier could be replaced by a calibrated rubric evaluator scoring trajectory segments against predefined behavioral criteria, with the rejection sampler accepting only trajectories exceeding a confidence threshold; we leave this extension to future work. 8.2 Broader Implications GCE complements scalable oversight approaches (debate, recursive reward modeling) by addressing how to structure evaluation so judgments are collected under deployment-representative conditions. Evaluation validity is safety-relevant: if evaluations fail to detect capability degradation and reward specification gaming, they provide unreliable signal about deployed system safety. The accessibility argument has a safety dimension: evaluation methodology advances fastest when reproducible across diverse research environments. 8.3 Research Directions Extended training (20+ iterations) would reveal whether the implicit curriculum reaches T5 composition. Cross-domain transfer to formal logic, code generation, and scientific computing requires only a new deterministic verifier. Scaling to 8B and 70B models would test whether resource reasoning emerges at scale [13]. 9 Conclusion We have argued that current LLM evaluation frameworks exhibit four systematic validity failures that make them structurally misaligned with the agentic deployment context, and that these failures compound in RLHFâwhere architectural requirements additionally impose a hardware barrier constraining evaluation research to well-resourced organizations. GCE responds to all four failure modes through interaction-grounded sampling, training-integrated continuous evaluation, and simulation-based agentic assessment. ISOPro demonstrates this framework is implementable at consumer-hardware scale: a deterministic verifier eliminates reward hacking by construction; CPU-updatable LoRA adapters eliminate the dual-model constraint; rejection sampling provides continuous evaluation integrated into training; and an implicit-curriculum replay buffer grounds the training distribution in the modelâs actual capability trajectory. The convergence between ISOPro and DeepSeek-R1âs GRPOâsystems developed independently at orders-of-magnitude different scalesâreflects a structural insight: for verifiable-reward domains, the verifier is the reward signal, and the learned reward model is an unnecessary intermediary. Recognizing reward hacking as an evaluation failure rather than a training failure is the first step toward addressing it systematically. Code and Data Availability. ISOPro is open-source and available at https://github.com/iso-ai/isopro. The scheduling problem generator, deterministic verifier, ablation configurations, and evaluation scripts are included. A patent application for the simulation evaluation platform is pending (USPTO). References Liang et al. [2022] P. Liang, R. Bommasani, T. Lee, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. Srivastava et al. [2022] A. Srivastava, A. Rastogi, A. Rao, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. Zheng et al. [2023] L. Zheng, W.-L. Chiang, Y. Sheng, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In NeurIPS, 2023. Ouyang et al. [2022] L. Ouyang, J. Wu, X. Jiang, et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022. Stiennon et al. [2020] N. Stiennon, L. Ouyang, J. Wu, et al. Learning to summarize with human feedback. In NeurIPS, 2020. Gao et al. [2022] L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210.10760, 2022. Lightman et al. [2023] H. Lightman, V. Kosaraju, Y. Burda, et al. Letâs verify step by step. arXiv preprint arXiv:2305.20050, 2023. DeepSeek-AI [2025] DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Bai et al. [2022] Y. Bai, A. Jones, K. Ndousse, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022. Denison et al. [2024] C. Denison, F. Barez, D. Duvenaud, et al. Sycophancy to subterfuge: Investigating reward tampering in language models. arXiv preprint arXiv:2406.10162, 2024. Messick [1989] S. Messick. Validity. In R. L. Linn, editor, Educational Measurement, pages 13â103. American Council on Education, 3rd edition, 1989. Brown et al. [2020] T. Brown, B. Mann, N. Ryder, et al. Language models are few-shot learners. In NeurIPS, 2020. Wei et al. [2022] J. Wei, Y. Tay, R. Bommasani, et al. Emergent abilities of large language models. TMLR, 2022. Hu et al. [2022] E. J. Hu, Y. Shen, P. Wallis, et al. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. Schulman et al. [2017] J. Schulman, F. Wolski, P. Dhariwal, et al. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Qwen Team [2024] Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. Hannun et al. [2023] A. Hannun et al. MLX: Efficient machine learning on Apple Silicon. Apple Machine Learning Research, 2023. Shinn et al. [2023] N. Shinn, F. Cassano, A. Gopinath, et al. Reflexion: Language agents with verbal reinforcement learning. In NeurIPS, 2023. Appendix A Activation-Guided vs. Random LoRA Layer Selection ISOProâs default targets the top-K layers identified by activation probing (layers 28â35 in the scheduling domain). To test whether this selection matters, we replaced it with uniformly random layer selection (same K=8K=8, same LoRA rank, same seeds). Random selection reached 40.7%±2.640.7\%± 2.6 mean accuracyâstatistically indistinguishable from the 39.8%±3.539.8\%± 3.5 of the activation-guided configuration. Loss curves, hit-rate ramps, buffer growth, and wall-clock time (89.0 vs. 90.9 min) were all within noise across every training iteration. We interpret this as evidence that at this scale (3B parameters, rank-16 LoRA, 8 of 36 layers, 6 iterations), LoRA placement is not load-bearing: the rank and training signal dominate, and placement heuristics may only separate from random at larger scale, higher rank, or on harder distributions. We retain activation-guided targeting in the default configuration because it adds negligible overhead, but we do not claim it as a contribution of this work.