Paper deep dive
The Informational Cost of Agency: A Bounded Measure of Interaction Efficiency for Deployed Reinforcement Learning
Wael Hafez, Cameron Reid, Amit Nazeri
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/20/2026, 1:39:09 AM
Summary
The paper introduces Bipredictability (P), an information-theoretic metric quantifying the efficiency of closed-loop interaction between an agent and its environment. It establishes a classical bound P โค 0.5 and demonstrates that responsive agency suppresses P to approximately 0.33, termed the 'informational cost of agency.' The authors propose the Information Digital Twin (IDT), a model-agnostic architecture that computes P from external interaction streams to monitor runtime reliability. Empirical results show IDT detects 89.3% of coupling degradations with 4.4x lower latency compared to reward-based monitoring.
Entities (8)
Relation Signals (6)
Information Digital Twin โ computes โ Bipredictability
confidence 96% ยท The Information Digital Twin (IDT), a model agnostic architecture that computes P from the external interaction stream
Bipredictability โ hasbound โ 0.5
confidence 95% ยท P admits a provable classical bound P equal, smaller than 0.5, derived from Shannon entropy subadditivity
Bipredictability โ measures โ interaction efficiency
confidence 95% ยท quantifies how efficiently a closed loop interaction between agent and environment converts uncertainty into shared predictability
Information Digital Twin โ outperforms โ reward-based monitoring
confidence 94% ยท detects 89.3% of coupling degradations against 44.0% for reward based monitoring, with 4.4 times lower latency.
Informational Cost of Agency โ causes โ suppression of Bipredictability
confidence 93% ยท responsive agency necessarily suppresses P below this ceiling, a structural prediction we term the informational cost of agency.
Bipredictability โ hasvalue โ 0.33
confidence 92% ยท Across 21 trained continuous control agents, we confirm this prediction empirically at P = 0.33 plus minus 0.02.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deployed reinforcement learning systems lack a principled runtime reliability theory. We close this gap by introducing Bipredictability, P, a closed form information theoretic metric that quantifies how efficiently a closed loop interaction between agent and environment converts uncertainty into shared predictability. P admits a provable classical bound P equal, smaller than 0.5, derived from Shannon entropy subadditivity, and responsive agency necessarily suppresses P below this ceiling, a structural prediction we term the informational cost of agency. Across 21 trained continuous control agents, we confirm this prediction empirically at P = 0.33 plus minus 0.02. The same suppression signature reproduces in language model dialogue, convolutional vision systems, and classical mechanical baselines, indicating that P captures a substrate independent property of agentic interaction rather than an algorithm specific artifact. The Information Digital Twin, IDT, a model agnostic architecture that computes P from the external interaction stream, detects 89.3% of coupling degradations against 44.0% for reward based monitoring, with 4.4 times lower latency. P provides the missing measurement layer for runtime reliability and closed loop self regulation in deployed autonomous systems.
Tags
Links
- Source: https://arxiv.org/abs/2603.01283v3
- Canonical: https://arxiv.org/abs/2603.01283v3
Trouble viewing inline? Open PDF directly โ
Full Text
43,846 characters extracted from source content.
Expand or collapse full text
1 The Informational Cost of Agency: A Bounded Measurement Layer for Deployed Reinforcement Learning AbstractโDeployed reinforcement learning systems lack a principled runtime reliability theory. We close this gap by introducing Bipredictability (ํ), a closed-form information-theoretic metric that quantifies how efficiently a closed-loop interaction converts uncertainty into shared predictability between agent and environment. ํ admits a provable classical bound ํ โค 0.5, derived from Shannon entropy subadditivity, and responsive agency necessarily suppresses ํ below this ceiling โ a structural prediction we term the informational cost of agency. Across 21 trained continuous-control agents, we confirm this prediction empirically at ํ = 0.33 ยฑ 0.02. The same suppression signature reproduces in language-model dialogue, convolutional vision systems, and classical mechanical baselines, indicating that ํ captures a substrate-independent property of agentic interaction rather than an algorithm-specific artifact. The Information Digital Twin (IDT), a model-agnostic architecture that computes ํ from the external interaction stream, detects 89.3% of coupling degradations against 44.0% for reward-based monitoring, with 4.4ร lower latency. ํ provides the missing measurement layer for runtime reliability and closed-loop self-regulation in deployed autonomous systems. 1 Introduction The deployment of reinforcement learning (RL) agents (Sutton & Barto, 2018) into real-world environments has outpaced the development of principled frameworks for their runtime monitoring and assurance. RL agents exhibit complex behaviors derived from a continuous observationโactionโoutcome loop, and once a policy is frozen and deployed, the agentโenvironment coupling that supported training-time performance can be disrupted by environmental shifts, sensor degradation, or actuator drift โ often silently, and often before any reward-visible failure (Dulac-Arnold et al., 2021; Josifovski et al., 2024; Levine et al., 2020). The field's most impressive demonstrations (Kaufmann et al., 2023; Radosavovic et al., 2023; Tang et al., 2024) were achieved without any principled runtime measure of whether the deployed policy remains coupled to its environment. Current reliability is managed through paradigms that address pieces of the problem but not the gap itself: out-of-distribution detection (Liang et al., 2018; Koh et al., 2021) and concept-drift methods (Greco et al., 2025) flag inputs that look unfamiliar; reward-based monitoring flags outcomes that look bad. Both are reactive, both require degradation to accumulate before deviating from baseline, and neither measures the closed-loop interaction itself. Model-internal diagnostics, where they exist, are frequently unavailable in third-party or black-box deployments. Wael Hafez Semarx Research LLC Alexandria, VA, USA. w.hafez@semarx.com Cameron Reid Semarx Research LLC Alexandria, VA, USA cameron.reid@semarx.com Amir Nazeri Semarx Research LLC Alexandria, VA, USA amir.nazeri@semarx.com 2 We introduce Bipredictability (ํ) as a substrate-independent metric that quantifies the fraction of the total observationโactionโoutcome uncertainty budget converted into shared predictability across the closed-loop interaction. Whereas a policy describes how the agent acts; ํ describes how well the agent and environment remain coupled. This makes ํ sensitive to coupling degradation regardless of whether the policy itself has changed. Unlike empowerment (Klyubin et al., 2005; Salge et al., 2014; Mohamed & Rezende, 2015) and active inference (Friston, 2009; Friston et al., 2017), which characterize the agent's potential influence or its drive to minimize prediction error, ํ measures the realized coupling against the interaction's full uncertainty budget at runtime, and is structurally bounded by the geometry of that budget. We establish that all classical interactions are subject to a structural ceiling ํ โค 0.5, derived from Shannon entropy subadditivity (Shannon, 1948). The transition from passive coupling to responsive agency โ where actions depend on observations โ necessarily suppresses ํ below this ceiling, a phenomenon we term the informational cost of agency. Our primary contribution is the demonstration that this cost is not an artifact of any specific implementation but a structural signature of agency. Empirical validation across 21 continuous-control agents trained with SAC and PPO in the MuJoCo HalfCheetah environment yields a consistent baseline of ํ = 0.33 ยฑ 0.02. This finding is corroborated by independent measurements across substrates that share no dynamical structure with continuous- control RL: multi-turn language-model dialogue (Hafez & Nazeri, 2026), convolutional vision systems (Nazeri & Hafez, 2025), and a non-agentic classical mechanical baseline (Hafez et al., 2026). The framework extends theoretically to quantum-entangled systems, where the classical bound generalises to ํ โค 1 under unitary evolution (Hafez et al., 2026). The convergence of agentic systems toward ํ โ1/3 across these heterogeneous settings indicates that ํ functions as a structural property of the closed-loop interaction itself, not a property of any specific algorithm. To operationalize this measurement for autonomous systems engineering, we present the Information Digital Twin (IDT), a model-agnostic architecture that computes ํ from the external interaction stream without access to policy weights, internal activations, or reward signals. Across 168 perturbation trials spanning eight perturbation types and two policy architectures, IDT-based monitoring detected 89.3% of coupling degradations versus 44.0% for reward- based monitoring, with 4.4ร lower median latency. The empirical results presented here rest on two foundations: a theoretical framework derived from first principles in a companion paper (Hafez et al., 2026), and an architecture โ the IDT โ that operationalizes the framework as a real-time measurement instrument. Section 2 summarizes the framework. Section 3 describes the IDT architecture. Section 4 presents results, including the cross-substrate convergence. Section 5 discusses implications for runtime reliability and self-regulation. Section 6 details experimental methods. 2 Bipredictability framework 2.1 Uncertainty resolution requires an information-theoretic framework Deployment monitoring is fundamentally a question about uncertainty resolution: whether observations and actions reduce uncertainty about outcomes, and whether outcomes constrain what the agent must have done. Information theory operationalizes this directly โ entropy quantifies uncertainty, mutual information quantifies its resolution. The minimal sufficient description of a closed-loop interaction requires three elements: what the agent perceives (ํ), what it does (ํด), and what results (ํโฒ). Information theory maps naturally onto this structure. Each variable carries uncertainty quantified by its Shannon entropy โ ํป(ํ),ํป(ํด), and ํป(ํโฒ) โ and the degree to which they resolve each other's uncertainty is captured by mutual information. Figure 1 makes this geometry explicit: each circle represents the entropy of one variable, their central overlap represents the shared predictability ํํผ(ํ,ํด; ํโฒ) realized across the full loop, and the total area represents the complete uncertainty budget ํถ available in the interaction. The deployment monitoring question then has a precise geometric form: what fraction of the total area is the overlap? 3 Conditional entropy quantifies what remains unresolved: ํป(ํโฒ|ํ,ํด) is the region of ํโฒ falling outside its overlap with (ํ,ํด) in Figure 1, and ํป(ํ,ํด|ํโฒ) is the region of (ํ,ํด) falling outside its overlap with ํโฒ. When the overlap is large relative to the total area, each side of the interaction reliably predicts the other; when it is small, the loop is informationally loose. Figure 1. Information-theoretic structure of the observationโactionโoutcome interaction. Each circle represents the entropy of one variable: observations ํป(ํ), actions ํป(ํด), and outcomes ํป(ํโฒ). The central overlap is the mutual information ํํผ(ํ,ํด;ํโฒ); non- overlapping regions correspond to conditional entropies. Note on visual representation: While the central overlap is labeled ํํผ(ํ,ํด;ํโฒ) for clarity, it serves as a visual proxy for the joint mutual information; mathematically, ํํผ(ํ,ํด;ํโฒ) corresponds to the entire region of ํโฒ shared with the (ํ,ํด) union. Bipredictability (ํ) is defined as the ratio of this shared information to the total informational union ํป ( ํ,ํด,ํ โฒ ) . This bidirectional uncertainty resolution is also what makes agency informationally costly: when actions depend on observations โ as they must in any responsive agent โ part of the uncertainty budget is consumed by the observationโ action dependency before any outcome is realized, reducing the fraction available for shared predictability. We term this the informational cost of agency. 2.2 Bipredictability measures interaction efficiency against a provable classical bound The classical bound ํ โค 0.5 and the structural suppression below this ceiling under observation-dependent action โ the informational cost of agency โ are derived from first principles in (Hafez et al., 2026). We restate the bound here for self-containment and test both predictions empirically in Section 4. The geometry of Figure 1 suggests a natural measure of interaction efficiency: the ratio of the central overlap to the total area. Formally, we define Bipredictability ํ as the fraction of the total uncertainty budget ํถ = ํป(ํ) + ํป(ํด) + ํป(ํโฒ) converted into shared predictability across the full loop: ํ= ํํผ(ํ,ํด;ํ โฒ ) ํป(ํ)+ํป(ํด)+ํป(ํ โฒ ) (1) At ํ = 0 the observationโaction pair and outcome are statistically independent. Higher values indicate tighter coupling. Because ํ is normalized by the full uncertainty budget rather than any single variable's entropy, it remains meaningful when interaction scope changes โ a property pairwise measures such as normalized mutual information do not share. ํ is related to but distinct from empowerment (Klyubin et al., 2005; Salge et al., 2014; Mohamed & Rezende, 2015), which quantifies the channel capacity ํผ(ํด; ํโฒ|ํ) of an agent's action channel and is typically maximized during training as an intrinsic motivation signal. ํ differs in three respects: it is normalized by the full interaction uncertainty budget rather than measured on a single channel, it is structurally bounded above by ยฝ rather than unbounded, and it is defined for runtime measurement of realized coupling rather than policy optimization. 4 A strict upper bound follows directly from Shannon entropy (Shannon, 1948; Cover & Thomas, 2006). Mutual information cannot exceed the entropy of either side, and by subadditivity: ํํผ ( ํ,ํด;ํ โฒ ) โคmin ( ํป ( ํ,ํด ) ,ํป ( ํ โฒ ) ) โคmin ( ํป ( ํ ) +ํป ( ํด ) ,ํป ( ํ โฒ ) ) (2) Let the total uncertainty budget be ํถ=ํป(ํ)+ํป(ํด)+ํป(ํ โฒ ). Under the constraint that ํถ is fixed, the ratio ํํผ/ํถ is maximised when both ํป(ํ)+ํป(ํด) and ํป(ํ โฒ ) are as large as possible while mutually constraining each other. The maximum occurs when ํป(ํ)+ํป(ํด)=ํป(ํ โฒ )=ํถ/2, yielding: ํ= ํํผ ํถ โค ํถ 2 C =0.5 ( 3 ) Thus, no classical interaction loop can convert more than half of its total uncertainty into shared predictability. Responsive agents, in which actions genuinely depend on observations, necessarily consume some of the budget in the ํโ ํด dependency, suppressing ํ below this ceiling. The 1/3 baseline observed across trained agents empirically quantifies this structural cost. This bound is structural โ it holds for any classical system representable over discrete variables, independent of domain, task, or agent architecture. The full derivation and saturation conditions are given in Hafez et al. (2026). The mechanism is that responsive action introduces a statistical dependency between ํ and ํด that consumes part of the uncertainty budget without contributing to shared predictability across the loop. The empirical baseline reported in Section 4.1 โ ํ=0.33ยฑ0.02 across trained agents โ confirms this prediction. 2.3 Bipredictability decomposes into forward, backward, and asymmetry components ํ measures overall interaction efficiency but detecting that coupling has degraded is only the first requirement for deployment monitoring โ understanding where the degradation originates is what enables intervention. A drop in ํ could reflect growing environmental unpredictability, loss of action legibility, or both simultaneously. To distinguish these, we analyze the two directions of uncertainty flow within the loop. โข Forward predictive uncertainty measures how much the outcome remains unresolved after the observation and action are known: ํปf = ํป(ํโฒ | ํ,ํด) (4) When ํปf increases, the environment responds in ways the agent cannot anticipate โ outcomes are no longer well-constrained by the stateโaction pair. โข Backward predictive uncertainty measures how much the observationโaction pair remains unresolved after the outcome is observed: ํปํ = ํป(ํ,ํด | ํโฒ) (5) When Hํ increases, many distinct observationโaction pairs lead to the same outcome โ the agent's internal distinctions are invisible from the environment's side, indicating loss of action legibility. Their difference defines predictive asymmetry: ํฅํป = ํปํ โ ํปํ ( 6 ) 5 In passive systems without responsive action, uncertainties are balanced and ฮH โ 0 โ consistent with double pendulum results reported in Hafez et al. (2026). Responsive agency disrupts this balance, inducing persistent asymmetry whose sign reveals the dominant source of coupling failure: environment-side disruptions drive ํปํ, while agent-side degradation drives ํปํ. Together, ํ,ํปํ,ํปํ, and ํฅํป form a diagnostic set โ P signals that something has changed, while the components localize where. 3 The Information Digital Twin (IDT): Real-time monitoring of Bipredictability Section 2 showed that ํ captures the shared information structure of the interaction loop, while its components, ํปํ ,ํปํ and ํฅํป, localize predictive failures. To serve as a deployment monitoring signal, these quantities must be computed online from the interaction stream, without access to internal model parameters or reward signals. We introduce the Information Digital Twin (IDT) (Fig. 2), an auxiliary architecture that runs alongside the deployed agent and computes ํ and its components directly from the (ํ,ํด,ํโฒ) stream. The IDT adopts the digital twin paradigm (Grieves & Vickers, 2017; Tao, Zhang & Nee, 2019) โ a real-time computational counterpart of a physical or computational system โ and specializes it for monitoring information-theoretic structure rather than physical state. Figure 2. Information Digital Twin (IDT) architecture. The IDT operates alongside the agentโenvironment loop, receiving copies of observations (ํ,ํโฒ) and actions (ํด). The ํ Calculator calculates Bipredictability ํ and predictive asymmetry ํฅํป from the interaction stream. The ํ Controller detects statistical deviations from baseline coupling. Dashed boxes indicate architecturally specified modulation pathways โ observation modulation and action modulation โ that are not experimentally validated in this work. The Adaptive Signal path connects controller output to the modulation interfaces. The IDT operates in three stages: online metric estimation (3.1), statistical detection of deviations from the learned baseline (3.2), and a modulation stageโarchitecturally specified but not yet experimentally validatedโthat would adapt the agentโs observation and action interfaces in response to sustained deviations (3.3). 3.1 ํท is computed online without access to model internals The IDT treats the deployed agent as a black box: it observes only the externally visible interaction stream and requires no access to policy weights, internal activations, or reward signals. At each time step, the agent selects an action ํด given its current observation ํ, the environment transitions, and a new observation ํโฒ is produced. The IDT intercepts copies of this (ํ,ํด,ํโฒ) tuple and estimates ํ and its diagnostic components through the following pipeline: At each timestep, the IDT records the (ํ,ํด,ํโฒ) tuple, accumulates tuples over a sliding window, computes the marginal entropies, joint mutual information, and conditional entropies from the windowed distributions, and derives ํ and ํฅํป for that window. The pipeline repeats at each window stride producing a time series of ํ and ํฅํป values that track coupling integrity as the agent operates. The pipeline itself is general: it applies to any system expressible as an (ํ,ํด,ํโฒ) interaction loop, with the estimation fidelity governed by the windowing and discretization parameters. 6 3.2 Coupling degradation is detected via multi-channel statistical deviation The time series produced by the estimation pipeline provides a continuous readout of coupling integrity, but raw values of ํ and ฮํป are not directly interpretable without a reference. The ํ controller (Fig. 2) establishes this reference during an initial calibration period under nominal operation by estimating the baseline mean ํ and standard deviation ํ for each metric. Subsequent values are evaluated against this learned baseline using the following protocol: The ํ controller calibrates baseline ฮผ and ฯ for each metric over a pre-perturbation window, then flags any post- perturbation window in which any metric โ ํ,ํฅํป,ํปํ, or ํปํ โ deviates beyond ยฑ3ํ. The union of channels constitutes the IDT detection signal; trials with no threshold crossing are recorded as undetected. The pattern of which metrics deviate, and in which direction, carries diagnostic information that differs across perturbation types. These diagnostic signatures are characterized in Section 4.4; operationalizing them as an automated attribution mechanism is future work. The ยฑ3ํ threshold is used because deviations in either direction indicate issues: in general terms, a decrease in ํ shows decoherence, while an increase indicates rigidity. Using four metrics improves detection, since different perturbations affect components differently. 3.3 ํท provides the prerequisite signal for closed-loop self-regulation A complete reliability architecture would not stop at detection and attribution. It would close the loop, using the measured deviation as feedback to adjust the agent's coupling with the environment in real time. The IDT specifies this modulation layer as a controller that receives (ํ,ํปํ,ํปํ,ํฅํป) deviations from the learned baseline and emits bandwidth adjustments at the interaction interface โ observation filtering when deviations are ํปํ-dominated, action damping when deviations are ํปํ-dominated โ without altering policy weights, preserving learned behavior while mitigating coupling degradation. The interface is defined; the control law is not. We do not claim experimental validation of this layer in the present paper. We claim that the measurement and attribution layers together establish the prerequisite signal that any such control law would require โ the analog, in deployment-reliability terms, of the measurable channel capacity that made principled communication-system design possible. Demonstrating closed-loop self-regulation built on this foundation is the natural next step. 4 Results 4.1 Trained agents exhibit stable coupling below the classical bound Before evaluating detection performance, we characterize the informational baseline of nominal agentโenvironment interaction. Under unperturbed operation, the 21 trained agents exhibit a mean Bipredictability of ํ=0.33ยฑ 0.02 and a mean predictive asymmetry of ฮํป=โ0.56ยฑ0.22, stable across the pre-perturbation episodes of all 168 evaluation runs. The baseline ํ โ 1/3 lies well below the classical upper bound of 0.5 (Hafez et al., 2026), confirming the theoretically predicted informational cost of agency: responsive, observation-dependent action necessarily suppresses interaction efficiency below the classical ceiling. This value is consistent with the agency-cost regime observed across other substrates (Section 4.5), suggesting the suppression reflects a structural property of observation-dependent action rather than a HalfCheetah-specific artifact. The negative ํฅํป shows that forward prediction (anticipating ํโ from ํ and ํด) carries greater residual uncertainty than backward inference (recovering ํ and ํด from ํ โฒ ), reflecting the asymmetry of active control: committed actions constrain backward inference more strongly than stochastic environmental responses constrain forward prediction. These baselines define the reference for perturbation-induced deviations. Each seedโs baseline mean and standard deviation are used to calibrate the ยฑ3ํ detection protocol. 7 4.2 IDT-based monitoring detects twice the perturbations that reward signals miss We evaluate whether monitoring Bipredictability from the interaction stream detects coupling degradation missed by reward-based monitoring. The ยฑ3ํ detection protocol is applied identically to IDT metrics (ํ,ํฅํป,ํปํ,ํปํ) and windowed episode reward across all 168 perturbation trials. IDT-based monitoring detected 89.3ยฑ15.1% of perturbations, compared with 44.0ยฑ26.1% for reward-based detection (paired t-test, ํ=21 seeds; ํก=7.95, ํ<10 โ6 , ํ=1.73). Detection rates were computed per seed as the proportion of perturbation trials detected and averaged across seeds. The improvement arises from two factors: IDT metrics respond to changes in interaction structure that may not immediately affect cumulative reward, and the union of four diagnostic channels provides broader coverage than a single reward signal. A detection is registered when any metric exceeds its ยฑ3ํ threshold; different perturbations activate different subsets of channels, but collectively they capture most conditions tested. Reward-based monitoring remains effective for perturbations that cause large, sustained performance drops. However, perturbations that degrade coupling without immediate reward collapse (e.g., moderate observation noise partially compensated by the policy) often go undetected by reward while producing clear deviations in ํ or its components. This โsilent degradationโ regime motivates the proposed framework. 4.3 The IDT detects coupling degradation 4.4ร faster than reward Beyond detecting more perturbations, timely detection is critical for deployment monitoring, as early warning enables intervention before irreversible performance degradation. Detection latency is measured as the number of observation windows between perturbation onset (episode 15) and the first window in which any metric exceeds its ยฑ3ํ threshold. Across all 168 perturbation trials, IDTโbased monitoring achieved a median detection latency of 42 windows, compared with 184 windows for rewardโbased detectionโa 4.4ร improvement. This gap reflects a fundamental signal difference: ํ responds to structural changes in the interaction at the transition level, whereas reward aggregates effects over episodes and requires degradation to accumulate before deviating. In deployment terms, earlier detection provides a larger window for corrective action, including operator alerts, fallback policies, orโwithin a future closedโloop architectureโreflexive modulation. 4.4 The diagnostic decomposition provides complementary detection channels The IDTโs detection advantage over reward arises from the complementary sensitivity of multiple information-theoretic channels rather than any single metric. Table 1 reports the individual detection rate and median latency for each monitored quantity, their union (IDT), and reward. No single metric dominates: individual detection rates cluster between 69โ73%, exceeding reward-based detection (44.0%) but falling well short of the union (89.3%). TABLE 1. Per-metric detection rate and median latency across 168 perturbation trials. IDT (union) registers detection when any metric exceeds ยฑ3ํ. ํ = Bipredictability; ํปํ = ํป(ํโฒ|ํ,ํด); ํปํ = ํป(ํ,ํด|ํโฒ); ํฅํป = ํปํ โ ํปํ. Reward uses the same ยฑ3ฯ protocol on windowed episodic return. Metric Detection Rate (%) Median Latency (windows) IDT (union) 89.3 42 ํ 73.2 74 ํปํ 70.8 69 ํปํ 69.6 75 ํฅํป 69.0 67 Reward 44.0 184 8 A similar pattern holds for latency, with individual medians between 67โ75 windows versus 42 for the union. This gapโapproximately 16 percentage points in coverage and 25 windows in latencyโdemonstrates that each channel detects perturbations missed by the others; the metrics are complementary, not redundant. This complementarity is further reflected in per-algorithm effect sizes. For SAC agents, information-theoretic metrics are approximately twice as sensitive as reward (mean Cohenโs ํ=0.93 โ 1.02 vs. 0.60), whereas PPO agents exhibit comparable sensitivity across all metrics (ํโ0.5). Nevertheless, the IDT outperforms reward-based detection for both algorithmsโthrough stronger individual signals for SAC and through the union of channels for PPO, where individual metrics alone are insufficient. 4.5 The agency-cost signature reproduces across substrates The classical bound ํ โค 0.5 is a structural property of Shannon entropy and applies to any closed-loop interaction with finite state spaces. The agency-cost suppression toward ํ โ 0.3 is empirical, and a single-domain confirmation cannot distinguish a structural regularity from a HalfCheetah-specific artifact. We therefore situate the present results within the broader cross-substrate evidence. Three independent validations, conducted under the same definition of ํ and the same protocol of computing it from external interaction streams, report converging values. In multi-turn language-model dialogue across three frontier models, ํ = 0.275 ยฑ 0.029, with 100% perturbation detection across contradiction, topic-shift, and non-sequitur injections (Hafez & Nazeri, 2026). In convolutional vision systems under input perturbation, the entropy and mutual-information components from which ํ is constructed exhibit the same structured sensitivity, yielding 90% noise-detection accuracy with 0% false positives (Nazeri & Hafez, 2025). As a non-agentic control, a double- pendulum classical mechanical system without observation-dependent action sits at ํ โ 0.48, near the theoretical ceiling โ the regime predicted for systems where the agency cost does not apply (Hafez et al., 2026). These are not commensurate experiments. They differ in dimensionality, dynamics, and computation. They share only the definition of ํ and the presence or absence of observation-dependent action. That such heterogeneous systems converge on the same regime structure โ agency-bearing systems suppressed to roughly one-third of the uncertainty budget, agency-free systems near the classical ceiling โ is the strongest available evidence that the signature observed here is substrate-independent rather than HalfCheetah-specific. A formal cross-substrate meta- analysis is beyond the scope of this paper, which focuses on the deepest single-domain validation. 5 Discussion The central finding of this work is that the information structure of the observationโactionโoutcome loop carries a deployment-relevant signal that reward-based monitoring cannot access. Across 168 perturbation trials in continuous control, IDT-based monitoring detected 89.3% of coupling degradations versus 44.0% for reward, with 4.4ร lower median latency. This advantage is structural rather than incidental. Reward aggregates task outcomes over episodes and requires degradation to accumulate before deviating; ํ measures the fraction of the available uncertainty budget converted into shared predictability at the transition level, responding to coupling changes that may never manifest as reward collapse. The gap between the empirical baseline ํ โ 1/3 and the classical ceiling ํ โค 0.5 is not a performance limitation โ it is the informational cost of agency, the structural consequence of observation-dependent action that makes closed-loop control possible. This interpretation is supported by the convergence of the same regime structure across radically different substrates, each measured with its own discretisation and estimation pipeline. A classical double pendulum, convolutional neural networks, and multi-turn language models all exhibit the same qualitative ordering: agentic systems suppressed to roughly one-third of the uncertainty budget, non-agentic systems near the classical ceiling. The framework extends theoretically beyond the classical regime โ quantum-entangled systems are predicted to reach ํ โค 1 under unitary evolution โ but this extension is mathematical rather than empirical. That such heterogeneous systems share only the definition of ํ and the presence or absence of observation-dependent 9 action โ yet yield a consistent signature โ strongly suggests that the nature of what ํ captures is invariant to the details of the estimation pipeline. Binning granularity and window length shift absolute scalar values, but the regime classification relative to the bound, and the separation between agentic and non-agentic dynamics, appear robust. It is important to distinguish the metric from its operationalization. The theoretical bound and the qualitative suppression signature are proven or observed across multiple domains; the specific detection rates (89.3%) and latency improvements (4.4ร) reported here are properties of a particular IDT configuration (three equal-width bins, 300-step windows, ยฑ3ํ thresholds) applied to a single continuous-control environment. A formal sensitivity analysis across bin counts, window sizes, and threshold choices remains future work. However, given that the underlying metric is structurally robust, the IDTโs detection advantage over reward โ which arises from measuring coupling efficiency directly at the transition level โ is unlikely to be an artefact of fine-tuning those parameters. The cross-substrate evidence provides strong independent support that the interpretive power of ํ does not hinge on a specific discretisation. Several limitations should be acknowledged. The full IDT pipeline was validated in a single continuous-control environment (HalfCheetah); multi-environment RL validation, while strongly indicated by the cross-substrate evidence, is not yet performed within the RL domain itself. The detection results are reported without a dedicated false-positive analysis on held-out unperturbed data, although the calibration on 14 unperturbed episodes per seed provides an implicit baseline stability estimate. Most importantly, this paper validates measurement and attribution; the modulation layer of the IDT, which would close the loop from monitoring to self-regulation, is architecturally specified but not experimentally demonstrated. What this paper establishes is the prerequisite without which any such control law would be impossible: a real-time, model-agnostic, structurally bounded measurement of agentโ environment coupling. The cost of operating without such an instrument is already visible โ training compute redone whenever deployment conditions shift, reliability strategies that detect failures only after they occur, and an absence of runtime guarantees that has become the binding constraint on real-world autonomous deployment. Bipredictability does not solve these problems. It provides the measurement layer required to begin solving them. 6 Methods 6.1 Online estimation of ํท from interaction streams Bipredictability is defined over discrete random variables. Real interaction streams are continuous, requiring discretization before ํ can be estimated. We apply z-score normalization and map continuous values into discrete bins, converting the raw (ํ,ํด,ํโฒ) stream into a sequence of transition symbols from which empirical joint and marginal distributions are constructed. Distributions are estimated over sliding windows of W transitions advanced by stride ฮด, producing a time series of ํ,ํปํ,ํปํ, and ํฅํป. A distinction worth making explicit: ํ is defined analytically as the ratio ํํผ(ํ,ํด; ํโฒ) / [ํป(ํ) + ํป(ํด) + ํป(ํโฒ)] and its bound ํ โค 0.5 is a mathematical consequence of Shannon entropy subadditivity, independent of any discretization resolution, binning strategy, or window size. The bound is a property of the joint distribution of the interaction loop; estimation choices affect only how that distribution is approximated. What requires empirical investigation is not the bound itself, but whether the agency-cost suppression โ the regime where agentic systems settle near ํ โ 1/3 โ is robust to the estimation pipeline. The cross-substrate evidence summarized in Section 4.5 bears on this directly. The same regime structure appears in continuous-control RL agents (this work), multi-turn LLM dialogue (Hafez & Nazeri, 2026), and the entropy and mutual-information components of convolutional vision systems (Nazeri & Hafez, 2025), while a non-agentic classical pendulum sits near the theoretical ceiling (Hafez et al., 2026). These systems differ in dimensionality, dynamics, and discretization scheme โ high-dimensional continuous state in RL and CNNs, naturally discrete tokens in language, low-dimensional continuous mechanics in the pendulum. That the regime separation between 10 agentic and non-agentic dynamics persists across these heterogeneous estimation pipelines indicates that the metric's interpretive power does not hinge on a particular operationalization. For the HalfCheetah experiments specifically, we fixed the discretization at three equal-width bins and a window of 300 timesteps after confirming that these choices produce stable entropy estimates. A formal sensitivity analysis across bin counts and window lengths in the RL setting is left for future work. 6.2 Discretisation, bounds, and estimation robustness The discretization pipeline interacts with the theoretical bound in a way worth making explicit. The bound ํ โค 0.5 is a property of the joint probability distribution; discretization produces an empirical distribution from which we compute an estimate ํ. By the data-processing inequality, coarse-graining cannot increase mutual information, so the mutual information estimated from discretized variables is a lower bound on the continuous mutual information. The sum of marginal entropies, by contrast, can shift in either direction with binning. Consequently, ํ is not guaranteed to be a strict lower bound on the continuous P for every binning scheme, but the ceiling ํ โค 0.5 remains valid for the discretized joint distribution itself, since the derivation in Section 2.2 applies to any discrete- variable representation. Any apparent violation of the 0.5 ceiling in an empirical estimate would therefore indicate an estimation artifact โ typically from insufficient window size โ rather than a failure of the bound. The observed baseline ํ โ 1/3 across our experiments lies well below 0.5, indicating that the suppression is genuine rather than a binning artifact. Moreover, when the same discretization scheme is applied to both agentic and non-agentic systems, any systematic bias from coarse-graining applies symmetrically to both and does not affect the regime classification. This is why the cross-substrate signature โ ํ โ 1/3 with responsive agency, ํ โค 0.5 without โ survives across radically different discretization choices. 6.3 Environment and agents We evaluate our framework on MuJoCo HalfCheetah-v4 (Todorov et al., 2012), a continuous-control benchmark with 17-dimensional observations and 6-dimensional torque actions. The task is to maximize forward velocity under control costs. HalfCheetah provides sufficient complexity for nontrivial information-theoretic estimation while remaining interpretable. Agents are trained using Soft Actor-Critic (SAC) (Haarnoja et al., 2018) and Proximal Policy Optimization (PPO) (Schulman et al., 2017), implemented in Stable-Baselines-3 (Raffin et al., 2021) with default hyperparameters. This pairing tests whether Bipredictability monitoring generalizes across policy architectures. SAC agents were trained to convergence (โ10ํโ17ํ ํํํกํขํํํ ) over ~3 ํ steps, while PPO agents converged faster (โ6ํโ 8ํ ํํํกํขํํํ ) over ~1.5 ํ steps. Convergence was defined by stable returns over a sustained evaluation window, after which policies were frozen. Using algorithms with different performance levels tests whether ํ captures interaction structure independent of raw task performance. In total, 11 SAC and 10 PPO seeds were evaluated, yielding 21 agents. Each evaluation comprised 50 episodes (50,000 steps each). Freezing policies ensures that all observed ํ deviations reflect perturbation-induced coupling changes rather than ongoing learning. The perturbation suite is described next. 6.4 Perturbation design To assess generalization across failure modes, we designed a perturbation suite spanning environment-side interaction changes and agent-side observation and action degradations, with type and severity varied to test detection sensitivity. Environment perturbations include gravity changes (one level) and external force impulses (two levels), while agent perturbations include observation noise (two levels) and action noise (three levels). The eight conditions are summarized in Table 2. 11 TABLE 2. Perturbation suite. Eight perturbation conditions spanning agent-side and environment-side changes applied to all 21 agents for 168 total trials. Category Perturbation Type Parameter Agent Actuator noise 1% Gaussian noise on actions Agent Actuator noise 3% Gaussian noise on actions Agent Actuator noise 4% Gaussian noise on actions Environment External force 5 N force applied to torso, x- axis Environment External force 10 N force applied to torso, x-axis Environment Gravity Gravity increased to 110% Agent Observation noise 1% Gaussian noise on observations Agent Observation noise 3% Gaussian noise on observations This suite does not exhaust the space of possible perturbations, which is effectively unbounded in continuous control settings. Instead, it samples qualitatively distinct failure mechanisms across multiple severity levels, supporting task-independent sensitivity rather than tuning to specific failure modes. Each perturbation is introduced at episode 15 of a 50-episode evaluation, yielding 14 unperturbed episodes for baseline calibration. All eight perturbation types are applied to each of the 21 agents, for a total of 168 trials. Information metric computation for these trials is described next. 6.5 Information metrics computation and comparison protocol Computing information-theoretic quantities from the HalfCheetah interaction stream requires discretizing 40 continuous variables per timestep (17 state, 6 actions, 17 next-state). We apply a three-step procedure: z-score normalization, discretization into three equal-width bins, and grouping variables by body part (front leg, back leg, torso) following the environmentโs kinematic structure. Per-variable bins within each group are concatenated to form composite symbols for (ํ,ํด,ํโ). This grouping preserves embodiment structure while keeping the joint distribution tractable. The three-bin choice was empirically validated: four bins produced unreliable entropy estimates at available window sizes, while quantile-based binning yielded flat, uninformative metrics. ํ,ํปํ,ํปํ, and ฮํป are computed over sliding windows of 300 timesteps with a stride of 50, yielding 991 windows per evaluation. All quantities use standard base-2 entropy formulas. For comparison, per-step reward is averaged over identical windows, and the same ยฑ3ํ detection protocol is applied to both IDT metrics and windowed reward, ensuring matched temporal resolution and fair comparison. REFERENCES 1. A. Nazeri, W. Hafez. "Entropy-Based Non-Invasive Reliability Monitoring of Convolutional Neural Networks." arXiv preprint arXiv:2508.21715 (2025). 2. A. Raffin et al., "Stable-Baselines3: Reliable reinforcement learning implementations," Journal of Machine Learning Research, vol. 22, no. 268, p. 1โ8, 2021. 3. A. S. Klyubin, D. Polani, and C. L. Nehaniv, "Empowerment: A universal agent-centric measure of control," in Proc. IEEE Congr. Evol. Comput., 2005, vol. 1, p. 128โ135. 4. C. E. Shannon, "A mathematical theory of communication," Bell System Technical Journal, vol. 27, no. 3, p. 379โ423, July 1948. 5. C. Salge, C. Glackin, and D. Polani, "Empowerment โ An introduction," in Guided Self-Organization: Inception, Springer, 2014, p. 67โ114. 6. C. Tang et al., "Legged locomotion in challenging terrains using ego-centric vision," IEEE Robotics and Automation Letters, 2024. 7. E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Mรผller, V. Koltun, and D. Scaramuzza, "Champion-level drone racing using deep reinforcement learning," Nature, vol. 620, no. 7976, p. 982โ987, Aug. 2023. 8. E. Todorov, T. Erez, and Y. Tassa, "MuJoCo: A physics engine for model-based control," in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2012, p. 5026โ5033. 12 9. F. Tao, M. Zhang, and A. Y. C. Nee, Digital Twin Driven Smart Manufacturing. Academic Press, 2019. 10. G. Dulac-Arnold et al., "Challenges of real-world reinforcement learning: Definitions, benchmarks and analysis," Machine Learning, vol. 110, p. 2419โ2468, 2021. 11. W. Hafez, and A. Nazeri, " Token Statistics Reveal Conversational Drift in Multi-turn LLM Interaction." Research Square preprint, 2026. https://doi.org/10.21203/rs.3.rs-9182401/v1 12. I. Radosavovic et al., "Real-world humanoid locomotion with reinforcement learning," arXiv preprint arXiv:2303.03381, 2023. 13. J. Josifovski et al., "Continual learning for robotics: A review," Robotics and Autonomous Systems, 2024. 14. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint arXiv:1707.06347, 2017. 15. K. Friston, "The free-energy principle: A rough guide to the brain?," Trends in Cognitive Sciences, vol. 13, no. 7, p. 293โ 301, 2009. 16. K. Friston, T. FitzGerald, F. Rigoli, P. Schwartenbeck, and G. Pezzulo. "Active inference: a process theory." Neural computation 29, no. 1 (2017): 1-49. 17. M. Grieves and J. Vickers, "Digital twin: Mitigating unpredictable, undesirable emergent behavior in complex systems," in Transdisciplinary Perspectives on Complex Systems, Springer, 2017, p. 85โ113. 18. Mohamed, S. and Rezende, D.J. "Variational information maximisation for intrinsically motivated reinforcement learning." Advances in Neural Information Processing Systems, vol. 28, 2015. 19. P. W. Koh et al., "Wilds: A benchmark of in-the-wild distribution shifts," in Proc. Int. Conf. Mach. Learn. (ICML), 2021, p. 5637โ5664. 20. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. Cambridge, MA, USA: MIT Press, 2018. 21. S. Greco, B. Vacchetti, D. Apiletti, and T. Cerquitelli. "Unsupervised concept drift detection from deep learning representations in real-time." IEEE Transactions on Knowledge and Data Engineering (2025) 22. S. Levine, A. Kumar, G. Tucker, and J. Fu. "Offline reinforcement learning: Tutorial, review, and perspectives on open problems." arXiv preprint arXiv:2005.01643 (2020). 23. S. Liang, Y. Li, and R. Srikant, "Enhancing the reliability of out-of-distribution image detection in neural networks," in Proc. Int. Conf. Learn. Representations (ICLR), 2018. 24. T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor," in Proc. 35th Int. Conf. Mach. Learn., 2018, p. 1861โ1870. 25. T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed. Hoboken, NJ, USA: Wiley-Interscience, 2006. 26. W. Hafez, C. Wei, R. Pena, A. Nazeri, and C. Reid, "A Mathematical Theory of Agency and Intelligence," preprint https://doi.org/10.21203/rs.3.rs-9216783/v1.