Paper deep dive
Human Supervision as an Information Bottleneck: A Unified Theory of Error Floors in Human-Guided Learning
Alejandro Rodriguez Dominguez
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/20/2026, 8:43:57 AM
Summary
The paper proposes a unified theory called Human-Bounded Intelligence (HBI), arguing that human supervision acts as an information-reducing channel that creates a strictly positive excess-risk floor for machine learning models. This limitation arises from annotation noise, preference distortion, and semantic compression. The theory is validated across six frameworks (operator theory, PAC-Bayes, information theory, causal inference, category theory, and RLHF) and empirically shown to be breakable by introducing auxiliary non-human signals like retrieval or program execution.
Entities (10)
Relation Signals (9)
Human supervision → actsas → Information-Reducing Channel
confidence 95% · human supervision... acts as an information-reducing channel that induces a strictly positive excess-risk floor
Human supervision → causes → Excess-Risk Floor
confidence 92% · induces a strictly positive excess-risk floor for any learner dominated by it
Human supervision → comprises → Annotation Noise
confidence 90% · structural decomposition into annotation noise, preference distortion, and semantic compression
Human supervision → comprises → Preference Distortion
confidence 90% · structural decomposition into annotation noise, preference distortion, and semantic compression
Human supervision → comprises → Semantic Compression
confidence 90% · structural decomposition into annotation noise, preference distortion, and semantic compression
Auxiliary Non-Human Signals → reduces → Excess-Risk Floor
confidence 88% · auxiliary non-human signals... increase effective supervision capacity and collapse the floor
Human-Bounded Intelligence → appliesto → Reinforcement Learning from Human Feedback
confidence 85% · game-theoretic analyses of reinforcement learning from human feedback... The theory explains why scaling alone cannot eliminate persistent human-aligned errors
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models are trained primarily on human-generated data and feedback, yet they exhibit persistent errors arising from annotation noise, subjective preferences, and the limited expressive bandwidth of natural language. We argue that these limitations reflect structural properties of the supervision channel rather than model scale or optimization. We develop a unified theory showing that whenever the human supervision channel is not sufficient for a latent evaluation target, it acts as an information-reducing channel that induces a strictly positive excess-risk floor for any learner dominated by it. We formalize this Human-Bounded Intelligence limit and show that across six complementary frameworks (operator theory, PAC-Bayes, information theory, causal inference, category theory, and game-theoretic analyses of reinforcement learning from human feedback), non-sufficiency yields strictly positive lower bounds arising from the same structural decomposition into annotation noise, preference distortion, and semantic compression. The theory explains why scaling alone cannot eliminate persistent human-aligned errors and characterizes conditions under which auxiliary non-human signals (e.g., retrieval, program execution, tools) increase effective supervision capacity and collapse the floor by restoring information about the latent target. Experiments on real preference data, synthetic known-target tasks, and externally verifiable benchmarks confirm the predicted structural signatures: human-only supervision exhibits a persistent floor, while sufficiently informative auxiliary channels strictly reduce or eliminate excess error.
Tags
Links
- Source: https://arxiv.org/abs/2602.23446v1
- Canonical: https://arxiv.org/abs/2602.23446v1
Trouble viewing inline? Open PDF directly →
Full Text
44,499 characters extracted from source content.
Expand or collapse full text
Human Supervision as an Information Bottleneck: A Unified Theory of Error Floors in Human-Guided Learning Alejandro Rodriguez Dominguez Alejandro Rodriguez Dominguez is Head of Quantitative Analysis at Miralta Finance Bank S.A., Spain, and a PhD Candidate with the Department of Computer Science, University of Reading, UK. aodriguez@miraltabank.com Abstract Large language models are trained primarily on human-generated data and feedback, yet they exhibit persistent errors arising from annotation noise, subjective preferences, and the limited expressive bandwidth of natural language. We argue that these limitations reflect structural properties of the supervision channel rather than model scale or optimization. We develop a unified theory showing that whenever the human supervision channel is not sufficient for a latent evaluation target, it acts as an information-reducing channel that induces a strictly positive excess-risk floor for any learner dominated by it. We formalize this Human-Bounded Intelligence limit and show that across six complementary frameworks—operator theory, PAC-Bayes, information theory, causal inference, category theory, and game-theoretic analyses of reinforcement learning from human feedback—non-sufficiency yields strictly positive lower bounds arising from the same structural decomposition into annotation noise, preference distortion, and semantic compression. The theory explains why scaling alone cannot eliminate persistent human-aligned errors and characterizes conditions under which auxiliary non-human signals (e.g., retrieval, program execution, tools) increase effective supervision capacity and collapse the floor by restoring information about the latent target. Experiments on real preference data, synthetic known-target tasks, and externally verifiable benchmarks confirm the predicted structural signatures: human-only supervision exhibits a persistent floor, while sufficiently informative auxiliary channels strictly reduce or eliminate excess error. I Introduction Large language models (LLMs) are trained on human-generated text [22], refined via Reinforcement Learning from Human Feedback (RLHF) [4, 19], and evaluated according to human preferences. Despite their capabilities, such systems inherit limitations of human supervision, including annotation noise, shortcut biases, and subjective distortions [8]. These induce persistent error patterns reflecting human preferences and semantic bottlenecks, raising a central question: Can a system trained solely on human-generated signals reliably exceed performance relative to the underlying task objective? Empirically, LLMs outperform humans on narrow benchmarks, yet pipelines relying exclusively on human-labeled or human-judged data exhibit reward hacking [26], preference drift and overoptimization [7], and degradation under iterative self-training [25]. These effects often persist with scale, suggesting a structural rather than optimization-based limitation. We provide a general explanation. Under mild assumptions on (i) human signal generation, (i) empirical risk minimization, and (i) information flow through the supervision pipeline, any learner dominated by human-generated signals is constrained by a strictly positive excess-risk floor. Human supervision—whether labels, rankings, demonstrations, or curated text—acts as an information-reducing channel of the latent task objective. Even with unlimited capacity, infinite data, and ideal optimization, a learner cannot recover information that never passes through this channel. We formalize this Human-Bounded Intelligence (HBI) limit across six theoretical perspectives: operator theory, PAC-Bayes [16], information theory [5], causal inference [20], category theory [15], and game-theoretic RLHF models [4]. In each framework, non-sufficiency of the supervision channel yields a strictly positive lower bound of the form lim infn→∞ℰ∗(fθn)≥γH(⋅)>0, _n→∞E (f_ _n)≥ _H^(·)>0, (1) where the constant depends on properties of the supervision channel but not on model scale or compute. Although numerical bounds differ across frameworks, their positivity arises from the same structural source: information loss induced by PHP_H. Moreover, supervision induces a structured decomposition BH=Bnoise+Bpref+Bsem,B_H=B_noise+B_pref+B_sem, (2) corresponding to annotation noise, preference distortion, and semantic compression. Empirically, we validate these predictions across three regimes: real human-preference data, synthetic tasks with known ground truth, and externally verifiable benchmarks. Human-only supervision exhibits a persistent floor; hybrid supervision with auxiliary signals that provide independent information about Y∗Y reduces the floor and eliminates it when the auxiliary channel becomes sufficient. The HBI limit applies only when human supervision is the primary information source about Y∗Y . Auxiliary non-human channels— including code execution [3], retrieval [14], and tool augmentation [23]—increase effective supervision capacity and can remove the floor by restoring information about the latent target. Hybrid systems therefore modify the supervision channel rather than merely scaling model capacity. Our contributions are: • A unified framework modeling human supervision as an information-reducing channel with a structured bias decomposition. • A Human-Bounded Intelligence (HBI) theorem establishing a strictly positive excess-risk floor under human-dominated supervision. • Instantiations across six independent theoretical frameworks demonstrating the same structural limitation. • A characterization of auxiliary channels that break the bound, defining human-only, human+model, and human+model+auxiliary regimes. • Empirical validation confirming floor persistence under human-only supervision and collapse under sufficient auxiliary information. The remainder of the paper is organized as follows. Section I reviews related work. Section I presents the formal framework and HBI theorem. Section IV derives lower bounds across six perspectives. Section V characterizes auxiliary-channel regimes. Section VI provides empirical validation. Section VII concludes. I Related Work Large language models trained on human-curated corpora [22] and refined via RLHF [4, 19] inherit limitations of human supervision, including annotation noise, shortcut bias, and preference distortion [8]. Empirical studies of reward overoptimization and objective misspecification [7, 26] demonstrate systematic gaps between learned reward models and latent task objectives, motivating theoretical accounts of supervision-induced bias. Learning with noisy labels [17] characterizes corruption in supervision signals, and shortcut learning analyses [8] show exploitation of spurious human-annotated features. These works typically model specific perturbations. In contrast, we treat human supervision itself as an information-reducing channel and derive a structural decomposition into annotation noise, preference distortion, and semantic compression independent of any particular corruption mechanism. Scaling-law analyses [12, 10] document smooth improvements with data and compute, while model-collapse results [25] show degradation under recursive self-training. Our results complement these findings: even with infinite data and ideal optimization, a supervision channel of bounded information capacity induces a nonzero excess-risk floor that scaling alone cannot remove. Inverse reward learning [18] formalizes ambiguity among reward functions consistent with observed feedback. Our causal and information-theoretic instantiations connect this ambiguity to supervision-channel non-invertibility, yielding lower bounds independent of model class or estimation procedure. Tool- and retrieval-augmented systems [23, 3, 14] introduce auxiliary evaluators such as program execution and search, supplying additional information about latent targets. Our auxiliary-channel analysis formalizes when such signals strictly increase mutual information and eliminate the human-bounded floor. More broadly, our formulation draws on information theory [5], causal identifiability [20], PAC-Bayesian generalization [16], and categorical structure [15]. We differ from prior work by establishing a supervision-induced excess-risk floor and showing that the same structural limitation arises independently across multiple theoretical frameworks. I Formal Framework and HBI Theorem We model supervision as a stochastic transformation from an unknown latent task-relevant mapping f∗:→f :X to a human-provided signal S, with the learner observing only (X,S)(X,S). The analysis does not assume metaphysical objectivity: Y∗Y may represent any latent evaluation target not fully revealed by human supervision. I-A Ground Truth, Human Channel, and Bias Let (,,μ)(X,A,μ) be a probability space and let f∗:→f :X be a measurable mapping. Define the latent evaluation variable Y∗=f∗(X).Y =f (X). (3) Human supervision is generated through a stochastic channel S∼PH(⋅∣X,Y∗),S P_H(· X,Y ), (4) which need not be sufficient for Y∗Y . In additive settings (e.g., ⊆ℝY with squared loss), one illustrative decomposition is S=Y∗+εH+bH(X)+(qH(Y∗)−Y∗),S=Y + _H+b_H(X)+(q_H(Y )-Y ), (5) where: • εH _H is zero-mean stochastic noise, • bHb_H is systematic preference distortion, • qHq_H is a possibly non-invertible semantic compression. This decomposition is heuristic; the theory below does not rely on additivity. Let ℓ denote the ground-truth loss and L the surrogate optimized during training. Define the population risks ℛ∗(f)=[ℓ(f(X),Y∗)],ℛH(f)=[L(f(X),S)].R (f)=E[ (f(X),Y )], _H(f)=E[L(f(X),S)]. (6) Define the induced human bias functional BH(f)=ℛH(f)−ℛ∗(f).B_H(f)=R_H(f)-R (f). (7) This functional depends only on the supervision channel PHP_H and the surrogate L. I-B Learners and Assumptions Let ℱ⊆ℒ2(μ)F ^2(μ) be a hypothesis class. Consider empirical risk minimization θ^n∈argminθ1n∑i=1nL(fθ(Xi),Si). θ_n∈ _θ 1n _i=1^nL(f_θ(X_i),S_i). (8) We impose the following structural assumptions. Assumption 1 (Human-dominated supervision) All information about Y∗Y available during training flows through the channel PH(⋅∣X,Y∗)P_H(· X,Y ). No auxiliary signal provides additional independent information about Y∗Y . Assumption 2 (Asymptotically optimal optimization) limn→∞ℛH(fθ^n)=inff∈ℱℛH(f)≕RoptH. _n→∞R_H(f_ θ_n)= _f R_H(f) R^H_opt. Assumption 3 (Strict minimizer separation) The minimizers of ℛHR_H do not coincide with those of ℛ∗R . Define γH≔inff∈argminℛH(ℛ∗(f)−Ropt∗). _H _f∈ _H (R (f)-R _opt ). Then γH>0 _H>0. Regularity Conditions We assume: 1. ℛHR_H and ℛ∗R attain their minima over ℱF; 2. ℛ∗R is lower semicontinuous under the topology in which predictors converge; 3. Any sequence fn\f_n\ satisfying ℛH(fn)→RoptHR_H(f_n)→ R^H_opt has accumulation points contained in argminℛH _H. These conditions are standard in statistical learning theory and isolate structural limitations of the supervision channel from optimization artifacts. I-C Excess Risk and the Human-Bounded Limit Define the ground-truth excess risk ℰ∗(f)=ℛ∗(f)−Ropt∗,Ropt∗=infg∈ℱℛ∗(g).E (f)=R (f)-R _opt, R _opt= _g R (g). (9) A learner is human-bounded if lim infn→∞ℰ∗(fθ^n)>0. _n→∞E (f_ θ_n)>0. Theorem 1 (Human-Bounded Intelligence (HBI)) Under Assumptions 1–3 and the regularity conditions, lim infn→∞ℰ∗(fθ^n)≥γH>0. _n→∞E (f_ θ_n)≥ _H>0. I-D Proof of Theorem 1 Proof: By Assumption 2, ℛH(fθ^n)→RoptH.R_H(f_ θ_n)→ R^H_opt. By regularity condition (3), every accumulation point of fθ^n\f_ θ_n\ lies in argminℛH _H. Let f∞f_∞ be any such accumulation point. Then f∞∈argminℛH.f_∞∈ _H. By Assumption 3, ℛ∗(f∞)−Ropt∗≥γH.R (f_∞)-R _opt≥ _H. Lower semicontinuity of ℛ∗R yields lim infn→∞ℛ∗(fθ^n)≥ℛ∗(f∞). _n→∞R (f_ θ_n) (f_∞). Therefore, lim infn→∞ℰ∗(fθ^n)≥γH. _n→∞E (f_ θ_n)≥ _H. ∎ IV Instantiations Across Six Frameworks We now show that six classical frameworks independently yield strictly positive excess-risk lower bounds under non-sufficiency of the supervision channel. In each case, the bound arises from the same structural decomposition of supervision-induced deviation, γH(⋅)=γnoise(⋅)+γpref(⋅)+γsem(⋅), _H^(·)= _noise^(·)+ _pref^(·)+ _sem^(·), (10) where the constants may differ across analytical perspectives. Although the numerical lower bounds obtained in each framework need not coincide, their positivity originates from the same structural source: information loss induced by PHP_H. IV-A Operator-Theoretic Limit We model the ground truth as a bounded linear operator T∗:L2(,μ)→L2(,ν),T :L^2(X,μ)→ L^2(Y,ν), (11) mapping input functions to output functions. Human supervision induces an operator THT_H capturing what is representable via PHP_H, and BH=TH−T∗.B_H=T_H-T . (12) Under ideal optimization, many infinite-width or kernel-limit analyses imply convergence Tθn→THT_ _n→ T_H [2, 1, 11, 13]. Theorem 2 (Operator-Theoretic HBI) Suppose Tθn→THT_ _n→ T_H in operator norm and BH≠0B_H≠ 0. Then limn→∞‖Tθn−T∗‖=‖BH‖ _n→∞ \|T_ _n-T \|=\|B_H\| (13) and, if the task excess risk is Lipschitz in ‖T−T∗‖\|T-T \|, there exists a task-dependent constant c>0c>0 such that γH≥c‖BH‖>0. _H\;≥\;c\,\|B_H\|>0. (14) Proof: We have Tθn−T∗=(Tθn−TH)+BH.T_ _n-T =(T_ _n-T_H)+B_H. (15) Taking norms and applying the triangle inequality, ‖Tθn−T∗‖≥‖BH‖−‖Tθn−TH‖,\|T_ _n-T \|≥\|B_H\|-\|T_ _n-T_H\|, (16) so lim infn→∞‖Tθn−T∗‖≥‖BH‖. _n→∞\|T_ _n-T \|≥\|B_H\|. Similarly, ‖Tθn−T∗‖≤‖BH‖+‖Tθn−TH‖,\|T_ _n-T \|≤\|B_H\|+\|T_ _n-T_H\|, (17) so lim supn→∞‖Tθn−T∗‖≤‖BH‖. _n→∞\|T_ _n-T \|≤\|B_H\|. Thus, the limit exists and equals ‖BH‖\|B_H\|. If the excess risk ℰ∗E satisfies ℰ∗(fθ)≥c‖Tθ−T∗‖E (f_θ)≥ c\|T_θ-T \| for some c>0c>0 (for example, for squared loss under an L2L^2–Lipschitz condition), the lower bound on γH _H follows. ∎ IV-B PAC-Bayesian Limit PAC-Bayes bounds [16, 24, 9] relate expected risk to empirical risk plus a complexity penalty. Let P be a prior and Q a posterior over hypotheses. Write LH(f)=L∗(f)+BH(f).L_H(f)=L (f)+B_H(f). (18) A standard PAC-Bayes inequality gives, with probability at least 1−δ1-δ, f∼Q[LH(f)]≤f∼Q[L^H(f)]+KL(Q∥P)+ln(1/δ)2n.E_f Q[L_H(f)] _f Q[ L_H(f)]+ KL(Q\|P)+ (1/δ)2n. (19) Theorem 3 (PAC-Bayes HBI) Let QH(n)Q_H^(n) be posteriors that concentrate on minimizers of LHL_H as n→∞n→∞, and assume that any minimizer fH∗f_H of LHL_H satisfies L∗(fH∗)≥inffL∗(f)+γHPACL (f_H )≥ _fL (f)+ _H^PAC (20) for some γHPAC>0 _H^PAC>0. Then lim infn→∞f∼QH(n)[L∗(f)]≥inffL∗(f)+γHPAC. _n→∞E_f Q_H^(n)[L (f)]\;≥\; _fL (f)+ _H^PAC. (21) Proof: By construction, QH(n)Q_H^(n) concentrates on the set of minimizers of LHL_H, so limn→∞f∼QH(n)[LH(f)]=inffLH(f). _n→∞E_f Q_H^(n)[L_H(f)]= _fL_H(f). By the assumption on minimizers, any such fH∗f_H obeys L∗(fH∗)≥inffL∗(f)+γHPAC.L (f_H )≥ _fL (f)+ _H^PAC. Since L∗L is continuous in an appropriate topology and QH(n)⇒δfH∗Q_H^(n) _f_H along some subsequence, we obtain lim infn→∞f∼QH(n)[L∗(f)]≥inffL∗(f)+γHPAC. _n→∞E_f Q_H^(n)[L (f)]≥ _fL (f)+ _H^PAC. ∎ This instantiation corresponds to the case where the human-aligned Gibbs posterior does not concentrate on ground-truth minimizers. IV-C Information-Theoretic Limit Under Assumption 1, the supervision pipeline induces the Markov chain Y∗→S→Θ.Y → S→ . By the data-processing inequality, I(Y∗;Θ)≤I(Y∗;S)=CHeff.I(Y ; )≤ I(Y ;S)=C_H^eff. (22) Let d:×→ℝ+d:Y×Y _+ be a distortion measure compatible with the task loss and define DΘ≔[d(Y∗,Y^Θ)].D_ [d(Y , Y_ )]. Let R(D)R(D) denote the rate–distortion function of Y∗Y under d. By rate–distortion theory, I(Y∗;Θ)≥R(DΘ).I(Y ; )≥ R(D_ ). (23) Combining with data processing, R(DΘ)≤CHeff.R(D_ )≤ C_H^eff. (24) Theorem 4 (Information-Theoretic HBI) Assume R(D)R(D) is strictly decreasing and continuous on the interval of interest. If CHeff<R(D∗),C_H^eff<R(D ), where D∗D is the minimal achievable distortion under full information, then lim infn→∞DΘ≥R−1(CHeff)≕DH>D∗. _n→∞D_ ≥ R^-1(C_H^eff) D_H>D . (25) If the task excess risk satisfies ℰ∗(f)≥c(DΘ−D∗)for some c>0,E (f)≥ c\,(D_ -D ) some c>0, then γH≥c(DH−D∗)>0. _H≥ c\,(D_H-D )>0. Proof: Rate–distortion theory implies R(DΘ)≤I(Y∗;Θ).R(D_ )≤ I(Y ; ). Data processing gives I(Y∗;Θ)≤CHeff.I(Y ; )≤ C_H^eff. Hence R(DΘ)≤CHeff.R(D_ )≤ C_H^eff. Since R(D)R(D) is strictly decreasing and continuous, it is invertible on its image, yielding DΘ≥R−1(CHeff).D_ ≥ R^-1(C_H^eff). If CHeff<R(D∗)C_H^eff<R(D ), strict monotonicity implies R−1(CHeff)>D∗.R^-1(C_H^eff)>D . The excess-risk lower bound follows from the distortion–risk inequality. ∎ IV-D Causal Non-Identifiability In a Structural Causal Model (SCM) [20, 21], X→Y∗→S→Θ,H→S,X→ Y → S→ , H→ S, (26) the supervision mechanism S=h(X,Y∗,H,US)S=h(X,Y ,H,U_S) is typically many-to-one in Y∗Y : different ground-truth outputs may receive the same human label or judgment. This non-invertibility makes f∗f non-identifiable from (X,S)(X,S). Assumption 4 (Human channel non-invertibility) There exists a measurable set A⊆A with positive probability such that for all x∈Ax∈ A there exist y1∗≠y2∗y_1 ≠ y_2 with PH(⋅∣X=x,Y∗=y1∗)=PH(⋅∣X=x,Y∗=y2∗).P_H(· X=x,Y =y_1 )=P_H(· X=x,Y =y_2 ). (27) Theorem 5 (Causal HBI) Under Assumption 4, the ground-truth mapping f∗f is not identifiable from human-supervised data on A. For any estimator sequence fθ^nf_ θ_n based on (X,S)(X,S), there exists a compatible SCM in which the ground-truth excess risk is bounded below by the Bayes risk on A, hence strictly positive whenever Y∗Y is nondegenerate on A. Proof: Non-invertibility implies that for x∈Ax∈ A and any estimator based on (X,S)(X,S), one cannot distinguish y1∗y_1 from y2∗y_2 . For 0–1 loss, the Bayes optimal classifier on A for two such labels has an error of at least minp,1−p \p,1-p\, where p is the conditional probability of Y∗=y1∗Y =y_1 . Thus, the excess risk on A is bounded below by a positive constant whenever both labels occur with nonzero probability. For general losses, the same argument applies with the corresponding Bayes risk. ∎ IV-E Category-Theoretic Formulation Let C denote a category of semantic task objects and ℋH a category of human-representable structures. Human supervision induces a functor FH:→ℋ.F_H:C . Let L:→L:C be an evaluation functor, where R is viewed as a category whose objects are real numbers and whose morphisms are identities. Define the equivalence relation c1∼c2⟺FH(c1)=FH(c2).c_1 c_2 F_H(c_1)=F_H(c_2). Theorem 6 (No-Factorization Lower Bound) The evaluation functor L factors through FHF_H (i.e., L=L~∘FHL= L F_H for some functor L~ L) if and only if L is constant on each equivalence class of ∼ . If there exist c1∼c2c_1 c_2 with L(c1)≠L(c2)L(c_1)≠ L(c_2), then any predictor depending only on FH(c)F_H(c) incurs irreducible excess loss at least γsem=12|L(c1)−L(c2)|. _sem= 12|L(c_1)-L(c_2)|. Proof: Factorization L=L~∘FHL= L F_H holds if and only if L(c1)=L(c2)L(c_1)=L(c_2) whenever FH(c1)=FH(c2),F_H(c_1)=F_H(c_2), which is the universal property of quotient objects. If L is not constant on some equivalence class, there exist c1∼c2c_1 c_2 with distinct loss. Any predictor defined solely on ℋH must assign identical outputs to c1c_1 and c2c_2. Therefore at least one of the two incurs loss at least half their difference, establishing the bound. ∎ IV-F RLHF as a Biased Fixed Point Preference-based methods such as RLHF [4, 19] optimize a human-aligned utility UH(π)=U∗(π)+BH(π),U_H(π)=U (π)+B_H(π), (28) where BH(π)=y∼π[BH(y)]B_H(π)=E_y π[B_H(y)]. Let π∗∈argmaxπU∗(π),πH∗∈argmaxπUH(π).π ∈ _πU (π), _H ∈ _πU_H(π). Theorem 7 (Biased Optimization Gap) Assume: 1. The policy space is compact and U∗U , BHB_H are continuous; 2. The mapping π↦BH(π)π B_H(π) is not constant. Then U∗(π∗)−U∗(πH∗)>0.U (π )-U ( _H )>0. Proof: Suppose equality holds. Then πH∗ _H also maximizes U∗U . Since it maximizes UHU_H, U∗(πH∗)+BH(πH∗)≥U∗(π)+BH(π)∀π.U ( _H )+B_H( _H )≥ U (π)+B_H(π) ∀π. Setting π=π∗π=π and using optimality of π∗π for U∗U yields BH(πH∗)≥BH(π∗).B_H( _H )≥ B_H(π ). By symmetry of the argument, BH(πH∗)=BH(π∗).B_H( _H )=B_H(π ). Repeating for arbitrary π implies BHB_H must be constant over the policy space, contradicting assumption (2). ∎ V Breaking the Human-Bounded Limit The HBI theorem applies only under Assumption 1, namely when all information about Y∗Y available during training flows exclusively through the human supervision channel PHP_H. If an auxiliary channel provides additional information about Y∗Y beyond what is contained in SHS_H, the excess-risk floor can strictly decrease or vanish. The limitation is therefore structural but conditional: it depends on the information geometry of the supervision pipeline. V-A Auxiliary Information and Hybrid Supervision Let SAS_A denote an auxiliary signal and SMS_M a model-generated signal. Hybrid supervision can be represented abstractly as Smix=Fα,β,γ(SH,SM,SA),α+β+γ=1,S_mix=F_α,β,γ(S_H,S_M,S_A), α+β+γ=1, (29) where F denotes an arbitrary measurable combination rule (e.g., weighted mixing, joint embeddings, multi-objective optimization, or concatenation). This representation is schematic and does not assume linearity. By the chain rule for mutual information, I(Y∗;SH,SM,SA) I(Y ;S_H,S_M,S_A) =I(Y∗;SH)+I(Y∗;SM∣SH) =I(Y ;S_H)+I(Y ;S_M S_H) +I(Y∗;SA∣SH,SM)≕Cmix, +I(Y ;S_A S_H,S_M) C_mix, (30) where CmixC_mix denotes the effective supervision capacity under hybrid signals. This decomposition requires no independence assumptions; it simply quantifies how additional channels increase information about Y∗Y . Proposition 1 (Auxiliary Channel Reduces the Floor) Suppose I(Y∗;SA∣SH)>0I(Y ;S_A S_H)>0. Then Cmix>I(Y∗;SH),C_mix>I(Y ;S_H), and the induced excess-risk floor under hybrid supervision, denoted γH+A(⋅) _H+A^(·), satisfies γH+A(⋅)≤γH(⋅), _H+A^(·)≤ _H^(·), with strict inequality whenever the rate–distortion function is strictly decreasing on the relevant interval. Moreover, if SAS_A is sufficient for Y∗Y (i.e., I(Y∗;SA∣SH)=H(Y∗∣SH)I(Y ;S_A S_H)=H(Y S_H)), then the floor collapses to zero. Proof: From (30), Cmix=I(Y∗;SH)+I(Y∗;SM∣SH)+I(Y∗;SA∣SH,SM).C_mix=I(Y ;S_H)+I(Y ;S_M S_H)+I(Y ;S_A S_H,S_M). Hence Cmix>I(Y∗;SH)C_mix>I(Y ;S_H) whenever I(Y∗;SA∣SH)>0I(Y ;S_A S_H)>0. By rate–distortion monotonicity, increasing channel capacity weakly reduces the minimal achievable distortion. Since excess risk is lower-bounded by distortion gap in the information-theoretic instantiation, the induced floor under hybrid supervision cannot exceed that under human-only supervision, and is strictly smaller under strict monotonicity. If SAS_A renders Y∗Y identifiable up to Bayes-optimal distortion, the achievable distortion equals the optimal distortion, and the excess-risk floor vanishes. ∎ Auxiliary information therefore mitigates γnoise(⋅) _noise^(·), reduces γpref(⋅) _pref^(·), and can eliminate γsem(⋅) _sem^(·) whenever it resolves structure not expressible through PHP_H. The bound is not architectural but informational: it disappears precisely when the supervision channel becomes sufficient. V-B Supervision Regimes Three regimes follow directly from this analysis. Human-only (H) When supervision is dominated by SHS_H, the induced floor satisfies γH(⋅)=γnoise(⋅)+γpref(⋅)+γsem(⋅), _H^(·)= _noise^(·)+ _pref^(·)+ _sem^(·), and the learner converges to the human-aligned solution. Hybrid Human+Model (H+M) Model-generated signals can reduce variance and partially correct annotation noise but do not introduce fundamentally new information about Y∗Y . Consequently, the induced floor satisfies γH+M(⋅)≤γH(⋅), _H+M^(·)≤ _H^(·), with noise effects attenuated while structural preference and semantic distortions may remain. Hybrid with Auxiliary Channels (H+M+A) When auxiliary channels provide independent information about Y∗Y , semantic compression can be removed and preference-induced gaps reduced, yielding γH+M+A(⋅)≤γH+M(⋅)≤γH(⋅), _H+M+A^(·)≤ _H+M^(·)≤ _H^(·), with γH+M+A(⋅)=0 _H+M+A^(·)=0 in the sufficient-channel case. Y∗Y SHS_HSMS_MSAS_AΘ +MH+M+A Figure 1: Conceptual information flow under human-only (H), hybrid human+model (H+M), and hybrid with auxiliary channels (H+M+A). Auxiliary channels introduce additional information about Y∗Y , increasing effective supervision capacity and reducing or eliminating the structural excess-risk floor. VI Experiments We empirically evaluate the structural predictions of the Human-Bounded Intelligence (HBI) theorem across three complementary regimes: (i) real human-preference data, (i) controlled synthetic tasks with known ground truth, and (i) externally verifiable objective benchmarks. We test whether hybrid supervision (α<1α<1) improves generalization, corruption robustness, scaling behavior, and distortion when R∗R is known. All metrics report pairwise accuracy on held-out comparisons with 95% confidence intervals across three seeds. VI-A Real Preference Data Experiments use Dahoas/full-h-rlhf [6] with 1000 training and 300 test pairs unless otherwise stated. Reward models (BERT-tiny, DistilRoBERTa-base, RoBERTa-base) are trained using Bradley–Terry loss. An auxiliary verifier (TinyLlama-1.1B-Chat-v1.0, 4-bit) provides SA(x,y)=−NLLLLM(x,y)S_A(x,y)=-NLL_LLM(x,y), and hybrid scoring is defined as Sα,λ=αSM+(1−α)λSAS_α,λ=α S_M+(1-α)λ S_A. α-Sweep Sweeping α∈0,0.25,0.5,0.75,1α∈\0,0.25,0.5,0.75,1\ shows that human-only supervision (α=1α=1) is never optimal (Table I). Hybrid gains range from +0.007 to +0.059 and are largest in lower-capacity models, consistent with structural bottleneck effects predicted by HBI. Model Human-only Best Hybrid BERT-tiny 0.399 ± 0.022 0.458 ± 0.031 DistilRoBERTa 0.433 ± 0.006 0.440 ± 0.043 RoBERTa-base 0.441 ± 0.018 0.465 ± 0.021 TABLE I: Human-only versus best hybrid supervision λ-Ablation Fixing α=0.5α=0.5, we sweep λ∈0.5,1.0,2.0λ∈\0.5,1.0,2.0\. Results (Table I) remain stable across λ, indicating that improvements arise from structural mixing rather than hyperparameter tuning. Model λ=0.5λ=0.5 λ=1.0λ=1.0 λ=2.0λ=2.0 BERT-tiny 0.451 0.458 0.451 DistilRoBERTa 0.437 0.436 0.431 RoBERTa-base 0.462 0.465 0.463 TABLE I: λ-ablation Noise Robustness Human labels are flipped with probability γ∈0,0.2,0.4γ∈\0,0.2,0.4\. As shown in Table I, hybrid supervision consistently mitigates degradation under corruption, reducing the noise component γnoise _noise predicted by the theoretical decomposition. Model γ Human-only Hybrid BERT-tiny 0.0 0.399 0.458 0.2 0.392 0.462 0.4 0.389 0.462 DistilRoBERTa 0.0 0.431 0.436 0.2 0.430 0.433 0.4 0.394 0.437 TABLE I: Robustness under human corruption Scaling Behavior Training size varies over N∈2000,4000,8000,16000N∈\2000,4000,8000,16000\. Results in Table IV and Fig. 2 show scaling reduces variance but does not eliminate the structural supervision gap: hybrid supervision matches or exceeds human-only performance across scales. Model N Human-only Hybrid BERT-tiny 2000 0.373 0.411 4000 0.410 0.409 8000 0.429 0.427 16000 0.437 0.461 DistilRoBERTa 2000 0.355 0.395 4000 0.406 0.409 8000 0.446 0.430 16000 0.441 0.447 RoBERTa-base 2000 0.349 0.384 4000 0.397 0.409 8000 0.444 0.428 16000 0.457 0.465 TABLE IV: Scaling with dataset size Figure 2: Real-data scaling behavior. Pairwise accuracy versus training size for human-only supervision (α=1α=1, blue) and hybrid supervision (α=0.5α=0.5, orange). Hybrid supervision matches or exceeds human-only performance across scales, while scaling alone does not eliminate the structural supervision gap VI-B Synthetic Known-Target Validation Synthetic experiments use a known reward R∗(x,y)=w⊤ϕ(x,y)R^*(x,y)=w φ(x,y), allowing direct measurement of alignment error and distortion norms. As shown in Table V and Fig. 3, distortion and alignment error increase monotonically toward human-only supervision (α=1α=1), confirming the predicted structural trajectory. α Accuracy Alignment Error Distortion Norm 0.00 0.510 0.329 0.632 0.25 0.513 0.534 0.670 0.50 0.510 1.172 1.136 0.75 0.502 2.797 2.658 1.00 0.493 5.197 4.965 TABLE V: Synthetic validation: pairwise accuracy, alignment error, and distortion norm as a function of α Figure 3: Synthetic distortion trajectory. Objective accuracy as a function of the human-weight parameter α in the known-target synthetic task. Distortion increases monotonically toward human-only supervision (α=1α=1), confirming the predicted structural alignment gap. VI-C Structural Supervision on GSM8K We evaluate HBI in an externally verifiable regime using GSM8K. Informative pairs consist of one correct and one incorrect solution generated by Mistral-7B-Instruct (4-bit). The auxiliary channel is defined as SA(x,y)=g(y)=Y∗(x)S_A(x,y)=1\g(y)=Y^*(x)\, where y is the generated completion, g(y)g(y) extracts the predicted answer, and Y∗(x)Y^*(x) denotes the ground-truth target. Hybrid scoring is Sα=αSH+(1−α)SAS_α=α S_H+(1-α)S_A. Objective accuracy as a function of α is reported in Table VI. Human-only supervision exhibits a persistent error floor, while hybrid supervision strictly dominates for all α<1α<1 and converges to perfect accuracy as α→0α→ 0, demonstrating auxiliary sufficiency. α Objective Accuracy 0.00 1.000 0.25 1.000 0.50 0.983 0.75 0.868 1.00 0.696 TABLE VI: Objective pairwise accuracy on GSM8K as a function of α VI-D HumanEval: Auxiliary Sufficiency and Normalization Effects On HumanEval [3], informative pairs consist of one functionally correct and one incorrect completion. The auxiliary channel encodes binary correctness SA(x,y)=passS_A(x,y)=1\pass\, while the human channel SHS_H is a learned stylistic reward model. Hybrid scores use shybrid=αz(SH)+(1−α)z(SA)s_hybrid=α z(S_H)+(1-α)z(S_A) with α=0.5α=0.5, where z(⋅)z(·) denotes batch z-score normalization. Across corruption levels γ∈0,0.2,0.4γ∈\0,0.2,0.4\, human-only and hybrid supervision both yield 0.481±0.0190.481± 0.019, while auxiliary-only achieves 1.0001.000, and Gaussian and shuffled null controls match human-only performance. This illustrates two boundary properties: (i) the human reward model exhibits a persistent structural floor relative to functional correctness, and (i) auxiliary sufficiency collapses the floor when correctness is directly revealed. The equality between human-only and hybrid performance arises from normalization-induced interaction suppression: because SAS_A is binary and perfectly separable, batch z-score normalization removes auxiliary variance within pairs, causing convex mixing to preserve the ranking induced by SHS_H. The absence of hybrid improvement therefore reflects a normalization artifact rather than a contradiction of HBI. Across real preference data, synthetic validation, and objective benchmarks (Tables I–VI, Figs. 2–3), results consistently support the central thesis: supervision-channel structure—not scale alone—determines error floors and distortion behavior, and auxiliary information expands effective learning capacity. VII Discussion and Conclusion We developed a unified theory of human-supervised learning across six frameworks—operator theory, PAC-Bayes, information theory, causal inference, category theory, and a game-theoretic analysis of reinforcement learning from human feedback—showing that when supervision is dominated by the human channel PHP_H, learning is constrained by a strictly positive excess-risk floor γH(⋅)=γnoise(⋅)+γpref(⋅)+γsem(⋅), _H^(·)= _noise^(·)+ _pref^(·)+ _sem^(·), arising from annotation noise, preference distortion, and semantic compression. Although the numerical bounds differ across analytical perspectives, their positivity reflects the same structural source: non-sufficiency of the supervision channel. The limitation is informational rather than architectural—scaling model size, data, or compute cannot recover information that never passes through PHP_H. Empirically, three regimes emerge. Human-only (H) supervision exhibits a persistent floor. Hybrid human+model (H+M) supervision reduces variance but retains structural distortions. When auxiliary channels provide independent information about Y∗Y (H+M+A), the floor weakly decreases and collapses under sufficient information. Experiments across real preference data, synthetic known-target tasks, and externally verifiable benchmarks confirm these structural signatures. In GSM8K, auxiliary correctness eliminates the floor, instantiating the prediction that increased channel capacity reduces distortion. In HumanEval, normalization-induced degeneracy shows that observable hybrid gains require non-degenerate auxiliary variance. The framework applies only when human supervision is the primary information source about Y∗Y . Auxiliary channels—such as program execution, retrieval, and verifiers—expand effective information capacity and can remove the human bottleneck when they supply independent signal. Hybrid systems therefore alter the supervision channel itself rather than merely improving optimization within it. Limitations include asymptotic optimization assumptions, semantic abstraction, and mid-scale experiments. These affect convergence rates but not the structural bound under information-reducing supervision. Future work includes modeling tool-assisted supervision, estimating γpref(⋅) _pref^(·) and γsem(⋅) _sem^(·) in real datasets, extending the information-theoretic analysis, and studying auxiliary-channel dynamics in tool-integrated agents. References [1] S. Arora, N. Cohen, and N. Golowich (2019) Fine-grained analysis of optimization and generalization for overparameterized models. In ICML, Cited by: §IV-A. [2] M. Belkin, D. Hsu, S. Ma, and S. Mandal (2019) Understanding deep learning requires rethinking generalization. Proceedings of the National Academy of Sciences 116 (32), p. 15849–15854. Cited by: §IV-A. [3] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, and W. Zaremba (2021-07) Evaluating large language models trained on code. p. . External Links: Document Cited by: §I, §I, §VI-D. [4] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei (2017) Deep reinforcement learning from human preferences. p. . Cited by: §I, §I, §I, §IV-F. [5] T. M. Cover and J. A. Thomas (1991) Elements of information theory. Wiley. Cited by: §I, §I. [6] R. Dahoas (2021) Full h-rlhf dataset. Note: https://huggingface.co/datasets/Dahoas/full-h-rlhfHuggingFace Dataset Cited by: §VI-A. [7] L. Gao, J. Schulman, and J. Hilton (2023-23–29 Jul) Scaling laws for reward model overoptimization. p. 10835–10866. Cited by: §I, §I. [8] R. Geirhos, J. Jacobsen, C. Michaelis, R. S. Zemel, W. Brendel, M. Bethge, and F. Wichmann (2020) Shortcut learning in deep neural networks. Nature Machine Intelligence 2, p. 665 – 673. Cited by: §I, §I, §I. [9] B. Guedj (2019) A primer on pac-bayesian learning. arXiv preprint arX1901.05353. Cited by: §IV-B. [10] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. W. Rae, and L. Sifre (2022) Training compute-optimal large language models. Red Hook, NY, USA. External Links: ISBN 9781713871088 Cited by: §I. [11] A. Jacot, F. Gabriel, and C. Hongler (2018) Neural tangent kernel: convergence and generalization in neural networks. Cited by: §IV-A. [12] Cited by: §I. [13] J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington (2019) Wide neural networks of any depth evolve as linear models under gradient descent. p. . Cited by: §IV-A. [14] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. p. 9459–9474. Cited by: §I, §I. [15] S. Mac Lane (1998) Categories for the working mathematician. Springer. Cited by: §I, §I. [16] D. McAllester (1999) PAC-bayesian model averaging. Cited by: §I, §I, §IV-B. [17] N. Natarajan, A. Tewari, I. S. Dhillon, and P. Ravikumar (2013-12) Learning with noisy labels. In Neural Information Processing Systems (NIPS)Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USAAdvances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USAAdvances in Neural Information Processing SystemsAdvances in Neural Information Processing SystemsAdvances in Neural Information Processing SystemsProceedings of the 40th International Conference on Machine LearningProceedings of the 36th International Conference on Neural Information Processing SystemsAdvances in Neural Information Processing SystemsAdvances in Neural Information Processing SystemsProceedings of the 37th International Conference on Neural Information Processing SystemsCOLTProceedings of the 37th International Conference on Neural Information Processing SystemsICLRProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL)NeurIPSNeurIPS, I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, R. Garnett, I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, R. Garnett, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, H. Lin, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, J. Scarlett, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, R. Garnett, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Proceedings of Machine Learning ResearchNIPS ’22NIPS ’23NIPS ’23, Vol. 3633302023233. Cited by: §I. [18] A. Y. Ng and S. J. Russell (2000) Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML ’00, San Francisco, CA, USA, p. 663–670. External Links: ISBN 1558607072 Cited by: §I. [19] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. E. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. J. Lowe (2022) Training language models to follow instructions with human feedback. ArXiv abs/2203.02155. Cited by: §I, §I, §IV-F. [20] J. Pearl (2009) Causality: models, reasoning, and inference. Cambridge University Press. Cited by: §I, §I, §IV-D. [21] J. Peters, D. Janzing, and B. Schölkopf (2017) Elements of causal inference: foundations and learning algorithms. MIT Press. Cited by: §IV-D. [22] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever (2019) Language models are unsupervised multitask learners. Cited by: §I, §I. [23] T. Schick, J. Dwivedi-Yu, R. Dessí, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom (2023) Toolformer: language models can teach themselves to use tools. Red Hook, NY, USA. Cited by: §I, §I. [24] M. Seeger (2002) PAC-bayesian generalization error bounds for gaussian process classification. Cited by: §IV-B. [25] I. Shumailov, Z. Shumaylov, Y. Zhao, Y. Gal, N. Papernot, and R. J. Anderson (2023) The curse of recursion: training on generated data makes models forget. CoRR abs/2305.17493. Cited by: §I, §I. [26] J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger (2022) Defining and characterizing reward hacking. p. 9460–9471. Cited by: §I, §I.