Paper deep dive
Policy Iteration with Human Feedback: Bringing Post-Training RL to In-context Learning
Minh-Ha Nguyen, Cathy Shyr
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/23/2026, 2:57:03 AM
Summary
The paper introduces Policy Iteration with Human Feedback (PIHF), a framework that applies generalized policy iteration principles to in-context learning using fixed-weight pretrained language models. Instead of updating model weights, PIHF iteratively revises a versioned natural-language policy and tool set (the artifact) based on feedback from a language-model critic and a clinical expert. The system evaluates these artifacts on ultra-rare-disease benchmarks, demonstrating significant improvements in diagnostic accuracy (Recall@1) across various executors, including GPT-5.4 and Qwen3.6-35B.
Entities (10)
Relation Signals (7)
PIHF → improves → Recall@1
confidence 95% · a PIHF-derived policy improved Recall@1 in one proprietary executor and three open-weight executors
PIHF → uses → In-context Learning
confidence 95% · PIHF uses a pretrained language model as its execution substrate and moves persistent revision to a versioned natural-language policy and tool set.
PIHF → appliedto → Ultra-rare-disease benchmarks
confidence 90% · Across cumulative ablations and ultra-rare-disease benchmarks, a PIHF-derived policy improved Recall@1
PIHF → evaluatedon → GPT-5.4
confidence 90% · Gains were 32.7 percentage points for GPT-5.4
PIHF → evaluatedon → Qwen3.6-35B
confidence 90% · Gains were 31.1 points for Qwen3.6-35B
Clinical Expert → reviews → PIHF
confidence 90% · A language-model critic and clinical expert review complete-panel reasoning and tool-use trajectories
Language-Model Critic → reviews → PIHF
confidence 90% · A language-model critic and clinical expert review complete-panel reasoning and tool-use trajectories
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Generative pretraining established reusable task representations; later work on language-based task conditioning and in-context learning showed that a fixed model could adapt its behavior from instructions and demonstrations. Policy Iteration with Human Feedback (PIHF) builds on this development and the recurrent evaluate-and-improve structure of generalized policy iteration. PIHF uses a pretrained language model as its execution substrate and moves persistent revision to a versioned natural-language policy and tool set. A language-model critic and clinical expert review complete-panel reasoning and tool-use trajectories to localize recurrent failures and form candidate revisions; the expert may reinterpret the evidence and retains authority over admission and rollback, while Recall@1 and Recall@5 validate outcomes after candidate execution. Across cumulative ablations and ultra-rare-disease benchmarks, a PIHF-derived policy improved Recall@1 in one proprietary executor and three open-weight executors spanning 3 to 49 billion active parameters. Gains were 32.7 percentage points for GPT-5.4 and 31.1 points for Qwen3.6-35B, a difference of 1.7 points. These results support the feasibility of using pretrained language models as fixed-weight execution substrates for expert-guided policy development in rare-disease diagnosis.
Tags
Links
- Source: https://arxiv.org/abs/2608.16831v1
- Canonical: https://arxiv.org/abs/2608.16831v1
Trouble viewing inline? Open PDF directly →
Full Text
31,036 characters extracted from source content.
Expand or collapse full text
Policy Iteration with Human Feedback: Bringing Post-Training RL to In-context Learning Minh-Ha Nguyen Affiliation: Department of Epidemiology, Vanderbilt University, Nashville, TN, USA Cathy Shyr Affiliation: Department of Pediatrics, Vanderbilt University Medical Center, Nashville, TN, USA Affiliation: Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN, USA Affiliation: Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA 16 August 2026 Abstract Generative pretraining established reusable task representations; later work on language-based task conditioning and in-context learning showed that a fixed model could adapt its behavior from instructions and demonstrations. Policy Iteration with Human Feedback (PIHF) builds on this development and the recurrent evaluate-and-improve structure of generalized policy iteration. PIHF uses a pretrained language model as its execution substrate and moves persistent revision to a versioned natural-language policy and tool set. A language-model critic and clinical expert review complete-panel reasoning and tool-use trajectories to localize recurrent failures and form candidate revisions; the expert may reinterpret the evidence and retains authority over admission and rollback, while Recall@1 and Recall@5 validate outcomes after candidate execution. Across cumulative ablations and ultra-rare-disease benchmarks, a PIHF-derived policy improved Recall@1 in one proprietary executor and three open-weight executors spanning 3 to 49 billion active parameters. Gains were 32.7 percentage points for GPT-5.4 and 31.1 points for Qwen3.6-35B, a difference of 1.7 points. These results support the feasibility of using pretrained language models as fixed-weight execution substrates for expert-guided policy development in rare-disease diagnosis. 1 Reinforcement-learning principles Reinforcement learning represents behavior by a policy and improves that policy by increasing expected return [9]. For language-model feedback learning, x∼x is a prompt sampled from a prompt distribution, and y∼πθ(⋅∣x)y _θ(· x) is a response sampled from the current policy. The scalar R(x,y)R(x,y) evaluates that prompt-response pair. Ziegler et al. and InstructGPT use a per-response log-ratio penalty (Equation 2 in each paper, with InstructGPT’s pretraining term omitted). Writing the expected log ratio as a KL divergence gives the objective stated directly by Rafailov et al. in their Equation 3 [12, 5, 8]: J(θ)=x∼y∼πθ(⋅∣x)[R(x,y)]−βx∼[KL(πθ(⋅∣x)∥πanc(⋅∣x))],β>0.J(θ)=E_ subarraycx \\ y _θ(· x) subarray\! [R(x,y) ]-β\,E_x \! [KL\! ( _θ(· x)\,\|\, _anc(· x) ) ], β>0. (1) The first term is the expected evaluator score of responses produced by the current policy and supplies the reward-seeking component of policy improvement. The second term penalizes departure from the anchor policy πanc _anc, and β sets the strength of that penalty. The evaluator R maps each prompt–response pair to the scalar signal used by the first term; in language-model feedback learning, it may be a learned reward model, a rule-based evaluator, or a human-derived score. The notation separates three kinds of variation. Sampling variation comes from the prompts x and responses y observed across rollouts. Learning changes θ and therefore changes the conditional response distribution πθ(⋅∣x) _θ(· x). Within one declared policy-update phase, the prompt distribution D, the evaluator R, the anchor πanc _anc, and the coefficient β are held fixed. Across tasks or training phases, D and R may change and thereby define a new environment and evaluation target. The objective has a closed-form exponential-tilt optimizer. Rafailov et al. give this language-model preference-optimization form in their Equation 4 [8]. Proposition 1 (KL-regularized response tilt). For a fixed x, suppose 0<Zβ(x)<∞0<Z_β(x)<∞. Maximizing the corresponding integrand of Equation 1 over conditional distributions supported by πanc _anc yields π∗(y∣x)=πanc(y∣x)expR(x,y)/βZβ(x),Zβ(x)=∑y′πanc(y′∣x)expR(x,y′)/β.π^*(y x)= _anc(y x) \R(x,y)/β\Z_β(x), Z_β(x)= _y _anc(y x) \R(x,y )/β\. (2) Equation 2 makes the improvement mechanism explicit. The anchor supplies the base response distribution, while expR(x,y)/β \R(x,y)/β\ reweights each response according to its evaluated quality. The normalizer Zβ(x)Z_β(x) converts those reweighted masses into a conditional distribution. Higher reward therefore shifts probability toward a response relative to other responses for the same prompt. Applying the score-function identity underlying REINFORCE [11, Section 4, Theorem 1] to the full regularized objective, and using y∼πθ(⋅∣x)[∇θlogπθ(y∣x)]=0E_y _θ(· x)[ _θ _θ(y x)]=0, gives ∇θJ=x∼y∼πθ(⋅∣x)[(R(x,y)−βlogπθ(y∣x)πanc(y∣x)−b(x))∇θlogπθ(y∣x)], _θJ=E_ subarraycx \\ y _θ(· x) subarray\! [ (R(x,y)-β _θ(y x) _anc(y x)-b(x) ) _θ _θ(y x) ], (3) where b(x)b(x) is any prompt-dependent, response-independent baseline. It leaves the expected gradient unchanged and can be chosen to reduce variance. The quantity in parentheses is the baseline-adjusted learning signal: R(x,y)R(x,y) contributes evaluated quality, and the log-ratio contributes anchor pressure. The score term ∇θlogπθ(y∣x) _θ _θ(y x) converts that signal into changes in response probability. Together, policy evaluation and this update instantiate the recurring evaluate-and-improve pattern of reinforcement learning. 2 Structural bridge to PIHF Generalized policy iteration organizes reinforcement learning as an interaction between policy evaluation and policy improvement [9, Section 4.6]. In weight-space language-model reinforcement learning, rollouts from πθ _θ expose current behavior, R(x,y)R(x,y) evaluates sampled prompt–response pairs, and optimization changes θ so that subsequent rollouts place more probability on higher-valued responses under the regularized objective. PIHF implements corresponding functions over an external artifact. Complete-panel trajectories and outcomes provide evaluation evidence. The critic and expert use that evidence to localize recurrent failures and form targeted revisions; candidate freeze, complete-panel comparison, and expert admission determine which artifact persists to the next iteration. The correspondence first separates policy representation from induced behavior. In weight-space reinforcement learning, θ denotes the learned model parameters. In PIHF, t indexes the iteration and At=(Pt,Tt)A_t=(P_t,T_t) denotes the external artifact, with natural-language policy PtP_t and available tools TtT_t. The representation-level correspondence is θ⟷At=(Pt,Tt).θ A_t=(P_t,T_t). (4) Given a prompt x, πθ(y∣x) _θ(y x) denotes the distribution over responses y. Given a clinical case z, πM,At(τ∣z) _M,A_t(τ z) denotes the distribution over complete trajectories τ induced when the frozen executor M applies AtA_t to z. The behavior-level correspondence is πθ(y∣x)⟷πM,At(τ∣z). _θ(y x) _M,A_t(τ z). (5) At the sample level, the prompt and response correspond to the case-instantiated prompt and complete trajectory: x x ⟷xt(z), x_t(z), (6) y y ⟷τ. τ. (7) PIHF separates two functions that a scalar reward may combine in weight-space RL. First, critic and expert review of panel trajectories localizes a recurrent failure to the policy stage or tool behavior that should change. This process feedback supplies credit assignment for the discrete artifact revision. Second, Recall@1 and Recall@5 measure terminal diagnostic outcomes after the frozen candidate executes the complete panel. Proposal formation, candidate freeze, outcome validation, and expert admission therefore implement the improvement cycle while keeping process feedback and outcome validation as distinct evidence streams. 3 In-context policy definition Definition 1 (In-context policy representation). Fix an executor model M. At iteration t, the mutable representation is At=(Pt,Tt),A_t=(P_t,T_t), (8) where PtP_t is the versioned natural-language policy and TtT_t is the set of tools available to the executor. For a clinical case z, the policy instantiates the executor prompt xt(z)=Prompt(Pt,z).x_t(z)=Prompt(P_t,z). (9) The symbol τ denotes one complete rollout trajectory: τ=(e1,…,eL,y^),τ=(e_1,…,e_L, y), (10) where eℓe_ is a recorded model output, tool call, or tool result, and y y is the final ranked differential. The artifact induces the trajectory distribution πM,At(τ∣z):=pM(τ∣xt(z);Tt). _M,A_t(τ z):=p_M\! (τ x_t(z);T_t ). (11) A complete trajectory may therefore span several model calls connected by intervening tool calls and results. The policy organizes execution into ordered stages Φt=(ϕt(1),…,ϕt(mt)). _t=( _t^(1),…, _t^(m_t)). (12) The stages instruct execution and index review. A critique localizes the earliest stage where τ departs from PtP_t and attributes the departure to a policy rule, tool use, or returned evidence. Conformance asks whether τ followed PtP_t. Adequacy asks whether PtP_t remains clinically sound and useful, using clinical evidence, panel outcomes, critic analysis, and expert judgment. RL parallel. The artifact AtA_t is the policy representation, and πM,At(τ∣z) _M,A_t(τ z) is its induced behavior. 4 Pretrained representations and in-context adaptation Generative pretraining produced representations that could be reused across language tasks. Radford et al. demonstrated this transfer by expressing diverse tasks through task-aware sequence interfaces and adapting a shared pretrained Transformer through supervised fine-tuning [6]. Their subsequent GPT-2 study expanded the role of reuse: language could encode the task, input, and output in one sequence and condition model behavior under fixed parameters [7]. Brown et al. then defined this fixed-weight inner loop as in-context learning, in which instructions, demonstrations, or both specify a task within the input sequence at inference [2]. This research line moved from adapting reusable representations through fine-tuning to eliciting task-conditioned behavior under fixed weights. Brooks et al. then connected fixed-weight contextual adaptation to reinforcement learning. In six small control tasks, their In-Context Policy Iteration algorithm appended real trajectories to an experience buffer, sampled prompt context from that buffer for a frozen model, and used model-generated rollouts for greedy action selection [1]. These results motivate a PIHF design hypothesis: a pretrained executor has task-relevant representations and knowledge that an explicit task policy can recruit to organize diagnostic behavior under fixed weights. PIHF uses this capacity as its execution substrate and makes the persistent object a versioned, expert-governed artifact At=(Pt,Tt)A_t=(P_t,T_t). The policy PtP_t supplies the textual task specification, and TtT_t supplies the available tools. Candidates are evaluated across the complete development panel, and each admitted artifact change guides subsequent cases. This policy-level reuse concentrates expert feedback on evaluated artifact changes and supplies PIHF’s sample-efficiency rationale. The corresponding execution and update scopes are execution under At: under A_t: τ∼πM,At(⋅∣z), τ _M,A_t(· z), M and At fixed, M and A_t fixed, (13) persistent PIHF update: PIHF update: At⟶At+1, A_t A_t+1, M fixed. M fixed. The first line summarizes a rollout under a frozen artifact. Each model call conditions M on the current model-visible context assembled under AtA_t; intervening tool interactions connect these calls into the complete trajectory τ. Across panel cases, z varies; conditional on each case, trajectory sampling varies. The executor M and current artifact AtA_t remain fixed throughout that evaluation. The second line is persistent improvement: panel evaluation and expert admission determine whether a revised artifact becomes At+1A_t+1. In-context learning names the sequence-bounded conditioning mechanism, while the versioned external artifact carries persistent state across rollouts and PIHF iterations. The development panel supplies the empirical evidence used in that admission decision. 5 PIHF operator and composed runs Let (P⋆,T⋆)=PIHF(M,Pinit,Tinit,dev).(P_ ,T_ )=PIHF (M,P_init,T_init,D_dev ). (14) The inputs are the frozen executor M, initial policy PinitP_init, initial tools TinitT_init, and complete development panel devD_dev. A declared invocation also fixes its comparison protocol and stopping condition. The output (P⋆,T⋆)(P_ ,T_ ) is the final admitted artifact when that invocation ends. The study used two composed invocations. First, public-policy development began from the initial artifact on a LIRICAL development panel: (PL,TL)=PIHF(ML,P0,T0,Ldev).(P_L,T_L)=PIHF (M_L,P_0,T_0,D_L^dev ). (15) Second, UDN development warm-started from the frozen LIRICAL artifact: (PU,TU)=PIHF(MU,PL,TL,Udev).(P_U,T_U)=PIHF (M_U,P_L,T_L,D_U^dev ). (16) Development exclusion is policy-specific. Claims about (PL,TL)(P_L,T_L) exclude LdevD_L^dev from its held-out evaluation, and claims about (PU,TU)(P_U,T_U) exclude UdevD_U^dev. The warm start evaluates procedural reuse and artifact adaptation in discrete evaluations. Unchanged cross-cohort transfer is a separate estimand. RL parallel. A warm start initializes a new improvement run from an admitted policy representation. The following sections unpack the complete-panel evaluation, proposal formation, candidate comparison, admission, and stopping operations within each invocation. 6 Development-panel evaluation Let the complete development panel be dev=(zi,yi∗)i=1n,D_dev=\(z_i,y_i^*)\_i=1^n, (17) with panel composition and inference settings frozen for a declared run. During each executor rollout, yi∗y_i^* and derived outcome signals are hidden from the executor. For a frozen artifact A, define Recall^k(M,A;dev)=1n∑i=1nyi∗∈Topk(y^i(M,A)),k∈1,5. Recall_k(M,A;D_dev)= 1n _i=1^n1\! \y_i^* _k( y_i(M,A)) \, k∈\1,5\. (18) Here yi∗y_i^* is the reference diagnosis for case ziz_i, and y^i(M,A) y_i(M,A) is the ranked differential produced by executor M under artifact A. The operator TopkTop_k returns its first k diagnoses. The indicator equals one when yi∗y_i^* appears among them and zero otherwise. Thus Recall^k Recall_k is empirical Recall@k, the fraction of the n development cases whose reference diagnosis appears within the first k positions. The hat on y^i y_i marks a model-produced prediction; the hat on Recall^k Recall_k marks the empirical value computed on the finite panel. Because Top1(y^i)⊆Top5(y^i)Top_1( y_i) _5( y_i), every Recall@1 success is also a Recall@5 success, and therefore Recall^1(M,A,dev)≤Recall^5(M,A,dev) Recall_1(M,A;D_dev)≤ Recall_5(M,A;D_dev). Recall@1 measures correct first-rank placement, while Recall@5 measures inclusion in the five-item differential. RL parallel. The indicator in Equation 18 is a case-level terminal outcome signal at cutoff k. Its empirical mean is computed after a frozen candidate has executed the complete panel and serves as a post-revision validation check of diagnostic accuracy. 7 Critic and expert proposal formation After executor outputs are sealed, let EtE_t denote the complete-panel record of trajectories, tool use, and retrospective outcomes at iteration t. A second LLM critic reviews EtE_t for recurrent reasoning and tool-use failures. It proposes an interpretation, localizes the implicated policy stage, and suggests evaluation measures, constraints, protected-win checks, and a candidate change to the policy, tools, or both. Denote this proposal material by utGu_t^G. The expert reviews the critic’s proposal against EtE_t and the incumbent artifact. The expert may accept or reject the critic’s interpretation, revise it, or replace it with a new interpretation supported by the panel evidence. The expert can likewise originate or revise the hypotheses, measures, constraints, protected-win checks, and artifact changes. Write this proposal-formation step as ut=Hform(utG,Et,At),u_t=H_form(u_t^G,E_t,A_t), (19) Here HformH_form denotes expert proposal formation. Its inputs are the critic proposal utGu_t^G, the panel evidence EtE_t, and the incumbent artifact AtA_t. Its output utu_t is the expert-authorized proposal for candidate testing; ut=∅u_t= ends that proposal path. The expert then defines the candidate protocol. Admission and rollback remain expert decisions after complete-panel evaluation. RL parallel. The critic plays the generative reward model (GRM) role by turning trajectory evidence into a reasoned evaluation; PIHF also asks it to propose a correction. The expert is the human evaluator who validates or revises that feedback and controls candidate authorization, admission, and rollback. Outcome information therefore occupies three separated roles: 1. Executor diagnosis. Outcomes remain hidden while the executor reasons and uses tools on each case. 2. Outer-loop failure selection. After outputs are sealed, outcome summaries may help the critic and expert prioritize recurrent failures and construct candidate revisions. 3. Terminal outcome validation. Recall@1 and Recall@5 are computed after each frozen candidate executes the complete panel. They test whether a reasoning-policy revision preserves or improves diagnostic accuracy before expert admission. 8 Candidate freeze and expert admission The expert proposal utu_t authorizes one edit δt _t to the incumbent artifact AtA_t. Applying and freezing that edit produces the candidate At′=Freeze(At⊕δt).A _t=Freeze(A_t _t). (20) Here ⊕ means “apply the edit.” The freeze fixes the policy and tool versions, development panel, inference settings, and output schema. The incumbent AtA_t and candidate At′A _t are then evaluated on the complete panel under the same protocol. The recall-preservation indicator is Itrecall=Recall^1(M,At′,dev)≥Recall^1(M,At,dev)∧Recall^5(M,At′,dev)≥Recall^5(M,At,dev).I_t^recall=1\! \ aligned & Recall_1(M,A _t;D_dev)≥ Recall_1(M,A_t;D_dev)\\[-1.99997pt] \;& Recall_5(M,A _t;D_dev)≥ Recall_5(M,A_t;D_dev) aligned \. (21) The indicator equals one only when both inequalities hold: the candidate preserves Recall@1 and Recall@5 relative to the incumbent. A regression in either endpoint makes the indicator zero. The qualitative expert indicator is Itexpert=clinical suggestions are sound, the revision addresses a generalizablepanel-supported pattern, and information-boundary rules are satisfied.I_t^expert=1\! \ array[]cclinical suggestions are sound, the revision addresses a generalizable\\ panel-supported pattern, and information-boundary rules are satisfied array \. (22) The expert records one binary verdict across these qualitative criteria after reviewing the candidate trajectories, complete-panel evidence, and protected prior wins. This indicator summarizes the final admission judgment; expert interpretation and proposal formation occur earlier in Equation 19. For the current candidate comparison, the candidate becomes the next incumbent only when both indicators equal one: At+1=At′,ItrecallItexpert=1,At,otherwise.A_t+1= casesA _t,&I_t^recallI_t^expert=1,\\ A_t,&otherwise. cases (23) Equation 23 governs candidate admission. When either indicator equals zero, the incumbent remains in force while the expert rejects the proposal or revises its interpretation, measures, constraints, or edit to form a new candidate. Every revised candidate is frozen and evaluated through the same two indicators. Rollback is a separate expert action: the expert selects a previously admitted safe checkpoint, freezes it as the restored incumbent, and forms subsequent proposals from that checkpoint. RL parallel. Candidate formation is the policy-improvement step. Complete-panel execution supplies post-revision outcome evidence, and expert admission determines whether the proposed policy representation persists. 9 Process-guided policy improvement and outcome validation Research on language-model reasoning distinguishes process-based feedback, which evaluates the reasoning process, from outcome-based feedback, which evaluates the final result. Uesato et al. instantiate this distinction within expert-iteration RL, treating each generated reasoning step as an action and comparing policy-improvement procedures driven by final-answer correctness, outcome-supervised reward models, and process-supervised reward models [10]. Lightman et al. compare process- and outcome-supervised reward models for best-of-N selection with a fixed generator. Their process labels identify the first incorrect step, providing more precise feedback and easing the reward model’s credit-assignment problem [3]. PIHF applies this functional separation at policy-stage and complete-trajectory granularity. The critic and expert inspect sealed reasoning and tool-use trajectories, localize a recurrent failure to the implicated stage or tool behavior, interpret its cause, and form a targeted revision of AtA_t. The expert may replace the critic’s interpretation and originate new proposal elements from the panel evidence. This stage-localized review assigns credit to the external policy component that should change and drives the policy-improvement step in Equation 19. Recall@1 and Recall@5 enter after the frozen candidate executes the complete panel. They are terminal outcome validation metrics: Recall@1 measures correct first-rank placement, and Recall@5 measures inclusion within the five-item differential. The recall-preservation indicator in Equation 21 tests whether the process-guided revision preserves diagnostic accuracy. Improvement in either metric provides downstream evidence of better diagnostic performance. PIHF therefore improves the reasoning and tool-use policy through process feedback, then validates the clinical outcome of that revision with Recall@1 and Recall@5. Each PIHF invocation declares its stopping condition in the run protocol. In the reported development study, iteration ended after diagnostic performance had plateaued for 10 or more completed iterations under finite compute [4]. This is the recorded stopping observation for that study invocation. A new invocation specifies its stopping condition before candidate evaluation begins. The last admitted artifact is then frozen for development-excluded evaluation. RL parallel. Process feedback supplies stage-localized credit assignment for policy improvement. Recall@1 and Recall@5 supply terminal outcome validation. Expert admission commits the resulting external policy as the next incumbent. 10 Portability, invariance, and attribution Let MmM_m index executor backbones, let A⋆A_ be one frozen composite artifact, and let A∅A_ denote the matched no-artifact condition. For endpoint k∈1,5k∈\1,5\, define the within-backbone benefit Δm,k=Recall^k(Mm,A⋆,transfer)−Recall^k(Mm,A∅,transfer). _m,k= Recall_k(M_m,A_ ;D_transfer)- Recall_k(M_m,A_ ;D_transfer). (24) Positive Δm,k _m,k across the declared backbones supports portability of the frozen artifact. A separate invariance estimand is the dispersion Dispk=maxmΔm,k−minmΔm,k.Disp_k= _m _m,k- _m _m,k. (25) Low dispersion supports similarity of benefit across backbones. Both estimands permit backbone-specific trajectories and absolute performance. Portability and invariance therefore require separate claims and separate uncertainty analyses. RL parallel. Holding A⋆A_ fixed while changing MmM_m evaluates how the same policy representation induces behavior under different executors. The identified object is the composite policy-and-tool artifact because policy content, tool access, and their coordination differ between A⋆A_ and A∅A_ . Current transfer evidence supports the composite artifact. Stronger attribution to the optimized reasoning process requires a content-only versus with-critique ablation that separates expert-written clinical content from the iteratively revised process structure. 11 Summary PIHF persistently improves a versioned in-context policy-and-tool artifact At=(Pt,Tt)A_t=(P_t,T_t) executed by a frozen pretrained model. A language-model critic reviews complete-panel reasoning and tool-use trajectories to identify recurrent failures and propose interpretations and changes. The expert can accept, revise, replace, or originate these interpretations and changes and retains authority over candidate formation, admission, and rollback. Stage-localized process feedback directs revision, while Recall@1 and Recall@5 validate diagnostic outcomes after each frozen candidate executes. In the liteOdyssey study, a PIHF-derived policy developed from 50 cases and executed with its tool set increased Recall@1 from 26.5% to 59.3% across 1,243 public rare-disease benchmark cases in the frontier proprietary executor and transferred across proprietary and open-weight executors [4]. For rare-disease diagnosis, PIHF converted scarce expert reasoning into an inspectable, revisable execution policy that could be reused across model backbones while model weights remained fixed. References [1] E. Brooks, L. Walls, R. L. Lewis, and S. Singh (2022) Large Language Models can Implement Policy Iteration. External Links: Document, Link Cited by: §4. [2] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020) Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33, External Links: Link Cited by: §4. [3] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2023) Let’s Verify Step by Step. External Links: Document, Link Cited by: §9. [4] M. Nguyen, E. Gray, B. A. Schuler, K. W. Byram, C. Yang, F. Ma, H. Xu, W. Su, C. Yan, W. Wei, A. Wright, L. Bastarache, J. Peterson, L. Li, S. Ma, U. D. Network, R. Hamid, T. A. Cassini, and C. Shyr (2026) Teaching agentic AI to learn expert reasoning for rare disease diagnosis. External Links: Document, Link Cited by: §11, §9. [5] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35, External Links: Document, Link Cited by: §1. [6] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever (2018) Improving Language Understanding by Generative Pre-Training. Technical report OpenAI. External Links: Link Cited by: §4. [7] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever (2019) Language Models are Unsupervised Multitask Learners. Technical report Technical Report GPT-2 Technical Report, OpenAI. External Links: Link Cited by: §4. [8] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn (2023) Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems 36, External Links: Document, Link Cited by: §1, §1. [9] R. S. Sutton and A. G. Barto (2018) Reinforcement Learning: An Introduction. Second edition, MIT Press (en-US). External Links: ISBN 9780262039246 Cited by: §1, §2. [10] J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins (2022) Solving math word problems with process- and outcome-based feedback. External Links: Document, Link Cited by: §9. [11] R. J. Williams (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8 (3-4), p. 229–256. External Links: Document, ISSN 0885-6125, Link Cited by: §1. [12] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving (2019) Fine-Tuning Language Models from Human Preferences. External Links: Document, Link Cited by: §1.