Paper deep dive
Iterative Policy Refinement through Semantic Rollout Analysis
Feiyu Gavin Zhu, Qi Xu, Zhifei Deng, Zhigang Hua, Luke Simon, Jean Oh, Reid Simmons
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/3/2026, 3:50:28 AM
Summary
The paper proposes a closed-loop framework for iteratively refining structured policies in imitation learning using LLM-guided semantic rollout analysis. By logging policy rollouts as tabular data and prompting an LLM to generate diagnostic code, the method identifies structural suboptimalities and automatically corrects them without human intervention. Experiments on car racing and door opening tasks demonstrate up to 15% performance improvement over zero-shot LLM-generated structures and 75% reduction in compute for reinforcement learning convergence.
Entities (11)
Relation Signals (7)
Iterative Policy Refinement Framework → evaluatedon → Car Racing
confidence 95% · Experiments on car racing and door opening tasks show that our approach improves...
Iterative Policy Refinement Framework → evaluatedon → Door Opening
confidence 95% · Experiments on car racing and door opening tasks show that our approach improves...
Tabular Rollout Analysis → partof → Iterative Policy Refinement Framework
confidence 95% · By logging rollouts as semantically meaningful tabular data... our method identifies suboptimalities
Iterative Policy Refinement Framework → uses → LLM
confidence 95% · We propose a closed-loop framework that iteratively refines structured policies using LLM-guided analysis of policy rollouts.
Iterative Policy Refinement Framework → improves → Imitation learning
confidence 90% · our approach improves imitation learning performance by up to 15%
Iterative Policy Refinement Framework → reduces → Reinforcement Learning
confidence 90% · requires 75% less compute to achieve the same reinforcement learning performance
Iterative Policy Refinement Framework → buildsupon → KIM
confidence 80% · Our representation follows KIM [31]: generated code specifies a sparse semantic structure
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Structured policies improve efficiency, robustness, and interpretability in imitation learning by introducing task-specific inductive bias, but existing structure generation methods rely either on extensive human input or on static domain knowledge encoded in LLMs, which may be inconsistent with the expert demonstrations. We propose a closed-loop framework that iteratively refines structured policies using LLM-guided analysis of policy rollouts. By logging rollouts as semantically meaningful tabular data and prompting the LLM to generate diagnostic analysis code, our method identifies suboptimalities in the policy structure and iteratively corrects them without requiring human instruction. Experiments on car racing and door opening tasks show that our approach improves imitation learning performance by up to 15% over zero-shot LLM-generated structures and requires 75% less compute to achieve the same reinforcement learning performance. These results demonstrate that tabular rollout analysis provides an effective feedback signal to align LLM-generated policy structures with expert demonstrations, and we can utilize it to generate good policy structures automatically.
Tags
Links
- Source: https://arxiv.org/abs/2610.01652v1
- Canonical: https://arxiv.org/abs/2610.01652v1
Trouble viewing inline? Open PDF directly →
Full Text
67,257 characters extracted from source content.
Expand or collapse full text
NEmo: Neuro-Symbolic Embodied Intelligence Iterative Policy Structure Refinement through Semantic Rollout Analysis Feiyu Gavin Zhu Affiliation: Carnegie Mellon University Email: feiyuz@cs.andrew.edu Qi Xu Affiliation: Meta Zhifei Deng Affiliation: Meta Zhigang Hua Affiliation: Meta Luke Simon Affiliation: Meta Jean Oh Affiliation: Carnegie Mellon University Reid Simmons Affiliation: Carnegie Mellon University Abstract Structured policies improve efficiency, robustness, and interpretability in imitation learning by introducing task-specific inductive bias, but existing structure generation methods rely either on extensive human input or on static domain knowledge encoded in LLMs, which may be inconsistent with the expert demonstrations. We propose a closed-loop framework that iteratively refines structured policies using LLM-guided analysis of policy rollouts. By logging rollouts as semantically meaningful tabular data and prompting the LLM to generate diagnostic analysis code, our method identifies suboptimalities in the policy structure and iteratively corrects them without requiring human instruction. Experiments on car racing and door opening tasks show that our approach improves imitation learning performance by up to 15% over zero-shot LLM-generated structures and requires 75% less compute to achieve the same reinforcement learning performance. These results demonstrate that tabular rollout analysis provides an effective feedback signal to align LLM-generated policy structures with expert demonstrations, and we can utilize it to generate good policy structures automatically. 1 Introduction Black-box deep learning models are highly non-linear and can exhibit unpredictable malicious behaviors [1] or inconsistent performance [20]. Recent work addresses this by using LLMs to generate task-specific policy structures that are sparse and semantically meaningful [31], improving the efficiency and robustness of imitation learning. However, these approaches rely heavily on human experts to provide domain knowledge or to continuously monitor and refine the generated structures. In principle, LLMs already encode substantial task-relevant knowledge from pretraining and could be used to instantiate policy structures. However, because LLMs do not directly observe the environment, the generated structures may encode modeling assumptions that conflict with expert demonstrations or overlook critical variable interactions. For example, an LLM may introduce a damping term that links steering to angular velocity to correct spins. However, in demonstrations, an expert may steer in the same direction as the angular velocity during sharp turns, causing the learned policy to violate the intended assumption and exhibit unwanted oscillatory behavior. To resolve such misalignment, we propose a closed-loop framework that alternates demonstration-based parameter fitting with LLM-guided structural refinement (Figure 1). The process resembles software debugging: a programmer specifies expected behavior, writes tests, and uses their results to locate and correct errors. Here, the LLM generates data analysis code to test expected policy behavior, runs it on tabular rollouts, and uses the results to revise the structure before fitting its parameters again. This debugging interface is possible because structured policies are sparse and semantically meaningful: each intermediate quantity has an interpretable role, and its dependencies can be traced to specific computations that the LLM can inspect and edit. For the turning example above, tests can examine whether the angular-velocity term damps rotation as intended and whether steering oscillates during turns. The results can guide updates to that term or its parameter constraints, such as using softplus to enforce an intended coefficient sign. Unlike prior methods that rely on visual [4] or textual feedback [25], our approach executes analysis code outside the LLM on complete rollout tables and returns only the results, allowing it to handle trajectories beyond the LLM’s context limit and data that are not easily visualizable. Our contributions are threefold: (1) LLM-driven automated policy structure generation and refinement without human expert input, (2) the use of tabular rollout analysis as environment feedback for policy updates, and (3) empirical evidence that structure refinement improves the efficiency of reinforcement learning from demonstrations in control tasks. Figure 1: Overview of policy refinement. 1. Generate the initial structure by querying an LLM with the task description. 2. Update parameters through imitation learning on the demonstrations. 3. Generate analysis code based on the desired behavior of the policy and its structure topology. 4. Collect tabular trajectories, recording observations, actions, and named latent values during policy rollouts. 5. Execute analyses and revise the structure based on the analysis results. The updated program returns to parameter fitting and another refinement cycle. 6. Reinforcement learning after the structure aligns with the demonstrations to go beyond the demonstration performance. 2 Related work 2.1 Structured policies and learning from demonstrations Since embodied policies can often be naturally expressed in code, many works have explored incorporating LLM code generation into policy representations. The most direct approaches generate executable policies, symbolic plans, or production rules from language instructions [16, 17, 14]. Other approaches generate intermediate components that define behavior indirectly, such as spatial value maps and language-guided state abstractions [10, 26, 25]. More recent graph-based approaches, including Graph-as-Policy (GaP), compose perception, planning, and control modules into interpretable computation graphs [3]. Demonstration-guided approaches use expert knowledge, narrated actions, or interactive corrections to construct interpretable representations of behavior [31, 13, 32]. Our representation follows KIM [31]: generated code specifies a sparse semantic structure, while continuous parameters are learned from demonstrations. We extend this separation by generating tabular data analysis code to identify misalignment and guide structural updates, reducing the need for human corrective instructions. Reinforcement learning from demonstrations improves learning efficiency by shaping rewards, augmenting replay data, or initializing policies and dynamics models [2, 8, 27, 7]. Our work uses demonstrations to fit parameters within successively refined structures, so subsequent reinforcement learning starts with both a learned policy and a demonstration-aligned structure. 2.2 LLM-guided program revision and execution feedback Classical neural architecture search automates structure design through differentiable, evolutionary, or learned search over model primitives [18, 11, 33]. LLM-guided methods instead use semantic knowledge and execution feedback to revise programs directly, including self-debugging, algorithm evolution, and coding-agent improvement [5, 22, 29]. Related methods revise reward functions or state representations using learning and environment feedback [21, 15, 25]. In robotics, programmatic search and agentic learning systems use rollout feedback to improve control programs, reusable skills, or the surrounding learning workflow [9, 19, 28]. Graph-based refinement extends this process to modular policy graphs, with GaP using simulation rehearsal to improve graph structure and parameters [3]. Our pipeline uses code generation in two complementary ways: one program defines the trainable policy structure, and another analyzes its tabular rollouts. The sparse semantic structure connects numerical feedback to specific computations that can be revised, while demonstrations fit the continuous parameters between revisions. This differs from searching only over complete executable policies or optimizing a reward function: the analysis code inspects the intermediate values of the policy itself to guide structural refinement. (a) Semantic computation graph target_speedspeedsubtractspeed_errorgainReLU → multiply→ clamp to [0,1][0,1]throttleObservationLatentActionWeight (trainable) Figure 2: A sample speed-control subgraph and its corresponding PyTorch implementation. Solid nodes distinguish observations (gray), semantic latents (blue), action outputs (green), and trainable weights (orange); dashed nodes denote operation nodes (some collapsed due to space constraints). The returned speed_error exposes the latent value for further rollout analysis. (b) Corresponding PyTorch code snippet ⬇ class SpeedControl(n.Module): def __init__(self): # Trainable weights self.target_speed = n.Parameter( torch.tensor(0.6)) self.gain = n.Parameter( torch.tensor(1.0)) def forward(self, speed): speed_error = self.target_speed - speed throttle = self.gain * speed_error.relu() return throttle.clamp(0, 1), "speed_error": speed_error 3 Semantic policy structure Following previous work [31], a structured policy computes actions from observations through a semantically meaningful latent space. Formally, each structured policy has a symbolic component ⟦α⟧=⟨(V,P),E⟩ α = (V,P),E : a bipartite graph whose variable nodes V represent intermediate values, operator nodes P compute transformations between them, and edges E connect variables and operators. Each structured policy also has a neural component Θ containing all learnable latent values (a subset of V), which determine the exact continuous output. We implement each structured policy as a task-specific PyTorch model [24], representing the graph topology in code and fitting its parameters through the autograd function. We denote the policy with structure ⟦α⟧ α and trained weights Θ by π⟨⟦α⟧,Θ⟩ _ α , , using ⟦α⟧ α for both the graph topology and its PyTorch implementation. Figure 2 shows a minimal speed-control example alongside its underlying implementation. The topology defines a proportional controller for maintaining constant speed, while imitation learning fits parameters such as target_speed and gain to the demonstration’s speed and responsiveness. This highlights the separation of symbolic and neural components: the symbolic structure is task-specific, specifying the intermediate steps that generate actions; continuous parameters are demonstration-specific, determining how the demonstration completes the task. Unlike post-hoc explanations of black-box models, latent semantics directly reveal expected policy behavior, while intermediate computations trace how actions are generated. We audit all the internal mechanisms that generate actions and use this information to detect structure-demonstration misalignment and propose structural refinement. 4 Iterative refinement through semantic analysis We draw inspiration from software development: programmers write tests for intermediate steps to flag potential issues and use the feedback to improve the software. We treat the structured policy similarly to develop through this iterative refinement process (Figure 1). 4.1 Generating the initial policy We generate ⟦α(0)⟧ α^(0) by querying the LLM with only the task description, whereas prior work requires demonstrator-provided knowledge of how demonstrations are generated. We use the superscript (i)(i) to denote the policy after i iterations. Relying on pretrained knowledge and an incomplete environment description, the LLM typically generates simple structures (e.g., Figure 3(a)) that may misalign with the structure underlying the demonstration. Subsequent refinement aims to close this structure-demonstration gap. 4.2 Fitting policy parameters from demonstrations Given the policy structure, we still need to tune its parameters to complete the policy. We use imitation learning to maximize the probability of demonstration actions given demonstration states. Formally, given a demonstration as a set of state-action pairs, =⟨s0,a0⟩,⟨s1,a1⟩,⋯D=\ s_0,a_0 , s_1,a_1 ,·s\ (1) we train the weights Θ(i) ^(i) of structure ⟦α(i)⟧ α^(i) with the negative log-likelihood loss [23] Θ(i)=argmaxΘs,a∈[−logP(a∣π⟨⟦α(i)⟧,Θ⟩(s))] ^(i)= _ E_s,a [- P (a _ α^(i) , (s) )] (2) In Figure 3(a), imitation learning fits centering_weight to determine how quickly the policy steers toward the track center. This highlights the division of labor: the LLM knows that a race car generally needs to stay near the track center and encodes this knowledge in the structure. However, it does not know how aggressively the car should recenter, so this information must be learned from demonstrations and reflected in the weights while keeping the structure unchanged. (a) Before: fixed centering strength Lateral offset x Centering weight multiply Centering contribution Other steering contributions add Steering command (b) After: centering amplified when off-track Lateral offset x Centering weight multiply Centering contribution Other steering contributions add Steering command Off-track indicator max(|x|−CLOSE (|x|- OPENhalf_track_width,0) half\_track\_width,0) Off-track strength k≥0k≥ 0 multiply, then +1+1 Recovery multiplier multiply (c) Recovery behavior analysis Expected Behavior: When the race car is far from the track center, it should steer back toward the center. Analysis Code: ⬇ # Filter: far from the track center off_track = abs(log.offset) > half_track_width frames = log[off_track] # Compare: steering toward center? toward = frames.steering * frames.offset > 0 fraction = mean(toward) Analysis Result: Only 14.9% satisfy the check (d) Recorded lateral offset and steering Figure 3: Example of off-track recovery structure refinement. (a) The steering structure before refinement (other nodes omitted); (b) The refined structure adds stronger steering when the race car is off-track; (c) The desired behavior of re-centering, corresponding test, and the result; (d) Recorded lateral offset and steering. 4.3 Generating analysis code from the policy structure By capturing semantics, the policy structure provides an interface for detailed inspection and localizing issues in each subgraph. As when generating unit tests in software development, we instruct the LLM to reflect on expected behaviors and generate corresponding tests. Given the task description and current structure ⟦α(i)⟧ α^(i) , the LLM proposes hypotheses and writes Python analysis functions Tj(i)T^(i)_j to test each of the hypotheses. Figure 3(c) illustrates one hypothesis: the policy should attempt to recenter after the car goes off-track. The policy structure connects this hypothesis to a specific editable computation: the centering contribution generated from lateral offset (subgraph shown in Figure 3 (a)). The test directly queries relevant variables (e.g., offset). We take advantage of the flexibility of Python syntax and have no limit on what kind of tests it can generate. The above example filters off-track time steps and measures the percentage of time steps in which the policy exhibits recovery behavior. As these tests specify general desired behavior, they can be generated directly from the policy structure, independently of actual rollouts. It shows a unique advantage: exposing semantic latents lets the LLM test specific questions, such as whether a steering contribution sufficiently influences off-track behavior, rather than inspecting only the aggregate action. During each iteration, the LLM generates multiple tests, each testing an individual behavior and subgraph of the policy. Other generated analyses examine action saturation, reconstruction errors, and speed regulation (Appendix E.1). 4.4 Collecting tabular trajectories from policy rollouts After training π⟨α(i),Θ(i)⟩ _ α^(i), ^(i) , we execute it and collect trajectories for analysis. At each time step, the policy takes in the current state sts_t and populates all the latent values z0,t(i),z1,t(i),…,zk,t(i)z^(i)_0,t,z^(i)_1,t,…,z^(i)_k,t in its computation graph ⟦α(i)⟧ α^(i) . Therefore, in addition to the output distribution ata_t, it also returns all of the intermediate latent states, such as lateral_offsett=0.1 lateral\_offset_t=0.1. We organize observations, latents, parameters, and actions into a tabular trajectory, X(i)[t,:]=[st,z0,t(i),z1,t(i),…,zk,t(i),Θ(i),at,st+1]X^(i)[t,:]=[s_t,z^(i)_0,t,z^(i)_1,t,…,z^(i)_k,t, ^(i),a_t,s_t+1] (3) where each row records the current state, intermediate latents, weights, action output, and next state. Figure 3(d) plots these two recorded columns over the full 1,000-step rollout. The traces show the raw values supplied to the recovery behavior code analysis. This shows the advantage of a semantic latent space: recorded trajectories contain not only states and actions but also the physical meaning of the intermediate computations that generate each action. 4.5 Executing the analysis code and revising the policy structure We execute tests T(i)T^(i) for desired behaviors on rollout X(i)X^(i) through a Python interpreter and collect their outputs. The tests can be applied to arbitrarily long rollouts or large rollout tables without exceeding the LLM’s context length. Their results indicate whether the policy meets each criterion and, if not, which part of its structure is suboptimal. In Figure 3(c), the test finds center-directed steering in only 14.9%14.9\% of off-track steps. It identifies a state-dependent discrepancy: the executed policy rarely selects center-directed steering where the analysis expects recovery when the race car is off the track. This motivates a structural update making the steering more aggressive when the race car is off the track. The LLM revises the structure using the task description, current structure, analysis code, and analysis outputs. Edits can change dependencies, operations, latent variables, coefficient constraints, and initialization. Figure 3(b) shows the recovery branch after the revision. The revised program adds a pathway explicitly handling off-track behavior. It computes an off-track indicator measuring excess lateral offset, scales it by a trainable nonnegative recovery strength, and adds one to form a multiplier. The additional operations make recovery strength depend explicitly on the state while retaining the other steering contributions. The program exposes the new indicator as latent_offtrack_amount, allowing subsequent data analysis code to inspect when this branch is active. The illustrated change creates a structural update, rather than merely changing a fitted parameter. Multiple analysis scripts may expose several issues in the structured policy during each update iteration, and the LLM is instructed to address all of them in that iteration. The revised structure then repeats the cycle of training, expected-behavior test generation, rollout inspection, and revision. 4.6 Reinforcement learning on the demonstration-aligned structure Convergence in the policy structure ⟦α⟧ α indicates no structural improvement is needed for the policy. To further improve performance, we apply a finite-length variant of GSPO [30] reinforcement learning to further optimize the continuous parameters Θ . Specifically, given k trajectories per step, we use a sliding window of length l to break each of them into tracklets of length l, then form groups such that the starting positions of the groups are similar. Appendix C.2 describes the implementation details. 5 Experiments and analysis We evaluate iterative refinement on Car Racing and Door Opening using one demonstration per task. The experiments test whether iteration improves policies, compare the complete pipeline with gradient learning and program evolution, and examine how semantic analysis guides refinement. 5.1 Environment setup and choice of baselines We choose two tasks with complementary challenges: Car Racing tests long-horizon behavior where small errors accumulate, while Door Opening requires distinct approach and pull stages with abrupt changes in appropriate actions at contact. Both tasks use continuous actions and low-dimensional semantic observations (details in Appendix D.2). A human using a joystick provides the Car Racing demonstration, and a motion planner provides the Door Opening demonstration. We compare with the following three baselines: 1. MLP: Unstructured neural policies: a two-layer unstructured MLP that can be learned through both imitation learning and reinforcement learning 2. Zero-shot Structure: Structured but non-adaptive LLM-generated policies: a structured policy generated by an LLM with the same prompt but without any refinement 3. MLES [9]: Fully LLM-driven programmatic evolution: LLM-guided evolutionary search in the policy space; the policy cannot be further trained with demonstrations or rewards. The refinement experiments use GPT-5-mini with high reasoning effort. For all imitation and reinforcement learning, we train to convergence. Method Demo LLM calls ↓ Resets ↓ Return ↑ Human demonstrations - - - 871.0±12.8871.0± 12.8 MLP + IL Yes 0 0 636.6±258.6636.6± 258.6 MLP + RL, low compute Yes 0 400 591.8±330.5591.8± 330.5 MLP + RL, high compute Yes 0 2000 850.2±101.4850.2± 101.4 Initial structure + IL Yes 1 0 766.5±100.4766.5± 100.4 Initial structure + RL, low compute Yes 1 400 657.0±37.3657.0± 37.3 Initial structure + RL, high compute Yes 1 2000 901.1±14.0¯ 901.1± 14.0 MLES No 2016 8064 847.1±132.5847.1± 132.5 MLES, limited queries No 16 64 240.0±130.3240.0± 130.3 Refined structure + IL (ours) Yes 9 4 882.0±29.33882.0± 29.33 Refined structure + RL, low compute (ours) Yes 9 404 900.5±14.2900.5± 14.2 Refined structure + RL, high compute (ours) Yes 9 2004 906.5±24.0906.5± 24.0 Table 1: Car Racing performance and resource usage. Returns are mean ± standard deviation over ten random seeds. Resets include refinement and training interactions, excluding final evaluation. 5.2 Iterative refinement and overall performance comparison Figure 4: Execution feedback improves policy refinement in Car Racing. Generated analysis avoids the large intermediate drop under direct table feedback. Shading shows standard deviations. Iteration improves performance Table 1 shows that structural refinement improves performance over the zero-shot structure with no refinement. Under imitation learning, the refined policy significantly outperforms the zero-shot model by 15%15\% (p≈0.007p≈ 0.007), with performance similar to human demonstrations. This verifies our hypothesis that refinement corrects the mismatch between the LLM’s pretrained task knowledge and the demonstration’s true structure. The zero-shot structure requires significantly more RL steps for similar performance (20002000 vs 404404 resets), showing that a better structure improves RL efficiency. The zero-shot structure is expected to eventually match performance because it is inconsistent with the expert demonstrations, rather than inherently wrong. This also explains why low-budget RL underperforms IL (p≈0.010p≈ 0.010): the policy must unlearn IL parameters at a local optimum. This also holds for Door Opening, where structural refinement produces a reach-and-pull policy that generalizes from one demonstration to unseen configurations (Figure 5). The initial policy’s linear-motion assumption causes upward drift; the next revision adds an arc toward the handle but no pull, before later refinement learns both approach and pull. Refinement reduces the minimum handle-to-target distance by approximately 80% (p<10−4p<10^-4). Semantic analysis of refinement. The generated analyses examine summary statistics, reconstruction errors, correlations and regressions, and temporal behavior to connect policy computations with their observed effects. These checks guide both changes to existing features and the addition of new interactions: repeated slip checks lead to a revised feature and a sign constraint, while the recovery behavior analysis motivates stronger centering when the car is off-track. Manual inspection also shows that refinement retains the initial policy’s ten strategy categories and adds two more, rather than replacing the controller with a collection of unrelated rules. Appendix A details the strategy and diagnostic coverage, and Appendix B presents the corresponding tests and structural updates. Iteration 0 0.548±0.00.548± 0.0 Iteration 1 0.545±0.0080.545± 0.008 Iteration 2 0.540±0.0230.540± 0.023 Iteration 3 0.110±0.0590.110± 0.059 Figure 5: Door Opening progresses through three revisions. The numbers below show the mean and std of distance from handle to target across 10 seeds; lower is better. Performance comparison with other baselines The refined structure outperforms the MLP in mean return under each matched learning condition. After IL, it achieves 38%38\% better performance than MLP (p≈0.019p≈ 0.019); at the low RL budget, it also achieves 52%52\% better performance (p≈0.021p≈ 0.021). At the high RL budget, the refined policy has Table 1’s highest mean return (906.5 vs 850.2 for MLP), but the difference is not statistically significant (p≈0.136p≈ 0.136). But with RL, the refined structure exceeds the human demonstrations with significance (p≈0.002p≈ 0.002), demonstrating that RL enables the policy to perform beyond the demonstrations. Combining structural refinement with gradient learning also achieves high returns with far fewer LLM queries than program evolution alone. With 16 queries, MLES reaches 240.0 return, compared with 900.5 for the refined policy with a single demonstration. MLES reaches similar performance at 847.1 with 2,016 queries, 224x224x the refined policy’s query count, and 8,064 environment resets. It therefore requires a much larger search budget to approach the performance of the refined pipeline. These comparisons show that combining symbolic structure with continuous updates from demonstrations is more efficient than either component alone. Feedback Input tokens (k) ↓ Output tokens (k) ↓ Trajectory table 110.1±1.6110.1± 1.6 17.9±1.317.9± 1.3 Analysis results 26.0±3.426.0± 3.4 37.5±2.937.5± 2.9 Table 2: Generated analysis uses fewer input tokens and more output tokens than direct trajectory-table feedback. Values summarize six refinement rounds, with population standard deviations. 5.3 Ablation on feedback type Execution feedback improves both the quality and trajectory of policy refinement (Figure 4). We compare generated analysis with two conditions: 1) direct trajectory-table feedback, providing the entire rollout table to the LLM, and 2) revision without execution feedback. Without closed-loop environment feedback, asking the LLM to improve the structure increases misalignment and causes return to collapse. Both feedback conditions reach high returns, but direct table feedback is more unstable, dropping to 415.4 at the second revision. The analysis-based condition remains above 839 across all four revisions, providing more consistent improvement along this refinement trajectory. Our data analysis method uses only 25%25\% of the input tokens required by table feedback (Table 2). Output usage doubles because the LLM generates analysis in addition to structure code, but the total number of processed tokens remains much lower. 6 Limitations Because tests are generated from the existing policy structure and table column names, the LLM may not anticipate every failure mode or generate tests for it. As a result, some problems may never be exposed or corrected. Additionally, current evaluations cover only low-dimensional tasks following the MDP assumption. A natural next step is to scale the method to more complex tasks with larger feature spaces or possibly a POMDP formulation [12]. The method may also benefit from a fine-tuned LLM and/or a more carefully designed, higher-level domain-specific language such as Scenic [6], which may be easier to update than PyTorch. 7 Conclusion We introduced iterative policy structure refinement through semantic rollout analysis. The method alternates demonstration-based parameter fitting with LLM-generated diagnostics and structural edits, linking execution and revision through semantically meaningful latents. The sparse, semantic structure allows the LLM to test specific relationships within a policy and use the results to guide structural revision, following the cycle of testing and revision used in code debugging. With one demonstration per task, refinement improves imitation learning, generalizes to unseen task configurations, and reduces subsequent reinforcement learning resets. Acknowledgments and Disclosure of Funding Feiyu is supported by the SoftBank Group–Arm Fellowship. This work is part of the Keio-CMU partnership. References [1] Jan Betley, Jorio Cocola, Dylan Feng, James Chua, Andy Arditi, Anna Sztyber-Betley, and Owain Evans. Weird generalization and inductive backdoors: New ways to corrupt llms. arXiv preprint arXiv:2512.09742, 2025. [2] Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In IJCAI, pages 3352–3358, 2015. [3] Kaiyuan Chen, Shuangyu Xie, Letian Fu, Justin Yu, William Pacini, Sandeep Bajamahal, Hudson Kim, Jaimyn Drake, Daehwa Kim, Haoru Xue, Jonathan Francis, Christian Juette, Peter Schaldenbrand, Muhammet Yunus Seker, Ruwan Wickramarachchi, Uksang Yoo, Guanzhi Wang, Adithyavairavan Murali, Balakumar Sundaralingam, S. Shankar Sastry, Spencer Huang, Yuke Zhu, Linxi Fan, and Ken Goldberg. GaP: A graph-as-policy multi-agent self-learning harness for variational automation tasks. arXiv preprint arXiv:2607.05369, 2026. URL https://arxiv.org/abs/2607.05369. [4] Letian Chen, Nina Moorman, and Matthew Gombolay. Elemental: Interactive learning from demonstrations and vision-language models for reward design in robotics. arXiv preprint arXiv:2411.18825, 2024. [5] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023. URL https://arxiv.org/abs/2304.05128. [6] Daniel J. Fremont, Tommaso Dreossi, Shromona Ghosh, Xiangyu Yue, Alberto L. Sangiovanni-Vincentelli, and Sanjit A. Seshia. Scenic: A language for scenario specification and scene generation. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, 2019. doi: 10.1145/3314221.3314633. URL https://people.eecs.berkeley.edu/~sseshia/pubdir/pldi19-scenic.pdf. [7] Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, and Aravind Rajeswaran. Modem: Accelerating visual model-based reinforcement learning with demonstrations. arXiv preprint arXiv:2212.05698, 2022. [8] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. [9] Qinglong Hu, Xialiang Tong, Mingxuan Yuan, Fei Liu, Zhichao Lu, and Qingfu Zhang. Multimodal LLM-assisted evolutionary search for programmatic control policies. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=OHFNJoNtjW. [10] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. VoxPoser: Composable 3D value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023. URL https://arxiv.org/abs/2307.05973. [11] Zheng Jian, Han Wenran, Zhang Ying, and Ji Shufan. Eenas: An efficient evolutionary algorithm for neural architecture search. In Asian Conference on Machine Learning, pages 1261–1276. PMLR, 2023. [12] Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1–2):99–134, 1998. doi: 10.1016/S0004-3702(98)00023-X. URL https://w.sciencedirect.com/science/article/pii/S000437029800023X. [13] Edward Kim, Daniel He, Jorge Chao, Wiktor Rajca, Mohammed Amin, Nishant Malpani, Ruta Desai, Antti Oulasvirta, Bjoern Hartmann, and Sanjit Seshia. Interactive program synthesis for modeling collaborative physical activities from narrated demonstrations, 2025. URL https://arxiv.org/abs/2509.24250. [14] James R Kirk, Robert E Wray, Peter Lindes, and John E Laird. Improving knowledge extraction from llms for task learning through agent analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18390–18398, 2024. [15] Pengyi Li, HAO Jianye, Hongyao Tang, Yifu Yuan, Jinbin Qiao, Zibin Dong, and Yan Zheng. R*: Efficient reward design via reward structure evolution and parameter alignment optimization with large language models. In Forty-second International Conference on Machine Learning, 2025. [16] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as Policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753, 2022. URL https://arxiv.org/abs/2209.07753. [17] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477, 2023. [18] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018. [19] Runyu Lu, Yubo Wu, Ethan Kou, Letian Fu, Wenli Xiao, Ajay Mandlekar, Yinzhen Xu, Guanya Shi, Ken Goldberg, Ang Chen, Mosharaf Chowdhury, Yuke Zhu, Linxi Fan, and Guanzhi Wang. ASPIRE: Agentic /skills discovery for robotics. arXiv preprint arXiv:2607.00272, 2026. URL https://arxiv.org/abs/2607.00272. [20] Zhimeng Luo, Lixin Wu, Adam Frisch, and Daqing He. Beyond overall accuracy: A psychometric deep dive into the topic-specific medical capabilities of 80 large language models. arXiv preprint arXiv:2509.24186, 2025. [21] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2310.12931. [22] Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025. URL https://arxiv.org/abs/2506.13131. [23] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 7(1–2):1–179, 2018. doi: 10.1561/2300000053. URL https://arxiv.org/abs/1811.06711. [24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://papers.neurips.c/paper_files/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html. [25] Andi Peng, Belinda Z Li, Ilia Sucholutsky, Nishanth Kumar, Julie A Shah, Jacob Andreas, and Andreea Bobu. Adaptive language-guided abstraction from contrastive explanations. arXiv preprint arXiv:2409.08212, 2024a. URL https://arxiv.org/abs/2409.08212. [26] Andi Peng, Ilia Sucholutsky, Belinda Z Li, Theodore R Sumers, Thomas L Griffiths, Jacob Andreas, and Julie A Shah. Learning with language-guided state abstractions. arXiv preprint arXiv:2402.18759, 2024b. [27] Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. In International Conference on Machine Learning, pages 34556–34583. PMLR, 2023. [28] Wenli Xiao, Jia Xie, Tonghe Zhang, Haotian Lin, Letian Fu, Haoru Xue, Jalen Lu, Yi Yang, Cunxi Dai, Zi Wang, Jimmy Wu, Guanzhi Wang, S. Shankar Sastry, Ken Goldberg, Linxi Fan, Yuke Zhu, and Guanya Shi. ENPIRE: Agentic robot policy self-improvement in the real world. arXiv preprint arXiv:2606.19980, 2026. URL https://arxiv.org/abs/2606.19980. [29] Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin Gödel Machine: Open-ended evolution of self-improving agents. arXiv preprint arXiv:2505.22954, 2025. URL https://arxiv.org/abs/2505.22954. [30] Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025. URL https://arxiv.org/abs/2507.18071. [31] Feiyu Zhu, Jean Oh, and Reid Simmons. Sample-efficient behavior cloning using general domain knowledge. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, pages 7254–7262, 2025. doi: 10.24963/ijcai.2025/807. URL https://w.ijcai.org/proceedings/2025/807. [32] Feiyu Gavin Zhu, Jean Oh, and Reid Simmons. InterPReT: Interactive policy restructuring and training enable effective imitation learning from laypersons. In Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction, 2026. doi: 10.1145/3757279.3785549. URL https://arxiv.org/abs/2602.04213. [33] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697–8710, 2018. Appendix Appendix A Semantic analysis of policies and diagnostics We analyze the strategies represented by the policies and the coverage of generated diagnostics to understand how rollout analysis guides structural refinement. A.1 Policy strategy coverage The policy structure makes both the controller’s strategies and the targets of its diagnostics explicit. We manually group computations in the Car Racing programs into speed-control, steering-control, and special-case strategies (Table 3). The initial structure contains ten categories; refinement retains all ten and adds steering toward the outside of curves and braking during sharp turns. MLES contains fifteen categories, including more rules for grass, curbs, off-track recovery, and stopping. Our structured policies instead concentrate on general relationships between task state and actions, with specific nuanced behavior handled by continuous parameters. These categories summarize broad control strategies rather than every operator or structural edit. Strategy MLES Initial Refined Speed Control speed during straights ✓ ✓ ✓ speed at track center ✓ ✓ ✓ slow at corners ✓ ✓ ✓ slow at sides / large offset ✓ ✓ ✓ slow when outside track ✓ slow at high angular velocity ✓ ✓ no concurrent acceleration / braking ✓ ✓ ✓ Steering Control toward track center ✓ ✓ ✓ toward outside of curves ✓ speed-based steering ✓ to counter angular velocity ✓ ✓ anticipate curvature ✓ ✓ ✓ Specific Cases avoid grass ✓ avoid curbs ✓ recover when off-track ✓ brake when turning sharply ✓ ✓ brake when track is lost ✓ brake when losing traction ✓ ✓ prevent full stop ✓ Total coverage 15 10 12 Table 3: Strategy categories identified by manual inspection of Car Racing policies. A.2 Diagnostic categories and coverage The generated analyses connect these strategies to measurable properties of execution through four diagnostic categories. Together they examine the values a policy computes, whether those values follow its definitions, how they relate to behavior, and how control signals evolve over time. Summary statistics. Means, standard deviations, ranges, and signs characterize observations, semantic latents, and trainable parameters. For example, the slip coefficient’s sign tests whether its contribution to target speed agrees with the intended speed-reducing strategy. Reconstruction errors. Analyses recompute a latent or action mean from its declared inputs and compare the result with the rollout value. These checks test arithmetic consistency and help distinguish a mismatched computation from variation introduced by action sampling or logging. In the slip example, accurate target-speed reconstruction establishes that the program executes the specified formula even while its learned coefficient violates the expected sign. Correlations and regressions. Pearson correlations and linear regressions measure relationships between observations, latents, and actions across the trajectory. For example, steering as a function of lateral offset tests whether the car tends to recenter itself, connecting a named strategy to its expression during execution. Temporal behavior and action limits. Other diagnostics examine abrupt changes in control signals and the frequency of action saturation. These checks identify behavior that may be obscured by aggregate return or trajectory-wide averages. Table 4 summarizes generated diagnostics across three refinement rounds, with 13, 10, and 10 test functions, respectively. Their coverage changes as the policy evolves: later rounds add checks of angular velocity, off-track behavior, and steering at sharp turns. This combination turns the semantic structure into an interface for testing and revision: each diagnostic identifies a policy quantity, a behavioral expectation, and a computation that can be edited. Appendix B shows how these connections guide updates to features, parameter constraints, and state-dependent interactions. Data Analysis Iter 1 Iter 2 Iter 3 Sanity Checks Weights stay constant ✓ ✓ ✓ Signs consistent with assumptions ✓ ✓ ✓ Summary statistics ✓ Actions Output reconstruction ✓ ✓ Latent reconstruction ✓ ✓ ✓ Action saturation ✓ ✓ ✓ Acceleration/braking consistency ✓ ✓ Jerkiness ✓ ✓ Behavior Steering corrects lateral offset ✓ Steering corrects heading error Slow at sharp turns ✓ ✓ Slow at edge of the track ✓ Slow when angular velocity is high ✓ Slow when wheels are slipping ✓ ✓ Off-track portion ✓ Coefficient Importance Extra steering at sharp turns ✓ Number of test functions 13 10 10 Table 4: Generated diagnostics adapt to the policy across three refinement rounds. Checks cover computational consistency, action behavior, and task-specific control relationships. Appendix B Policy refinement examples The following examples illustrate how generated diagnostics guide policy changes in Car Racing. B.1 Off-track recovery behavior The recovery behavior analysis filters rollout steps where the absolute lateral offset exceeds half_track_width, then checks whether steering points toward the track center under the recorded sign convention. In the 1,000-step rollout, 74 steps are off-track and only 14.9% of these have center-directed steering. This identifies a state-dependent failure that aggregate return does not locate: the centering contribution is present, but rarely determines the steering direction when recovery is needed. The revision adds an off-track indicator and a trainable nonnegative recovery strength, forming a multiplier that increases centering beyond the track boundary (Figure 3). The other steering contributions remain in place. B.2 Car Racing programs, diagnostics, and rollouts Table 5 follows the slip-to-target-speed branch through three revisions. Repeated sign checks lead from an unconstrained wheel-mean feature to a compressed positive residual with a nonpositive contribution to target speed. Round Slip feature Fitted wqw_q Diagnosis and revision 0 Mean wheel reading +0.1729+0.1729 Positive coefficient violates the intended sign; replace wheel mean with a relative proxy. 1 u¯−v u-v +3.8367+3.8367 Positive coefficient persists; rectify the residual. 2 max(u¯−v,0) ( u-v,0) +6.6077+6.6077 Sign anomaly persists; constrain the coefficient to be nonpositive. 3 p/(1+αp)p/(1+α p) −0.00390-0.00390 Compression and wq=−|η|w_q=-|η| preserve the branch’s speed-reducing role after fitting. Table 5: Successive revisions align the slip proxy’s learned contribution with its intended speed-reducing role. Figure 6 compares 1,000-step rollouts of the initial and third-round policies. The shared axis limits expose changes in the slip proxy’s scale and the resulting target-speed trajectories. Figure 6: Recorded slip and target-speed values before refinement (left) and after three revisions (right), with shared axis limits. The slip example uses a generated sign check together with an arithmetic reconstruction check from the same round. Listing 1 shows the slip-coefficient check. Listing 1: Abbreviated generated sign check from round 0. ⬇ checks = "weight_speed_slip": ("negative", lambda v: v < 0.0), for col, (expect, fn) in checks.items(): if col not in df.columns: _append(f"col: MISSING") continue mean_val = _safe_get(df, col).mean() ok = fn(mean_val) _append(f"col: mean=mean_val:.6f, expected expect -> ’PASS’ if ok else ’FAIL’") The target-speed calculation satisfies the reconstruction check, but its positive slip coefficient fails the expected sign check (Listing 2). Listing 2: Target-speed reconstruction and slip-sign diagnostics at round 0. ⬇ === test_2_action_consistency === desired_speed reconstruction: mean_abs_diff=2.296791e-08, max_abs_diff=7.852650e-08, frac_mismatch=0.000 === test_4_signs_and_bounds === weight_speed_slip: mean=0.172909, expected negative -> FAIL The generated explanation interprets the wheel mean as a speed-related signal and proposes comparing it with vehicle speed. Later analyses revisit the coefficient after fitting the revised proxy, leading to rectification and an explicit sign constraint. Appendix C Implementation details C.1 Refinement procedure and execution Algorithm 1 summarizes the data flow and permitted interaction branches. Here Σ(C) (C) denotes the table-column schema exposed by the environment and the policy’s logging interface. Analysis-code generation depends on this schema and the policy program, independently of the numerical rollout values. Algorithm 1 Refinement through semantic rollout analysis 0: Task specification S, demonstrations D, environment 1: C←LLM(S,generation specification)C (S,generation specification) 2: while the refinement procedure continues do 3: θ←Fit(C,)θ (C,D); π^←πC,θ π← _C,θ 4: T←LLM(S,C,Σ(C),diagnostic prompt)T (S,C, (C),diagnostic prompt) 5: X←RolloutLog(C,θ)X (C,θ) 6: repeat 7: r←Execute(T,X)r (T,X) 8: b←LLM(S,C,T,r,revision context)b (S,C,T,r,revision context) 9: if b requests additional diagnostics then 10: T←T← additional analysis code 11: end if 12: until b proposes a revised program or stops refinement 13: if b stops refinement then 14: break 15: end if 16: C←C← revised program from b 17: end while 18: return π π; optionally apply post-refinement RL The refinement loop has a configurable iteration budget, with a default maximum of six rounds and early termination through the LLM’s stop response. Additional tests run on the current table before the next revision decision. If fitting a revised program fails, the LLM can regenerate it using the preceding policy and diagnostic context, with up to five attempts. C.2 Post-refinement reinforcement learning We optimize a clipped policy-gradient objective using group-relative reward-to-go and window likelihoods. For trajectory i, let TiT_i be its length and let ℐtI_t contain trajectories valid at time t. We compute Rti=∑u=tTi−1γu−trui,Ati=Rti−μtσt+10−8,R_t^i= _u=t^T_i-1γ^u-tr_u^i, A_t^i= R_t^i- _t _t+10^-8, (4) where μt _t and σt _t are the population mean and standard deviation over ℐtI_t. The likelihood ratio multiplies action likelihoods over a window of L valid steps. In log-probability form, ρit(θ)=exp[∑u=t+L−1(logπθ(aui∣sui)−logπθold(aui∣sui))]. _it(θ)= \! [ _u=t^t+L-1 ( _θ(a_u^i s_u^i)- _ _old(a_u^i s_u^i) ) ]. (5) For valid window starts, the clipped objective is (θ)=i,t[min(ρit(θ)Ati,clip(ρit(θ),1−ϵ,1+ϵ)Ati)].J(θ)=E_i,t\! [ \! ( _it(θ)A_t^i,clip( _it(θ),1-ε,1+ε)A_t^i ) ]. (6) An entropy bonus encourages exploration, with its coefficient decaying linearly over training. The optimizer defaults are γ=0.98γ=0.98, clipping threshold ϵ=0.2ε=0.2, Adam learning rate 0.0070.007, 20 sampled episodes per update, three optimization epochs, and initial entropy coefficient 0.01; the default Car Racing likelihood window is 150 steps. Unlike GSPO’s length-normalized sequence ratio [30], Eq. (5) uses the product of likelihoods within each window. Trajectories share an initial condition and are grouped by time step for advantage estimation. Appendix D Policy-generation prompts and interfaces The prompts specify the interface between the LLM, the trainable policy, and the execution logger. D.1 Shared generation specification The shared specification asks for meaningful intermediate computations, trainable coefficients, and an instrumentation dictionary. It also specifies action-distribution outputs and initialization based on expected signs and scales. * The model should represent the causal structure of a good policy for the task. Each intermediate step in the policy should have semantic meanings and the learnable parameters should only connect the variables that are correlated (or have a causal relationship) as opposed to a generic MLP where all the hidden nodes are connected. * For a discrete output space, the model should predict a (potentially unnormalized) distribution among those discrete actions. For a continuous action space, predict the parameters of a multivariate normal distribution for the action (action mean and the lower triangle matrix L such that Sigma = LˆT). Make sure the $L$ is a lower Cholesky matrix. If you think the actions should not be correlated then the L can be a diagonal matrix for simplicity. * Along with the action distribution prediction, also return an `info_dict` which is a dictionary that records all the intermediate steps and the weights. This will be aggregated into a table for further analysis. For the weights, name the key `weight_variable_name` and for the intermediate steps, name the key `latent_variable_name`. If an intermediate feature has multiple dimensions, break it down to multiple keys. But keep the name semantically meaningful. For example, instead of `latent_1`, `latent_2`, ... use `latent_feature_name` instead. Each value in the dictionary should be of shape `(1,)` or `(B,)` where B is the batch size. * Keep the model simple. * Represent new features as linear combinations of old features when possible. If you are very certain that no bias term is needed (i.e., the feature value should be 0 if all inputs are 0), then don’t include the bias term to make it easier to learn. * There might be cases where a linear combination is not sufficient; then you may use other operations, such as multiplication, to represent the interaction between two features. * Register all weights and bias terms as `n.Parameter` with `required_grad=True`. Don’t use `register_buffer`. * Don’t name a parameter using PyTorch keywords such as `self.half`. * When initializing the values for each weight and bias, set a value based on positive or negative correlation (e.g., if the input feature is negatively correlated, then set its weight to -0.1) instead of using random initialization. Pay attention to the scale of the input features and output actions and set the magnitude of the numbers accordingly. * There should be no magic number in the `forward` function. Constants like 0, 1, and 0.5 are fine. * Make sure all operations are differentiable so that the parameters can be learned by gradient descent. * You may use any of the functions defined in PyTorch (e.g., `torch.logical_and`, `torch.clamp`, `torch.abs`, `torch.square`, etc.). * Refrain from using `torch.sigmoid` or `torch.tanh` to avoid the vanishing gradient problem. * Make sure the gradient can flow back to the parameters. Avoid in-place operations (always use a new variable name instead) or constructing new tensors (always use `torch.stack` or `torch.cat` instead). * Assume the inputs are batched. # Make sure the model handles `torch.nan` gracefully. D.2 Environment descriptions and observation schemas Environment descriptions provide the semantic interpretation of each input and output coordinate. Car Racing supplies track tiles and vehicle indicators; Door Opening supplies current and previous gripper/door information and a target position. Figure 7 shows the two environments. Figure 7: Car Racing and Door Opening: continuous control from semantic observations. Car Racing. The following schema specifies the inputs to policy generation and diagnostic construction. # Features The input to the model is two tensors representing the tracks and other indicator information: 0. (float32, (B, L, 4)) tiles. Where $B$ is the batch size and $L=12$ is the number of tiles on the track ahead of the race car, and the last dimension contains $(x, y, θ, border)$ where $x$ is the signed lateral position of the tile (positive means the race car is on the left of the center of the tile), $y$ is the signed longitudinal position of the tile, $θ$ is the relative heading of the tile to the race car (positive means the road is heading to the left), and $border$ is either 0 or 1, indicating whether the tile is a sharp corner (curvature exceeds some threshold). 1. (float32, (B, 7)) indicators. Where the first column is the speed $v$ (normalized from [0, 100] to [0, 1]), the second column is the current (absolute) heading of the race car $θ$ (normalized from [-pi, pi] to [-1, 1]), the third column is the angular velocity $ω$, and the rest are the abs sensor readings (approximate wheel omega) from each wheels (in the order of front left, front right, rear left, rear right). # Output Space A tensor of shape (B, 3) where each element corresponds to steer, accelerate, and brake controls of the race car, respectively. These controls are continuous. The steer value should be in the range of $(-1, 1)$, where -1 is full left, 0 is no steering, and 1 is full right. Accelerate is in the range of $(0, 1)$, where 0 is no acceleration and 1 is full acceleration. Brake is also in the range of $(0, 1)$, where 0 is no braking and 1 is full braking. # Objective and Reward The objective is to complete the track as fast as possible without going off the track. A positive reward is given whenever the race car moves to a new tile on the track and a small negative reward is given at each step as time penalty. # Additional Notes * The track is a loop. * The track is about 0.4 unit wide. When the closest tile’s x is at -0.2 or 0.2 it means the race car is on the edge of the track. Beyond that it’s grass. * The race car is approximately 3 tiles long. * The first tile is the closest tile to the race car, and the rest are in the order of tiles ahead. * The arguments of the forward function should only be self, tiles, and indicators. * Estimate a variance for each output dimension to account for uncertainty. * Clip the output to the desired range. * Name your model "RacecarPolicy". Door Opening. The description includes current and previous observations, allowing a generated policy to use information about recent motion. # Features The input to the model is a tensor representing the position of the robot’s gripper and the door at the current and previous time step. Features: B x 39 where B is the batch size 0-2. The x, y, z coordinates of the gripper at the current time step. 3. A scalar value that represents how open/close the gripper is at the current time step. 4-6. The x, y, z coordinates of the door. 7-10. The quaternion (x, y, z, w) representing the orientation of the door. 11-17. Not used 18-20. The x, y, z coordinates of the gripper at the previous time step. 21. A scalar value that represents how open/close the gripper was at the previous time step. 22-24. The x, y, z coordinates of the door at the previous time step. 25-28. The quaternion (x, y, z, w) representing the orientation of the door at the previous time step. 29-35. Not used 36-38. The x, y, z coordinates of the target position for the door to be opened to. # Output Space A tensor of shape (B, 4) where the first three elements are the x, y, z motion of the gripper relative to its current position, and the last element is the gripper open/close command (1 for open, 0 for close). # Objective and Reward The objective is to reach for the handle on the door and pull the door open to a desired angle. The reward is given by the distance to the desired angle, when the gripper first hold onto the handle, and with a large positive reward for successfully opening the door. A small negative reward for each time step taken. # Additional Notes * The door revolves around a vertical axis. * The door handle is horizontal. * The gripper always points down. * Name your model "OpenDoorPolicy". Appendix E Diagnostic and revision prompts The diagnostic and revision prompts connect policy semantics to numerical execution evidence. The LLM first proposes analyses using the code and schema, then receives their results as the basis for a possible edit. E.1 Hypothesis generation and analysis execution The system prompt asks the LLM to consider nominal behavior and possible anomalies before writing analysis code. It requests explicit hypotheses, defensive functions over a dataframe, and output in a common results file. The full numerical rollout is an input to these generated functions rather than part of the initial diagnostic-generation context. You are trying to find patterns in data and use it to enhance a task-specific PyTorch model that represents the causal structure of the policy. Specifically, you have access to a `pd.DataFrame` that records the model’s intermediate steps as well as the observations from the environment during some policy rollouts. The model has a sparse, task-specific structure and its weights are trained using Imitation Learning from one demonstration. More specifically, the model were created with the following specifications: SPECS You are going to do the following: 1) Analyze the task domain. List the important factors / behaviors of successfully completing the task. Think about what some suboptimal behaviors for the task may be in general, not just specifically for the given model structure. 2) Analyze the model definition and determine what it is for. Write a summary on what each weight and computation is used for, and how you can verify that the learned values are sensible. Essentially, you should form some hypotheses on the data by analyzing the structure of the model. Define what some expected nominal behavior is and what may be some anomalies for both the policy and the environment (e.g., as the policy is getting rolled out, how do you expect the observations to change). 3) For each expected behavior and potential anomaly, formulate some hypothesis that you can test given the rollout data. Make sure your tests cover all of the potential issues. Also, formulate some tests that will help you analyze the behavior of the policy. As the policy is trained with potentially incompatible demonstrations, it may perform very badly. So, you should verify if the policy demonstrates some of the suboptimal behaviors that you have reflected earlier. 4) Write scripts that help you process the data and test the hypotheses. You might find many time series feature processing techniques, data filtering, as well as statistical tests useful. These scripts should help you find the anomalies. Each of your functions should only take in `df` as the input, and should print out all the results to `test_results.txt`. Put all of your tests in a `run_all(df)` function. Remember to include all relevant imports. You may use `numpy, scipy, pandas, statsmodels, scikit-learn` among others, as well as custom helper functions. Keep your code defensive and try to handle errors gracefully. Once the scripts finish running, you can see the result of `test_results.txt`. Focus on the model design. Assume that there is no bug in the code execution and no missing values in the dataframe. 5) Ultimately, the results of these scripts should inform you of the problem of the current policy structure that allow you to propose updates to the original model structure (e.g., create a new latent variable, change to another operation, enforce a bound on the learned parameters, etc.) such that it performs better after another session of Imitation Learning training. When you are writing the test scripts, keep in mind that: * Action sampling is handled outside of model using `torch.distributions`. Specifically, discrete action spaces will be sampled with `Categorical(logits=action_probs).rsample()` and continuous action spaces will be sampled with `MultivariateNormal(action_mean, scale_tril=action_cov_tril).rsample()`. * Actions will be automatically clipped to desired ranges after sampling. So it is normal for the final output action to have a different variance than the model output `action_cov_tril`. Use the following return format: [Task Analysis] [Model Summary] [Expected Nominal Behavior] [Potential Anomalies and Tests] [Hypotheses and Tests] [Code Implementation] ```py import ... def test_1(df): """Brief description of test 1""" try: # actual test implementation except Exception as e: # log the error e into the same result file. def run_all(df): ... ``` The per-policy template supplies the environment description, model definition, and available columns. [Environment Description] ENV_DESCRIPTION [Policy Structure] ```py POLICY_CODE ``` [DataFrame Columns] DF_COLUMNS E.2 Revision, additional tests, and termination The revision prompt offers three responses: provide an updated program, request more diagnostic tests, or indicate that no further action is needed. It allows changes to the computations and parameterization while preserving the model interface and logging outputs. test_results.txt ```text TEST_RESULTS ``` Analyze the test results and compare them with the model summary, expected nominal behavior, and potential anomalies. Formulate some hypotheses on why the model may diverge from expected nominal behavior and think about what in the environment might be missing in the current structure. You may assume that the model already converged under Imitation Learning so that the weights already maximize the similarity to expert actions given the current structure. Based on these analysis you can choose one of the following three options: 1) If you have a clear idea on how to improve the model, then explain the reasons and suggest the updates. Keep in mind that you are trying to design the model to learn better with Imitation Learning. You may add a new function like `torch.abs`, introduce a new latent variable, initialize the weights to different values, or add any differentiable operations. Keep the model simple to ensure learning stability. Remember not to edit variables in-place since that will confuse the gradient descent step. Keep the model explainable by prioritizing latent variables with semantic information as opposed to generic MLPs. No need to sample the actions when predicting the action mean, the sampling will be taken care of outside of the model. But you can control the sampling process by updating the output `scale_tril` matrix (e.g, adding upper/lower bounds, setting some variables to be independent, etc.) Include the description of the entire new model in the "Model Summary" section, not just the changes. Do not forget to return all weights and the intermediate steps (latent variables) in the return dictionary. Make sure each value in the `info` dict is either of shape (B,) or a scale. If there are multiple dimensions for a variable then break it down to multiple keys and use their physical meaning as keys (for example use "latent_action_steer" instead of "latent_action_0"). Do not forget to include all relevant imports and keep the function interface the same. Use the following format: [Analysis of Test Results] [Model Update Suggestions] [Model Summary] [Updated Model] ```py import torch from torch import n # other relevant imports class ModelName(n.Module): def __init__(self): ... def forward(self, tiles, indicators): ... return actions, cov_matrix, info ``` 2) If you think the current test results are inconclusive, explain why and suggest additional tests or data that could help clarify the situation. Do not include the previously generated tests. Only include the additional tests. As a reminder you may only write to `test_results.txt` and should not create other files. Use the following format: [Analysis of Test Results] [Additional Tests] ```py import ... def additional_test_1(df): """Brief description of additional test 1""" ... def run_all(df): ... ``` 3) If you find no limitations and the model seems to be functioning well, confirm this using the following format: [Analysis of Test Results] [Done - No Further Action Needed]