Paper deep dive
PID-Gated Continual Prompt Adaptation Under Noisy Feedback
Zhentao Xu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deploying LLM agents in non-stationary environments requires continually revising their instructions as the task drifts, yet rewriting after every reported mistake can turn noisy feedback into persistent errors. This raises an often-overlooked scheduling question in prompt adaptation: when should feedback trigger a lasting instruction update, and at what editor-call cost? Taking inspiration from proportional-integral-derivative (PID) control, we introduce a family of lightweight gates that react to the current error, accumulate evidence over time, and roll back deterioration after an edit. We benchmark these gates against both simple baselines and advanced prompt optimizers (TextGrad and GEPA), over a diverse suite of drifting tasks, LLM parameter sizes, feedback-noise levels, and random seeds. PID-gated continual prompt adaptation yields a more favorable Pareto trade-off between task performance (high accuracy, low error) and editor-call budget than the baselines considered, pointing to control-theoretic scheduling as a simple, general tool for robust prompt adaptation under unreliable feedback.
Tags
Links
- Source: https://arxiv.org/abs/2610.16129v1
- Canonical: https://arxiv.org/abs/2610.16129v1
Trouble viewing inline? Open PDF directly →
Full Text
33,442 characters extracted from source content.
Expand or collapse full text
Continual Learning in the Era of Foundation Models and Embodied Agents PID-Gated Continual Prompt Adaptation Under Noisy Feedback Zhentao Xu Affiliation: Independent Researcher Affiliation: Sunnyvale, CA, USA Email: frankxu@umich.edu Abstract Deploying LLM agents in non-stationary environments requires continually revising their instructions as the task drifts, yet rewriting after every reported mistake can turn noisy feedback into persistent errors. This raises an often-overlooked scheduling question in prompt adaptation: when should feedback trigger a lasting instruction update, and at what editor-call cost? Taking inspiration from proportional-integral-derivative (PID) control, we introduce a family of lightweight gates that react to the current error, accumulate evidence over time, and roll back deterioration after an edit. We benchmark these gates against both simple baselines and advanced prompt optimizers (TextGrad and GEPA), over a diverse suite of drifting tasks, LLM parameter sizes, feedback-noise levels, and random seeds. PID-gated continual prompt adaptation yields a more favorable Pareto trade-off between task performance (high accuracy, low error) and editor-call budget than the baselines considered, pointing to control-theoretic scheduling as a simple, general tool for robust prompt adaptation under unreliable feedback. We open source our code at https://github.com/zhentao-xu/pid-gated-agent-learning. 1 Introduction Consider an agent guessing a hidden integer under noisy high/low feedback: it holds a belief about the plausible range [L,U][L,U] that must tighten as evidence accumulates, yet a single mislabeled “higher” reply can widen the wrong bound and lock in errors for many rounds. This is the general dilemma of adapting instructions in a changing environment: revising after every reported mistake can turn a useful rule into a persistent source of errors, while waiting leaves the agent following an outdated rule. The challenge is deciding when a mistake warrants a lasting change. Language models offer several ways to learn from feedback: revising an answer [11], retaining verbal experience across attempts [18], or optimizing prompts through textual feedback [26] and candidate search [24, 1]. In each, however, the scheduling decision is fixed: revisions fire after every attempt or after every observed error subject to a call ceiling, offering no principled way to withhold an edit when the feedback itself might be wrong. We instead ask a complementary scheduling question: when should feedback trigger a persistent instruction update? This connects prompt adaptation to concept drift [7] and continual learning [23]. Proportional-integral-derivative (PID) control [3] suggests three complementary responses: reacting to current error, accumulating evidence over time, and monitoring deterioration after an edit [4]. We translate this intuition into discrete editing gates and compare them against online TextGrad and GEPA using one LLM-based editor and structured predictors. Across our empirical study (Section 5, Table 1, Figure 2), the PID gates collectively yield a more favorable Pareto trade-off between task performance and editor-call budget than the baselines. 2 Related Work Self-evolving LLM agents and noisy evaluators. Methods that improve LLM behavior from natural-language feedback split by what they update. In the textual loop, Self-Refine [11], Reflexion [18], TextGrad [26], OPRO [24], and GEPA [1] revise prompts, chains-of-thought, or candidate instructions in response to observed errors, and skill-library agents such as Voyager [20] and SkillWeaver [28] accumulate reusable textual skills. In the parameter loop, STaR [27], Self-Rewarding Language Models [25], RLHF [13], and Constitutional AI [5] convert feedback into weight updates. Our PID gates target the textual loop, where every edit is a stochastic LLM rewrite of a persistent artifact, so the scheduling question of when to invoke the editor is especially acute. The feedback signal itself is unreliable in both regimes: LLM-as-judge suffers position, verbosity, and self-preference biases [29, 22, 14], sycophancy [17], and models cannot reliably self-correct [8]. This motivates evidence-gated editing rather than reacting to every reported error. Control theory for learning and continual adaptation. Feedback control has long informed machine learning: SGD-momentum as PI control [2], KL-weight scheduling in VAEs as PID regulation [16], constrained RL as a PID-Lagrangian [19], and deep-learning optimizers as continuous-time PID [6]. Our work instantiates PID intuition outside the parameter space of a differentiable model, on natural-language artifacts driven by a stochastic LLM editor. On the continual-learning side, catastrophic forgetting [12] and the stability-plasticity dilemma have driven decades of research [15, 21, 23, 7]. Classical parameter-space defenses (elastic weight consolidation [9], replay memory [10]) have natural prompt-space analogues in the buffered evidence and rollback rules used here. 3 Method: PID-Gated Adaptation 3.1 System overview Figure 1 lays out the five-component closed loop. At each step t, a drifting environment presents input xtx_t with hidden target yty_t; the predictor emits y^t=f(θt,xt) y_t=f( _t,x_t) under the current prompt θt _t; a noisy sensor returns an observed label y~t y_t that disagrees with yty_t at rate η; the comparator then produces an observed error ete_t (binary [y^t≠y~t]1[ y_t≠ y_t] for categorical tasks, or a bucketed distance for numeric ones) and, on mismatch, a natural-language verbalization vtv_t. A PID controller reads the histories eτ,vτ≤t\e_τ,v_τ\_τ≤ t and decides whether to invoke the LLM editor g, which returns θt+1=g(θt,vτ) _t+1=g( _t,\v_τ\). Because g is a stochastic actuator with no gradient and θ is a persistent state whose bad edits compound, the controller’s role is scheduling (when and on what evidence to call g) rather than choosing the prompt itself; the goal is to track yty_t at low editing cost while remaining robust to corruption in y~t y_t. In the guess-number game, for instance, θt _t encodes the agent’s belief about the plausible value range as integer bounds [L,U][L,U]; on each turn (xtx_t), the LLM predictor f emits a numeric guess (y^t y_t) conditioned on this belief; the environment scores it against a hidden target (yty_t) and returns high/low/correct feedback with a distance bucket (y~t y_t) that the sensor may flip; the comparator packages this into (et,vt)(e_t,v_t), e.g., “guess was too low, distance 10–19”; when the controller fires, the LLM editor g rewrites [L,U][L,U], tightening the belief for θt+1 _t+1. yty_tSensor(label flip η)≠ controllerP: reactI: accumulateD: roll backEditor g(stochastic LLM)Prompt θ(plant state)Predictor fθf_θy^t y_txtx_tyty_ty~t y_tet,vte_t,\,v_ty^t y_t Figure 1: The regulated textual feedback loop. Signals enter from outside: target yty_t from the left, input xtx_t from above. The Sensor flips the label with probability η (labels are either categorical, as in Parity, or bucketed categorical, as in Guess number / moving target). The comparator produces the observed error ete_t (binary for Parity, bucketed distance for numeric tasks); when the controller fires, editor g rewrites the prompt θ. 3.2 PID-inspired Gating We map each classical PID component onto a discrete editing gate. All three gates share the same eligibility check (unused call budget, error evidence since the last reset, no mandatory pause) and invoke the same LLM editor g, which receives the current prompt and up to K recent observed examples (including correct ones; K=8K=8 in our experiments); invalid proposals retain the prompt and every model call counts against the budget. The gate controls only when g is called, never the content of θt+1 _t+1. Proportional Gate (P). The proportional term reacts to the current error: the Proportional Gate fires whenever the observer detects an error, i.e., et=1e_t=1 in the binary case or et≥τPe_t≥ _P when the error is a non-negative bucketed distance, and eligibility holds. It is the most responsive schedule while the most vulnerable to isolated corruption in y~t y_t. at=[et≥τP] 1[eligible at t].a_t=1[e_t≥ _P]\,1[eligible at t]. (1) Integral Gate (I). The integral term accumulates evidence over time. The Integral Gate maintains a decayed error sum and resets AtA_t to zero after any attempted edit. At=αAt−1+et,at=[At≥τI] 1[eligible at t].A_t=α A_t-1+e_t, a_t=1[A_t≥ _I]\,1[eligible at t]. (2) Derivative Gate (D). The derivative term watches for post-edit deterioration. On firing, the Derivative Gate snapshots the current prompt, holds the replacement fixed for W observations, and restores the snapshot if the post-edit error mean rises by at least τD _D over the pre-edit mean. Full update order is in Appendix A. rt=[e¯post−e¯pre≥τD].r_t=1[ e_post- e_pre≥ _D]. (3) 4 Experiments 4.1 Tasks We design three simple but representative tasks to test the self-evolving agent’s capability across different output types (categorical and numeric) and drift patterns (sudden reversal, abrupt state shift, and smooth continuous drift). Together they probe how quickly the agent detects change, how it accumulates evidence under noisy feedback, and how gracefully it tracks a target that never stops moving. Hidden targets, boundaries, and initial rules are seed-randomized and never shown to the editor, so the study isolates LLM-mediated policy editing from end-to-end agent reasoning. • Parity (categorical, sudden rule flips). Inputs are integers 1–10; the hidden rule assigns Red/Blue by parity, and the mapping reverses near steps 30, 60, and 90. Feedback is a categorical color, corrupted independently with probability η. Stresses categorical rule detection under discrete reversals. • Guess number (numeric, abrupt state shifts). The agent guesses a hidden integer in [1,100][1,100] that shifts abruptly and periodically; feedback is high/low/correct paired with a distance bucket. Stresses numeric adaptation when the target state jumps. • Track moving target (numeric, smooth continuous drift). The hidden integer follows a smooth continuous trajectory (e.g., sinusoidal, rectangular, or exponential); our implementation uses a sinusoid completing ∼1.5 \!1.5 cycles over 120 steps, with no discrete change points. Stresses continuous tracking without visible boundaries a controller could anchor to. 4.2 Experimental Setup We evaluate seven policies (No Edit, Fixed Cadence, Proportional (P), Integral (I), Derivative (D), TextGrad, GEPA) on the three tasks under three feedback-corruption levels η∈0,0.1,0.2η∈\0,0.1,0.2\ with T=120T=120 steps per trajectory, using the same LLM as both predictor and editor; results are mean ± std over 5 seeds per cell. No Edit keeps the initial prompt unchanged for the full trajectory (zero edits, zero editor calls) and serves as a no-adaptation reference; Fixed Cadence fires an edit every 15 steps regardless of the error signal and serves as a data-independent control. Seeded inputs, hidden targets, and controller settings are paired across corruption levels for matched comparison. Evaluation metrics are task-specific: classification accuracy for Parity (higher is better), mean absolute error (MAE) for Guess number (lower is better), and root-mean-squared error (RMSE) for Track moving target (lower is better). Model calls are counted at every generation stage; tokens and deterministic candidate evaluations are reported separately. TextGrad and GEPA use the default parameters from their published libraries; interface and accounting details are given in Appendix B. We report primary results on Qwen3-14B (Table 1) and a scaling-up ablation on Qwen3.8-27B (Table 2). 5 Results and Analysis Figure 2: Cost-quality snapshot at η=0.1η=0.1 on Qwen3-14B (one marker per method, mean over 5 seeds). Higher accuracy and lower MAE/RMSE are better; fewer calls are cheaper. Points near the top-left (or bottom-left, for MAE/RMSE) indicate a better cost-quality trade-off. Table 1: Primary results on Qwen3-14B at η=0.1η=0.1 (10%10\% feedback corruption, 5 random seeds). Bold marks the best mean per column, excluding No Edit and Fixed Cadence. Parity Guess number Track moving target Policy Acc.(%) ↑ Calls ↓ MAE ↓ Calls ↓ RMSE ↓ Calls ↓ No Edit 47.8±5.847.8± 5.8 0.0±0.00.0± 0.0 17.61±3.0517.61± 3.05 0.0±0.00.0± 0.0 19.42±2.0519.42± 2.05 0.0±0.00.0± 0.0 Fixed Cadence 68.5±10.168.5± 10.1 7.0±0.67.0± 0.6 12.17±3.5412.17± 3.54 8.0±0.08.0± 0.0 19.70±4.1019.70± 4.10 8.0±0.08.0± 0.0 TextGrad 88.0±4.488.0± 4.4 34.8±13.134.8± 13.1 5.20±1.475.20± 1.47 105.0±21.1105.0± 21.1 8.33±1.578.33± 1.57 165.0±3.3165.0± 3.3 GEPA 79.7±6.479.7± 6.4 44.4±7.644.4± 7.6 9.18±1.559.18± 1.55 142.4±18.3142.4± 18.3 10.12±0.9110.12± 0.91 159.6±9.2159.6± 9.2 Proportional Gate (P) 90.2±3.690.2± 3.6 20.2±5.120.2± 5.1 4.70±2.764.70± 2.76 63.8±14.363.8± 14.3 7.14±1.427.14± 1.42 110.2±5.5110.2± 5.5 Integral Gate (I) 89.7±2.589.7± 2.5 10.0±3.010.0± 3.0 4.38±1.154.38± 1.15 39.2±8.039.2± 8.0 6.79±2.386.79± 2.38 53.2±1.253.2± 1.2 Derivative Gate (D) 83.3±4.983.3± 4.9 7.4±2.17.4± 2.1 10.66±2.1910.66± 2.19 18.8±1.218.8± 1.2 9.80±0.849.80± 0.84 20.0±0.020.0± 0.0 Table 2: Scaling-up ablation on Qwen3.8-27B at η=0.1η=0.1 (10%10\% feedback corruption, 5 random seeds). Bold marks the best mean per column, excluding No Edit and Fixed Cadence. Parity Guess number Track moving target Policy Acc.(%) ↑ Calls ↓ MAE ↓ Calls ↓ RMSE ↓ Calls ↓ No Edit 47.8±5.847.8± 5.8 0.0±0.00.0± 0.0 17.61±3.0517.61± 3.05 0.0±0.00.0± 0.0 19.42±2.0519.42± 2.05 0.0±0.00.0± 0.0 Fixed Cadence 61.0±15.761.0± 15.7 7.2±0.77.2± 0.7 14.59±4.4814.59± 4.48 7.8±0.47.8± 0.4 20.03±5.5020.03± 5.50 8.0±0.08.0± 0.0 TextGrad 87.3±5.987.3± 5.9 34.8±13.934.8± 13.9 3.92±1.393.92± 1.39 99.6±20.299.6± 20.2 7.03±1.227.03± 1.22 160.8±8.8160.8± 8.8 GEPA 79.7±6.479.7± 6.4 44.4±7.644.4± 7.6 8.27±1.758.27± 1.75 107.0±10.6107.0± 10.6 10.31±0.9810.31± 0.98 161.6±3.5161.6± 3.5 Proportional Gate (P) 89.2±3.789.2± 3.7 22.6±6.422.6± 6.4 3.93±1.343.93± 1.34 67.6±19.567.6± 19.5 6.45±1.206.45± 1.20 108.0±3.9108.0± 3.9 Integral Gate (I) 89.7±3.789.7± 3.7 10.0±3.710.0± 3.7 4.19±1.614.19± 1.61 39.8±10.139.8± 10.1 6.34±1.496.34± 1.49 52.4±2.052.4± 2.0 Derivative Gate (D) 82.3±4.382.3± 4.3 8.0±1.98.0± 1.9 8.85±1.278.85± 1.27 19.2±1.019.2± 1.0 10.25±2.1410.25± 2.14 19.8±0.419.8± 0.4 Table 1 summarizes the primary comparison on Qwen3-14B at η=0.1η=0.1: on every task, a PID gate attains the top quality metric among learned schedules (90.2%90.2\% accuracy on Parity for Proportional; 4.384.38 MAE on Guess number and 6.796.79 RMSE on Track for Integral), while all PID gates issue fewer LLM calls than TextGrad and GEPA. The Integral Gate occupies a particularly efficient operating point: it matches or beats TextGrad on every quality metric while using roughly 2.72.7–3.5×3.5× fewer editor calls, and the Derivative Gate uses the fewest calls on each task at a modest quality trade-off. The same cost-quality ordering replicates on a larger LLM (Table 2): on Qwen3.8-27B, the Integral Gate takes the top Parity accuracy (89.7%89.7\%) and the top Track RMSE (6.346.34) at roughly 3×3× fewer calls than TextGrad, with Guess-number MAE within 0.30.3 of the best. Absolute quality tracks editor capacity, but the Pareto ordering between PID gates and the optimizer baselines is preserved. Figure 2 visualizes the 14B snapshot: PID points sit up and to the left of the TextGrad/GEPA cluster, i.e., comparable or higher quality at substantially lower cost. This ordering is consistent across all three noise levels η∈0,0.1,0.2η∈\0,0.1,0.2\ (Appendix C, Figure 3): the PID gates occupy a better cost–quality frontier than TextGrad and GEPA under both clean and corrupted feedback. 6 Conclusion and Future Work We introduced PID-gated scheduling (Proportional, Integral, Derivative) that decides when to invoke LLM prompt editors for continual prompt adaptation under drifting tasks and noisy feedback. Across three drifting tasks, the PID gates collectively yield a more favorable Pareto trade-off between task performance and editor-call budget than the baselines considered, typically matching the strongest optimizer at a fraction of its call cost (Section 5, Tables 1–2, Figure 2). Future work includes the how-much question (edit magnitude), for which a natural extension is an “edit gain” that limits the magnitude of the change from θt _t to θt+1 _t+1 so one controller regulates both frequency and magnitude; and online auto-tuning of the gate thresholds, borrowing established methods from classical control. References [1] L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. G. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab (2026) GEPA: reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: Appendix B, §1, §2. [2] W. An, H. Wang, Q. Sun, J. Xu, Q. Dai, and L. Zhang (2018) A PID controller approach for stochastic optimization of deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. [3] K. J. Åström and T. Hägglund (1995) PID controllers: theory, design, and tuning. 2nd edition, Instrument Society of America. Cited by: §1. [4] K. J. Åström and R. M. Murray (2008) Feedback systems: an introduction for scientists and engineers. Princeton University Press. Cited by: §1. [5] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. El Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan (2022) Constitutional AI: harmlessness from AI feedback. arXiv preprint arXiv:2212.08073. Cited by: §2. [6] S. Chen, J. Liu, P. Wang, C. Xu, S. Cai, and J. Chu (2024) Accelerated optimization in deep learning with a proportional-integral-derivative controller. Nature Communications 15 (1), p. 10263. Cited by: §2. [7] J. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, and A. Bouchachia (2014) A survey on concept drift adaptation. ACM Computing Surveys 46 (4), p. 44:1–44:37. Cited by: §1, §2. [8] J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou (2024) Large language models cannot self-correct reasoning yet. In International Conference on Learning Representations (ICLR), Cited by: §2. [9] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell (2017) Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114 (13), p. 3521–3526. Cited by: §2. [10] D. Lopez-Paz and M. Ranzato (2017) Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. [11] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark (2023) Self-Refine: iterative refinement with self-feedback. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2. [12] M. McCloskey and N. J. Cohen (1989) Catastrophic interference in connectionist networks: the sequential learning problem. In Psychology of Learning and Motivation, Vol. 24, p. 109–165. Cited by: §2. [13] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. [14] A. Panickssery, S. R. Bowman, and S. Feng (2024) LLM evaluators recognize and favor their own generations. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. [15] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter (2019) Continual lifelong learning with neural networks: a review. Neural Networks 113, p. 54–71. Cited by: §2. [16] H. Shao, S. Yao, D. Sun, A. Zhang, S. Liu, D. Liu, J. Wang, and T. Abdelzaher (2020) ControlVAE: controllable variational autoencoder. In International Conference on Machine Learning (ICML), Cited by: §2. [17] M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, N. Cheng, E. Durmus, Z. Hatfield-Dodds, S. R. Johnston, S. Kravec, T. Maxwell, S. McCandlish, K. Ndousse, O. Rausch, N. Schiefer, D. Yan, M. Zhang, and E. Perez (2024) Towards understanding sycophancy in language models. In International Conference on Learning Representations (ICLR), Cited by: §2. [18] N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2. [19] A. Stooke, J. Achiam, and P. Abbeel (2020) Responsive safety in reinforcement learning by PID lagrangian methods. In International Conference on Machine Learning (ICML), Cited by: §2. [20] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar (2024) Voyager: an open-ended embodied agent with large language models. Transactions on Machine Learning Research (TMLR). Cited by: §2. [21] L. Wang, X. Zhang, H. Su, and J. Zhu (2024) A comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (8), p. 5362–5383. Cited by: §2. [22] P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, L. Kong, Q. Liu, T. Liu, and Z. Sui (2024) Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §2. [23] T. Wu, L. Luo, Y. Li, S. Pan, T. Vu, and G. Haffari (2024) Continual learning for large language models: a survey. arXiv preprint arXiv:2402.01364. Cited by: §1, §2. [24] C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen (2024) Large language models as optimizers. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. [25] W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. Weston (2024) Self-rewarding language models. In International Conference on Machine Learning (ICML), Cited by: §2. [26] M. Yuksekgonul, F. Bianchi, J. Boen, S. Liu, P. Lu, Z. Huang, C. Guestrin, and J. Zou (2025) Optimizing generative AI by backpropagating language model feedback. Nature 639, p. 609–616. External Links: Document Cited by: Appendix B, §1, §2. [27] E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman (2022) STaR: bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. [28] B. Zheng, M. Y. Fatemi, X. Jin, Z. Z. Wang, A. Gandhi, Y. Song, Y. Gu, J. Srinivasa, G. Liu, G. Neubig, and Y. Su (2025) SkillWeaver: web agents can self-improve by discovering and honing skills. arXiv preprint arXiv:2504.07079. Cited by: §2. [29] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, Cited by: §2. Appendix A Update Order and Gate Characterization Algorithm 1 PID-inspired gate editing (Proportional, Integral, Derivative). One pass over T steps; the gate G selects the trigger condition and whether the post-edit regression check is armed. The Derivative gate reuses the Integral trigger (A≥τIA≥ _I) and adds a pre/post error-mean regression check that can roll back the edit. 1: prompt θ; editor g; gate G∈P,I,DG∈\P,I,D\; decay α; thresholds τP,τI,τD _P, _I, _D (τI _I is shared by I and D); guard window W; history size K; call cap B; horizon T 2: A←0A← 0; ℋ←∅H← ; ℛ←∅R← (recent-error queue, length W); pending ←∅← (holds (θsnap,e¯pre,post-window)( _snap, e_pre,post-window) for D); cooldown←0← 0; calls←0← 0; evidence← ⊳ True once any error has been seen since the last reset 3: for t=0,…,T−1t=0,…,T-1 do 4: Predict y^t=f(θ,xt) y_t=f(θ,x_t); observe y~t y_t; set et=[y^t≠y~t]e_t=1[ y_t≠ y_t] (categorical) or a bucketed distance (numeric) 5: Append (xt,y^t,y~t,et)(x_t, y_t, y_t,e_t) to ℋH; retain the latest K entries 6: restored←Falserestored← False ⊳ set if a D-gate rollback just happened; skips accumulator update 7: if G=DG=D and pending ≠∅≠ then⊳ regression check 8: Append ete_t to pending’s post-edit window 9: if W post-edit observations collected then 10: if e¯post≥e¯pre+τD e_post≥ e_pre+ _D then 11: Restore θ←θsnapθ← _snap; A←0A← 0; evidence← ; restored←Truerestored← True 12: end if 13: pending←∅← 14: end if 15: end if 16: Append ete_t to ℛR (drop oldest if |ℛ|>W|R|>W) 17: if not restored then 18: A←αA+etA←α A+e_t; if et≥1e_t≥ 1 then evidence← 19: end if 20: if cooldown >0>0 then 21: cooldown← −1-1; continue 22: end if 23: fire←(G=P∧et≥τP)∨(G∈I,D∧A≥τI)← (G=P e_t≥ _P ) (G∈\I,D\ A≥ _I ) 24: if fire and evidence and calls <B<B then 25: if G=DG=D and |ℛ|=W|R|=W then 26: pending←(θ,e¯pre=1W∑ℛ,empty post-window)← (θ,\ e_pre= 1W ,\ empty post-window ) 27: end if 28: θ←g(θ,ℋ)θ← g(θ,H); calls← +1+1 29: A←0A← 0; evidence← ; if G=DG=D then cooldown←W← W 30: end if 31: end for Default settings (shared across gates): α=0.9α=0.9, W=5W=5, K=8K=8, B=180B=180. Gate-specific: τI=1.5 _I=1.5 (used by I and D), τD=0.15 _D=0.15 (D only), τP=1 _P=1 (P only, binary-error case). Appendix B Published Optimizers as Online Baselines TextGrad [26] and GEPA [1] use their pinned official packages, with inference routed exclusively through the common LLM-based editor. Both trigger after two observed errors since the previous invocation and require at least three remaining calls. Each restarts from the currently deployed prompt and receives at most eight past/current observations. Neither accesses clean evaluation metrics or future observations. These adaptations compare online use of the algorithms with measured resource costs, rather than reproducing their original task suites or claiming that all implementations are equivalent. TextGrad. Version 0.1.8 uses an explicit engine for each of three stages: TextLoss critiques the current structured prompt against the task specification and observed feedback; backward produces textual gradient feedback; TextualGradientDescent proposes an updated variable. Gradient memory is zero. Loss and backward stages request concise critiques of at most 100 words; the update retains the package’s required variable tags. Deterministic prediction is outside the autograd graph: the optimized variable is the structured policy prompt itself. Missing tags or an invalid structured proposal retain the deployed prompt and consume all three calls. GEPA. Version 0.1.4 starts a fresh search population from the deployed prompt, uses instance-level Pareto parent selection, and permits at most three model reflections per invocation. Accepted proposals must strictly improve on a common training minibatch before full validation. Alternating history entries form disjoint training and validation subsets when at least four observations are available, with the newest in training; smaller histories reuse the same observations in both. Minibatches contain at most three training examples. Optional merging is disabled, and perfect-score minibatches may skip reflection. Validation is past noisy evidence, not an independent estimate of future task performance. The common client counts all reflection calls. Candidate-example evaluations include rejected and malformed candidates and are logged separately. A nominal metric-evaluation limit provides an additional termination condition for skipped reflections; the official search checks it between iterations, so realized evaluations can exceed that nominal limit. Every proposal, selected parent, fitness evaluation, validation score, and final selection is retained. An invocation containing an invalid proposal may still deploy a different valid candidate; invalid-proposal count is therefore distinct from final prompt changes. Unexpected model or library failures halt the run rather than being relabeled successful no-ops. Appendix C Robustness Across Feedback-Noise Levels Figure 3 extends the primary comparison (Figure 2) by plotting all three noise levels η∈0,0.1,0.2η∈\0,0.1,0.2\ together, with marker face shade encoding noise (hollow → half-filled → solid). Across every corruption level and on every task, the PID gates sit on a better cost–quality frontier than TextGrad and GEPA: they reach comparable or higher quality at a substantially lower editor-call cost. The Integral Gate, in particular, holds its operating point across the whole noise range, matching or beating the strongest baseline at a fraction of its call count. This replicates the Pareto conclusion of Section 5 and shows that the ordering is not an artifact of the single noise level highlighted in the main text. Figure 3: Cost–quality tradeoff across all three feedback-noise levels on Qwen3-14B (mean over 5 seeds). Marker face shade encodes noise: hollow η=0η=0, half-filled η=0.1η=0.1, solid η=0.2η=0.2. Lines connect the three noise points for each method. Points near the top-left (bottom-left for MAE/RMSE) indicate a better cost–quality trade-off. PID gates sit on a better frontier than TextGrad and GEPA at every noise level. Appendix D Notation Only recurring symbols are listed. One-shot symbols keep their meaning inline. The letter g denotes the LLM editor throughout. Table 3: Recurring symbols. Symbol Meaning t,Tt,\,T Step index and horizon; T=120T=120 steps per trajectory. xt,ytx_t,\,y_t Input and hidden ground-truth target; yty_t never reaches the editor. y^t,y~t y_t,\, y_t Predictor output y^t=f(θt,xt) y_t=f( _t,x_t) and observed (possibly corrupted) feedback. ete_t Observed error: [y^t≠y~t]1[ y_t≠ y_t] for categorical tasks, or a bucketed distance for numeric ones. θt _t Current prompt (plant state); the editor’s target of change. f,gf,\,g LLM predictor y^t=f(θt,xt) y_t=f( _t,x_t) and LLM editor θt+1=g(θt,ℋ) _t+1=g( _t,H) (invoked when the gate fires). ℋ,KH,\,K Rolling editor history: up to K=8K=8 most recent observed examples. At,α,τIA_t,\,α,\, _I Decayed error accumulator At=αAt−1+etA_t=α A_t-1+e_t; decay α=0.9α=0.9; threshold τI=1.5 _I=1.5. τP,τD _P,\, _D Proportional-gate threshold (τP=1 _P=1 in the binary case); Derivative-gate restoration threshold τD=0.15 _D=0.15. W Guard window: number of observations used for the pre- and post-edit error means (W=5W=5). B Nonbinding call cap per trajectory (B=180B=180). η Feedback corruption probability (label-flip rate), η∈0,0.1,0.2η∈\0,0.1,0.2\.