Paper deep dive
Incremental Retrieval-Augmented Generation for Human-in-the-Loop Decision Support
Federico Mazzoni, Andrea Failla
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/12/2026, 6:14:13 AM
Summary
The paper introduces iRAG, a human-in-the-loop decision support system that combines Retrieval-Augmented Generation (RAG) with Hybrid Decision-Making (HDM). Unlike static RAG systems, iRAG builds its Knowledge Base (KB) incrementally from human decisions. It employs a Fading Empirical Accuracy (FEA) metric to transition between three states: Silent Observer, Skeptical Contestator, and Deferring Surrogate, allowing the system to evolve from passive observation to autonomous decision-making with human oversight. The system addresses concept drift through temporal decay in retrieval relevance.
Entities (10)
Relation Signals (9)
iRAG → uses → Hybrid Decision-Making
confidence 95% · We introduce iRAG, a human-in-the-loop decision-support system combining Retrieval-Augmented Generation (RAG) with Hybrid Decision-Making.
iRAG → uses → Retrieval-Augmented Generation
confidence 95% · We introduce iRAG, a human-in-the-loop decision-support system combining Retrieval-Augmented Generation (RAG) with Hybrid Decision-Making.
iRAG → governedby → Fading Empirical Accuracy
confidence 92% · The transitions are governed by Fading Empirical Accuracy (FEA)
Skeptical Learning → istypeof → Hybrid Decision-Making
confidence 90% · Skeptical Learning (SL) belongs to the human-driven paradigm [30, 29, 3, 22, 17].
Learning-to-Defer → istypeof → Hybrid Decision-Making
confidence 90% · The representative machine-driven paradigm is Learning-to-Defer (LtD)
iRAG → transitionsto → Skeptical Contestator
confidence 90% · iRAG can operate in three states... Skeptical Contestator (SC)
iRAG → transitionsto → Silent Observer
confidence 90% · iRAG can operate in three states... Silent Observer (SO)
iRAG → transitionsto → Deferring Surrogate
confidence 90% · iRAG can operate in three states... Deferring Surrogate (DS)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We introduce iRAG, a human-in-the-loop decision-support system combining Retrieval-Augmented Generation (RAG) with Hybrid Decision-Making. Unlike conventional RAG systems, relying on static Knowledge Bases (KBs), our proposal addresses settings where the KB is initially empty and is progressively populated through human decisions. The system stores previous decisions as retrievable precedents and moves from silent observation to evidence-grounded contestation and, finally, to autonomous handling with abstention. Against an independent baseline, iRAG matches final-decision error under 10% drift without retrieval decay and reduces it under 40% drift across all configurations. Retrieval decay improves performance on changed knowledge while increasing error on stable knowledge, evidencing a stability--plasticity trade-off rather than a general advantage. Human oversight is an architectural provision of the workflow; its usability with real human users remains to be evaluated.
Tags
Links
- Source: https://arxiv.org/abs/2610.12666v1
- Canonical: https://arxiv.org/abs/2610.12666v1
Trouble viewing inline? Open PDF directly →
Full Text
50,306 characters extracted from source content.
Expand or collapse full text
Incremental Retrieval-Augmented Generation for Human-in-the-Loop Decision Support Federico Mazzoni Affiliation: Independent Researcher Email: mazzoni.federico1@gmail.com Andrea Failla Affiliation: Institute of Information Science and Technologies (ISTI), Affiliation: National Research Council (CNR), Italy Email: andrea.failla@isti.cnr.it Abstract We introduce iRAG, a human-in-the-loop decision-support system combining Retrieval-Augmented Generation (RAG) with Hybrid Decision-Making. Unlike conventional RAG systems, relying on static Knowledge Bases (KBs), our proposal addresses settings where the KB is initially empty and is progressively populated through human decisions. The system stores previous decisions as retrievable precedents and moves from silent observation to evidence-grounded contestation and, finally, to autonomous handling with abstention. Against an independent baseline, iRAG matches final-decision error under 10% drift without retrieval decay and reduces it under 40% drift across all configurations. Retrieval decay improves performance on changed knowledge while increasing error on stable knowledge, evidencing a stability–plasticity trade-off rather than a general advantage. Human oversight is an architectural provision of the workflow; its usability with real human users remains to be evaluated. Keywords: Large Language Models, Retrieval-Augmented Generation, Hybrid Decision-Making, Skeptical Learning, Learning-to-Defer, Concept Drift 1 Introduction Large Language Models (LLMs) are trained on large text corpora to generate contextually appropriate natural language. When embedded in a larger software system to invoke tools, maintain a state, and/or act on behalf of a user, an LLM constitutes the core of an agent [26]. The behaviour of an LLM-based agent depends on its training data and, at inference time, on the context it receives: the user’s input, any externally retrieved information, and a system prompt defining the agent’s role and constraints [28]. Therefore, a frozen LLM can be adapted to different tasks without retraining by defining its context appropriately. Retrieval-Augmented Generation (RAG) exploits this property systematically. Rather than relying on knowledge encoded during training, an LLM agent retrieves relevant passages from an external Knowledge Base (KB) at query time, and includes them in its context to produce a suitable response [5], grounding LLM outputs in verifiable, domain-specific evidence. However, standard RAG formulations assume a static KB, compiled offline before deployment [10], but the knowledge could be built incrementally through the decisions, progressively populating an initially empty KB through human deliberations. The coordination of human and machine decisions over time has been extensively studied in the Machine Learning literature under the umbrella of Hybrid Decision-Making (HDM) [20], with two main paradigms: either the machine is the primary decision-maker, deferring to the human when necessary (like Learning-to-Defer, LtD) [8], or the human makes decisions, and they are assisted by a model which has learned from their past behaviour (like Skeptical Learning, SL) [30, 29]. Bridget [19] unifies the two paradigms by moving between Human-in-Command and Machine-in-Command phases. The transitions are governed by Fading Empirical Accuracy (FEA), a temporally weighted variant of the Empirical Accuracy from SL, i.e., the agreement between the human annotator and the machine. By discounting older decisions, FEA provides an estimate of the model reliability. The original paper only proposes this idea; to the best of our knowledge, neither FEA nor the phase transition has been tested empirically. Moreover, HDM systems have dealt with concept drift, which can also affect a RAG system built around an incremental, dynamic KB. When decision criteria evolve, older documents may misrepresent the current criteria, silently degrading the quality of RAG-grounded generations. A static formulation of RAG is therefore ill-suited to domains where the KB must be built incrementally, and where the currency of retrieved evidence determines decision quality. In this work, we propose iRAG, an LLM-based system bridging RAG and HDM to support human experts when the KB must be built from their own deliberations. The system is named after both its incremental nature, and the close relationship it has with its user(s). Our proposal rests on two main ideas. First, the KB is not a pre-compiled corpus, but an incremental repository of previous human decisions, structured to retrieve semantically relevant precedents for similar cases. Second, the relationship between the system and the user evolves over time. At first, unless iRAG abstains, its decisions are compared with the human ones to update the system’s FEA. Once the FEA exceeds a threshold, the system moves from a silent observer to an active interlocutor, presenting disagreements to the user (who retains the final decision) with evidence-grounded explanations retrieved from the KB. If FEA further improves, iRAG can move from a SL-based paradigm, in which it challenges the human, to a LtD-inspired one, in which it takes autonomous decisions and consults the human only when uncertain. Effectively, iRAG empirically tests the concept proposed by [19]. The contributions of this paper are as follows: • We formalise incremental deliberative RAG, i.e., RAG over a KB that grows as a direct consequence of the decisions supported by the system. • We define the system behaviour, from silent observation to SL-based contestation and, finally, to LtD-inspired autonomy. • We evaluate iRAG against an independent Controller-free RAG-with-defer rollout. Our claims are deliberately system-level: we study iRAG under controlled decision-maker and acceptance policies without real human involvement, following previous controlled evaluations with simulated decision-makers or oracle-mediated in the SL [29, 3, 17] and LtD [18, 23, 21] literatures. The rest of the paper is organised as follows. Section 2 surveys related work across RAG and HDM systems. Section 3 formalises the problem setting and the proposed architecture. Section 4 presents the experimental evaluation. Section 5 concludes with a discussion of limitations and future directions. 2 Related Work In this section, we review the literature on Hybrid Decision-Making (HDM) and Retrieval-Augmented Generation (RAG). Related topics are also discussed. HDM Systems. Hybrid Decision-Making refers to systems where final decisions result, entirely or in part, from the collaboration between a human decision-maker and a Machine Learning model [25]. HDM is especially relevant in high-stakes domains where full automation may be neither safe nor desirable. The literature identifies two broad families, corresponding to Learning-to-Abstain and Learning Together: machine-driven HDM, where the model makes most decisions and defers to the human under specific conditions, and human-driven HDM, where the human retains the primary role and the machine proposes corrections or improvements [20]. The representative machine-driven paradigm is Learning-to-Defer (LtD) [16, 18, 14], an extension of Learning-to-Reject [4, 31]. Instead of simply abstaining when uncertain, the model routes ambiguous instances to a human supervisor. A limitation of machine-driven HDM is that the human is consulted at the machine’s discretion and consequently plays a largely passive role. [2] addresses this by providing natural-language suggestions from an integrated LLM to guide the human towards a predicted answer while preserving their final say. Skeptical Learning (SL) belongs to the human-driven paradigm [30, 29, 3, 22, 17]. The human is the primary decision-maker, while the model learns incrementally from their decisions and acts as a surrogate. When its prediction differs from the human decision, it computes a skeptical score, combining epistemic uncertainty and prediction confidence. If it exceeds a threshold, the model presents its prediction as an alternative that the human can accept or reject; the surrogate is then updated using the final human decision. In this way, it can identify inconsistent or erroneous decisions and help the human remain consistent over time. All SL variants are affected by early-decision bias: the surrogate is influenced by early decisions and assumes that the user acts in good faith. SL implementations differ in how the surrogate itself is updated. [29] retrains the model from scratch after each interaction, whereas [3, 17] employ Incremental Learning (IL) [24] which updates the model with new samples rather than retraining it. Systems that learn incrementally deal with concept drift: a change in the underlying data distribution over time [15, 12]. As such, while exposed to new data, IL models need to maintain a good stability-plasticity trade-off, where stability refers to the ability to retain previous knowledge while learning something new, and the plasticity to modify existing information in light of new data [9, 13]. RAG Architecture. Retrieval-Augmented Generation (RAG) was introduced by [10] as a general architecture for knowledge-intensive NLP tasks. Instead of relying only on knowledge encoded in the parameters of a LLM model, a RAG system employs embedding similarity to retrieve passages from an external KB at inference time and includes them in the generation context. This allows the KB to be updated without retraining the model and grounds its outputs in verifiable evidence [5]. The original formulation treats the KB as static: the document index is built offline and remains fixed during training and inference [10]. Subsequent work has attempted to improve the retrieval mechanism, e.g., with reranking techniques re-ordering retrieved documents, thus improving the relevance of the passages passed to the main LLM [1, 7, 27]. For example, RePAQ [11] retrieves from a large QA-pair memory by matching the embedded input question against stored question embeddings and returning the answer associated with the nearest question. TG-RAG [6] deals with updating the KB, by introducing temporal graph structures to handle time-sensitive knowledge, enabling incremental updates as new documents arrive. These approaches share the assumption that the KB grows by ingesting external document streams, such as updated domain corpora. iRAG departs from this assumption: its KB grows from human deliberations, represented as structured records of decisions made over time. 3 System Overview Figure 1: iRAG’s three states. Transition between them is governed by Fading Empirical Accuracy. We indicate with H and M the human user and the LLM employed by the system, and with ti∈Tt_i∈ T the i-th input document from a corpus T, e.g., a support ticket. The system also employs an auxiliary model MAM_A, which assesses whether a candidate decision covers a reference decision. MAcov(tC,tR)M_A^cov(t_C,t_R) is true if the content of tRt_R is explicitly stated in or entailed by tCt_C without contradicting it or adding additional claims (wording and structure may, however, differ). Given tit_i, H produces the decision document tH,it_H,i, while M produces tM,it_M,i. If the available evidence is insufficient, M abstains by returning an empty tM,it_M,i. The final decision tF,it_F,i corresponds to either tH,it_H,i or, during the later stages of the system operation, tM,it_M,i. After each interaction, the system stores the input and its final decision directly as a question–answer record ri=(ti,tF,i)r_i=(t_i,t_F,i). The set of records forms the KB D, whose current size is denoted by m=|D|m=|D|: D=rj=(tj,tF,j)j=1m.D=\r_j=(t_j,t_F,j)\_j=1^m. (1) Retrieval is based on the input component of each record (e.g., the question, as in [11]): the stored input tjt_j is embedded through an embedding model E: DE=E(tj)j=1m,D_E=\E(t_j)\_j=1^m, (2) and each record is associated with a temporal index corresponding to its insertion order: r1r_1 is the earliest record and rmr_m the most recent. Since one question–answer record is stored for every input, m increases by one at each interaction, regardless of the system state. We distinguish m from v, the number of reliability observations collected by the system. An observation is normally recorded whenever M does not abstain and its decision can therefore be compared with the final human decision. A new negative observation is also recorded when the human flags an autonomous decision as incorrect. Consequently, m and v may evolve at different rates. To generate tM,it_M,i, iRAG first computes ei=E(ti)e_i=E(t_i) and its cosine similarity with each stored-input embedding E(tj)∈DEE(t_j)∈ D_E: ci,j=cos(ei,E(tj))∈[−1,1].c_i,j= \! (e_i,E(t_j) )∈[-1,1]. (3) We use the native positive cosine magnitude as the semantic relevance score, thus excluding negatively correlated records: si,j+=max(0,ci,j)∈[0,1].s^+_i,j= (0,c_i,j)∈[0,1]. (4) We then weight si,j+s^+_i,j by an exponential decay term based on the number of records inserted into the KB after rjr_j: Si,j=si,j+⋅λRAG(mi−j),S_i,j=s^+_i,j· _RAG^\,(m_i-j), (5) where mim_i is the size of the KB when tit_i is processed, and λRAG∈(0,1] _RAG∈(0,1] controls how quickly the retrieval relevance of older question-answer records is discounted. The decay depends on the number of records added after rjr_j, rather than on elapsed real time. To avoid retrieving recent but unrelated records, we require a positive raw cosine similarity satisfying ci,j>0c_i,j>0 and ci,j≥θsemc_i,j≥ _sem. Ranking by Si,jS_i,j, including the application of temporal decay, is performed only among records satisfying this constraint. By construction, Si,j→0S_i,j→ 0 as mi−j→∞m_i-j→∞ whenever λRAG<1 _RAG<1, making older records progressively harder to retrieve. With this soft temporal forgetting, similar older records remain available in the KB (e.g., for audit), and may still be retrieved if the only relevant source of information. iRAG thus retrieves the top-K records according to Si,jS_i,j and includes their question and final-answer components in the context of M. iRAG can operate in three states, depicted in Figure 1: Silent Observer (SO), Skeptical Contestator (SC), and Deferring Surrogate (DS). Initially, D=DE=∅D=D_E= and the system is in the SO state. The KB, its embeddings, the reliability estimate, and the current state persist across interactions. The system tracks the Fading Empirical Accuracy (FEA) of M, an HDM metric introduced in [19], here defined as FEAv(M)=∑k=1vλFEAv−kδk∑k=1vλFEAv−k.FEA_v(M)= _k=1^v _FEA^\,v-k _k _k=1^v _FEA^\,v-k. (6) Following SL literature, FEA measures agreement with final human decisions rather than objective task accuracy; treating it as evidence for autonomy assumes that final human decisions are sufficiently reliable. The coefficient λFEA∈(0,1] _FEA∈(0,1] is independent of λRAG _RAG: the former advances only when a reliability observation is recorded, whereas the latter advances at every KB insertion and periods without human validation do not alter FEA, which fades across observed evidence rather than elapsed interactions. Here, v is the number of reliability observations collected so far and δk∈0,1 _k∈\0,1\ is the value of the k-th observation. When a model decision is validated against the final human decision: δk=[MAcov(tM,tF)]. _k=1 [M_A^cov(t_M,t_F) ]. (7) When the human flags an autonomous decision as incorrect, a new observation with δk=0 _k=0 is recorded. Following SL [29, 19, 17], agreement is measured against the human’s final, possibly revised decision. By definition, FEAv(M)∈[0,1]FEA_v(M)∈[0,1], while smaller values of λFEA _FEA assign progressively less weight to older observations. In practice, FEAv(M)FEA_v(M) is maintained via two running sums rather than recomputed from scratch at each step (with A0=W0=0A_0=W_0=0): Av A_v =λFEAAv−1+δv, = _FEAA_v-1+ _v, (8) Wv W_v =λFEAWv−1+1, = _FEAW_v-1+1, FEAv(M) _v(M) =AvWv(if Wv>0, else 0), = A_vW_v (if $W_v>0$, else $0$), Silent Observer. After M generates tMt_M, the system checks whether it abstained. If tMt_M is non-empty, MAM_A compares it with tHt_H, a new observation δ is recorded, and FEA(M)FEA(M) is updated. Throughout this state, tF←tHt_F← t_H: the human decision is always taken as final, regardless of the model output, and tMt_M is not shown to H. The system transitions to SC when FEA(M)>αFEA(M)>α and v≥Nv≥ N. Skeptical Contestator. M first produces tMt_M. If it abstains, tF←tHt_F← t_H and no reliability observation is recorded. Otherwise, MAM_A assesses whether tMt_M covers tHt_H. tHt_H is provisionally taken as tFt_F. If MAcov(tM,tH)M_A^cov(t_M,t_H) holds, the system records δ=1δ=1. Otherwise, tMt_M is presented to the human as an alternative. If the human revises their decision, the resulting tFt_F replaces its provisional value. The system then updates FEA(M)FEA(M) after recording δ=[MAcov(tM,tF)]δ=1 [M_A^cov(t_M,t_F) ]. The system returns to SO if FEA(M)<βFEA(M)<β, and may transition to DS if FEA(M)>γFEA(M)>γ. We require β<α≤γβ<α≤γ, preventing the system from oscillating between SO and SC after a single disagreement and ensuring that autonomy is not granted at a lower reliability threshold than contestation. Explicit human authorisation may trigger the transition to DS independently of FEA(M)FEA(M) and N. Table 1: Comparison between Incremental-Learning-based HDM and iRAG. Aspect HDM iRAG Adapted object Predictive model External retrieval KB Base model Updated incrementally as observations arrive LLM parameters remain fixed New observation Updates model parameters, statistics, or structure Appended as an explicit input–decision record Past observations Absorbed into the model state and need not be revisited Retained explicitly and remain individually retrievable Adaptation to change Model updates, drift adaptation, or pruning Temporal weighting of retrieval relevance Dependence on history Inference depends primarily on the learned model state Naive retrieval compares against all stored records Auditability If original observations are retained separately Previous decisions remain explicit KB records Deferring Surrogate. M becomes the primary decision-maker: for each t, M produces tMt_M, and H is consulted if M abstains, in a Learning-to-Defer fashion. Otherwise, tMt_M becomes the final decision. Autonomous decisions do not update FEA(M)FEA(M), since they are not compared with a corresponding human decision. However, H may review and override an autonomous decision. If the decision is flagged as incorrect, a new observation with δ=0δ=0 is recorded and FEA(M)FEA(M) is updated. The system returns to SO if the updated FEA(M)FEA(M) is lower than β. Otherwise, it remains in DS unless the human explicitly retakes control. Like similar LtD systems, DS operates under selective feedback, with human labels observed only after deferral or review. As such, FEA may remain stale during extended autonomous operation. Periodic review is therefore required if FEA is used for continued authorization of autonomy. 3.1 Comparison with HDM systems Algorithm 1 summarises the complete interaction loop of iRAG. As mentioned, the system shares its principles with established HDM approaches, particularly Skeptical Learning, Learning-to-Defer, and the framework proposed in [19], but implements adaptation through an evolving retrieval context. Whereas HDM approaches based on retraining or Incremental Learning update the predictive model using newly observed decisions, iRAG keeps the parameters of its LLM fixed and appends each input and its final decision to an external KB. Retrieved records then inform subsequent decisions without additional model training. The use of natural-language decisions, rather than a limited set of decision labels defined a priori, also changes how agreement and deferral are implemented. iRAG employs an auxiliary model to assess whether a proposed answer covers the final human decision, making its reliability estimate dependent on the quality of this semantic assessment. Abstention serves a distinct purpose, allowing the model to defer when the evidence retrieved from the KB is insufficient. These mechanisms preserve key HDM principles while introducing dependencies on retrieval quality and auxiliary-model judgements. The distinction between iRAG and HDM systems is summarised in Table 1. Both adapt sequentially to newly observed decisions, but they differ in what is updated and in how past observations remain available to the system. Note that if a HDM system is not employing IL, the model is always re-trained from scratch for each new observation (e.g., [29]). Algorithm 1 iRAG: Incremental Retrieval-Augmented Generation Input : T; β<α≤γβ<α≤γ; N; K,θsem,λRAGK, _sem, _RAG; λFEA _FEA Output : Final decisions (tF,i)i=1|T|(t_F,i)_i=1^|T| 1 D,DE←[]D,D_E←[\,]; ←SOstate← SO; 2 v,A,W,FEA←0v,A,W,FEA← 0; 3 foreach ti∈Tt_i∈ T do 4 m←|D|m←|D|; ei←E(ti)e_i← E(t_i); 5 ←j∈1,…,m:cos(ei,DE[j])>0∧cos(ei,DE[j])≥θsemJ←\j∈\1,…,m\: (e_i,D_E[j])>0 (e_i,D_E[j])≥ _sem\; 6 ←topKj∈(D[j];Si,j)C _j (D[j];S_i,j) using Eq. (5); 7 tM←M(ti,)t_M← M(t_i,C); 8 δ←⊥δ← ; 9 if =DSstate=DS then 10 if tM=∅t_M= then 11 tF←H(ti)t_F← H(t_i); 12 else 13 tF←tMt_F← t_M; 14 if H flags tMt_M as incorrect then 15 tF←H(ti,tM)t_F← H(t_i,t_M); δ←0δ← 0; 16 else 17 tH←H(ti)t_H← H(t_i); tF←tHt_F← t_H; 18 if tM≠∅t_M≠ then 19 b←MAcov(tM,tH)b← M_A^cov(t_M,t_H); 20 if =SC∧¬bstate=SC b then 21 Present (tM,)(t_M,C) to H; 22 tF←H(ti,tM)t_F← H(t_i,t_M); 23 δ←[MAcov(tM,tF)]δ 1[M_A^cov(t_M,t_F)]; 24 else 25 δ←[b]δ 1[b]; 26 if δ≠⊥δ≠ then 27 v←v+1v← v+1; 28 A←λFEAA+δA← _FEAA+δ; 29 W←λFEAW+1W← _FEAW+1; 30 FEA←A/WFEA← A/W; 31 if =DS∧(H∨(δ=0∧FEA<β))state=DS (retake_H (δ=0 <β)) then 32 ←SOstate← SO; 33 else if ≠DS∧ℎHstate≠ DS _H then 34 ←DSstate← DS; 35 else if =SO∧v≥N∧FEA>αstate=SO v≥ N >α then 36 ←SCstate← SC; 37 else if =SC∧FEA<βstate=SC <β then 38 ←SOstate← SO; 39 else if =SC∧FEA>γstate=SC >γ then 40 ←DSstate← DS; 41 Append (ti,tF)(t_i,t_F) to D; append eie_i to DED_E; 42 tF,i←tFt_F,i← t_F; 4 Evaluation iRAG was evaluated on a synthetic benchmark simulating customer-support tickets for a CRM-like product undergoing quarterly releases, with simulated human profiles. This evaluation does not substitute for a human-participant study, which we defer to future work. The experiment separates the effects of retrieval and FEA decay on adaptation, authority allocation, and final-decision accuracy. The code with instructions to replicate the experiments is available on: https://github.com/FedericoMz/iRAG. Dataset. We created a synthetic dataset representing four quarters Q1,…,Q4Q_1,…,Q_4 of product evolution. Each quarter contains 500500 tickets across five topics (Billing, Integrations, Permissions, Reporting, and Onboarding) and tagged as easy, normal, or hard. A deterministic pipeline (available for audit in the project repo) generated the product documentation, the tickets, a gold answer consistent with the documentation of the corresponding quarter, and a perturbed answer for each human profile. The first quarter contains no drift and is shared by both dataset variants. In Q2Q_2–Q4Q_4, either 50 (10%) or 200 (40%) tickets are near-duplicates of earlier tickets whose correct answer changed after a release. These tickets form the drift subset; the others the stable subset. The two variants contain, respectively, 150150 and 600600 drift tickets, among the 2,0002,000 nominal tickets. Human Profiles. We simulate three decision-makers by perturbing the gold answers, following the methodology used in prior HDM evaluations [17, 18, 21]: the CEO, who is always correct; the Domain Expert (DE), reliable within one declared topic; and the Intern, reliable on easy tickets and systematically defaults to the previous quarter’s answer on drift tickets. Hard tickets and a randomly selected 50% of out-of-domain normal tickets are assigned to the CEO; in-domain easy and normal tickets are given to the DE; out-of-domain easy tickets are answered by the Intern; and the remaining out-of-domain normal tickets are assigned to the DE acting outside their expertise, set as Billing. The random assignment is resampled independently in each repetition. Following the original SL evaluation, where a related oracle-mediated resolution was used [29], in SC a suggestion is accepted if it covers the gold answer (as checked by an auxiliary LLM call). This is an experimental control policy, simulating cooperative users who accept a disputed proposal if judged as correct. Experimental Protocol. Quarters are processed in order to preserve drift, while tickets within each quarter are shuffled. Every run begins with a trusted bootstrap: all Q1Q_1 tickets are assigned to the CEO and iRAG is locked in SO while constructing the KB and updating FEA. Normal state transitions and informed-mixture assignment begin in Q2Q_2. The first 100 tickets of every subsequent quarter are also assigned to the CEO, and state transitions are suspended until this review window ends. If a quarter begins in DS, iRAG first returns to SC. This periodic review provides recent trusted decisions, potentially including drift tickets because the within-quarter order is random, and mitigates the early-decision bias of SL-based systems. We independently set λRAG _RAG and λFEA _FEA either to 0.998610.99861 (D) or to 11 (ND), yielding four decay combinations at each drift level. The decaying value gives retrieval a half-life of approximately one quarter (500 KB insertions) and FEA a half-life of approximately 500 reliability observations. This factorial design separates retrieval plasticity from adaptation of the reliability estimate. We set θsem=0.7 _sem=0.7, K=5K=5, (α,β,γ)=(0.70,0.55,0.80)(α,β,γ)=(0.70,0.55,0.80), and N=30N=30. Both M and MAM_A use Amazon Nova 2 Lite through Amazon Bedrock with temperature zero, chosen for reproducibility of the decision policy; sensitivity to sampling temperature is left to future work11 1 Although M and MAM_A share the same underlying model, their calls are stateless and use distinct prompts and output schemas. Across the experiments, MAM_A rejected 8,159/16,0648,159/16,064 disputed proposals (50.79%) and classified 30,102/151,11230,102/151,112 non-abstaining, gold-evaluated model answers as incorrect (19.92%). It rejected 830/1,077830/1,077 answers that exactly reproduced a dataset-labelled incorrect human response (77.07%), while accepting all 19,06219,062 exact matches to the gold answer. This audit provides no evidence of blanket preference for M’s outputs.. Questions were embedded with qwen3-embedding:4b through Ollama. Each configuration is repeated ten times using the same initial seed and independent within-quarter permutations. Figure 2: Mean trajectories across ten repetitions for all eight configurations. Rows compare coupled decay, RAG-only decay, FEA-only decay, and no decay; columns compare 10% (left) and 40% (right) drift. Shaded bands show one population standard deviation, vertical dotted lines separate quarters, and horizontal dashed lines mark the state thresholds. Each column repeats the corresponding independent, non-decaying Controller-free RAG-with-defer trajectory. Evaluation Measures. The final-decision error is the proportion of final answers not covering the gold answer. The human-only baseline is the counterfactual error obtained by always retaining the initially assigned human answer. Our Controller-free RAG-with-defer baseline is instead executed as an independent end-to-end simulation. It follows the same bootstrap and review protocol: the CEO finalizes all Q1Q_1 tickets and the first 100 tickets of each later quarter. On every remaining ticket, M uses the same retrieval and abstention rules as iRAG; a non-abstaining answer is final, whereas an abstention defers to the informed-mixture-assigned human. The baseline constructs its own KB exclusively from its own final decisions, uses non-decaying retrieval (λRAG=1 _RAG=1), and has no SO/SC/DS controller or FEA. We run it with the same ten seeds; consequently, one independent baseline is shared by all four iRAG configurations within each drift dataset. We report final-decision error separately for stable and drift tickets. The abstention rate is computed conditionally on the tickets processed in SO, SC, or DS. The SC proposal-acceptance rate is the number of accepted proposals divided by the number of explicit accept/reject decisions, excluding model abstentions and model answers already covering the human answer. Finally, model-finalized tickets is the proportion of all nominal tickets for which the autonomous model supplies the final answer. Table 2 reports the mean and sample standard deviation across repetitions. For the primary outcomes, paired 95% Student-t confidence intervals and two-sided exact sign-flip tests with Holm correction are reported below when relevant. Per-run metrics, counts, statistical tests, plots, and seeds are provided in the project repository. Table 2: Mean ± sample standard deviation across ten repetitions. Rates are percentages except for FEA; final-state entries are repetition counts. R and F denote RAG and FEA decay, respectively; R:ND is the incremental-RAG benchmark (λRAG=1 _RAG=1). Controller-free RAG uses λRAG=1 _RAG=1, so its dataset-level result spans all four columns. 10% 40% Metric R:D F:D R:D F:ND R:ND F:D R:ND F:ND R:D F:D R:D F:ND R:ND F:D R:ND F:ND Human-only baseline 11.47±0.5211.47± 0.52 17.22±0.6817.22± 0.68 Controller-free RAG-with-defer error 5.50±0.625.50± 0.62 14.03±0.7314.03± 0.73 Final-decision error 7.59±0.977.59± 0.97 6.99±1.206.99± 1.20 5.75±0.425.75± 0.42 5.52±0.415.52± 0.41 10.52±0.6810.52± 0.68 9.62±1.439.62± 1.43 10.36±0.8710.36± 0.87 9.51±0.829.51± 0.82 Abstain (SO) 15.73±0.6515.73± 0.65 15.68±0.5915.68± 0.59 14.97±0.3614.97± 0.36 15.17±0.3215.17± 0.32 15.71±0.7115.71± 0.71 15.70±0.6215.70± 0.62 14.97±0.3614.97± 0.36 15.12±0.3615.12± 0.36 Abstain (SC) 2.63±0.722.63± 0.72 2.28±0.582.28± 0.58 0.03±0.100.03± 0.10 0.02±0.080.02± 0.08 2.83±0.542.83± 0.54 2.27±0.402.27± 0.40 0.08±0.090.08± 0.09 0.02±0.070.02± 0.07 Abstain (DS) 3.36±0.633.36± 0.63 2.01±0.522.01± 0.52 0.08±0.150.08± 0.15 0.12±0.120.12± 0.12 3.01±0.863.01± 0.86 2.32±0.992.32± 0.99 0.03±0.070.03± 0.07 0.00±0.000.00± 0.00 Accepted proposals (SC) 52.39±2.2952.39± 2.29 56.09±3.6156.09± 3.61 33.95±10.4833.95± 10.48 53.73±7.6953.73± 7.69 50.39±2.8450.39± 2.84 53.26±2.9753.26± 2.97 40.23±2.7740.23± 2.77 45.55±2.6745.55± 2.67 Stable-ticket error 6.08±0.626.08± 0.62 5.51±0.975.51± 0.97 3.45±0.373.45± 0.37 3.20±0.383.20± 0.38 5.02±0.905.02± 0.90 3.94±1.363.94± 1.36 3.17±0.773.17± 0.77 2.65±0.552.65± 0.55 Drift-ticket error 26.20±5.8726.20± 5.87 25.20±5.6525.20± 5.65 34.07±2.9234.07± 2.92 34.07±2.0034.07± 2.00 23.33±1.4223.33± 1.42 22.87±1.8722.87± 1.87 27.12±1.8127.12± 1.81 25.50±1.8825.50± 1.88 Model-finalized tickets 43.34±4.3643.34± 4.36 36.45±7.3036.45± 7.30 55.93±1.6155.93± 1.61 50.70±1.8850.70± 1.88 35.80±4.6735.80± 4.67 27.42±10.7427.42± 10.74 32.09±3.5032.09± 3.50 26.12±2.9926.12± 2.99 Final FEA 0.8±0.00010.8± 0.0001 0.8±0.0020.8± 0.002 0.81±0.0040.81± 0.004 0.81±0.0050.81± 0.005 0.8±0.000080.8± 0.00008 0.8±0.000020.8± 0.00002 0.78±0.010.78± 0.01 0.79±0.0050.79± 0.005 Final state (DS/SC) 10/0 10/0 10/0 10/0 10/0 10/0 1/9 0/10 4.1 Contribution of RAG Decay Figure 2 reports the FEA and cumulative-error trajectories for the complete factorial design. During the CEO-controlled Q1Q_1, iRAG’s final-decision and human-only error remain zero by construction while FEA accumulates evidence about model behaviour. The initially wide FEA bands narrow as observations accumulate. From Q2Q_2 onward, the configurations diverge under normal assignment and state transitions; the local changes around later quarter boundaries reflect the injection of recent CEO-reviewed decisions before transitions resume. The independently rolled-out Controller-free RAG-with-defer trajectory is repeated across the four panels for the corresponding drift dataset, rather than recomputed from each iRAG trajectory. Against the principal incremental-RAG benchmark (λRAG=1 _RAG=1), retrieval decay produces the stability–plasticity trade-off. Holding FEA decay fixed, it lowers drift-ticket error by 7.877.87–8.878.87 points under 10% drift (paired 95% CIs [−11.10,−4.63][-11.10,-4.63] and [−12.16,−5.57][-12.16,-5.57]) and by 2.632.63–3.783.78 points under 40% drift ([−4.00,−1.27][-4.00,-1.27] and [−5.29,−2.28][-5.29,-2.28]). All four drift-ticket improvements remain significant after Holm correction. Retrieval decay simultaneously raises stable-ticket error by 2.312.31–2.622.62 and 1.291.29–1.851.85 points, respectively, with all four paired CIs above zero. The mechanism is consistent with the scoring rule: down-weighting older precedents reduces interference from superseded decisions, but also weakens still-valid evidence when no change has occurred. The aggregate result depends on how many tickets can benefit from this plasticity. The 10% dataset contains 150150 drift and 1,8501,850 stable tickets. Per repetition, retrieval decay therefore prevents approximately 1212–1313 drift errors but introduces approximately 4343–4949 stable errors, increasing aggregate error by 1.481.48–1.841.84 points. The 40% dataset contains 600600 drift and 1,4001,400 stable tickets: approximately 1616–2323 avoided drift errors are offset by 1818–2626 additional stable errors, leaving only 0.110.11–0.160.16 points of aggregate difference, with both CIs including zero. Thus, the smaller percentage-point improvement on individual drift tickets at 40% drift has greater system-level relevance because it applies to four times as many changed tickets. The incremental-RAG benchmark retains the lowest mean aggregate error, whereas retrieval decay consistently provides stronger resistance to concept drift. Retrieval decay is therefore not universally superior, but a controllable exchange of stability for plasticity whose utility grows with the prevalence and cost of change. 4.2 Contribution of FEA Decay FEA decay has a different, primarily operational effect. It does not directly alter which precedents are retrieved; instead, it reduces the influence of older agreement observations on the reliability estimate controlling state transitions. Holding retrieval decay fixed, FEA decay increases the model-finalized proportion by 5.245.24–6.896.89 points under 10% drift and 5.985.98–8.388.38 points under 40% drift, with every paired CI above zero. Over 2,0002,000 tickets, this corresponds to approximately 105105–138138 additional model-finalized decisions per repetition under 10% drift and 120120–168168 under 40% drift. FEA decay should therefore be interpreted primarily as a change in the timing and extent of delegation, rather than as a direct improvement to retrieval quality. The accuracy consequence depends on the environment. Under 10% drift, FEA decay changes final-decision error by only 0.240.24–0.600.60 points and both CIs include zero. Under 40% drift, it increases error by 0.850.85 points in the incremental-RAG benchmark (95% CI [0.21,1.49][0.21,1.49]) and by 0.900.90 points with retrieval decay ([0.19,1.61][0.19,1.61]), although only the former remains significant after within-contrast Holm correction. This is consistent with an authority-exposure effect: earlier or longer residence in DS makes a larger share of model errors final, even though the decay operation itself does not directly change the retrieved evidence. The final states make the threshold sensitivity visible. All 10% repetitions finish in DS. Under 40% drift, both retrieval-decay conditions finish in DS in all ten repetitions, compared with one of ten for FEA decay alone and none for the fully non-decaying benchmark. Yet the terminal mean FEA values occupy a narrow range around the predefined γ=0.80γ=0.80. Small continuous differences can therefore produce large differences in the discrete DS/SC count, and the final state alone does not reveal how long a run spent under autonomous authority. We consequently treat final-state frequencies as threshold-dependent operational outcomes and rely on continuous FEA, error, and model-finalization rates as the principal evidence. This motivates future sensitivity analysis of α, β, and γ rather than interpreting the observed state split as a threshold-independent property of the system. 4.3 Overall Contribution of iRAG Controller-free RAG-with-defer supplies a fully independent comparison. Its non-decaying KB is updated from its own decisions, so the comparison includes the downstream effect of authority allocation on later retrieval rather than replaying iRAG’s realised trajectory. It obtains 5.50±0.625.50± 0.62% final-decision error under 10% drift and 14.03±0.7314.03± 0.73% under 40% drift. Because this baseline always uses λRAG=1 _RAG=1, the same dataset-level value is compared with all four iRAG configurations; only the R:ND columns hold the retrieval-decay setting fixed. Under 10% drift, Controller-free RAG-with-defer is numerically best. The two R:ND iRAG settings obtain 5.525.52–5.755.75%, and their paired differences from the baseline are small (0.020.02–0.250.25 points) with both 95% CIs crossing zero. With retrieval decay, iRAG’s aggregate error increases to 6.996.99–7.597.59%: its drift-ticket error is lower than the baseline (25.2025.20–26.2026.20% versus 31.6731.67%), but its stable-ticket error is higher (5.515.51–6.086.08% versus 3.383.38%). Thus, at low drift, iRAG matches the simpler baseline without retrieval decay, while temporal plasticity incurs the stability cost identified above. The comparison reverses under 40% drift. Every iRAG configuration scores 9.519.51–10.5210.52%, reducing error by 3.523.52–4.534.53 points relative to Controller-free RAG-with-defer; all paired 95% CIs have lower bounds of at least 2.922.92 points, and all ten matched seeds favour iRAG in every configuration. Here iRAG improves both stable-ticket error (2.652.65–5.025.02% versus 7.847.84%) and drift-ticket error (22.8722.87–27.1227.12% versus 28.4728.47%). It also allows the model to finalize only 26.1226.12–35.8035.80% of tickets, compared with 59.9859.98% for the controller-free baseline, showing that the gain coincides with more selective autonomy rather than more model exposure. Taken together, the results bound iRAG’s contribution more precisely. The controller does not yield an unconditional advantage: under mild drift, an independent non-decaying RAG-with-defer policy is already competitive. Under stronger drift, however, iRAG’s state-dependent allocation of authority produces a substantial end-to-end improvement over both that baseline and the human-only policy. The factorial design then separates the remaining mechanisms: retrieval decay governs the stability–plasticity trade-off, while FEA decay governs the speed and extent of delegation. Per-run metrics, counts, statistical tests, plots, and seeds are provided in the project repository. 5 Conclusion We have presented iRAG, an incremental deliberative RAG system in which the knowledge base is constructed directly from the decisions supported by the system. iRAG operationalises a progressive collaboration between humans and LLMs: it first observes human decisions, then contests unsupported disagreements, and finally assumes conditional autonomy. In doing so, it combines an auditable, continuously updated RAG memory with explicit transitions between human-in-command and machine-in-command operation. The experiments provide the first empirical evaluation of FEA-driven transitions. Against independently rolled-out Controller-free RAG-with-defer, iRAG is statistically indistinguishable in the mild-drift, non-decaying setting and clearly more accurate under stronger drift. This is achieved by granting the model less final authority, establishing a synergy between the system and the human user(s) (defined as a co-evolutionary relationship in [17]). The complete factorial design separates the two forms of temporal decay: retrieval decay produces the stability–plasticity trade-off, consistently improving changed knowledge while sacrificing some stable accuracy, whereas FEA decay chiefly increases the amount of delegated authority. Under stronger drift, the retrieval benefit almost entirely offsets its stability cost and enables every retrieval-decay repetition to return to autonomous operation. iRAG therefore provides a concrete, auditable mechanism for managing adaptation and authority in an incremental RAG system while retaining human oversight, model abstention, and periodic reassessment. By design, iRAG also inherits assumptions from the HDM systems it combines. From SL, it inherits reliance on human decisions, and therefore early-decision bias (the reason for the CEO bootstrap and periodic trusted review in our experiments), as well as an empirical accuracy measured through agreement rather than objective correctness. From LtD, the Deferring Surrogate state inherits the risk of over-reliance on the machine and selective feedback during autonomous operation. From Bridget, it inherits user-specified, potentially arbitrary thresholds for state transitions, whose sensitivity is visible in the final-state results. These are explicit design assumptions inherited from the original paradigms and limit the conclusions that can be drawn from the present evaluation. Future work can mitigate them through learned or task-calibrated thresholds, broader and independent reliability checks, and adaptive human-review schedules. Moreover, as mentioned, M and MAM_A share the same underlying model family; although our audit (see fn. 1) found no evidence of blanket self-preference, a fully independent auxiliary judge would strengthen this validation further. Additionally, our experiments assign a substantial share of tickets to the oracle decision-maker; future works should vary the quantity, to establish how iRAG behaves with a fallible bootstrap expert or narrower review windows. This study establishes the feasibility of the architecture under controlled conditions and provides a reproducible basis for its human-centred validation. As the next step, we are planning a workshop with human participants to evaluate the transition and contestation mechanisms in practice and to begin the participatory design of the iRAG interface. The workshop will examine how users understand system states, inspect retrieved evidence, respond to contested decisions, and take back or delegate control. References [1] A. Asai et al. (2024) Self-rag: learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations, Vol. 2024, p. 9112–9141. Cited by: §2. [2] D. Banerjee et al. (2024) Learning to guide human decision makers with vision-language models. arXiv preprint arXiv:2403.16501. Cited by: §2. [3] A. Bontempelli et al. (2020) Learning in the wild with incremental skeptical gaussian processes. In IJCAI, p. 2886–2892. Cited by: §1, §2. [4] C. Cortes et al. (2016) Learning with rejection. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, p. 67–82. Cited by: §2. [5] Y. Gao et al. (2023) Retrieval-augmented generation for large language models: a survey. arXiv preprint arXiv:2312.10997. Cited by: §1, §2. [6] J. Han et al. (2025) RAG meets temporal graphs: time-sensitive modeling and retrieval for evolving knowledge. arXiv preprint arXiv:2510.13590. Cited by: §2. [7] Z. Jiang et al. (2023) Active retrieval augmented generation. In Proceedings of the 2023 conference on empirical methods in natural language processing, p. 7969–7992. Cited by: §2. [8] S. Joshi et al. (2021) Pre-emptive learning-to-defer for sequential medical decision-making under uncertainty. CoRR abs/2109.06312. Cited by: §1. [9] D. Kim et al. (2023) On the stability-plasticity dilemma of class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 20196–20204. Cited by: §2. [10] P. Lewis et al. (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, p. 9459–9474. Cited by: §1, §2. [11] P. Lewis et al. (2021) PAQ: 65 million probably-asked questions and what you can do with them. Transactions of the Association for Computational Linguistics 9, p. 1098–1115. Cited by: §2, §3. [12] Z. Li et al. (2020) Incremental learning imbalanced data streams with concept drift: the dynamic updated ensemble algorithm. Knowledge-Based Systems 195, p. 105694. Cited by: §2. [13] Y. Liang et al. (2023) Adaptive plasticity improvement for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 7816–7825. Cited by: §2. [14] J. Liu et al. (2022) Incorporating uncertainty in learning to defer algorithms for safe computer-aided diagnosis. Scientific reports 12 (1), p. 1762. Cited by: §2. [15] J. Lu et al. (2018) Learning under concept drift: a review. IEEE transactions on knowledge and data engineering 31 (12), p. 2346–2363. Cited by: §2. [16] D. Madras et al. (2018) Predict responsibly: improving fairness and accuracy by learning to defer. Advances in Neural Information Processing Systems 31. Cited by: §2. [17] F. Mazzoni et al. (2024) A frank system for co-evolutionary hybrid decision-making. In IDA (2), Lecture Notes in Computer Science, Vol. 14642. Cited by: §1, §2, §3, §4, §5. [18] H. Mozannar et al. (2020) Consistent estimators for learning to defer to an expert. In ICML, Proceedings of Machine Learning Research, Vol. 119, p. 7076–7087. Cited by: §1, §2, §4. [19] R. Pellungrini et al. (2025) Bridging the gap in hybrid decision-making systems. In HHAI 2025, p. 41–48. Cited by: §1, §1, §3.1, §3, §3. [20] C. Punzi et al. (2024) AI, meet human: learning paradigms for hybrid decision making systems. arXiv preprint arXiv:2402.06287. Cited by: §1, §2. [21] D. Tailor et al. (2024) Learning to defer to a population: a meta-learning approach. In International Conference on Artificial Intelligence and Statistics, p. 3475–3483. Cited by: §1, §4. [22] S. Teso et al. (2021) Interactive label cleaning with example-based explanations. In NeurIPS, p. 12966–12977. Cited by: §2. [23] R. Verma et al. (2023) Learning to defer to multiple experts: consistent surrogate losses, confidence calibration, and conformal ensembles. In International Conference on Artificial Intelligence and Statistics, p. 11415–11434. Cited by: §1. [24] L. Wang et al. (2023) A comprehensive survey of continual learning: theory, method and application. CoRR abs/2302.00487. Cited by: §2. [25] T. Wang et al. (2021) Hybrid predictive models: when an interpretable model collaborates with a black-box model. J. Mach. Learn. Res. 22, p. 137:1–137:38. Cited by: §2. [26] X. Wang et al. (2024) Beyond the limits: a survey of techniques to extend the context length in large language models. arXiv preprint arXiv:2402.02244. Cited by: §1. [27] S. Yan et al. (2024) Corrective retrieval augmented generation. Cited by: §2. [28] J. Zamfirescu-Pereira et al. (2023) Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, p. 1–21. Cited by: §1. [29] M. Zeni et al. (2019) Fixing mislabeling by human annotators leveraging conflict resolution and prior knowledge. ACM IMWUT 3 (1), p. 32:1–32:23. Cited by: §1, §1, §2, §3.1, §3, §4. [30] W. Zhang (2019) Personal context recognition via skeptical learning. In IJCAI, p. 6482–6483. Cited by: §1, §2. [31] X. Zhang et al. (2023) A survey on learning to reject. Proc. IEEE 111 (2), p. 185–215. Cited by: §2.