Paper deep dive
AgileLog: A Forkable Shared Log for Agents on Data Streams
Shreesha G. Bhat, Tony Hong, Michael Noguera, Ramnatthan Alagappan, Aishwarya Ganesan
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/18/2026, 1:36:50 AM
Summary
AgileLog is a novel shared log abstraction designed to support AI agents in streaming data systems. It introduces 'forking' primitives, specifically 'continuous forks' (cForks), to provide performance isolation, safe handling of agentic writes, and realistic sandboxing. The implementation, Bolt, utilizes a diskless architecture with hierarchical log indexing to ensure forks are cheap and performance-isolated.
Entities (5)
Relation Signals (3)
Bolt â implements â AgileLog
confidence 100% ¡ We design Bolt, an implementation of the AgileLog abstraction
AgileLog â provides â Continuous Forks
confidence 100% ¡ AgileLog offers a novel form of fork called continuous forks
AI Agents â interactwith â Streaming Data
confidence 90% ¡ AI agents use LLM reasoning to accomplish high-level tasks specified in natural language over streaming data.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In modern data-streaming systems, alongside traditional programs, a new type of entity has emerged that can interact with streaming data: AI agents. Unlike traditional programs, AI agents use LLM reasoning to accomplish high-level tasks specified in natural language over streaming data. Unfortunately, current streaming systems cannot fully support agents: they lack the fundamental mechanisms to avoid the performance interference caused by agentic tasks and to safely handle agentic writes. We argue that the shared log, the core abstraction underlying streaming data, must support creating forks of itself, and that such a forkable shared log serves as a great substrate for agents acting on streaming data. We propose AgileLog, a new shared log abstraction that provides novel forking primitives for agentic use cases. We design Bolt, an implementation of the AgileLog abstraction, that uses novel techniques to make forks cheap, and provide logical and performance isolation.
Tags
Links
- Source: https://arxiv.org/abs/2604.14590v1
- Canonical: https://arxiv.org/abs/2604.14590v1
Trouble viewing inline? Open PDF directly â
Full Text
95,451 characters extracted from source content.
Expand or collapse full text
AgileLog: A Forkable Shared Log for Agents on Data Streams Shreesha G. Bhat,Tony Hong,Michael Noguera,Ramnatthan Alagappan,Aishwarya Ganesan University of Illinois Urbana-Champaign Abstract.In modern data-streaming systems, alongside tradi- tional programs, a new type of entity has emerged that can interact with streaming data:AI agents. Unlike traditional programs, AI agents use LLM reasoning to accomplish high- level tasks specified in natural language over streaming data. Unfortunately, current streaming systems cannot fully sup- port agents: they lack the fundamental mechanisms to avoid the performance interference caused by agentic tasks and to safely handle agentic writes. We argue that theshared log, the core abstraction underlying streaming data, must support creatingforks of itself, and that such aforkable shared log serves as a great substrate for agents acting on streaming data. We propose AgileLog, a new shared log abstraction that provides novel forking primitives for agentic use cases. We design Bolt, an implementation of the AgileLog abstraction, that uses novel techniques to make forks cheap, and provide logical and performance isolation. 1 Introduction Shared logs [35, 51, 70, 86] are the fundamental building blocks that underpin data-streaming systems [8, 19, 75, 105]. Typically, upstream sources ingest data into the shared log, which orders and makes the data durable. Many downstream applications then consume the data from the shared log and process it for many purposes like real-time analytics [99], ML inference [129], search [126], and others [33, 60]. With the advances in reasoning capabilities of large lan- guage models (LLMs), a new kind of entity has emerged that can interact with streaming data:AI agents. Agents differ from âtraditionalâ entities in how they operate. The behavior of a traditional application interacting with streams is defined in advance by programmers: it follows fixed logic to produce and consume records, process the consumed data, and gener- ate outputs. In contrast, AI agents can accomplish high-level tasks on streaming data, specified in natural language. Rather than following fixed instructions, the agent uses LLM reason- ing to determine on-the-fly what sequence of steps will help achieve its goal. The agent then exercises its ability to invoke external tools [84, 111] to execute the LLM-informed steps on streams. Today, many practical streaming systems allow AI agents to read and write streaming data [14, 41, 102, 120]. Agents interacting with streaming data unlock a variety of use cases. For example, agents allow users who do not write code to run ad-hoc data-analysis tasks on streaming data specified in natural language [79, 103, 108]. Similarly, agents can use LLM intelligence for ambiguous tasks on unstructured streaming data [43â45]. Agents can also use LLM capabilities to generate contextually relevant test events and inject them into streams for testing purposes [31, 79]. Agents, however, introduce new problems for streaming systems. First, agents, unlike traditional tasks, are exploratory in nature: they explore many steps and paths to accomplish a task [59, 84]. Further, with the ease of building agents, many agents would operate on the system simultaneously [49, 84]. Combined, these two aspects cause agents to place significant load, creating performance interference for other workloads, which the system must mitigate. Second, writes performed by agents can be incorrect because LLM-informed actions can be inaccurate [78, 84, 114]; this causes correctness issues for applications consuming the data. Thus, the system must isolate and validate agentic writes before allowing them to take effect. Further, agents may explore many write paths and choose one. The system must thus safely allow such exploration. Finally, to enable coding and testing agents to build and test stream-processing applications, the system must provide sandboxes with realistic data. Unfortunately, todayâs streaming systems lack mechanisms to meet these needs. The above problems are not specific to streaming systems; agents create similar issues for any data system they inter- act with [84]. To address some of these problems, a few re- cent databases [6, 12, 48], object stores [10, 11], and lake- houses [122] provideforkingas a primitive to create isolated copies of the data system on which agents can safely operate. Inspired by these systems, we argue that,shared logs, the core storage abstraction underlying streaming data, must also support creatingisolated forks of itselfto better support AI agents operating on streaming data. With such a forkable shared log, each agent can operate on its own fork, while tra- ditional applications operate on the main shared log, enabling performance isolation. Forks also enable safe handling of agentic writes: an agentâs writes can be isolated on a fork, and then integrated after validation into the main log. Similarly, an agent can explore many write paths in parallel on different forks, isolated from each other and the main log. Finally, each fork naturally is a coding/testing sandbox with real data. Based on this, we propose AgileLog, a new shared log abstraction that offers forking as a core primitive. A fork in AgileLog is acheap,logically separate, andperformance- isolatedchild copy of the main shared log that can be read and appended (just like its parent), and in turn, itself be forked. Besides being the first forkable shared log, a key novelty in AgileLog is a new form of forks that we callcontinuous forks. The usual notion of forks in prior forkable data systems is 1 arXiv:2604.14590v1 [cs.DC] 16 Apr 2026 that the parent and the child are disconnected from each other at the fork point: while the parent and child share the state up to the fork point, after the fork, the writes on the parent are not visible to the child and vice versa. However, this is insufficient for agents acting on real-time, streaming data. Consider an agent performing streaming analytics on a fork. For the agent to keep producing up-to-date results, the fork must continue to see the real-time data from the parent log that is ingested even after the fork point. For this purpose, AgileLog offers continuous forks(or cForks). A cFork shares the history with its parent up to the fork point but, importantly,continuously inherits all further appendson the parent as well. A cFork can also be appended to, but those records remain private to the fork. cForks thus provide a uniqueunidirec- tional write isolation: records ingested into the parent are visible to the child butnotthe other way around. Thus, agents performing writes can do so on a cFork, safely isolated from the parent, and consumers of the parent will not see those records. A key property of cForks is that the records appended on a cFork are linearizably interleaved with the inherited par- ent records. This is useful for correctly interleaving agentic writes on a fork with non-agentic writes on the parent. In use cases like testing, agentic writes are synthetic and thus remain private to the cFork forever. However, in use cases where agents produce actual data, the writes need to be integrated into the main log. AgileLog thus provides a way topromotea cFork (after necessary validation) as the parent. Promote is also useful with exploration: agents can explore different write paths on different cForks and finally promote one. In addition to cForks, AgileLog also offers a (regular) severed fork, where the child and parent are disconnected after the fork point. Overall, AgileLogâs primitives help satisfy the needs of various agentic use cases over streaming data. We build Bolt, an implementation of the AgileLog abstrac- tion. Bolt must satisfy two basic requirements. First, creating forks must be cheap and quick. Second, forks must be iso- lated in performance: agentic tasks on child forks should not interfere with workloads on the parent. Since forks must be cheap, any design that copies data to create a fork is a non- starter. To avoid the copy, one could host a fork on the same set of storage servers that host the main shared log so that the fork and the parent can share the same underlying data. This design, however, will suffer from performance interference. We instead realize that the recent âdisklessâ shared-log architecture [47, 72, 133] offers a promising foundation for Bolt. Diskless shared logs usestateless brokersthat make records durable oncloud object stores(e.g., S3) instead of local disks. A fault-tolerantmetadata layerthen sequences the durable records, and maintains the log metadata: an index that maps log positions to shared-storage objects. Bolt adopts this diskless architecture to make forks cheap and performance- isolated. Specifically, Bolt creates a fork withzero-data-copy by instantiating a forkâs metadata to be the same as that of the parent log, making the fork point to the same objects on shared storage as the parent. Bolt then serves reads and appends to the fork on a separate broker; because the parent and forks are hosted on separate brokers and because cloud object stores scale well, Bolt cleanly isolates the performance of the parent and the forks. While the diskless architecture provides a good base, Bolt must address important challenges. First, even just copying metadata to create a fork can be slow. Bolt makes fork cre- ations truly low-latency by avoiding even metadata copies via a newhierarchical log indextechnique. Another major challenge is to implement cForks correctly and efficiently. A cFork must continuously inherit records from the parent and must linearizably interleave them with its own appends. This must be efficient: even with many cForks, there should be little impact on the parentâs performance. Bolt implements cForks by carefully sequencing themetadata of parentâs new recordsinto thechildâs metadata, without data copies. Bolt optimizes this idea with new techniques liketail-only updates andlazy-tail propagation. The end result is that a parent log can have many cForks with little to no performance impact. Our experiments show that Bolt can create forks quickly. Bolt also effectively isolates the performance of workloads on the parent log from agentic tasks. Boltâs techniques help maintain the parent logâs high performance even with 100s of cForks. We build three real agents powered by LLMs: an ad-hoc analytics agent for IoT data, a stream-processor testing agent, and a supply-chain-management agent. Bolt provides performance isolation and enables safe handling of agentic writes in these applications. In contrast, Kafka, a popular shared log for data-streaming, suffers from interference (e.g., 14Ăand 130Ăhigher mean and p99 latencies), and consumer failures when agents produce problematic records. Contributions.This paper makes four contributions. â˘We articulate how a forkable shared log serves as a better substrate for agents interacting with streaming data. â˘We present AgileLog, the first forkable shared log. Ag- ileLog offers a novel form of fork called continuous forks and primitives for integration and exploration of agentic writes. â˘We build Bolt, an AgileLog implementation that realizes cheap and performance-isolated forks by adopting a disk- less architecture. It uses novel techniques like hierarchical log indexes, tail-only updates, and lazy-tail propagation. ⢠We show that AgileLog benefits real agentic applications. 2 Background We describe the role of shared logs in data streaming, and explain the trend of AI agents interacting with streaming data. 2.1 Shared Logs and Data Streaming Shared logs [34â36, 51, 70, 86, 92] offer an abstraction of a durable and linearizably [67] ordered sequence of records. Shared logs expose a simple API. Clients canappendrecords, upon which the records are ordered and made durable. Clients 2 canreadrecords at given positions in the sequence. Shared logs are the core storage abstraction underlying data-streaming systems [36, 74, 75, 132, 141]. These systems support many applications like fraud monitoring [33, 95], real-time analytics [99] and search [126], and live infer- ence [83, 129]. In all these applications, upstream sources ingest data into the shared log. Downstream applications then consume the data from the shared log and process it. Shared logs have gained significant attention in research and many re- search implementations [35, 51, 70, 85, 86] exist today. Many practical shared logs exist as well. Kafka [27] is perhaps the most widely used, but others [28, 30, 61, 98] are also popular. 2.2 AI Agents Interacting with Streaming Data Recent improvement in LLMsâ reasoning abilities [65, 134] has enabled the widespread adoption ofAI agents. AI agents are systems that couple LLMsâ reasoning with the agency to take actions, such as invoking external tools, maintaining state, and coordinating with other agents [111, 138]. The ability to invoke external tools, in particular, has en- abled AI agents to interact with data systems. With this ability, AI agents today can interact with databases [48, 84, 89, 115], object stores [10, 11], lakehouses [122], and warehouses [32, 125, 128]. They can perform a variety of data tasks like re- trieval, analysis, and manipulation [38, 49, 54, 84, 97]. Adop- tion of agents for data tasks is rapidly growing; for example, a recent survey shows that many companies now use agents for data tasks and that a vast majority of databases within or- ganizations are created and operated upon by AI agents [49]. Unsurprisingly, agents have also started interacting with streaming data systems. Today, many practical systems sup- port agentic access to streaming data [14, 41, 102]. They do so via the Model Context Protocol (MCP) [26, 68]: each system provides an MCP server through which agents can read from and write to data streams [1, 13, 20, 79, 80, 107]. 3Agents on Streams: Use Cases and Problems 3.1 Real Use Cases AI agents operating on streaming data enable an array of use cases. We now present these use cases, which we compiled by analyzing real scenarios that practical systems aim to support. Agentic Data Analytics.Agents can analyze streaming data, enabling non-programmers (e.g., sales personnel) to âtalkâ to data via natural language [38, 48, 54]. Agents can construct on-the-fly dashboards, answer windowed streaming queries, and compute metrics over time [103, 108], as well as handle ad-hoc questions [79] like âwhat are the trending products in the last 24 hours?â or âshow revenue impact of the latest fea- ture deploymentâ [103]. In these scenarios, with the help of an LLM, the agent decomposes tasks and iteratively determines next steps. For example, based on LLM interaction, an agent may first probe streams to identify contents, sample records to infer schema, then read records for analysis from the target streams. Dashboards and streaming-query tasks require con- tinuously fetching new records, while point-in-time queries require access only up to a specific point in the stream. Real-Time Context Access.As many applications become agentic, they require the latest events happening in an orga- nization to get real-time context [50, 55, 71, 110] to make accurate decisions [42, 123]. For example, an agentic order- fulfillment application requires access to real-time inventory status [55]. As in the previous use case, agents may iteratively probe and read streams to construct the required context. Agentic Stream Processing.Stream processors are long- running programs that consume records from streams, act on them, and continuously write outputs to other streams. Traditionally, stream processors are hand-coded programs (e.g., Flink jobs [2] or Pulsar functions [3]). However, LLM- powered agents can also act as stream processors, particularly for tasks over unstructured, heterogeneous data. For example, an agent can act as a content-moderation processor to iden- tify inappropriate images [45]. Here, the agent would invoke the LLM on each image from the input stream and write its reasoning and decision to an output stream. Downstream con- sumers (either traditional or agentic) then act on the output stream. Similar examples of agentic stream processing for customer support [44], supply chain [56], and fraud-risk anal- ysis [43] exist as well. Unlike prior use cases, where agents only read streams, here, they both read and write to streams. Agentic Coding & Testing of Stream Applications.Instead of having agents themselves operate on streams, one could now use AI coding agents to develop (regular) applications that interact with streams [13, 80, 121]. Agents can synthesize stream processors or streaming SQL from prompts [80]. To build the application, the coding agent interacts with streams: it repeatedly reads the streams (to infer schema and under- stand the data) and iteratively fixes the code. Such iterative development existed even with hand-coded stream applica- tions [16], but the rate of such iteration will grow with agents. Agents can also be used to test streaming applications. LLMs excel at test generation [23, 39, 96] and thus agents with access to code can generate contextually relevant test cases [17]. Agentic testing is already applied to semantic testing of stream processors: the agent generates and injects test events into the stream, and checks if a stream proces- sor behaves correctly [31, 79, 106]. For example, agents can inject synthetic transactions to test if a fraud-detection model [33, 95] catches evolving fraud patterns [37, 69]. Such testing can be applied to hand-coded programs or during itera- tive agentic development. Agents also enable schema testing, where events with new schemas are injected to check if proces- sors can handle them. Further, agents can also perform âwhat- ifâ counterfactual testing [84]. With LLM aid [82], agents can do what-if tests on real-time data by simulating and inject- ing different event sequences and observing how downstream outputs change. Testing agents both read and write streams, but the written data is synthetic. 3 3.2 Agents on Streams: Problems Need for Performance Isolation.On the surface, AI agents may appear to be yet another set of clients interacting with streaming data. However, agents have distinguishing char- acteristics than usual applications. As identified by recent work [59, 84], agentic tasks are exploratory and speculative in nature: agents generate multiple courses of action to accom- plish a task. This exploration may occur within a single agent or via an orchestrator spawning sub-agents, each exploring a distinct hypothesis or path. In either case, these paths can be partial attempts at the task, probes to understand the data, suboptimal solutions, or validation of prior steps. While such exploration helps improve the overall accuracy of the agent, it leads to a large number of requests being issued, potentially with suboptimal access patterns [117, 140], which places sig- nificant load on the underlying system [62, 63, 84]. Given the ease of building and deploying agents, many agents will operate on a system simultaneously, multiplying this load. This will negatively interfere with traditional production ap- plications, especially latency-critical ones. However, current streaming systems do not provide mechanisms to isolate the performance of traditional workloads from agents. Need for Safely Handling Agentic Writes.Allowing agents to directly perform writes to a data system can be risky [78, 84, 114, 115]. This is because LLMs can generate inaccurate actions, causing agents to write incorrect data to streams. This, in turn, can lead to incorrect downstream computations based on those writes. Thus, the system must isolate agentic writes from the original stream and validate them (e.g., by a human in the loop or a âreviewerâ agent [115, 116]). Only after such validation, the agentic writes must take effect on the stream and consumers must be able to see them. However, current streaming systems do not provide mechanisms to do this. Further, an agent may explore many write paths either be- cause of its inherent exploratory nature or because it uses sub- agents to explicitly explore different paths. Take the content moderation example. Here, many sub-agents, each powered by a different LLM [64], may explore different ways to ana- lyze the input and produce the output. It must thus be possible to explore these different paths in an isolated manner. Even- tually, when a path is chosen as the desired one (e.g., based on result accuracy), it must be possible for the writes in that path to become part of the actual stream. However, todayâs streaming systems do not offer such support for exploration. Need forRealisticSandboxes.Coding and testing agents re- quire sandboxes. Testing agents require injecting test events, which must remain isolated from the production stream. To- day, agents, for this purpose, create separate synthetic streams. While this provides isolation, testing is done purely over syn- thetic data. However, practitioners often desire testing using real data [15, 18, 57, 124]. With streaming data, testing is most effective when executed against realistic production data which carries inherent temporal context that helps uncover corner cases. For instance, in the fraud-detector testing use case, the agent must be able to interleave the generated fraudu- lent test-events within a realistic transaction stream. Similarly, coding agents can produce more accurate code with realis- tic data. However, today, creating such isolated yet realistic sandboxes with production streaming data is not possible. 4 A Forkable Shared Log Abstraction To better support agents operating on streaming data, we argue that the shared log, the core storage abstraction underlying streaming data, must support creatingisolated forks of itself. AgileLog is a shared log abstraction that offers such forking capability. A fork in AgileLog is acheap,logically separate, performance-isolatedchild copy of the main shared log. With AgileLog, each agent can operate on its own fork, while traditional workloads operate on the main or root log. This provides performance isolation for workloads on the main log. AgileLog also enables safe handling of agentic writes. Writes performed by an agent on a fork are isolated from the root log, which can be validated before allowing them to take effect on the root log. Further, AgileLog enables simultaneous exploration of different write paths on separate forks. Finally, each fork acts as a sandbox with real data. This section first describes the AgileLog abstraction (§4.1) and then how AgileLog satisfies the needs of use cases from §3.1 (§4.2). The next section (§5) describes the implemen- tation of the AgileLog abstraction that enables forks to be created cheaply and ensures logical and performance isola- tion. 4.1AgileLog Abstraction, Interface, and Fork Semantics An AgileLog instance has a root log, which can be forked to create child copies. Each child can be appended and read (just like its parent), and can itself be forked. Figure 1 shows AgileLogâs interface. AgileLog augments the append and read calls with two new calls for forking: cFork and sFork; these return another AgileLog instance, which can be further forked to create deeper forks. Promote enables writes on a fork to take effect on its parent and squash deletes a fork. Continuous Forks.Typically, a fork means that the parent and the child will be disconnected from each other at the fork point, and data modifications that happen either on the parent or the child will remain private to them. Forks provided by prior databases [4, 5, 12, 48] offer this regular fork semantics. With streaming systems, live data keeps flowing in and agents need to operate on the live stream. Thus, it is insuffi- cient to provide the regular fork, where the child stops seeing new data ingested on the parent after the fork point. Take the real-time streaming query or dashboards examples from the agentic data-analytics use case. To keep producing up-to-date results, the fork where the agent runs must continue to see the appends on the parenteven after the fork point. Thus, Ag- ileLog supports a novel form of fork that we callcontinuous forksor cFork. A cFork shares the history with the parent 4 interface AgileLog: // Traditional shared-log API calls. Position append(Record r); List<Record> read(Position from, Position to); // Continuous fork; indicate if this fork is promotable. AgileLog cFork(promotable = false); // Severed fork; optionally can fork from a past offset. AgileLog sFork(optional Position past); bool promote(); // Promote this (promotable) cFork. void squash(); // Delete this fork. Figure 1:AgileLog Interface. log up to the fork point, but it alsocontinuously inheritsnew appends at the parent after the fork point. While appends in the parent are visible to children, children may want to insert their own records into their forks. Take the testing use case; here, the agent simulates fraud scenarios by injecting fraudulent transactions. By operating on a cFork, the agent can see the original transaction stream and also inject fraud events. However, these events must remain private to the child. cFork provides this required isolation: writes on children forks remain private to them and consumers of the parent will not see them. Thus, cForks enable a unique form of isolation that we callunidirectional write isolation: writes on the parent are visible to the children, but not the other way around. This is in contrast to regular forks provided by prior data systems, where the isolation is bidirectional. Since a cFork inherits records from its parent and also has its own records, how should these records be ordered on the cFork? AgileLog guarantees that appends on a cFork are linearizably [67] interleaved with those on the parent. That is, if an appendAfinishes on the parent and thenBis appended to a cFork, thenBis guaranteed to appear afterAin the cFork. In some use cases, appends on a cFork remain private to it forever (e.g., a cFork created for testing). However, in some cases, the agentic writes must be reflected on the main log; to enable this, AgileLog allows a cFork to be promoted as the parent (as we discuss below). If such promotion is desired, then the cFork must be created with thepromotableflag set. Figure 2(a) and (b) show cFork semantics. In 2(a), the cFork acts as a live read-only copy of the parent. In 2(b), in addition, records are inserted in the live copy (e.g.,Zon green fork) that are linearizably interleaved with the parent records. Severed Forks.AgileLog also offers regular forks via a sFork call, where the child is âseveredâ from the parent after the fork point. By default, a severed fork is created from the current log tail of the parent (at the instant sFork is called); however, it can also be created from a past offset of the parent. In either case, a severed fork shares the parentâs history up to the fork point and it stops seeing further parent updates. Some use cases may not perform writes on a severed fork as shown in 2(c), where the fork acts as a read-only snapshot of the parent. Some use cases write to a severed fork as shown in 2(d). The new records on the fork could be entirely unrelated to the parentâs data (like the yellow fork in 2(d)); or, they could be some perturbations of the parent values (like the green fork). (c) severed forks w/o writes P Q Dâ Eâ fork (b) continuous forks with writes ... A B A BC ... (a) continuous forks w/o writes A BC DE Z ... A BC ... DE A BC ... DE A BC ... DE A BC ... DE A BC ... D ... A B A B Câ ... (d) severed forks with writes ... A BC ... DE R streaming queries, dashboards, real-time context, coding agents testing agents, agentic stream processing point-in-time queries, analytics over past events testing agent: what-if counterfactual testing root fork root ... D E root validated promote ... agentic stream processing ... yellow fork chosen DE X D EX ... DE ... root DXE D YE root ... (e) promoting a continuous fork ... DXE root root fork fork fork fork fork fork A BC ... DE X (i) (i) agentic stream processing that explores multiple writes paths Figure 2:AgileLog Primitives.The yellow (upward) and green (down- ward) arrows are the fork points at which the yellow and green forks were created from the (purple) root log. Writes on the forks are shown in a darker shade. Gray boxes show use cases where a particular primitive is useful. Squash.Squash deletes a fork and no further operations can be performed on it. For example, it can used to dispose unused sandboxes. AgileLog disallows squash on the root log. Promote.To support use cases where agentic writes must be reflected on the parent, AgileLog offers a promote API. A promote call on a cFork (created with thepromotableflag) of a parent makes that fork become the parent itself. Future oper- ations that refer to the parent will now essentially occur on the promoted child and ones that refer to the child are disallowed. sForks or non-promotable cForks cannot be promoted. cFork and promote together enable safe integration of agen- tic writes. First, agentic writes can be isolated by performing them on a cFork. The cFork can then be validated and pro- moted. Since AgileLog guarantees linearizable interleaving of agentic writes on the cFork with the parent logâs records, after promotion, the parent log will have the records in the correct order. If validation fails, the cFork is squashed without any problems to the main log or consumers operating on it. One option instead of using a cFork would be to have the agents write to a temporary log, validate the writes there, and then append to the main log. However, this allows only non- contextual, stateless checks of the current agentic writes (e.g., if the schema of the records is correct). In many cases, how- ever, validation requires state: the current agentic writes must be validatedalong withthe records in the main log before the fork and other records on the main log after the fork (ingested by non-agentic producers). cForks and promote enable such stateful validation. Specifically, the cFork will contain the previous records and the writes from non-agentic producers linearizably interleaved with the current agentic writes. Thus, the cFork has the required temporal context for validation. Validation can be done, for instance, by running a copy of the stateful consumer on the cFork to check if it produces desired 5 results. If so, the cFork is promoted; otherwise it is squashed. Even if only stateless validation is required, cFork and pro- mote provide a neat API to validate and reflect agentic writes on the main log, instead of having to manage temporary logs. Figure 2(e)(i) shows how cFork and promote safely handle agentic writes. The agent performs its writes on the yellow fork, which is promoted as the parent after validation. Creating a promotable cFork introduces some restrictions on its parent. These restrictions arise because, after promotion, the parent will have additional records: ones that were inserted into the cFork. Thus, allowing reads on the parent beyond the fork point (before a promotion) could make those reads invalid (after a promotion). For this reason, AgileLog does not allow reads on the parent beyond a promotable cForkâs fork point until the fork is promoted or squashed. For ex- ample, in 2(e)(i), consumers on the parent cannot read en- triesDand beyond. However, once the cFork is promoted, those entries can be read. Further, while appends to the par- ent can continue, AgileLog cannot return indexes for those appends beyond the promotable cForkâs fork point. This is because, after promotion, the indexes would change (e.g., in 2(e)(i),Eâs position in root will change after promotion). Not returning indexes works in practice as many applications anyway do not use them [86]. Lastly, AgileLog also stops operations beyond the promotable cForkâs fork point on other non-promotable cForks of the parent until a decision is made. Note that these restrictions apply only when there are active promotable cForks. cFork and promote also enable agentic exploration with writes. The agent can simultaneously explore different paths on different promotable cForks and finally promote one. With many promotable cForks, only the first promote will succeed and AgileLog internally squashes others. As shown in 2(e)(i), the agent explores two paths and promotes the yellow fork. 4.2 How Different Agentic Use Cases use AgileLog Agentic Data Analytics. Agents for streaming queries and dashboards can see the most up-to-date data by operating on a cFork of the main log as shown in Figure 2(a). Agents for ad-hoc queries donât need the latest data and so can work on a sFork (2(c)). By running these exploratory tasks on forks, the root logâs performance would remain unaffected. Real-Time Context Access. Agents that require fresh real- time context can obtain it via a continuous fork (2(a)). Agentic Stream Processing.Agentic stream processors (e.g., for content moderation) use cFork and promote to safely in- tegrate agentic writes as shown in Figure 2(e)(i). The agent can periodically create and write to a new cFork, validate and promote it, and repeat. If the agent explores many paths (e.g., content-moderation with different models), it can explore each path on a cFork and promote one (2(e)(i)). Coding and Testing Agents.Coding agents can operate on a (read-only) cFork (2(b)). A testing agents can operate on a non-promotable cFork, where it can safely inject test events linearizably interleaved with the real data (2(b)). It can explore many test cases in parallel on different cForks. For what-if testing, the agent can create many sForks to simulate alternate worlds (2(d)). The yellow fork in 2(d) explores a completely different event sequence after the fork point. The green fork, in contrast, explores values based off of the parentâs data (e.g., sensor values that are off by 5% in an IoT stream). To do this, the agent creates a cFork, reads from it, perturbs the read values, and writes them into an sFork. 5 Bolt Design Bolt is an implementation of the AgileLog abstraction. Bolt aims to satisfy a few high-level requirements. First, creating a fork must be acheapandlow-latencyoperation. Second, a fork must beisolated in performancefrom the parent; this ensures that resource-intensive agentic workloads on forks do not degrade the performance of workloads on the parent log. Finally, Bolt mustefficiently scaleto many forks. 5.1Design Rationale, Challenges, Techniques Overview One option to realize forks is to use mirroring tools [29, 53, 81, 109] that can continuously mirror a shared log. To create a fork, one can use these tools to mirror the parent logâs data to the child log. Typically, these tools mirror data to a separate cluster with dedicated storage and compute. Thus, forks would be isolated in performance from the parent. However, this approach requires copying data, which makes creating and maintaining forks prohibitively slow and expensive. Another option is to build forks atop traditional shared-log designs [27, 36, 51, 86, 104] that use a set of replicated storage servers (or brokers) that store data on local disks. Here, a forked child can be hosted on the same servers as the parent log and thus can share the underlying data on disk without copying. However, this would violate our performance-isolation requirement due to resource contention at the servers. Diskless as a Substrate.We instead realize that the recent disklessarchitecture in shared logs [22, 72, 88, 133] provides a good substrate for Bolt. In contrast to traditional designs that replicate data on the local disks on a set of brokers, disk- less shared logs make brokers stateless and delegate storage to scalable shared object storage (e.g., AWS S3 [119], Azure Blobs [90]). The shared-storage layer provides the required data durability. A fault-tolerant metadata layer sequences durable records â and maintains an index to map log positions to shared-storage objects. The diskless architecture is increas- ingly popular for its two benefits [130, 139]. First, it avoids inter-fault-domain replication traffic between brokers, which is expensive in clouds. Second, it frees operators from man- aging disks and helps elastically scale the system. Bolt adopts the diskless architecture to make forks zero- data-copy and quick, and isolate performance (§5.3). At a high level, Bolt creates a fork by having the forked childâs â While some shared logs [51, 86] also separate durability and ordering, they arenât diskless: their storage servers use local disks, not scalable cloud stores. 6 metadata point to the same objects on shared storage as the parent, without copying data. Further, Bolt serves operations to the created fork on a separate broker from that of its par- ent, avoiding interference to the parent. Since cloud object stores can scalably serve many brokers without becoming a bottleneck [25], Bolt provides performance isolation between workloads on the main log and the agents on forks. Challenges.While diskless architectures provide a good foun- dation, Bolt must solve some critical challenges. First, how to make fork creations truly low-latency? Copying even the metadata can incur high latency. Second, how to implement cForks that continuously inherit new records ingested on the parent while linearizably interleaving appends on the child with the ones on the parent? Third, even with many cForks, how to inherit updates with little to no impact on parentâs performance? Finally, how to realize promotes? Techniques Overview.Bolt addresses these challenges via various techniques. Bolt avoids metadata copy using ahierar- chical log indexthat enables swift fork creations (§5.4). Bolt realizes cForks as a metadata-layer operation (without data copies) by sequencing the metadata of new records on the par- ent into the childâs metadata (§5.5). To scale to many cForks, Bolt proposestail updatesandlazy-tail propagation(§5.5.1, §5.5.2). Finally, Bolt also realizes promotes as a metadata- layer operation without data copies (§5.6). 5.2 Diskless Shared Logs: A Primer Before describing Bolt, we provide a primer on diskless de- signs. Figure 3 shows a typical diskless shared log [47, 72, 88, 133]. The system consists of stateless brokers, a metadata layer, and cloud object storage. A single diskless instance can hostmultiple logs(a unit of total order), each with a unique identifier. The metadata layer is a fault-tolerant group that implements state-machine replication (SMR) [112] using Paxos [77] or Raft [93]. It maintains anindexandtailfor each log. The index maps log positions to locations of records in the object store; the tail is the next free position in the log. Appends.Appends to a log can be routed to any broker (step a1 in Figure 3). The broker batches several records from many clients (a2) and writes a large object to the object store (a3). Note that an object can contain records from different logs. Once the object write completes, the broker informs the meta- data layer of the object identifier (O) and the metadata for each record withinO(i.e., the log identifier and byte ranges within O) (a4). The metadata layer then sequences these records by logging this metadata to its consensus log and executing them as SMR operations. Specifically, for each record inOfor a logL, it assigns a position inLstarting atLâs current tail, and updatesLâs index and tail (a5). It returns these positions (a6), and the broker acknowledges the clients (a7). Reads.Reads to a log can be routed to any broker (r1). The broker contacts the metadata layer (r2), which then looks up the index to fetch the object identifier and byte range for the requested position (r3, r4). The broker uses this metadata to Broker (a1) L.append O start (s)end (e) (a4) (O, (..., (L, s, e), ...)) (a2) (a3) put(O) 42 L.tail = 43 Metadata Layer (a5) index & tail update (a6) (..., 42 , ...) Object Store (a7) pos 42 client client (r1) L.read(42) (r2) L.lookup(42) (r4) (O, s, e) (r3) index lookup for 42 (r5) get(O) (r7) (r6)O[s, e] L.index: (O, s, e) record meta pos batch Broker Figure 3:Diskless Primer.The figure shows how appends and reads work (for entries in log L). OnlyLâs metadata is shown in the metadata layer. retrieve the record from the store (r5, r6) and return it (r7). 5.3 Forks in Bolt: Overview We now describe how Bolt implements the basic forking functionality atop the diskless architecture. Creating a Fork.Conceptually, Bolt treats a fork as an in- dependent log with its own total order and metadata. Upon a fork of a parent logP, Boltâs metadata layer creates a new child logCand initializesCâs metadata to be the same as that ofP. Thus,Câs metadata points to the same data objects on the shared storage asP, resulting in zero-data-copy fork creation. Concretely, when an agentic client needs a fork to operate on, it issues a fork call to any broker which gets routed to the metadata layer. The fork operation is sequenced in the con- sensus log of the metadata layer as an SMR operation. When the operation executes, the metadata layer creates a new child logC, and initializesCâs index and tail to be the same asPâs index and tail at this instant. §5.4 discusses how Bolt does this initial fork creation efficiently to reduce latencies. To track the continuous-inheritance relationships between a cFork and its parent, Bolt maintains aninheritance forest (a collection of trees) at the metadata layer. Initially, there could be a number ofrootlogs; these are the base logs. For each such root log, Bolt maintains aninheritance tree, which captures the continuous forks starting from that root. When a cForkCis created from a root logP, a new tree node is created forCas a child ofP. If subsequent cForks are created fromC, those will be added as children ofCin that inheritance tree. Since a severed fork does not continuously inherit updates from its parent, when a severed forkSis created, a new tree is created withSas the root. If cForks are created from this severed fork, those forks will be added as children of this root. Bolt finally assigns a broker distinct from those hosting the root logs to serve the created fork to avoid performance interference. Bolt maintains a small pool of brokers to avoid spinning up new ones upon fork creation. The log identifier of the created fork, i.e.,C, is returned to the client. Clients can invoke future operations on the fork using this identifier. After a Fork.After a fork is created, appends and reads to the forkCare handled by the separate broker. At the metadata 7 layer, appends toCare sequenced atCâs current tail and added toCâs index and remains logically isolated from its parent. Reads at the metadata layer are handled by readingCâs index. For severed forks, any append on the parent or the child gets sequenced only within that corresponding log index, logically isolated from each other. This is because severed forks pro- vide bidirectional write isolation. However, to support cFork, appends to a log must not only be sequenced and recorded within the index for that log, but for all logs present in its inheritance subtree. §5.5 describes how Bolt achieves this. 5.4 Zero-Metadata-Copy Fork Creation To initialize a forkâs metadata, one could copy the parentâs metadata. While this is faster than copying data, copying even the metadata incurs high latency and memory overhead in the metadata layer. Further, when the metadata copy happens, metadata operations on the parentâs index would have to wait behind it, impacting the performance of appends on the parent. Bolt thus avoids physically copying even the metadata to create a fork. Bolt can do so because indexes of the fork and the parent up to the fork point would never change subse- quently as logs are fundamentallyappend-only. Thus, Bolt just makes the childâs index point to the parentâs index for po- sitions up to the fork point, making forks zero-metadata-copy. To realize this, Bolt uses a data structure that we callhi- erarchical log index(or HLI). With HLI, upon a fork, Bolt initializes the childCâs index as an empty map, butCâs tail is initialized as the parentPâs tail (or as the specified offset+1if a past offset is given when creating a sFork). Future appends toCwill be sequenced atCâs tail. During reads, index lookups for positions lower than the smallest position inCâs index are looked up inP. IfPis also a fork created from another log Q, it will result in a recursive lookup inQâs index. In this way, HLI avoids metadata duplication for log prefixes shared between parents and their children when creating forks. 5.5 Supporting Continuous Forks We now explain how Bolt implements cForks, where the child must see new records appended to the parent even after the fork point. Bolt realizes that continuous inheritance can be implemented as ametadata-layer-only operation, without any data copies. Specifically, it is sufficient to propagate parentâs metadata updates to the childâs metadata. This metadata prop- agation indicates the presence of new records on the parent to the child. Since operations on the child also flow through the metadata layer, those operations will see the effect of the new parent records, helping establish a linearizable order across records on the child and those in the parent. BoltNaiveCF: Naive Metadata-based cFork.One approach to realize cFork is that whenever the metadata layer sequences a record in logP, it synchronously updates the indexes and tails of each descendantDâD P , whereD P is the set of all descendants inPâs inheritance subtree. Thus, any further ap- pend toDwill be linearized afterPâs append. We refer to the (iv) G.index.lookup(1) â return g1.meta 1 posrec metaloc.cnt r0.meta1 (i) R.index.lookup(2) (i) subtract local count 1 from 2 (i) recurse and lookup 1 in G posrec meta 0 loc.cnt g0.meta1 1 g1.meta 2 Green (parent) Red (cFork of G) R.index G.index posrec meta 1 r0.meta g1.meta R.index (i) R.index.lookup(2) â return g1.meta (a) BoltNaiveCF (b) Bolt posrec meta 0g0.meta 1g1.meta G.index G.tail=3 R.tail=5 G.tail=3 R.tail=5 g0 r0 g2 g0 0123 positions â 3r1.meta 2 4 r1 g1 g1 2g2.meta3 2g2.meta g2 3 r1.meta 2 g2.meta 4 Figure 4:Continuous Inheritance.Gis cForked to getR(blue line is the fork point). (a) and (b) show how BoltNaiveCF and Bolt implement cForks, respectively. Lookup of position 2 in R is also shown (red boxes). version of Bolt that realizes cForks in this manner as Bolt- NaiveCF. Figure 4(a) shows how BoltNaiveCF works. Here, logGis cFork-ed to createRwhenGâs tail was 1. With zero- metadata-copy fork creation, onlyRâs tail to set to 1, without copying the index entries toR. After the fork, recordr0is inserted inR. Later, recordg1is inserted inG. This results in insertingg1âs index entry into bothGandRâs index. Thus, whenr1is appended toR, it gets correctly linearized afterg1. Although records inGappear at different positions inR, by referring to the same objects on the shared storage, continuous inheritance can be realized without any data copies. BoltNaiveCF, unfortunately, has two drawbacks. First, when an index entrymis sequenced into a logP, it insertsm into the index of everyDâD P . This exacerbates the memory overhead in the metadata layer (nĂoverhead withnforks). Second, updating the index of all descendants in the critical path would impact the parentâs append latency and through- put. This can be especially detrimental when agents create many cForks of a root log (e.g., during agentic exploration). 5.5.1 Continuous Inheritance via Tail Updates To alleviate the above problems, Bolt makes a key observation. Upon the append of a record to a parent, the effect of the append can be reflected on the descendants byupdating only their log tailsandnot their log indexes. Concretely, upon an append ofrat anyP, Bolt inserts record metadata forr only intoPâs index. However, instead of also doing so for all DâD P , Bolt only updates their tails. This tail update ensures that any future append to a descendantDâD P will reflect the presence ofr, thereby achieving the effect of linearizably insertingrintoD. Bolt thus completely avoids any metadata duplication, i.e., each recordrâs metadata is only inserted into one log index, the one whererwas originally appended to. Bolt realizes the tail-update technique by augmenting HLI. Figure 4(b) shows how the augmented HLI avoids duplicating metadata even when continuously inheriting records. After the fork and insertion ofr0inR, when recordg1is inserted in G, Bolt avoids copyingg1âs index entry intoRâs index, but in- stead only updatesRâs tail to 3, reflectingg1âs presence. Thus, 8 whenr1is appended toR, it gets correctly linearized afterg1 at position 3 inR;Râs tail is now 4.g2is then appended toG and again onlyRâs tail is updated (to 5). Lookups. With tail updates, when looking up a position in child, if the corresponding record was inherited, then it wonât have an entry in the childâs index (e.g., lookup for position 2 inRâs index in 4(b) sinceg1was inherited). Despite this, Bolt must be able to retrieve the metadata for such positions. At a high level, the metadata for inherited records must be looked up in the parentâs index. However, the challenge is that an inherited record may appear at different positions in the parent and children. In 4(b),g1is at position 2 inRbut at 1 in G. How should Bolt determine to which position in the parent does the requested position in the child map to? Observe that g1âs position inG(i.e., 1) can be obtained by subtracting from g1âs position inR(i.e., 2), the number oflocally appended recordstoRprior tog1(1 record). Thus, to lookup inherited records, HLI augments the entry for a positionxin a logLâs index with another field: a cumulative count of records locally appended toLup to and includingx. The lookup works as follows. If a requested positioniis in a logLâs index, then Bolt simply returns that metadata entry. If not, Bolt retrieves the locally appended entries in Lbeforei: it finds the largest position smaller thaniinLâs index and gets the corresponding local countl. It then looks upiâlinLâs parent, recursing further ifiâlis also an inherited position inLâs parent. For example, in 4(b), position 2 is not inRâs index. Thus, Bolt finds the largest position <2inR, i.e., the 1 st record. By subtracting local count of 1 from the requested position 2, Bolt looks upRâs parentGâs index at position 1 which correctly returnsg1âs metadata. In comparison, BoltNaiveCF (4(a)), performs a single lookup on Râs full index, but does so at the cost of metadata duplication. 5.5.2 Lazy Tail Propagation Even with tail updates,eagerlyiterating over each descendant and updating its tail in the critical path of an append to the parent would affect the performance of the parent. Instead, Bolt uses alazymechanism. Here, Bolt updates a descendant Dâs tail only when there is an operation onDthat requires querying its current tail (i.e., an append toDor a read to positions beyond the currently reflected tail inD). Efficient Tail Updates.Bolt realizes tail updates and lazy propagation efficiently even with wide or deep forks using a data structure that we call Lazy Tail Tree (LTT). Logically, LTT is an inheritance tree of tails, i.e., each log with its cur- rent tail is a node, with edges between parents and inherited children. But, physically, LTT is an Euler Tour of this logical tree stored in a balanced binary search tree (BST) [66]. An Euler TourEof a treeTenumerates the nodes in the order they would be seen during a depth first traversal ofT[136]. Euler tours have a neat property: every subtree rooted at a node ofTappears as a contiguous range withinE. Thus, a subtree update inTconverts to a range update on the BST. Bolt then implements range updates lazily on the BST via standard lazy-propagation techniques in logarithmic time [87]. Similarly, point queries for tails on the BST take logarithmic time. Thus, forks could be arbitrarily deep or wide, but the complexity remains logarithmic in the number of logs. 5.6 Promotes and Squash Promote.In Bolt, promoting a (promotable) cFork can change the parent log beyond the fork point. Thus, parent reads cannot be allowed beyond it. Similarly, no appends or reads beyond this point can be allowed on any non-promotable descen- dant. To achieve this, Bolt maintains the earliest such position earliest-fp(fork point of the earliest promotable cFork that still exists) for each log. Both direct log reads and recur- sive reads from non-promotable descendants that go beyond earliest-fpare blocked. To block appends on non-promotable descendants, Bolt augments the lazy tail tree with the ability to block or unblock a subtree through the same lazy-propagation techniques as earlier. Further, since promotes can change the positions on the parent log, Bolt stops returning the append indexes afterearliest-fp. Bolt implements promotes via metadata copies, without any data copy. To promote a child forked at positionfp, metadata entries from the child afterfpare copied over to replace en- tries beyondfpin the parent. Unlike fork creation, where the history will be long and thus metadata copy can be slow, copy- ing metadata on a promote is a reasonable design choice since only metadata entries after the fork point must be copied. Squash.Squashing a log clears the metadata-layer state for it and all its children recursively. Apart from deallocating the indexes, Bolt also removes the corresponding nodes from the lazy tail tree. Lastly, squashing a promotable cFork also updates the parentâsearliest-fp, and unblocks it if the squashed log is the only existing promotable cFork of its parent. 5.7 Implementation We implement Bolt in C++ (~21K LOC). We use eRPC [73] for communication and MinIO [91] as the shared-storage layer, but any S3-compatible store works. The metadata layer uses a Raft implementation [137]. We equip brokers with a local object cache to improve reads. While Bolt hosts forks on separate brokers from those of their parents for perfor- mance isolation, it co-locates many forks of a parent on the same broker for two reasons. First, it enables cache reuse (e.g., across many analytics agents). Second, grouping many forks to a broker reduces the overhead on the metadata layer. That said, applications have the option to inform Bolt (via a configuration) if a fork must be hosted on a separate broker. 6 Evaluation Our evaluation answers the following questions: ⢠Does Bolt enable low-latency fork creation? (§6.1) â˘Does Bolt isolate the performance of workloads on root logs from those running on their forks? (§6.2) 9 10 â1 10 0 10 1 10 2 10 3 10 4 10 5 10 6 10 â1 10 0 10 1 10 2 10 3 10 4 10 5 10 6 10 7 Fork Latency (us) #Log Entries BoltMetaCpy Bolt Figure 5:Fork Latency. 0 4 8 12 16 20 â2 0 2 Throughput (KOps) Time relative to fork start (s) BoltMetaCpy Bolt Forks Begin Figure 6:Parent Throughput during Fork Creation. 0 0.5 1 1.5 2 2.5 3 KafkaBolt E2E Latency Norm. (ms) w/o analyticsw/ analytics (a) Isolation from Reads 0 5 10 15 20 25 30 35 avgp99 E2E Latency (ms) w/o cForkw/ cFork (b) Bolt: Isolation from Appends Figure 7:Performance Isolation with Bolt. ⢠Can Bolt support many cForks without impacting the per- formance of workloads running on the parents? (§6.3) ⢠How do the techniques to implement cForks help? (§6.4) ⢠What is the metadata memory overhead in Bolt? (§6.5) ⢠What is the overhead of HLIâs recursive lookups? (§6.6) ⢠Are promotes of cForks fast in Bolt? (§6.7) ⢠Does Bolt benefit real agentic applications? (§6.8) Setup.We experiment on an xl170 [7] CloudLab [52] cluster. We use nine nodes for MinIO and three replicas in the meta- data layer. Each broker runs on its own node. We use 4KB records. Since no existing shared log offers forks, we mostly compare Bolt to its many variants. In some experiments, we compare to Kafka, a popular shared log for data-streaming. 6.1 Fork Creation Latencies We first measure fork latency. We compare Bolt against Bolt- MetaCpy, a variant that creates forks by copying the metadata of the parent to instantiate the child index. Figure 5 shows fork latency with varying lengths of the parent log. In both systems, the created forks are assigned to a pre-existing bro- ker different than that of the parent. By avoiding any data or metadata copy, Bolt has low latencies (~50us), regardless of the log length. Although BoltMetaCpy avoids data copies, it incurs high latency to copy metadata (e.g., 100ms with 25M entries, three orders of magnitude slower than Bolt). Copying metadata not only hurts fork latency but also par- entâs performance. This is because record appends to the par- ent must sequence the metadata into the parentâs index, which is affected by the copy to create the fork. Figure 6 shows this: during the creation of 100 forks (like it happens at the start of an agentic exploration), with BoltMetaCpy, the parentâs throughput suffers when forks are being created; in contrast, with Bolt, the parentâs performance remains unaffected. 6.2 Performance Isolation We next show that Bolt provides performance isolation for a latency-critical (lc) workload from a read-heavy workload resembling agentic data analysis. The lc-workload appends and reads records at the tail. The analysis workload reads records in bulk from the same log (up to an offset). In Bolt, the analysis task creates and runs on asForkthat uses its own broker but shares the parentâs data via shared storage. Fig- ure 7(a) shows the end-to-end (e2e) latency of the lc-workload with and without the analysis task running alongside it. In Bolt, the lc-workload experiences no interference from the 0 5 10 15 20 25 0 20 40 60 80 100 120 140 Latency (ms) Throughput (KOps) 0 cForks 10 cForks100 cForks (a) 1 Root Log 0 5 10 15 20 25 0 20 40 60 80 100 120 140 Latency (ms) Throughput (KOps) (b) 32 Root Logs Figure 8:Performance Impact on Parent with many cForks. analysis task. We also run this experiment with Kafka. Since Kafka has no notion of forks, the analysis task must share data by running on the same brokers as the lc-workload. Thus, the lc-workloadâs latency degrades by 2.5Ădue to resource contention at the broker. We note that this behavior is not specific to Kafka but any shared log with stateful brokers or storage servers that store data on local disks [36, 51, 86]. We next show that the lc-workload is performance-isolated even with appends to the forks. We do not compare to Kafka, since it has no notion of forks that allow appends that are logically isolated from the root log. The lc-workload appends to the root at 13KOps/s. We create a cFork that also appends its own records at 13KOps/s. Figure 7(b) shows lc-workloadâs e2e latency with and without the cFork. Even with appends on the fork, the e2e latency (mean and p99) are unaffected. 6.3 Performance of Parent with Many cForks We now examine if the root log remains performance-isolated even when it has many cForks. We run an append-only work- load on the root with 0, 10, and 100 cForks, and plot the root logâs append throughput vs. latency. As shown in Figure 8(a), even with many cForks which must continuously inherit ap- pends from the root, the root log shows no performance degra- dation relative to the case with no forks. Figure 8(b) shows the case where the system hosts 32 root logs (mimicking real deployments where many topic-partitions are hosted on a diskless instance). We then create 0, 10, and 100 cForks per root log. Even with many root logs and each having many forks, root logsâ performance does not degrade in Bolt. 6.4 cFork Techniques Ablation We now evaluate the different techniques that Bolt uses in the metadata layer to implement cForks. We compare Bolt with (i) BoltNaiveCF, which copies metadata entries from the parentâs index to the child indexes on every parent append; (i) 10 0 20 40 60 80 100 120 0 10 20 30 40 50 60 Latency (us) Parent Metadata Tput (KOps) BoltNaiveCF BoltETBolt (a) 10 cForks 0 200 400 600 800 1000 0 10 20 30 40 50 60 Latency (us) Parent Metadata Tput (KOps) (b) 100 cForks 10 â1 10 0 10 1 10 2 10 3 10 4 10 5 0 10 20 30 40 50 60 Latency (us) Parent Metadata Tput (KOps) (c) 1000 cForks Figure 9:cFork Techniques Ablation. 0 5 10 15 20 0 1 2 3 4 5 6 7 Read Latency (us) cFork Nesting Depth BoltNaiveCFBolt Figure 10:Metadata Lookup vs. cFork Depth. 10 â1 10 0 10 1 10 2 10 3 10 4 10 5 10 6 10 â1 10 0 10 1 10 2 10 3 10 4 Promote Latency (us) #Promoted Records Bolt DC (Kafka) Figure 11:Promote La- tency. Bolt-ET (eager-tail), which improves over BoltNaiveCF by updating only the tails but does so eagerly and without the lazy tail tree (LTT). The difference in these variants emerge only when the data-layer throughput is high enough to stress the metadata layer. However, producing this metadata load would require more brokers and MinIO storage nodes than we have machines. Thus, we measure the metadata-layer throughput in isolation. We emulate two sets of brokers sending requests to the metadata layer: one set of brokers that host the root log which append records to the log; another set that hosts the cForks that perform reads on the cForks. Figure 9 shows the root logâs metadata performance with 10, 100, and 1000 cForks. First, BoltNaiveCF performs well with 10 cForks, but its approach of copying the index en- tries to all children becomes a bottleneck with 100 and 1000 forks, resulting in low metadata-layer performance. Once the metadata layer bottlenecks, the overall systemâs performance would not scale further. Second, Bolt-ETâs tail-only updates help it closely match the performance of Bolt for 10 and 100 forks. However, with 1000 forks, eagerly updating even just the tail becomes the bottleneck. In contrast, Boltâs metadata layer continues to scale well even with 1000 forks. This is because, unlike Bolt-ET, Bolt updates tails lazily (only upon a tail read on the cFork) and efficiently (via LTT). 6.5 Metadata Memory Overhead We now examine the metadata memory overhead of Bolt and BoltNaiveCF. We measure the memory occupied to support 1000 cForks of a root log to which 1M records are appended. BoltNaiveCF requires 4.4GB to maintain the cForks. In con- trast, Bolt avoids metadata duplication, requiring only 8MB. Boltâs techniques thus are not only critical for high metadata- layer performance but also to reduce memory overhead. 6.6 Recursive Metadata Lookup Overhead With HLI augmented with tail updates, Bolt may require re- cursive lookups to find the metadata for a given log position (Figure 4(b)). We measure this overhead and compare it to BoltNaiveCF that doesnât require recursive lookups. We cre- ate nested cForks with 1M records per level and query the deepest cFork, forcing recursion to the root. Figure 10 plots the metadata lookup latency against nesting depth. Bolt has minimal degradation: only 5.2% slower at depth 7 (â¤2Îźs overhead). Since this occurs in the metadata layer, which con- stitutes only a small fraction of the client-perceived latencies (which are inms), the end impact is negligible. 6.7 Performance of Promote We now evaluate promotes. As figure 11 shows, metadata- copy-based promote incurs only 10-100s ofÎźs, even with many records. In contrast, a temporary-log-based approach that copies data to promote (described in §4.1) with Kafka (âDC (Kafka)â), is much slower than Bolt. Thus, apart from the functionality limitations (i.e., no stateful validation, no linearizable interleaving of agentic and non-agentic records) the data-copy-based approach also suffers from high latencies. 6.8 Evaluation with Real Agentic Applications We demonstrate Boltâs benefits for real agentic tasks by build- ing three agentic applications: an ad-hoc analytics agent for IoT sensor data, a stream processor testing agent, and an supply-chain restocking agent. Together, these applications exercise all AgileLog interface calls. All applications follow a single-agent architecture, and are built atop OpenCode [94] and use the Gemini-2.5-Pro LLM [40] for reasoning. Analytics Agent (sFork).We build an agent to perform ad- hoc analysis of an IoT stream that has records with different metrics (temperature, humidity) and sensor status. The agent is given an open-ended task: âlook for anomalies in the first 1M recordsâ. The agent is provided a tool to read records from the shared log and a SQL tool [101] to analyze the read data. The agent can issue multiple parallel tool calls, each investigating a different hypothesis. We pre-populate 1M records in the stream with some anomalous values. The agent runs alongside a latency-critical (lc) workload that monitors newly ingested sensor readings. With Bolt, the agent creates a severed fork and reads records from the sFork. In Kafka, the agent directly queries the data stream. We find that the agent is able to form complex hypotheses, discover injected spikes, and correlate them with metrics. For determinism, we capture one run of the agent and replay the trace on both systems. Figure 12(a) and (b) shows the lc-workloadâs e2e latencies. The shaded regions are the periods where the agent reads the stream, with annotations (nQ) indicatingnparallel in- vestigations; the non-shaded regions are the thinking phases, where the agent interacts with the LLM to evaluate results and plan next steps. With Kafka, the lc-workload suffers from latency spikes during query execution due to contention. In Bolt, lc-workload experiences no interference. Figure 12(c) shows the latencies during the two phases. In Kafka, in the query-execution phase, the lc-workload experiences 14Ăand 130Ăhigher mean and p99 latencies, respectively, compared 11 0.1 1 10 100 1000 0 50 100 150 200 E2E Latency (ms) Time (s) Kafka Query ExecAgent Think 8Q3Q9Q5Q (a) Kafka Latency over Time 0.1 1 10 100 1000 0 50 100 150 200 250 E2E Latency (ms) Time (s) Bolt Query ExecAgent Think 8Q3Q9Q5Q (b) Bolt Latency over Time 10 -1 10 0 10 1 10 2 10 3 10 4 avgp99 Bolt avgp99 Kafka E2E Latency (ms) Agent ThinkQuery Exec 1.0x 1.0x 14.0x 130.8x (c) Mean and Tail Latency Figure 12:Ad-hoc Analytics Agent: Performance Isolation for a Latency-Critical Workload 0.1 1 10 100 1000 0 10 20 30 40 50 60 70 E2E Latency (ms) Time (s) Bolt Tests w/cFAgent Think 6T6T6T Figure 13:Real-time E2E La- tency w/ Testing Agent. 0 5 10 15 20 25 30 35 40 45 50 55 60 Throughput (KOps) Time (s) Kafka Append 0 5 10 15 20 25 25 30 35 40 45 50 55 60 Throughput (KOps) Time (s) Bolt ForkPromoteSquash Figure 14:Supply-Chain Agent: Consumer Throughput. to the thinking phase. Bolt has no interference. Stream Processor Testing Agent (non-promotable cFork). We build a stream processor that performs aggregates in tum- bling windows [100] of 5ms. We task an agent with testing this stream processor under corner cases such as late, mal- formed, and duplicate records, while running alongside a real- time workload. The agent is provided a tool that can create non-promotable cForks of the root log, inject agent-generated test records into them, run the stream processor on the cFork, squash the cFork, and output execution traces (errors/failures). The agent can invoke this tool in parallel to run multiple tests concurrently. We note that this agent successfully triggers many buggy scenarios. Figure 13 plots e2e latency of the real- time workload with agentic tests running alongside. Shaded regions are test-execution phases (with annotation showing number of parallel tests); non-shaded regions are thinking phases. Despite tests running alongside, the real-time work- load experiences no latency degradation. This shows that cForks provide a sandboxed environment with real data but with no impact to the workloads on the root log. Supply-Chain Agent (cFork + Promote/Squash).We build a supply-chain agent [56]. The supply-chain stream contains order events for items and restock events that replenish supply. The order events are ingested by non-agentic producers. The agent evaluates historical trends to pro-actively issue restock events for items that are expected to be in demand. This stream is processed by a non-agentic downstream application. In Bolt, the agent is provided a tool that creates apromotable cFork, adds restock events into it, and validates it by running a stateful copy of the downstream application on it, before promoting or squashing the cFork. In Kafka, the agent directly writes restock events to the main stream. To simulate an agent mistake, we inject schema error into an agent-created event. Figure 14 plots the downstream processorâs throughput. In Kafka, an agent mistake crashes the consumer application. Bolt avoids any crashes through the promotable cFork, which helps safely validate agentic writes before incorporating it into the main stream. Since Bolt prevents reads from the main stream when a promotable cFork exists, there are throughput dips. However, once the cFork is either promoted or squashed, it unblocks the downstream consumer, allowing it to quickly catchup. Optionally, once the agent is thoroughly vetted, it can be allowed to directly write to main stream and avoid the throughput dips. Promotable cForks provide a safe approach to quickly deploy agents that write to production streams without worrying about production application failures. 7 Related Work Shared Logs.Prior shared logs change the interface for virtualization [34], low-latency [36, 86], and flexible order- ing [58, 85]. However, none provide the forking capability. While FuzzyLog [85]implicitlyallows forked histories during network partitions, forking is not a first-class primitive and it is not designed for agentic use. To our knowledge, AgileLog is the first shared log abstraction with forking as a core primitive to support agents. Like diskless designs, prior shared logs like Scalog [51] and Boki [70] also separate data and metadata. However, their storage servers are stateful and utilize local disks, which makes them prone to storage-layer contention like Kafka (§6.2). Bolt avoids this via its diskless design. Fork as a Core Primitive.Beginning with OS process forks, forking as a first-class operation has been well explored in many contexts [24, 76, 127, 135]. Prior storage systems have also proposed forking (or sometimes referred to as branching) primitives [21, 46]. More recently, to support AI agents, a few databases [6, 48, 84], object stores [10, 11], lakehouses [122], and file-systems [131] have realized the need for and adopted forking/branching as a first-class primitive. However, no shared log today provides forks. Further, unlike these sys- tems, AgileLog provides a novel form of continuous forks. Fork vs. Branch.Branching, unlike forking, is desired when divergent paths are intended to be merged [118]. A few databases such as Dolt [9] offer mergeable branches, but most do not [113]. Tardis [46] is an earlier system that uses a branch-on-conflict mechanism with merge capability for weakly-consistent systems. AgileLogâs promote is a restricted merge where updates from a single forked child are effectively merged into the parent. A more general notion of merge that can merge updates from multiple forks into the parent would not preserve linearizable ordering of appends. Further, our motivating use cases did not require such a capability. 12 8 Conclusion AgileLog is a new shared log that offers forking as a core primitive, enabling agents to operate over forks of a shared- log-based stream in a safe and isolated manner. As AI agents increasingly operate over data systems, it is imperative to build systems abstractions that enable safe agent interactions without interference to traditional applications. Our work takes a step in this direction for streaming data systems. References [1]A Model Context Protocol (MCP) server for Apache Kafka.https://github.com/tuannvm/kafka-mcp- server. [2] Apache Flink: Stateful Computations over Data Streams.https://flink.apache.org/. Apache Soft- ware Foundation. [3] Apache Pulsar Functions Overview.https://pulsar. apache.org/docs/next/functions-overview/. Apache Software Foundation. [4] Branching in Supabase to Test and Preview Changes. https://supabase.com/docs/guides/deployment/ branching. [5]Branching Workflows to Test Queries in Neon.https: //neon.com/docs/guides/branching-test-queries. [6]Build Versioning / Checkpoints for your Agent.https: //neon.com/branching/branching-for-agents. [7]Cloudlab Hardware.https://docs.cloudlab.us/ hardware.html. [8]Confluent, Inc.https://en.wikipedia.org/wiki/ Confluent. Accessed: 2025-08-28. [9]Dolt is Git for Data!https://github.com/dolthub/ dolt. [10] FluidStorage:Forkable, Ephemeral, Durable Infrastructure for the Age of Agents.https: //w.tigerdata.com/blog/fluid-storage- forkable-ephemeral-durable-infrastructure- age-of-agents. [11] Fork Buckets Like You Fork Code | Tigris Object Storage.https://w.tigrisdata.com/blog/fork- buckets-like-code/. [12]Forks of a Database with Dolt.https://docs.dolthub. com/concepts/dolthub/forks. [13] IntroducingtheStreamNativeMCPServer: ConnectingStreamingDatatoAIAgents. https://streamnative.io/blog/introducing-the- streamnative-mcp-server-connecting-streaming- data-to-ai-agents. [14] Lenses IO.https://lenses.io/. [15]Marble is Shipping Faster with Neon Branch- ing.https://neon.com/blog/marble-is-shipping- faster-with-neon-branching. [16] Netflix Data Mesh: Streaming SQL in Data Mesh.https://netflixtechblog.com/streaming- sql-in-data-mesh-0d83f5a00d08. [17] NVIDIAHephaestus:BuildingAIAgents toAutomateSoftwareTest CaseCreation. https://developer.nvidia.com/blog/building- ai-agents-to-automate-software-test-case- creation/. [18]ProductionTestingwithDarkCanaries. https://w.linkedin.com/blog/engineering/ infrastructure/production-testing-with-dark- canaries. [19] StreamNative, Inc.https://streamnative.io/. [20]pulsar-mcp-server: A Model Context Protocol Server for Apache Pulsar.https://github.com/nodece/ pulsar-mcp-server, 2025. [21]Marcos Kawazoe Aguilera, Susan Spence, and Alis- tair C Veitch. Olive: Distributed Point-in-Time Branch- ing Storage for Real Systems. InProceedings of the 3rd Symposium on Networked Systems Design and Im- plementation (NSDI â06), San Jose, CA, May 2006. [22] Aiven. Aiven Inkless: Diskless Topics for Apache Kafka.https://aiven.io/inkless. [23]Nadia Alshahwan, Jubin Chheda, Anastasia Finogen- ova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. Automated Unit Test improvement using Large Language Models at Meta. InCompanion Proceed- ings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024. [24]Chloe Alverti, Stratos Psomadakis, Burak Ocalan, Shashwat Jaiswal, Tianyin Xu, and Josep Torrellas. CXLfork: Fast Remote Fork over CXL Fabrics. InPro- ceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS â25, 2025. [25]Amazon Web Services, Inc.Best practices design patterns: optimizing Amazon S3 perfor- mance.https://docs.aws.amazon.com/AmazonS3/ latest/userguide/optimizing-performance.html. [26] AnthropicandCommunityContributors. Model Context Protocol Specification.https: //modelcontextprotocol.io/specification. 13 [27] Apache. Kakfa.http://kafka.apache.org/. [28] Apache. Pulsar.https://pulsar.apache.org/. [29]Apache Software Foundation. Kafka mirroring (mir- rormaker).https://cwiki.apache.org/confluence/ pages/viewpage.action?pageId=27846330, 2017. [30]AWS. Amazon Kinesis.https://aws.amazon.com/ kinesis/. [31] Ahmed Azraq and Moises Dominguez Garcia. Build- ing an Event-Driven Agentic AI System with Apache Kafka on Confluent Cloud and watsonx Orches- trate.https://developer.ibm.com/tutorials/event- driven-agentic-ai-system-confluent-watsonx- orchestrate/, 2026. [32] Nooshin Bahador. Transparent, Evaluable, and Acces- sible Data Agents: A Proof-of-Concept Framework. arXiv preprint arXiv:2509.24127, 2025. [33]Gourav Singh Bais. How to detect fraudulent clicks in a real-time ad system.https://w.redpanda.com/ blog/detect-fraudulent-clicks-real-time-ads. [34]Mahesh Balakrishnan, Jason Flinn, Chen Shen, Mihir Dharamshi, Ahmed Jafri, Xiao Shi, Santosh Ghosh, Hazem Hassan, Aaryaman Sagar, Rhed Shi, et al. Vir- tual Consensus in Delos. InProceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI â20), Banff, Canada, November 2020. [35] Mahesh Balakrishnan, Dahlia Malkhi, Vijayan Prab- hakaran, Ted Wobber, Michael Wei, and John D. Davis. CORFU: A Shared Log Design for Flash Clusters. In Proceedings of the 9th Symposium on Networked Sys- tems Design and Implementation (NSDI â12), San Jose, CA, April 2012. [36] Shreesha G. Bhat, Tony Hong, Xuhao Luo, Jiyu Hu, Aishwarya Ganesan, and Ramnatthan Alagappan. Low End-to-End Latency atop a Speculative Shared Log with Fix-Ante Ordering. InProceedings of the 19th USENIX Conference on Operating Systems Design and Implementation (OSDI â25), Boston, MA, July 2025. [37]Caf.io.How Often Should Fraud Detection RulesbeReviewed toMaintainAccuracy? https://w.caf.io/post/how-often-should- fraud-detection-rules-be-reviewed-to-maintain- accuracy. [38] Calliope. AI-Powered Data Exploration on SQL, and NoSQL DBs.https://calliope.ai/. [39]Arda Celik and Qusay H. Mahmoud. A review of large language models for automated test case generation. Machine Learning and Knowledge Extraction, 7(3), 2025. [40]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Mar- cel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reason- ing, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025. [41]Confluent. Confluent Streaming Agents.https://w. confluent.io/product/streaming-agents/. [42] Confluent, Inc. The Confluent Real-Time Context En- gine: Power AI Agents and Apps With Instant Intel- ligence.https://w.confluent.io/press-release/ real-time-context-engine/, 2025. [43]Confluent, Inc.Streaming agents examples and tutorials with confluent cloud.https: //docs.confluent.io/cloud/current/ai/streaming- agents/streaming-agents-examples.html# financial-fraud-detection, 2026. [44] Confluent, Inc.Streaming agents examples withconfluentcloudâcustomersupport agent.https://docs.confluent.io/cloud/ current/ai/streaming-agents/streaming-agents- examples.html#customer-support-agent, 2026. [45]Confluent, Inc.Streaming Agents Examples â Step 1: Create Content Analysis Agents.https: //docs.confluent.io/cloud/current/ai/streaming- agents/streaming-agents-examples.html#step-1- create-content-analysis-agents, 2026. [46] Natacha Crooks, Youer Pu, Nancy Estrada, Trinabh Gupta, Lorenzo Alvisi, and Allen Clement. Tardis: A branch-and-merge approach to weak consistency. In Proceedings of the 2016 International Conference on Management of Data, SIGMOD â16, page 1615â1628, New York, NY, USA, 2016. Association for Computing Machinery. [47]Yashwanth Dasari. New in Confluent Cloud: Tableflow, Freight Clusters, Apache Flink AI Enhancements, and More.https://w.confluent.io/blog/2025-q1- confluent-cloud-launch/#freight-clusters-q1 , 2025. [48]Tiger Data. Native Search and Retrieval on Databases with Tiger Data.https://w.tigerdata.com/blog/ postgres-for-agents. 14 [49]Databricks.State of AI Agents 2026.https: //w.databricks.com/sites/default/files/2026- 01/State-of-AI-Agents-2026-Final.pdf, 2026. [50] DeltaStream, Inc.Two Paths to Context: When GenAI Agents Need a Real-Time Context En- gine like DeltaStream â and When They Donât. https://w.deltastream.io/blog/two-paths-to- context-when-genai-agents-need-a-real-time- context-engine-like-deltastream-and-when-they- dont/, 2025. [51] Cong Ding, David Chu, Evan Zhao, Xiang Li, Lorenzo Alvisi, and Robbert Van Renesse. Scalog: Seamless Reconfiguration and Total Order in a Scalable Shared Log. InProceedings of the 17th Symposium on Net- worked Systems Design and Implementation (NSDI â20), Santa Clara, CA, February 2020. [52] Dmitry Duplyakin, Robert Ricci, Aleksander Mar- icq, Gary Wong, Jonathon Duerig, Eric Eide, Leigh Stoller, Mike Hibler, David Johnson, Kirk Webb, Aditya Akella, Kuangching Wang, Glenn Ricart, Larry Landweber, Chip Elliott, Michael Zink, Emmanuel Cecchet, Snigdhaswin Kar, and Prabodh Mishra. The design and operation of CloudLab. InProceedings of the USENIX Annual Technical Conference (ATC), pages 1â14, July 2019. [53]Uber Engineering.uReplicator: Improvement of apache kafka mirrormaker.https://github.com/ uber/uReplicator, 2016. [54]Fabi. Fabi: AI Data Analysts.https://w.fabi.ai/ product/analyst-agent. [55] Sean Falconer. Introducing Real-Time Context Engine: Simplified Context Engineering With Real-Time, Pro- cessed Data for AI.https://w.confluent.io/blog/ introducing-real-time-context-engine-ai/. [56]Matthew Finio and Amanda Downie. AI Agents in Supply Chain.https://w.ibm.com/think/topics/ ai-agents-supply-chain, 2024. [57]Flagsmith. How to Test Safely in Production with Feature Flags?https://w.flagsmith.com/blog/ testing-in-production. [58]Dimitra Giantsidi, Emmanouil Giortamis, Nathaniel Tornow, Florin Dinu, and Pramod Bhatotia. FlexLog: A Shared Log for Stateful Serverless Computing. In Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Comput- ing, pages 195â209, 2023. [59] Ioana Giurgiu and Michael E. Nidd. Supporting Dy- namic Agentic Workloads: How Data and Agents In- teract.arXiv preprint arXiv:2512.09548, 2025. [60]Ken Goodhope, Joel Koshy, Jay Kreps, Neha Narkhede, Richard Park, Jun Rao, and Victor Yang Ye. Building LinkedInâs Real-time Activity Data Pipeline.IEEE Data Eng. Bull., 35(2):33â45, 2012. [61] Google.Pub/Sub.https://cloud.google.com/ pubsub. [62]Thibaut Gourdel.Agents meet databases: The future of agentic architectures.https: //thenewstack.io/agents-meet-databases-the- future-of-agentic-architectures/, 2025. [63] Confluent Technology Strategy Group. 2026 Confluent Predictions Report.https://assets.confluent.io/ m/62a1ef550db9e6a9/original/2026-Predictions- Program-DPF-EB.pdf, 2026. [64] JelaniHarper.ConfluentâsReal-Time AgentsBuildonKafkaStreamingData. https://thenewstack.io/confluents-real-time- agents-build-on-kafka-streaming-data/, 2025. [65] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Measuring Massive Multitask Language Under- standing.arXiv preprint arXiv:2009.03300, 2020. [66] Monika Rauch Henzinger and Valerie King. Random- ized dynamic graph algorithms with polylogarithmic time per operation. InProceedings of the Twenty- Seventh Annual ACM Symposium on Theory of Com- puting, STOC â95, page 519â527, New York, NY, USA, 1995. Association for Computing Machinery. [67]Maurice P. Herlihy and Jeannette M. Wing. Lineariz- ability: A Correctness Condition for Concurrent Ob- jects.ACM Trans. Program. Lang. Syst., 12(3), July 1990. [68]Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. Model Context Protocol (MCP): Landscape, Se- curity Threats, and Future Research Directions.arXiv preprint arXiv:2503.23278, 2025. [69]Inscribe.ai. How we continuously update our fraud detectors.https://w.inscribe.ai/blog/how-we- continuously-update-our-fraud-detectors. [70] Zhipeng Jia and Emmett Witchel.Boki: Stateful Serverless Computing with Shared Logs. InProceed- ings of the 28th ACM Symposium on Operating Sys- tems Principles (SOSP â21), Virtual, October 2021. [71]Jared Jones and Yves Laurent. Unleash real-time agen- tic ai with streaming agents on confluent cloud and couchbase.https://w.couchbase.com/blog/real- time-agentic-ai-confluent-cloud. 15 [72]Kafka.KIP-1150: Diskless Topics for Kafka. https://cwiki.apache.org/confluence/display/ KAFKA/KIP-1150%3A+Diskless+Topics. [73]Anuj Kalia, Michael Kaminsky, and David Andersen. Datacenter RPCs can be general and fast. InProceed- ings of the 16th Symposium on Networked Systems Design and Implementation (NSDI â19), Boston, MA, February 2019. [74]Martin Kleppmann and Jay Kreps. Kafka, samza and the unix philosophy of distributed data.IEEE Data Eng. Bull., 38:4â14, 2015. [75] Jay Kreps.The log: What every software engi- neer should know about real-time dataâs unifying abstraction.https://engineering.linkedin.com/ distributed-systems/log-what-every-software- engineer-should-know-about-real-time-datas- unifying, 2013. [76]John Alistair Kressel, Hugo Lefeuvre, and Pierre Olivier.ÎźFork: Supporting POSIX fork Within a Single-Address-Space OS. InProceedings of the ACM SIGOPS 31st Symposium on Operating Systems Prin- ciples, pages 18â35, 2025. [77]Leslie Lamport. Paxos Made Simple.ACM Sigact News, 32(4):18â25, 2001. [78]James Leng.Safe Agent Writes in Workbench Demo.https://w.dolthub.com/blog/2025-11-25- safe-agent-writes-video/, 2025. [79]Lenses.io. AI-Assisted Engineering for Data Stream- ing with Lenses MCP.https://lenses.io/kafka-ai. [80]lenses.io.Lenses MCP: A New Era in AI En- ablement for Streaming Application Development. https://lenses.io/blog/2025/10/lenses-mcp-new- era-in-ai-enablement-for-streaming-app-dev/. [81]Lenses.io. Kafka to kafka replication - lenses k2k. https://lenses.io/kafka-replication/, 2024. [82] Beibin Li, Konstantina Mellou, Bo Zhang, Jeevan Pathuri, and Ishai Menache. Large language mod- els for supply chain optimization.arXiv preprint arXiv:2307.03875, 2023. [83]Christina Lin. Real-time predictions for ML apps with Redpanda Data Transforms (powered by WASM). https://w.redpanda.com/blog/real-time- predictions-machine-learning-applications- wasm, April 2024. [84] Shu Liu, Soujanya Ponnapalli, Shreya Shankar, Sepanta Zeighami, Alan Zhu, Shubham Agarwal, Ruiqi Chen, Samion Suwito, Shuo Yuan, Ion Sto- ica, et al. Supporting Our AI Overlords: Redesign- ing Data Systems to be Agent-First.arXiv preprint arXiv:2509.00997, 2025. [85]Joshua Lockerman, Jose M Faleiro, Juno Kim, Soham Sankaran, Daniel J Abadi, James Aspnes, Siddhartha Sen, and Mahesh Balakrishnan. The fuzzylog: A par- tially ordered shared log. InProceedings of the 13th USENIX Conference on Operating Systems Design and Implementation (OSDI â18), Carlsbad, CA, November 2018. [86] Xuhao Luo, Shreesha G. Bhat, Jiyu Hu, Ramnatthan Alagappan, and Aishwarya Ganesan. LazyLog: A New Shared Log Abstraction for Low-Latency Applications. InProceedings of the 30th ACM Symposium on Op- erating Systems Principles (SOSP â24), Austin, TX, October 2024. [87]Maintainers of cp-algorithms. Segment Trees: Range Updates (Lazy Propagation).https://cp-algorithms. com/data_structures/segment_tree.html#range- updates-lazy-propagation. [88] Matteo Merli, Sijie Guo, Penghui Li, Hang Chen, and Neng Lu.Ursa: A lakehouse-native data streaming engine for kafka.Proc. VLDB Endow., 18(12):5184â5196, August 2025. [89]Microsoft. Build AI Agents with Azure Database for PostgreSQL and Azure AI Agent Service. https://techcommunity.microsoft.com/blog/ adforpostgresql/build-ai-agents-with-azure- database-for-postgresql-and-azure-ai-agent- service/4392616, 2025. [90]Microsoft.Azure Blob Storage documenta- tion.https://learn.microsoft.com/azure/storage/ blobs/, 2026. [91] MinIO. MinIO: A high-performance, S3-compatible object storage.https://github.com/minio/minio. [92] Micah Murray, Wen Zhang, Aisha Mushtaq, Natacha Crooks, Aurojit Panda, and Scott Shenker. Designing a Datacenter-wide Distributed Shared Log. InProceed- ings of the 2025 Workshop on Hot Topics in Operating Systems, pages 195â201, 2025. [93]Diego Ongaro and John Ousterhout. In Search of an Understandable Consensus Algorithm. In2014 USENIX Annual Technical Conference (USENIX ATC 14), Philadelphia, PA, June 2014. [94]OpenCode. OpenCode: The open source AI coding agent.https://opencode.ai/. 16 [95]ArtemOppermann.Detectingfraud in realtimeusingRedpandaandPinecone. https://w.redpanda.com/blog/fraud-detection- pipeline-redpanda-pinecone. [96] Juan Altmayer Pizzorno and Emery D Berger. Coverup: Effective high coverage test generation for python. arXiv preprint arXiv:2403.16218, 2024. [97]PowerDrill.Enterprise use-cases for Agentic AI.https://powerdrill.ai/blog/ai-agents-for- data-analysis-and-visualization. [98]Pravega. Pravega â A Reliable Stream Storage System. https://pravega.io/. [99]Shyam Purkayastha.Building a real-time data processing pipeline for IoT.https: //w.redpanda.com/blog/analyzing-iot- telemetry-data-apache-spark. [100]Quix.A guide to windowing in stream pro- cessing.https://quix.io/blog/windowing-stream- processing-guide. [101] Mark Raasveldt and Hannes MĂźhleisen. Duckdb: an embeddable analytical database. InProceedings of the 2019 International Conference on Management of Data, SIGMOD â19, page 1981â1984, New York, NY, USA, 2019. Association for Computing Machinery. [102]RedPanda. Redpanda: The Agentic Data Plane.https: //w.redpanda.com/. [103] RedPanda.Remote MCP Server Overview. https://docs.redpanda.com/redpanda-cloud/ai- agents/mcp/remote/overview/. [104] Redpanda Data. Kafka Broker: Architecture, Func- tions, and Concepts.https://w.redpanda.com/ guides/kafka-architecture-kafka-broker, 2025. [105] Redpanda Data, Inc. Redpanda Data â streaming data platform compatible with Apache Kafka.https://w. redpanda.com/. [106]Redpanda Documentation.Mcp tool patterns. https://docs.redpanda.com/redpanda-cloud/ai- agents/mcp/remote/tool-patterns/. [107]Redpanda Documentation.Model context proto- col (mcp).https://docs.redpanda.com/redpanda- cloud/ai-agents/mcp/. [108] Redpanda, Inc.MCP Tool Patterns & Stream Processing in Redpanda Cloud Remote MCP. https://docs.redpanda.com/redpanda-cloud/ai- agents/mcp/remote/tool-patterns/#stream- processing, 2026. [109]Salesforce.Mirus: Cross-data-center replication for apache kafka.https://github.com/salesforce/ mirus, 2018. [110]CraigSanchez.Agentic aimeetsreal- time data with streaming agents.https: //w.vectara.com/blog/agentic-ai-meets-real- time-data-with-streaming-agents. [111]Timo Schick, Jane Dwivedi-Yu, Roberto DessĂŹ, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv preprint arXiv:2302.04761, 2023. [112]Fred B. Schneider. Implementing Fault-tolerant Ser- vices Using the State Machine Approach: A Tutorial. ACM Comput. Surv., 22(4):299â319, December 1990. [113] Tim Sehn.So you Want Database Branches? https://w.dolthub.com/blog/2024-09-18- database-branches/, 2024. [114]Tim Sehn.Agent Mode in the Dolt Work- bench.https://w.dolthub.com/blog/2025-11-12- agentic-dolt-workbench, 2025. [115]TimSehn.Dolt for Agentic Workflows. https://w.dolthub.com/blog/2025-03-17-dolt- agentic-workflows/, 2025. [116]Tim Sehn. So you want an AI Database?https://w. dolthub.com/blog/2025-12-09-ai-database/, 2025. [117] Tim Sehn.So you want an AI Database? Isola- tion.https://w.dolthub.com/blog/2025-12-09- ai-database/#isolation, 2025. [118]Tim Sehn.So you want Database Forks? https://w.dolthub.com/blog/2022-07-29- database-forks/, 2025. [119]AmazonWebServices.Amazons3. https://aws.amazon.com/s3/. [120]StreamNative.Diskless Topics in Apache Pulsar. https://streamnative.io/blog/diskless-topics- in-apache-pulsar/. [121] StreamNative. StreamNative MCP Server: Create, De- ploy, and Test Python Pulsar Functions Using an LLM. https://w.youtube.com/watch?v=9JDHL-WaCXs. [122]Jacopo Tagliabue and Ciro Greco. Safe, Untrusted, âProof-Carryingâ AI Agents: Toward the Agentic Lake- house.arXiv preprint arXiv:2510.09567, 2025. [123]Midhat Tilawat. What are Context-Aware Agents in AI?https://w.allaboutai.com/ai-glossary/ context-aware-agents/, 2025. 17 [124]Tricentis.Testing in production.https://w. tricentis.com/learn/production-testing, 2026. [125]Typedef Team.Agentic Analytics on Big- Query: Turning SQL Warehouses into AI Ana- lysts.https://w.typedef.ai/resources/agentic- analytics-bigquery, December 2025. [126] Rajkumar Venkatasamy.Building a real-time search application with Redpanda and ZincSearch. https://w.redpanda.com/blog/real-time-data- search-redpanda-zincsearch. [127] Manuel VĂśgele, Christopher Thomas, and Timo HĂśnig. Spork: A posix_spawn you can use as a fork. InPro- ceedings of the 2025 Workshop on Hot Topics in Oper- ating Systems, HotOS â25, page 96â102, 2025. [128]OstapVykhopen, ViktoriaSkorik, Maxim Tereschenko, and Veronika Solopova.Beyond Text-to-SQL:AutonomousResearch-Driven Database Exploration with DAR.arXiv preprint arXiv:2512.14622, 2025. [129]Kai Waehner.Real-Time Model Inference with Apache Kafka and Flink for Predictive AI and GenAI. https://w.kai-waehner.de/blog/2024/10/01/ real-time-model-inference-with-apache-kafka- and-flink-for-predictive-ai-and-genai/, 2024. [130] Kai Waehner.The Rise of Diskless Kafka: Re- thinking Brokers, Storage, and the Kafka Protocol. https://w.kai-waehner.de/blog/2025/08/11/ the-rise-of-diskless-kafka-rethinking-brokers- storage-and-the-kafka-protocol/, 2025. [131]Cong Wang and Yusheng Zheng. Fork, Explore, Com- mit: OS Primitives for Agentic Exploration.arXiv preprint arXiv:2602.08199, 2026. [132]Guozhang Wang, Lei Chen, Ayusman Dikshit, Ja- son Gustafson, Boyang Chen, Matthias J. Sax, John Roesler, Sophie Blee-Goldman, Bruno Cadonna, Apurva Mehta, Varun Madan, and Jun Rao. Consis- tency and completeness: Rethinking distributed stream processing in apache kafka. InProceedings of the 2021 International Conference on Management of Data, SIGMOD â21, page 2602â2613, New York, NY, USA, 2021. Association for Computing Machinery. [133]WarpStream. The Diskless, Kafka Compatible Data Streaming Platform.https://w.warpstream.com/. [134]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-Thought Prompting Elicits Reason- ing in Large Language Models.Advances in Neural Information Processing Systems, 35, 2022. [135]Xingda Wei, Fangming Lu, Tianxia Wang, Jinyu Gu, Yuhan Yang, Rong Chen, and Haibo Chen. No Provi- sioned Concurrency: Fast RDMA-codesigned Remote Fork for Serverless Computing. In17th USENIX Sym- posium on Operating Systems Design and Implemen- tation (OSDI 23), pages 497â517, Boston, MA, July 2023. USENIX Association. [136]Wikipedia contributors. Euler Tour Technique.https: //en.wikipedia.org/wiki/Euler_tour_technique, 2025. [137]Willem Thiart, and other open-source contributors. willemt/raft.https://github.com/willemt/raft. [138] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing Reasoning and Acting in Language Mod- els. InThe 11th International Conference on Learning Representations, 2022. [139]Filip Yonov.The 3 Es of Diskless Kafka BYOC.https://aiven.io/blog/diskless-apache- kafka-guide-byoc, 2025. [140] Yazhuo Zhang, Jinqing Cai, Avani Wildani, and Ana Klimovic. Rethinking web cache design for the ai era. InProceedings of the 2025 ACM Symposium on Cloud Computing, SoCC â25, page 535â542, New York, NY, USA, 2026. Association for Computing Machinery. [141]Zhiting Zhu, Zhipeng Jia, Newton Ni, Dixin Tang, and Emmett Witchel. Impeller: Stream Processing on Shared Logs. InProceedings of the EuroSys Con- ference (EuroSys â25), Rotterdam, Netherlands, April 2025. 18