Paper deep dive

FlexiFlow: Bandit-based Model Switching in ML Workflows

Abhilash Jindal, Todd Nief, Bhanu Prakash Vangala, Shankaradithyaa V, Tvisha Malik, Anshik Sahu, Aaron Schein, Amitabh Chaudhary, Tanu Malik

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 91

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%

Last extracted: 10/7/2026, 9:03:17 AM

Summary

The paper introduces FlexiFlow, a dataflow system that dynamically switches between alternate machine learning models in multi-step workflows to improve accuracy and efficiency. It employs a novel multi-armed bandit approach, specifically Bayesian strategies like Thompson Sampling and a hybrid method, to learn optimal model configurations based on runtime, assertion pass rates, and workflow structure. Experiments demonstrate up to 23% accuracy improvement and 48% efficiency gain over sequential runs.

Entities (12)

FlexiFlow · system · 100%Multi-armed bandit · algorithm · 95%ID Card ML workflow · usecase · 90%IIT Delhi · organization · 90%Thompson Sampling · algorithm · 90%University of Chicago · organization · 90%University of Missouri · organization · 90%EasyOCR · model · 85%Tesseract · model · 85%Faster R-CNN · model · 80%SSD · model · 80%YOLO · model · 80%

Relation Signals (10)

Abhilash Jindal → affiliatedwith → IIT Delhi

confidence 95% · Abhilash Jindal ... Affiliation: IIT Delhi, India

Todd Nief → affiliatedwith → University of Chicago

confidence 95% · Todd Nief ... Affiliation: University of Chicago, USA

FlexiFlow → uses → Multi-armed bandit

confidence 95% · FlexiFlow learns to rank models using a novel multi-armed bandit approach

FlexiFlow → improves → workflow accuracy

confidence 90% · switching models improves workflow accuracy by up to 23%

FlexiFlow → provides → efficiency gain

confidence 90% · switching models at runtime while reusing intermediate results provides higher accuracy, but also 48% efficiency gain

Thompson Sampling → isinsufficientfor → switching models in ML workflows

confidence 85% · We show that the standard Thompson sampling approach is insufficient for switching models in ML workflows.

EasyOCR → isalternativeto → Tesseract

confidence 80% · if EasyOCR fails on a given input, the system can switch to an alternative like Tesseract.

YOLO → isusedin → image cropping

confidence 80% · the cropping step could use models like YOLO

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Model optimizations help improve inference performance and accuracy of ML workflows. However, relying on a single model to perform inference across all data batches often fails to maximize accuracy and thus overall performance. In many cases, alternate models could perform better on specific subsets of data where a primary model underperforms. Our experiments with real ML workflows indeed show that switching models improves workflow accuracy by up to 23%. Yet, current systems lack the ability to adaptively switch between models based on performance, forcing users to manually test models in sequence. We present FlexiFlow, a dataflow system that dynamically switches between alternate models when the current model exhibits low accuracy. FlexiFlow learns to rank models using a novel multi-armed bandit approach that accounts for model runtimes, probability of passing user-defined assertions, and the computational structure of the ML workflow. We show that the standard Thompson sampling approach is insufficient for switching models in ML workflows. In contrast, our proposed approaches are effective and scales to complex real-world ML workflows. Experiments show that switching models at runtime while reusing intermediate results provides higher accuracy, but also 48% efficiency gain compared to sequential workflow runs.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

87,539 characters extracted from source content.

Expand or collapse full text

FlexiFlow: Bandit-based Model Switching in ML Workflows Abhilash Jindal email: ajindal@cse.iitd.ac.in Affiliation: IIT Delhi, India , Todd Nief email: tnief@uchicago.edu Affiliation: University of Chicago, USA , Bhanu Prakash Vangala email: bv3hz@missouri.edu Affiliation: University of Missouri, USA , Tanu Malik email: tanu@missouri.edu Affiliation: University of Missouri, USA , Shankaradithyaa V email: shankaradith@iisc.ac.in Affiliation: IISc Bangalore, India , Tvisha Malik email: cs1210085@cse.iitd.ac.in Affiliation: IIT Delhi, India , Anshik Sahu email: cs1210577@cse.iitd.ac.in Affiliation: IIT Delhi, India , Amitabh Chaudhary email: amitabh@uchicago.edu Affiliation: University of Chicago, USA and Aaron Schein email: schein@uchicago.edu Affiliation: University of Chicago, USA Abstract. Model optimizations help improve inference performance and accuracy of ML workflows. However, relying on a single model to perform inference across all data batches often fails to maximize accuracy and thus overall performance. In many cases, alternate models could perform better on specific subsets of data where a primary model underperforms. Our experiments with real ML workflows indeed show that switching models improves workflow accuracy by up to 23%. Yet, current systems lack the ability to adaptively switch between models based on performance, forcing users to manually test models in sequence. We present FlexiFlow, a dataflow system that dynamically switches between alternate models when the current model exhibits low accuracy. FlexiFlow learns to rank models using a novel multi-armed bandit approach that accounts for model runtimes, probability of passing user-defined assertions, and the computational structure of the ML workflow. We show that the standard Thompson sampling approach is insufficient for switching models in ML workflows. In contrast, our proposed approaches are effective and scales to complex real-world ML workflows. Experiments show that switching models at runtime while reusing intermediate results provides higher accuracy, but also 48% efficiency gain compared to sequential workflow runs. 1. Introduction With the increasing use of machine learning in production systems, engineers often encounter practical problems in deploying and maintaining ML models and workflows. Traditionally, models are often learned in notebooks or scripts, serialized (i.e., “pickled”), and embedded within web servers—a.k.a deployed—to expose prediction APIs. However, deployment often reveals a range of issues, particularly related to model performance and accuracy. A recurring concern is the issue of soft failures—situations where the model does not crash but delivers incorrect or degraded predictions, often due to factors like data drift (Gao et al., 2008). These soft failures compromise prediction accuracy and, in turn, the overall effectiveness of the deployed system. When workflows consist of multiple steps and models, such degradations must be promptly identified and addressed. Developers commonly rely on trace logs to identify problematic data batches, but exhaustively testing alternative model configurations is resource-intensive and time-consuming. Figure 1. ID Card ML workflow. Consider a deployed ML workflow, visualized in Figure 1, for identity verification in KYC (Know Your Customer) scenarios using identification cards (e.g., driver’s licenses, passports). The workflow processes an image or scan of the ID card and outputs a structured record with fields such as name, date of birth, ID number, and expiration date. Each step in this pipeline—such as image cropping, color channel selection, OCR, card classification, filtering, and field extraction—may be implemented using different ML models. For instance, the cropping step could use models like YOLO (Redmon et al., 2016), SSD (Liu et al., 2016), or Faster R-CNN (Ren et al., 2015), while OCR could be performed by tools such as Tesseract, EasyOCR, Google Vision API, AWS Textract, or Microsoft Read API. In practice, only one single model is typically selected for each step based on evaluation over training data. However, this static selection often fails to maintain high accuracy across all real-world inputs during deployment. If a single model underperforms, the entire workflow may need to be re-executed from the beginning with an alternative model, significantly increasing the management burden and latency. One approach to mitigating soft model failures on specific inputs is to dynamically switch between models based on their performance. For example, in the OCR step of an ID card processing pipeline, if EasyOCR fails on a given input, the system can switch to an alternative like Tesseract. However, this requires the system to assess relative model performance—not just using aggregate accuracy metrics, but by detecting soft failures (e.g., invalid outputs or low confidence) on individual inputs. Even when estimates of model performance are known, a key challenge remains: determining the optimal order in which to try models to minimize latency while maintaining correctness. Recently, these challenges have been addressed, albeit independently, in the machine learning domain. Model assertions (Kang et al., 2020) provide a data-independent way to assess model performance. In the context of ID card processing, such assertions might include checks like validating that the extracted date of birth is a legitimate calendar date, ensuring the age is under 120 years, or applying confidence thresholds to predictions. Separately, explore-exploit strategies (Sutton and Barto, 2018) offer a principled method for determining which configurations to use first. These approaches balance exploiting configurations that have performed well historically with exploring new alternative configurations, adapting dynamically based on observed rewards. To our knowledge, no existing ML dataflow system integrates both model assertions and explore-exploit strategies to guide model switching decisions. Integrating model assertions with explore-exploit strategies requires a non-trivial coordination of correctness and efficiency, as the system must adapt to assertion violations while maintaining acceptable performance. In this paper, we present FlexiFlow a dataflow system for multi-step ML workflows that learns to perform model selection and switching when models (soft) fail on assertions. FlexiFlow allows a user to configure each step of an ML workflow with multiple model choices. To determine the best model configuration, i.e., combination of model choices at each step, we equip FlexiFlow with a Bayesian statistics-based optimizer that samples from the posterior reward distributions of model choices, enabling the estimation of the most effective model configuration for a given input. If a model configuration still fails assertions, to explore alternative configurations, FlexiFlow rolls back to a previous model operator and efficiently tests a sequence of new configurations. We demonstrate that a variety of explore-exploit Bayesian strategies are possible within the runtime to enhance performance. Our findings indicate that while the space of all possible workflow configurations can be large, Bayesian strategies can quickly find good configurations. 1.1. Our Contributions Figure 2 shows the different choices and assertions specified for an instance of the ID card ML workflow. In the ChannelPicker step, five color channel choices are made available: all channels (all), blue only (b), green only (g), red only (r), and a combination of blue and green channels (bg). In the OCR step, the user provides choice between two engines: Tesseract and EasyOCR. Additionally, the workflow allows for different classifier choices in the classifier step. The figure also depicts assertion checks on extracted data. As shown, there are a total of 20 different workflow configurations, corresponding to 5, 2, and 2 operator choices at various steps. The configuration space grows quickly as the number of choices increases or as more steps are added. Although an ML workflow might select models or configurations based on performance during training, these choices often degrade when deployed, due to shifts in the data distribution. To remain effective, the system must adapt to changing conditions at inference time. This motivates the use of online learning approaches where, rather than training the model selection process all at once on a fixed dataset, the system incrementally updates its model selection process with the arrival of each new example or batch of examples. A common approach to online learning is the multi-armed bandit (MAB), where the goal is to choose among k configurations by balancing exploration and exploitation to maximize expected reward. However, FlexiFlow departs from the standard MAB setting in two key ways: First, while there are k overall workflow configurations, each configuration is composed of multiple steps (or operators), and each step has its own set of model choices. This implies that a distinct MAB problem exists at each stage of the workflow. This MAB problem likely includes interactions across the decision at each step, resulting in a bandit problem that does not fit neatly into the standard contextual or hierarchical bandit structure. Second, unlike traditional MAB settings where each trial is independent, in FlexiFlow, if a given configuration fails (e.g., violates a model assertion), it should not require restarting the entire workflow. Instead, the system should be able to ‘rollback’ to the point of failure and selectively modify only the relevant operator choice. For instance, as shown in Figure 2, if a classification step yields incorrect results, the system should be able to switch classifiers without re-running upstream components like OCR or preprocessing. These distinctions highlight the need for a bandit-based approach that leverages the computational structure of the workflow graph, enabling localized reconfiguration and efficient exploration of the model choice space for improved overall performance. We make the following contributions: Figure 2. ID Card workflow in FlexiFlow. Choices for each workflow step are indicated below in red italics. Soft failure of assertions necessitates switching (dashed arrows) between model choices. • A novel programming model with operator choices and assertions: We introduce primitives within an existing dataflow language to support the expression of both operator choices and assertions. The language runtime, to support multiple choices, incorporates an adaptive rollback mechanism that reverts to specific operators when an assertion fails. This rollback mechanism is guided by an optimizer (described next), allowing for rollbacks to intermediate nodes within the workflow and not just the root step. To facilitate incremental computation after rollback, we leverage past work on dataflow error handling (Jindal et al., 2024). • Bayesian strategies for automatic error correction: We propose two multi-armed bandit-based strategies to learn an optimal sequence of configurations: – The first configuration strategy uses Thompson sampling with posterior distributions over model configurations, balancing exploitation of high-payoff configurations that pass the most assertions with exploration of alternative configurations. Execution time is explicitly modeled as part of the optimization strategy. – The second hybrid strategy exploits the structure of the choice-based programming model by running separate multi-armed bandit optimizations at each workflow step, tracking cumulative payoffs for configurations and updating posteriors for operator choices on high-payoff model configurations. Using microbenchmarks and real applications, we show that the hybrid (second) strategy provides a learning advantage. • A research prototype: We present FlexiFlow, an experimental Python prototype that supports efficient rollbacks, incremental computation, and the proposed language extensions and optimization strategies. We evaluate FlexiFlow with three types of ML workflows: document processing, image classification, and question-answering. For each workflow, we describe its formulation in FlexiFlow, the range of choices and assertions, and the performance of our proposed strategies. Our key findings are: – Adding operator choices in workflows can improve F1F_1 scores by upto 23% compared to workflows without them. We find that this is because different configurations, with different ML models, often have uncorrelated failures. – It is imperative for strategies to learn runtimes of operator choices along with their probabilities of passing assertions. Further, by making them aware of the rollback approach of FlexiFlow, they can quickly learn good configurations. The rest of the paper is organized as follows: Section 2 presents the FlexiFlow architecture and describes the programming model consisting of operator choices, assertions, and rollbacks. Section 3 formulates the configuration selection learning problem in terms of assertions and rollbacks. In Section 4 we describe Bayesian strategies that estimate the probability of choosing a configuration with the highest reward. In Section 5 we present our implementation of FlexiFlow and evaluation on microbenchmarks and four real ML pipelines. We discuss our results in Section 6, present related work in Section 7, and conclude in Section 8. 2. The FlexiFlow Execution Model FlexiFlow is defined within the context of a dataflow language that enables users to express complex machine learning (ML) workflows. We begin by reviewing the fundamentals of such a language and its data model, and then introduce two novel programmatic constructs provided by FlexiFlow: operator choices and workflow assertions. A basic dataflow language, such as Spark-SQL (Armbrust et al., 2015) or SCOPE (Zhou et al., 2012), models an ML workflow as a directed acyclic graph (DAG). In this DAG, nodes represent operators, each of which performs a data transformation; edges represent data dependencies between operators (i.e., the flow of intermediate datasets). Let d denote an input data item (e.g., a row or an image), and let opop denote a map transformation function implemented by an operator. Then, the semantics of an operator are defined as: op(d)→d’op(d)→ d in which d′d is the output data item produced by applying opop to d. The execution of a dataflow program thus proceeds by propagating data from source nodes to sink nodes, applying each operator as data becomes available. The runtime supports pipeline parallelism (Hennessy and Patterson, 2017): partial outputs from parent operators can be streamed to child operators without waiting for the entire input to be processed. Typical operators are data preprocessing operators such as extracting structured rows from a file, processing them, combining rows, or outputting to disk or user-defined operators such as processing video frames or images into structured data. Each operator consumes a well-defined input schema and produces a corresponding output schema as shown in Figure 3. The runtime verifies schema compatibility along edges in the DAG to ensure compositional correctness. In particular, operators are map operators that apply the operator independently to each data item, and currently do not include logical operators such as joins, and aggregates. Figure 3. FlexiFlow data model. Some operators are omitted for clarity. FlexiFlow augments this basic model by introducing three core constructs: operator choices and workflow assertions as language constructs, and rollback as a runtime construct. In FlexiFlow, each map operator opop may be instantiated by one of multiple alternative implementations Let op1,op2,…,opk\op^1,op^2,…,op^k\ denote the k possible implementations. The operator is now modeled as a non-deterministic transformation, selected at runtime: opi(d)→d’iop^i(d)→ d ^i, where i∈1,…,ki∈1,…,k That is, for a given input d, the runtime selects an instance choice opiop^i and produces the corresponding output d′id i. Figure 2 also shows (in italics) multiple choices available for each operator. In addition to operator choices, FlexiFlow allows users to define assertions on intermediate or final outputs of the workflow. These are Boolean-valued functions ϕ⁡(d′)∈true,falseφ(d )∈\true,false\ that act as semantic checks (e.g., validating that a date of birth is valid or that confidence scores exceed a threshold). In general there can be many assertions at different locations of a DAG, but for ease of exposition we assume a single assertion represented as a sink node in a DAG. Assertions provide a signal of correctness and are used by the runtime to detect soft failures and guide, as we describe in the next section, operator re-selection. Finally, when an assertion fails for a given input, FlexiFlow’s runtime rolls back to an operator. Figure 2 shows that ID card can possibly rollback to Channel Picker, OCR or Classifier operators. We describe further implementation details of how to rollback in Section 5. Architecture. Figure 4 shows the overall architecture of FlexiFlow with the three constructs: operator choices, assertions, and rollback, which introduce in FlexiFlow the decision of which specific operator to rollback to and which operator choice to pick next, when an assertion fails. To address this, FlexiFlow employs a runtime-aware, rollback-aware, learning-based optimizer that learns from past successes and failures at both the operator level and across combinations of operator choices as described next. Figure 4. FlexiFlow Architecture. Optimizer maintains estimates for each operator choice and set of operator choices. 3. The FlexiFlow Learning Problem The choices available at each operator in FlexiFlow multiplicatively increase the space of ML workflow configurations. Let an ML workflow consist of M operators, where each operator i, with i∈1,…,Mi∈\1,…,M\, has kik_i possible choices. Then, the total number of workflow configurations is given by: (1) K=∏i=1MkiK= _i=1^Mk_i For example, the ID card processing workflow has M=3M=3 operators— ChannelPicker, OCR, classifier—each having 5, 2, and 2 choices, respectively. One possible configuration within the workflow is [“red", “Tesseract", “class 1"]. In total, the workflow has 5×2×2=205× 2× 2=20 configurations to choose from. For a given data item, FlexiFlow initially selects a configuration from the set of K configurations, with the hope that the selected configuration passes the assertion. If this selection fails the assertion, the system selects another configuration, continuing until the assertion is satisfied. This process forms a sequence of configurations: S=(a1,…,aj),j≤KS=(a_1,…,a_j),j≤ K. In the worst case, the sequence can be as long as K, meaning all configurations of an ML workflow must be tested. However, the objective of FlexiFlow is to determine the optimal sequence S∗S^* that minimizes the expected execution cost, as each additional configuration incurs an overhead of re-running the operators. We observe two properties of any sequence S: • First, in any sequence of length j, the current configuration aja_j is tried if all previous configurations from index 1 to j−1j-1 have failed the assertion. • Second, if two consecutive configurations share a longer prefix of common operator choices, the rollback execution is faster. Consider the sequence ([“red”, “EasyOCR”, “class 2”], [“red”, “EasyOCR”, “class 1”]) and the sequence ([“red”, “Tesseract”, “class 2”], [“red”, “EasyOCR”, “class 1”]). The first sequence, due to longer common prefix, leads to a lower re-execution cost, improving overall efficiency. Given these properties, the goal in FlexiFlow is to identify the best configuration sequence S∗S^* that minimizes the expected runtime, formulated as: (2) ∑j∈1..KP[⋂i<jai¯]×aj−1→aj, _j∈\1..K\P [ _i<j a_i ]×T_a_j-1→a_j, in which, P⁡[⋂i<jai¯]P [ _i<j a_i ] is the probability that all previous configurations a1,…,aj−1\a_1,…,a_j-1\ fail the assertion, and aj−1→ajT_a_j-1→a_j is rollback execution cost from configuration aj−1a_j-1 to aja_j. Expression 2 is of limited practical use, since failure probabilities are unknown for new data items, and expected runtimes in deployment often deviate from training phase estimates due to differences in data distributions. So we next describe how FlexiFlow efficiently identifies S∗S^* from a large space of configuration sequences using an online, Bayesian explore-exploit learning strategy. 4. FlexiFlow’s Bayesian Online Optimizer In this section, we develop a variety of optimizers, each of which uses a Bayesian explore-exploit learning strategy from the multi-armed bandit framework (Sutton and Barto, 2018). The optimizers estimate success probabilities and runtimes to minimize total expected runtimes (Equation 6). Below, we provide a brief background on the Bernoulli bandit problem, followed by FlexiFlow online optimizers. Background. Consider a scenario where an operator has K possible choices. Each action a∈1,…,Ka∈\1,…,K\ has an unknown success probability θk _k meaning that each time action a is chosen, it yields a binary reward r. r is 1 with probability θa _a (success) and r is 0 with probability 1−θa1- _a (failure). The objective of the bandit framework is to learn the success probabilities (θ1,…,θK)( _1,…, _K) through experimentation, while maximizing cumulative rewards over N trials, where N≫KN K. Suppose we have an operator with three possible actions. Lets say over 2005 trials, these actions were selected as follows: Action 1 was chosen 1000 times, yielding 600 successes (θ1=0.6θ_1=0.6), Action 2 was chosen 1000 times, yielding 400 successes (θ2=0.4θ_2=0.4) , and Action 3 was chosen only 5 times, yielding 2 successes (θ3=0.4θ_3=0.4, but with high variance due to the small sample size). Figure 5 illustrates the probability density functions (PDFs) for these estimated success probabilities. The variance in the estimates decreases as the number of trials per action increases. In an ϵε-greedy approach, the learning agent explores by sampling actions uniformly at random with probability ϵε, where ϵε is fixed a priori. During exploitation (with probability 1−ϵ1-ε), the agent repeatedly selects the action with the highest estimated mean reward. For example, if action 1 consistently has a higher mean than actions 2 and 3, it will be chosen repeatedly during exploitation, while during exploration all three actions are sampled uniformly. Allocating an equal number of experimental trials to each action during exploration results in poor performance of an ϵε-greedy approach (Russo et al., 2020). Thompson Sampling (TS) fundamentally addresses this limitation of ϵε-greedy by updating the probabilities of success of actions using Bayes’ rule. In Thompson Sampling, each action a has an independent prior belief over each θa _a, which is modeled as a Beta distribution: (3) f⁡(θa,αa,βa)=1B⁡(αa,βa)θkαa−1(1−θa)βa−1f( _a; _a, _a)= 1B( _a, _a)\> _k _a-1\ (1- _a) _a-1 where the beta function B⁡(αa,βa)B( _a, _a) is a normalizing constant. As observations are gathered, each action’s posterior distribution is updated according to a simple rule: (4) (αa,βa)←(αa,βa),ifxt≠a(αa,βa)+(rt,1−rt),ifxt=a( _a, _a)← cases( _a, _a),&if\ x_t≠ a\\ ( _a, _a)+(r_t,1-r_t),&if\ x_t=a cases in which rtr_t represents the actual reward in the trial tht^th that attempts action xtx_t. When αk=βk=1 _k= _k=1, the prior probability of θk _k is uniform over [0, 1]. For each data item, only the Beta parameters (αa _a, βa _a) of the selected action a are updated based on the observed reward rtr_t. If the action succeeds (rtr_t = 1), αk _k increases, otherwise βk _k increases. As the number of trials grows, the distribution mean of the Beta distribution αk(αk+βk) _k( _k+ _k), which represents the ratio of success to total trials, approaches the empirical mean of successes. Instead of explicitly alternating between exploration and exploitation phases (as in ϵε-greedy), Thompson Sampling samples from the posterior distribution of each action’s success probability θa _a. Actions with higher success probability estimates are sampled more frequently, but uncertainty in estimates naturally encourages exploration. Figure 5. Probability density functions over mean rewards 4.1. FlexiFlow Optimizer for Single Operator with Multiple Choices Traditional Thompson Sampling (TS) updates success probabilities based on the observed reward from a single action/operator choice at each trial, accumulating cumulative rewards over time. However, FlexiFlow introduces a key difference: when an operator choice fails, alternative choices are repeatedly attempted via rollbacks until a successful configuration is found. These rollbacks introduce additional runtime costs, as each re-execution incurs the overhead of rerunning the operator choice. To address this, we first describe a modified version of Thomson Sampling (TS), termed Operator-level TS (OTS), in which only the reward function is changed and is modeled with expected runtimes of probability of success or failure of the assertions. We consider OTS for a single-operator scenario with multiple choices. We then extend this algorithm to include the computational structure of the workflow in operator and choices along with rollbacks in a multiple operator, multi-choice scenario. Operator-Level TS (OTS) In Listing 1, each operator choice a∈Aa∈ A has an unknown probability of passing θa _a, modeled as a beta distribution, and an unknown runtime distribution, modeled as a log-normal distribution. The domains of each distribution: beta distribution has 0-1 and log-normal has 0−inf0- , make them suitable for modeling successes and runtimes (Sutton and Barto, 2018). To estimate these parameters before selection, the following steps are performed: We first sample the probability of passing θa _a from a Beta distribution (Line 5). We then compute the expected number of runs nan_a to pass an assertion, which is the inverse of the probability of success (Line 6). We then sample operator execution runtimes from a log-normal distribution nan_a times, and compute a normalized expected runtime to pass the assertion (Line 7). Finally, the total estimated runtime for operator choice a is given in Line 8, and for each a the runtimes is stored in T. Once the estimated runtimes for all available operator choices are computed, the algorithm iteratively selects, executes, and updates operator choices until the example passes. The operator with the lowest estimated runtime is chosen as at∗a_t^* in Line 14. The data example xix_i is processed using at∗a_t^*, and its reward (success/failure) and the runtime is observed (Line 16). After each operator execution, the algorithm updates its internal estimates: it refines the success probability using a Beta distribution update (Line 18) and adjusts the log-normal runtime distribution parameters (μa,σa2)( _a, _a^2) based on the observed runtime yty_t using the Bayesian update rule (DeGroot and Schervish, 2011) in UpdateLogNormalParams (Line 20). If an operator choice fails, the corresponding action is removed from consideration, and the algorithm selects another alternative (Line 22) i.e., it incorporates rollback. This process repeats until the example successfully passes the assertion. Algorithm 1 Single Operator-based Thompson Sampling (X, A) 1: # Iterate over N examples 2: for xi∈Xx_i∈ X do 3: Ai←A_i← A ⊳ Initialize action list for current example 4: for a∈Aia∈ A_i do 5: Sample θa∼Beta(αa,βa) _a ( _a, _a) ⊳ Sample probability of passing 6: na←⌈1/θa⌉n_a← 1/ _a ⊳ Compute expected number of runs to pass 7: Sample τj∼Log-normal(μa,σa2) _j -normal( _a, _a^2) for j=1,…,naj=1,…,n_a ⊳ Sample runtime for each expected run 8: a←1na∑j=1naτjT_a← 1n_a _j=1^n_a _j ⊳ Compute mean sampled runtime 9: a^←a⋅1θa T_a _a· 1 _a ⊳ Calculate estimated runtime 10: Store a T_a in T 11: end for 12: 13: while example not passed do 14: at∗←argminaa∈Ai^a_t^*← a∈ A_i argmin$_a$\> T ⊳ Select best available action (fastest runtime); Subscript t indexes the processing attempt 15: 16: Process xix_i using at∗a_t^*; observe pass/fail rtr_t and runtime yty_t 17: 18: (αat∗,βat∗)←(αat∗+rt,βat∗+1−rt)( _a_t^*, _a_t^*)←( _a_t^*+r_t, _a_t^*+1-r_t) ⊳ Update Beta distribution for at∗a_t^* 19: 20: UpdateLogNormalParams(at∗,yt)UpdateLogNormalParams(a_t^*,y_t) 21: 22: Ai←Ai∖at∗A_i← A_i \a_t^*\⊳ Remove chosen action from available actions 23: end while 24: end for 4.2. FlexiFlow Optimizers for Multiple Operators with Multiple Choices Thus far, we have focused on the case of a single operator, but in practice, workflows consist of multiple operators, each offering several choices. One approach to learning in the multiple operator, multiple choice scenario is to assume operator-level independence and select the best choice for each operator individually. If the resulting configuration fails the assertion, the system proceeds to the next best configuration. For example, in case of ID card processing, we have three operators A=ChannelPickerA=Channel\ Picker, B=OCRB=OCR and C=ClassifierC=Classifier each with 5, 2, and 2 choices. To start, we determine the best operator choice (lowest expected runtime) independently at each of the three operators. Let’s say configuration K1=[A3,B2,C2]K_1=[A_3,B_2,C_2] where A3A_3, B2B_2, C2C_2, represent choices within their respective operators, is the best operator choice configuration i.e., resulting in the lowest expected runtime of 10s. However, this configuration also fails the assertion. If K2=[A3,B2,C1]K_2=[A_3,B_2,C_1] and K3=[A1,B2,C1]K_3=[A_1,B_2,C_1] are other available configuration choices that result in 11s and 10.5s, then per OTS in Listing 1 we will choose configuration K3K_3, which has the next lowest runtime. Naively applying OTS at each operator independently has two drawbacks. First, assuming operator level independence may not be true. For example, in ID card processing, a specific channel color may be necessary for EasyOCR engine to successfully extract text but that specific channel always causes Tesseract to fail. Second, choosing each operator independently is not rollback-cost-aware, i.e. in our example, minimizing overall configuration level runtime results in choosing K3K_3 as the next best configuration even though the cost of incrementally running K2K_2 may be the lowest. Note that K1K_1 and K2K_2 share an operator prefix and the rollback cost from K1K_1 to K2K_2 maybe lower than overall cost of incurring 10.5s for running the full configuration. We fix these two drawbacks next. Configuration-level TS (CTS). One option is to relax operator level independence and learn at the entire configuration-level. Additionally, failed configurations perform a conditional rollback, i.e., rollback to a configuration which is estimated to have the smallest increase in runtime from the current configuration. Listing 2 describes learning at the configuration level using FlexiFlow-based TS where the set of all actions A are the set of all K configurations. For each configuration, we maintain a Beta distribution with parameters α and β and obtain the estimated runtime of a configuration T. Each configuration is sampled in SelectAction to obtain a configuration with the lowest runtime, conditional on the current best choice. The example is processed with this configuration and actual rewards and operator-level runtimes are observed. The Beta distribution parameters are updated similarly to Listing 1 Line 18, but runtime μ and σ are updated for each operator choice in the current best configuration. If the example fails to pass assertions, the next best configuration with the lowest incremental increase in runtime is chosen in SelectAction. This is because our objective in Equation 2 is to minimize the cost of rollbacks, at−1→at/θtT_a_t-1→ a_t/ _t. Algorithm 2 Job Configuration Selection(X, A) for xi∈Xx_i∈ X do # Estimate rewards and runtimes for all actions in A ^←EstimatePassProb(A) θ (A) ^←EstimateRuntimes(A,^) T (A, θ) while example not passed do # Note: at∗a_t^* is conditional on the rollback cost from at−1∗a_t-1^* at∗←SelectAction(^,at−1∗)a_t^* ( T,a_t-1^*)⊳ at∗a_t^* is the chosen action Process xix_i using at∗a_t^*; observe pass/fail rtr_t and runtime yty_t UpdateRewardParams(a∗,rt,yt)UpdateRewardParams(a^*,r_t,y_t) Ai←Ai∖at∗A_i← A_i \a_t^*\ ⊳ Remove chosen action. end while end for There are two limitations of the above learning strategy—First, the number of configurations K can be potentially large–in experiments ID processing requires learning over nearly 500 choices! Second, the configuration-level does not account for the influence of individual operators on the overall performance of a configuration. We describe a novel hybrid strategy in which we determine the best configuration based on operator level estimates. Hybrid TS (HTS). A drawback of CTS is that we may have a very large number of configurations, making it difficult to fully explore the configuration space. We would like to use the results at the operator level to inform our selection of which configuration to try next. For example, in ID processing, let’s say configuration K1=[A3,B2,C2]K_1=[A_3,B_2,C_2], is the best operator choice configuration resulting in 10s of runtime but the assertion fails for this configuration. Further, lets say we have also tried a few other configurations, such as: [A3,B2,C1],[A3,B1,C2],[A2,B1,C3],[A4,B1,C2][A_3,B_2,C_1],[A_3,B_1,C_2],[A_2,B_1,C_3],[A_4,B_1,C_2], and these configurations have also resulted in failure. Thus observing at the operator level, in all trials, operators A3A_3 failed thrice in total, B2B_2 failed twice, and C2C_2 failed thrice. However, we also know the entire configuration of [A3,B2,C2][A_3,B_2,C_2] also failed once. Even though we have only tried this configuration once, we have some idea of how it will perform based on the operators that make it up. Thus we will compute an empirical β parameter (inspired by an empirical Bayes approach (Berger and Berger, 1985)) for the configuration as: (3−1)+(2−1)+(3−1) (3-1)+(2-1)+(3-1) i.e. we sum the failures from individual operators, but we subtract the failure of the entire configuration to avoid double counting. Similarly, we also compute an empirical α parameter for the configuration based on successes. Finally, we set a hyperparameter γ that influences how much to use the operator level information when calculating our final α and β parameters for the configuration. We “allocate" γ to the job configuration level parameters using a weighted average of successes and failures at the operator level. Formally, let k be a job configuration in a workflow with M operators. Let α’kα _k and β’kβ _k be the counts of successes and failure for the overall job configuration k. Let αkjα^j_k and βkjβ^j_k be the counts of successes and failures for each operator j in configuration k. We first calculate empirical Beta parameters for the configuration k, using our rule of summing the successes and failures for each operator j that makes up configuration k, but subtracting off each time that the operator was used previously as part of configuration k to avoid double counting: (5) αkem=∑j=1m(αkj−α’k),βkem=∑j=1m(βkj−β’k)α em_k= _j=1^m (α^j_k-α _k ), β em_k= _j=1^m (β^j_k-β _k )\\ We then calculate αk _k and βk _k, which are our hybrid parameters defining the reward distribution for job configuration k under the hybrid strategy. For this, we sum the count of success and failures for job configuration k (α’kα _k and β’kβ _k) with our empirical prior but normalize the empirical prior to sum to one and control its influence using the hyperparameter γ. (6) αk=α’k+γαkemαkem+βkem,βk=β’k+γβkemαkem+βkem _k=α _k+γ α em_kα em_k+β em_k, _k=β _k+γ β em_kα em_k+β em_k The hybrid strategy updates parameters after processing each data item as is done previously in other strategies. It also chooses the next best configuration similar to the conditional rollback described earlier. From a pseudocode perspective, our hybrid strategy only changes the function UpdateRewardParams in Listing 2 which updates the Beta distribution parameters as described in Equations 5 and 6. Summary. We summarize the different strategies for FlexiFlow optimizers in Table 1: • OTS (Operator-level Thompson Sampling) maintains probability and runtime parameters at the operator level. This approach does not work for multiple operator, multiple choice workflows. • CTS (Configuration-level Thompson Sampling) maintains parameters at both operator-level (to estimate runtime) and configuration-level (to estimate probability of passing). Actions are decided at the configuration level. • HTS (Hybrid Thompson Sampling) maintains probability of passing estimates both at the operator and configuration-level and uses an empirical Bayes-inspired approach to allow operator estimates to inform configuration-level selections. OTS, HTS, and CTS are equivalent for single operator, multiple choice workflows. Table 1. Comparison of different Thompson Sampling based FlexiFlow strategies. C is the total count of operator choices across operators and K is the count of configurations. Strategy Action Level Operator Params Config Params Params Count Rollback-Aware Single operator with multiple choices OTS Operator [0.2em]αj,βj _j, _j μj,σj2 _j, _j^2 O⁡(C)O(C) Multiple operators with multiple choices CTS [0.2em] Config μj,σj2 _j, _j^2 αk,βk _k, _k O⁡(C+K)O(C+K) ✓ HTS [0.2em] Config [0.2em] αj,βj _j, _j μj,σj2 _j, _j^2 [0.2em] αk′,βk′ _k , _k αkem,βkem _k^em, _k^em O⁡(C+K)O(C+K) ✓ 5. Evaluation Implementation. FlexiFlow implements operator choices, assertions, and online optimizers in Python. FlexiFlow is based on Scope (Zhou et al., 2012) which provides a parallel dataflow language. It is written in ∼ 20k lines of Python code. Code and experiment data presented in this paper is available (41). A choice is programmatically expressed within an operator as follows: ⬇ class OCR(Processor): def __call__(self, input_row: Row, …): img = input_row["IMG"] yield Row("OCR": [lambda img=img: easy_ocr(img), lambda img=img: tesseract(img)], …) In this implementation, the OCR operator encapsulates two possible choices—EasyOCR and Tesseract—which are represented as lambda functions. This means that either EasyOCR or Tesseract can be instantiated at any given time. Assertions are specified using functions that return False in case of an error, True otherwise. For example, the following assertion checks that the name should not be empty and should start with an upper case character: ⬇ def check(self, input_row: Row): name = input_row["name"] if len(name) == 0 or not name[0].isupper(): return False, "invalid name" return True, None While building a workflow, users can specify the optimizer. A complete workflow using operators , assertions, and optimizer is shown below. This workflow is instantiated with the HTS optimizer. ⬇ jb = JobBuilder(HTS(..)) n = jb.extract(ImgReader(..)) .process(ChannelPicker(..)) .process(OCR(..)) # … job = jb.union(inps=[alb, esp, …]) .assrt(predicate=check) .output(CsvWriter(..)) .build(..) All observed metadata such as runtimes corresponding to operator choices and success and failures of the assertion are maintained in Redis. Redis also maintains the ancestor row lineage information. The rollback uses row-level lineage and employs fast incremental computation described in more detail in (Jindal et al., 2024). Image (Deng et al., 2009)ClassifyDeiT tiny (Touvron et al., 2021),ViT (Wu et al., 2020)confidence > 0.65Class(a) IC: Imageclassification (2 configs)Question, Context (Rajpurkar et al., 2018)QATinyBert (Jiao et al., 2019),RoBERTa (Liu et al., 2019)confidence > 0.65Answer(b) SCQA: Small-contextquest. answer. (2 configs) (a) Question, Context (with distractions) (Yang et al., 2018)Embedding Modelall-MiniLM-L6-v2,all-mpnet-base-v2,all-MiniLM-L12-v2,bge-m3,paraphrase-multilingual-MiniLM-L12-v2,paraphrase-multilingual-mpnet-base-v2,LaBSEVectorStore(350, 8), (400, 7), (560, 5),(700, 4), (1400, 2)Retrieversimilarity, mmrQA Modeltinyroberta,distilbert-base-cased,roberta-base,mobilebert-uncased,google-bert,minilm-uncased,xlm-roberta-largeconfidence > 0.65Answer(c) LCQA: Large-contextquest. answer. (490 configs) (b) ID card image (Bulatov et al., 2021)CropContrastoriginal, increasedSharpnessoriginal, increasedColor Balanceoriginal, decreasedChannel Pickeroriginal, red, green, blue, red+green, red+blue, blue+greenOCRTesseract, EasyOCRClassifierclass-1, class-2σclass=ESP _class=ESPESP Extractσclass=ALB _class=ALBALB Extract…Uniondate of birth < date of issue ∧⋯ ·sfirst name, last name, dob, ..(d) ICP: ID cardprocessing (224 configs) (c) Figure 6. Real ML workflows for evaluating FlexiFlow. Experiment setup. We run all the experiments on a server with Intel Xeon Gold 6330 and an Nvidia A40 GPU. We evaluate whether the optimizers proposed in this paper (OTS, CTS, HTS in Table 1) can effectively lower runtimes. We perform an ablation study by selectively removing learning capabilities from the proposed optimizers. In particular, we ablate OTS to OTS-R, which does not model operator runtimes (μa,σa _a, _a in Algorithm 1); it only learns the probability that each operator choice passes the assertion and assumes equal runtimes for all choices. We ablate CTS to CTS-RB, which is not rollback-aware and selects the next configuration without considering the transition cost from the previously attempted one, i.e., SelectAction in Algorithm 2 does not consider at−1∗a^*_t-1. ML Workflows. We evaluate FlexiFlow using four real-world ML workflows and two synthetic workflows as shown in Figure 6 and Figure 7. Real workflows. The four real-world workflows are chosen from popular online sources described below. Each workflow includes the three core constructs—operator choices, assertions, and rollbacks—as illustrated in Figure 6. The original versions of these workflows did not include published operator choices. Instead, each configuration involving a specific model operator was published independently. The real input datasets used by the workflows are cited at the beginning of each, while the type of output produced is described at the end. IC . This ML workflow is based on the HuggingFace image classification pipeline. The workflow processes images from the ImageNet (Deng et al., 2009) dataset. There is a single classify operator that defines two model choices: a lightweight DeiT tiny (Touvron et al., 2021) model and a heavier ViT (Wu et al., 2020) model. The assertion checks if the confidence of the model prediction is above a specified threshold. SCQA . This ML workflow is structurally similar to the previous one and is based on the HuggingFace question answering pipeline. It answers questions from the SQuaD (Rajpurkar et al., 2018) dataset containing a single QA operator that defines two model choices: a lightweight TinyBERT (Jiao et al., 2019) model and a heavier RoBERTa (Liu et al., 2019) model. The assertion is similar to the one in IC. Our experiments will reveal that even though these workflows are structurally similar, FlexiFlow strategies can learn different optimal sequences of configurations which depends on runtimes and probabilities of passing assertions for different choices. LCQA . This workflow is for multi-hop question answering and is based on the popular benchmark HotpotQA, in the "distractor" mode (Yang et al., 2018). Briefly, each question is given paragraphs from 10 different Wikipedia entries as context. Exactly two of the paragraphs contain the necessary supporting facts that together allow one to determine the answer.11 1 Having to analyze facts across two separate paragraphs is what makes this task ”multi-hop.” The remaining paragraphs are deliberate distractions. The answers are usually one to three words long, and each predicted answer is evaluated against the ground truth using the F1F_1 metric.22 2 Precision is defined as the ratio of the number of tokens common to both the prediction and the ground truth to the number of tokens in the prediction. Recall and F1F_1 are defined similarly. See, e.g., (Wikipedia contributors, 2025). Our workflow is similar to a standard RAG-based workflow using the LangChain orchestration framework (see, e.g., (LangChain Development Team, 2025)). Its essential components are as follows: All the provided paragraphs are first concatenated and then split into chunks. An embedding vector is generated for each chunk using one of 7 embedding models (the first operator; see Figure 6(c) for model choices). The size of each chunk is chosen from 5 different options (second operator; called type of VectorStore). The embeddings of the question and the chunks are then used to retrieve the most relevant chunks, again using one of two retrieval mechanisms (third operator). The number of chunks retrieved can also vary, but for implementation reasons, this is a fixed number for each VectorStore option. So, e.g., the VectorStore option (350,8)(350,8) is the option in which the size of each chunk is 350350 characters and 88 of the most relevant chunks are returned by the retriever. The relevant chunks are fed into one of 77 question-answering (QA) models (fourth and last operator).33 3 Our models are general purpose QA models commonly used in HuggingFace/LangChain and not specially fine-tuned to identify supporting facts in the HotPotQA dataset. Finally, our assertion is that the confidence score computed by the QA model should meet a minimum threshold, which was set to 0.650.65 in our experiments. This workflow has a total of 490490 configurations. ICP . This ML workflow is inspired from articles describing KYC pipelines (Bui, 2021) deployed in the industry and open KYC Github repositories (Kunwar, ; Usman, ). This workflow extracts fields from ID card scans from the MIDV-2020 dataset (Bulatov et al., 2021). It crops the images to make them upright and then does various image cleaning operations like changing contrast, sharpness, color balance, and picks color channels to make the images more amenable to OCR. Then, it uses OCR engines and country-specific field extractor models to extract fields like first name, last name, and date of birth. Extracted fields are evaluated against ground truth using the F1F_1 metric. Image cleaning operations are made optional by keeping the original image as one of the operator choices. The OCR operator specifies two different OCR engines as choices. The classifier operator specifies top-2 predictions from a ResNet (He et al., 2015) classifier as choices. The assertions verify several data integrity constraints, such as ensuring that the extracted name is not null and that the extracted date of birth precedes the extracted date of issue. This workflow has a total of 224224 configurations. Choicec = 1 (0.15s),2 (0.2s), 3 (0.2s),4 (0.5s), 5 (1s)Assertionp=0.16*c(a) SC: single operator, multiple choice (5 configs) (a) Choice 1c1 = 0(0.2s),1(0.1s), 2(0.2s),3(0.2s), 4(1s)Choice 2c2 = 0(0.2s),1(0.1s), 2(0.2s),3(0.2s), 4(1s)Assertionp=0.8*(c1⊕ 2)/7(b) MC: multiple operator, multiple choice (25 configs) (b) Figure 7. Synthetic workflows for evaluating FlexiFlow strategies. Synthetic workflows. Since we do not know the optimal sequence of configurations for real-world ML workflows, we also evaluate FlexiFlow strategies with synthetic workflows, as illustrated in Figure 7, so that we can compare FlexiFlow performance to an optimal strategy. We calculate each workflow’s optimal sequence of configurations, based on Equation 2, to evaluate whether and how quickly FlexiFlow strategies learn to provide optimal runtimes. Each synthetic workflow contains one or two choice operators, a single assertion, and the ability to rollback to any choice operator. The pass probability of the assertion depends on the selected choice/(s). In SC, the probability of passing the assertion is kept as 0.16c0.16c, where c is the choice index. Therefore, even the best choice has a probability of 0.80.8 i.e., it does not always pass the assertion. This mimics the behavior of models in real world ML workflows. Each choice is associated with a runtime cost. For instance, choice c=1c=1 in SC incurs a runtime of 0.15 seconds. We chose these values to mimic real world situations where heavier models are often more accurate but also take more time. For example, choice 5 has a higher probability (0.80) of passing the assertion than choice 4 (0.64), but choice 5 takes double the time (1s) to run than the choice 4 (0.5s). For MC, the assertion is based on the combination of choice index values. We again keep the maximum probability as 0.80.8 so that none of the configurations always pass the assertion. To stress test the learning task of FlexiFlow strategies, we keep the choices in the two operators c1c1 and c2c2 dependent on each other by setting the probability 0.8×(c1⊕c2)/70.8×(c1\; \;c2)/7, where ⊕ denotes the bitwise XOR operation. For example, c1=4c1=4 has a 0.80.8 probability of passing assertions when c2=3c2=3 but has a 00 probability with c2=4c2=4. As shown in the figure, each choice operator provides 5 possible choices. While the number of configurations can be varied, we kept a maximum of 25 configurations so that the optimal sequence of configurations could be computed by a brute-force approach with sequence pruning. We omit the details of this brute-force approach due to space limitations. Prior knowledge of optimal ordering allows us to evaluate whether and how quickly FlexiFlow strategies converge to the same runtime as the optimal strategy. We also ran our experiments with other runtimes and pass probabilities in the synthetic workflows. Since the overall results were not much different, we skip them here due to space constraints. Evaluation Goals and Baselines. We evaluate FlexiFlow on three fronts: (1) Do FlexiFlow’s learning strategies quickly learn optimal runtimes for diverse ML workflows? This evaluation helps to assess the learning rate of the single operator and multi operator learning strategies. We evaluate this question using both synthetic and real workflows. In synthetic workflows, the optimal sequence serves as the baseline for comparison. In real workflows, since the optimal sequence is not available, we assess how quickly does a strategy converge to stable regret in the presence of a large number of configurations i.e., determine the scalability of strategies. We also present case studies to show how optimizers learn efficiently even in workflows with hundreds of configurations. (Section 5.1) (2) Do FlexiFlow’s programming abstractions–operator choices and assertions–improve accuracy compared to the single best configuration? We evaluate this on real ML workflows by separately running each workflow with individual configurations and comparing each configuration’s accuracy with the accuracy provided by FlexiFlow. (Section 5.2) (3) Finally, how does FlexiFlow’s performance compare with other similar dataflow systems? For this goal, we use real workflows and compare their performance on FlexiFlow and state-of-the-art dataflow systems Spark (Zaharia et al., 2012) and Flink (Carbone et al., 2017). (Section 5.3) 5.1. Performance of FlexiFlow optimizers In this section, we evaluate whether the optimizers proposed in this paper (OTS, CTS, HTS in Table 1) can lower runtimes for both synthetic and real-world ML workflows. We perform ablation studies by also evaluating OTS-R (OTS without runtime modeling) and CTS-RB (CTS without rollback awareness). 5.1.1. Performance of optimizers on synthetic workflows For each synthetic workflow, we first run it multiple times with its optimal sequence of configurations to get a series of optimal runtimes. Due to probabilistic assertions, choices may fail and cause rollbacks. Hence, the runtimes vary across different runs. Next, with a given optimizer, we run the workflow multiple times, with the optimizer updating itself after each run, to get a series of optimizer runtimes. We then compare the series of optimizer runtimes against that of optimal runtimes. Since runtimes vary across each run, we show runtime trends by plotting a sliding-window runtime average. In addition, we plot the cumulative regret curves for each optimizer. Cumulative regret is measured as the cumulative difference between the achieved runtime and the optimal runtime. Plotted as a series, cumulative regret curves show how fast an optimizer approaches the optimal runtime; a good learning strategy will show a “log-like shape" in its cumulative regret curve with a rapidly decreasing slope. Note that because of probabilistic assertions, learning strategies can sometimes get lucky and finish faster than the optimal strategy, leading to negative regret. (a) SC (b) MC Figure 8. Modeling runtime and being rollback aware is important for strategies to learn. SC We evaluate OTS and OTS-R for SC. We see from Figure 8a that the runtime-agnostic OTS-R is unable to reach the optimal runtime. This is because OTS-R simply maximizes probability of passing assertions. It learns to run the sequence c=[5,4,3,2,1]c=[5,4,3,2,1], but the optimal sequence for SC is c=[3,2,4,1,5]c=[3,2,4,1,5] due to different runtimes of choices. The runtime-aware multi-armed bandit OTS is able to learn the optimal order of choices maintaining a much lower regret than OTS-R. MC We next evaluate CTS, HTS, and CTS-RB for MC. From Figure 8b, we observe that the CTS-RB, that does not take rollback order into account, cannot learn the optimal order. This is because this learning strategy just picks configurations with the lowest expected runtime without considering the computational structure of the workflow. For example, in the last run, CTS-RB tries (c1=3,c2=4)(c_1=3,c_2=4) followed by (c1=4,c2=2)(c_1=4,c_2=2). In the same run, both CTS and HTS try (c1=4,c2=1)(c_1=4,c_2=1) followed by (c1=4,c2=0)(c_1=4,c_2=0). Switching from (c1=4,c2=1)(c_1=4,c_2=1) to (c1=4,c2=0)(c_1=4,c_2=0) avoids re-executing c1=4c_1=4. This shows that rollback awareness is essential for learning in workflows with multiple choice operators. Further, we observe in Figure 8b that HTS learns much faster than CTS, resulting in lower cumulative regret. This implies in HTS augmenting the probability estimations for configurations with the estimations for individual choices makes learning more effective. Our key observations are as follows: (i) accurately modeling both runtime and the probability of assertion success is crucial for determining the optimal execution order; (i) while the OTS strategy is effective for workflows involving a single choice operator, workflows with multiple choice operators require strategies that are aware of potential rollbacks; and (i) the hybrid strategy HTS—combining configuration-level learning with operator-level insights—achieves faster convergence compared to the configuration-only CTS approach. 5.1.2. Performance of optimizers on real-world ML workflows In this section, we shift our focus to real-world workflows and analyze how these strategies impact runtime performance. Specifically, we determine the effect of OTS on on runtimes of single operator workflows, IC and SCQA, and the impact of HTS on runtimes of multiple operator workflows, LCQA and ICP. Each workflow is executed with 1 input per run. After each run, we unload all the ML models from the GPU memory and stop all workers. Since, the optimal sequence of configurations for these real workflows is unknown, we evaluate how quickly learning strategies converge by reporting their cumulative runtime average. We also analyze the final configuration sequence learned by optimizers after convergence. OTS Convergence Behavior on Real Workflows. OTS learns effectively within the first 50 runs and consistently maintains low runtime for both IC and SCQA (figures omitted due to space constraints). For IC, OTS learns to favor the lightweight DeiT-Tiny model over the heavier ViT model. DeiT-Tiny loads and runs inference in 2.0 seconds, compared to 2.4 seconds for ViT, and satisfies the assertion in 82% of cases—making it a strong first choice. However, for SCQA, OTS learns to prioritize the heavier RoBERTa model over the lighter TinyBERT. The optimizer observes that the runtime difference is minimal—1.4s for TinyBERT versus 1.5s for RoBERTa. Moreover, the lightweight model often fails to satisfy the assertion, making the heavier model the better choice. (a) LCQA (b) Cards Figure 9. FlexiFlow optimizers can effectively improve runtimes for real-world ML workflows. HTS Convergence Behavior on Real Workflows. Figure 9a shows that the HTS runtime for the LCQA workflow converges after about 1000 runs. In its final run, the optimizer selects the configuration <all-MiniLM-L6-v2, (700, 4), mmr, google-bert>44 4 Recall a configuration for LCQA specifies operator choices for ⟨ Model, VectorStore chunk size and number of chunks retrieved, Retriever, QA Model⟩ ; also see Figure 6 (c).. Interestingly, this choice is not the one with the highest scores—its α = 36.4 55 5 Recall that HTS sums up a fraction of α and β values from the operator level. They may therefore not be an integer. and the empirical mean of 0.46, rank only third among the 490 configurations. However, because HTS is rollback aware, it explores other potential configurations probabilistically. Thus, after this initial configuration, the optimizer systematically explores the configurations as follows: (i) it tests 6 different QA models; (i) it changes the retriever from mmr to similarity in the next 7 configurations; (i) it runs with the chunk size 400 for the next 14 configurations, and, finally (iv) it changes the embedding model to LaBSE. HTS Converges Similarly but Learning Behavior Differs in the ICP Workflow. Figure 9b shows HTS also improves runtimes for the ICP workflow, though the learned configuration sequence behaves quite differently from LCQA. In LCQA, multiple operators contribute non-trivially to the total runtime. But in ICP, only the OCR operator has a substantial runtime; all the other choices take less than 80ms. Further, it learns that Tesseract passes 9 times less often than EasyOCR. Consequently, instead of following a bottom-up strategy as in LCQA, HTS adapts to the runtime profile of ICP as follows: It delays trying Tesseract until the 17th configuration in the final sequence; The first configuration disables most image-cleaning steps; When that fails, it tries the next prediction from the classifier since it does not need another model inference; Next, it tries modifying sharpness and later switches to a red+green color channel. Overall, we find that FlexiFlow optimizers learn efficiently even in workflows with hundreds of configurations. By leveraging assertion success rates, operator runtimes, and workflow structure, the optimizers adapt their strategies—favoring lightweight models when feasible and selecting more reliable ones when necessary. 5.2. Accuracy of FlexiFlow Figure 10. Effectiveness of using operator choices in ML workflows. FlexiFlow achieves a higher F1F_1 score than the best configuration. We now examine how and why FlexiFlow delivers more accurate results by considering the question answering workflow LCQA and the document processing workflow ICP. We compare the accuracy of FlexiFlow with that of individual workflow configurations run separately on identical input. Instead of individually running the large number of configurations, we run a reduced version of the workflows. For LCQA, we use two of the seven embedding models: all-MiniLM-L6-v2 and LaBSE, and two of the seven QA models: xlm-roberta-large and tinyroberta, totaling 40 configurations. For ICP, we remove contrast, sharpness, and color balance image cleaning operators, and remove red+blue and red+green options from ChannelPicker, totaling 20 configurations (refer Figure 6(d)). FlexiFlow version also runs these reduced workflows. FlexiFlow achieves an F1F_1 score of 0.480.48 (see Figure 10 (a)), a relative improvement of 23% over the best single underlying configuration—⟨ -MiniLM-L6-v2, (350, 8), similarity, xlm-roberta-large⟩ , which only achieves a score of 0.390.39.66 6 We chose general purpose QA Models rather than ones specially fine-tuned for HotpotQA because our primary objective here is to evaluate the performance of FlexiFlow w.r.t. a given a set of configurations. As such, our models score lower than the state-of-the-art for this dataset. As the figure shows, FlexiFlow is better in both precision and recall. The reason for the significant improvement in performance is that different underlying configurations are uncorrelated—they differ in the examples they get wrong. For instance, Figure 11 shows that the configuration with the best F1F_1 metric gets 174 questions correct (using the exact match metric). Among the remaining questions, a different configuration ⟨ , (560,5), similarity, tinyroberta⟩ gets 169 questions correct. 77 7 Observe that the former configuration had the best F1F_1 score, while the latter configuration had a higher exact match score. ⟨ -MiniLM-L7-v2, (350, 8),similarity, xlm-roberta-large⟩ ⟨ , (560, 5),similarity, tinyroberta⟩ 11416960 Figure 11. Failure modes of LCQA configurations are uncorrelated. Further, the estimated confidence score is a reliable indicator of the likelihood of the configuration being correct, which is exploited by FlexiFlow to switch to a different configuration when needed. To illustrate, here is a question and the corresponding answers provided by 4 of the configurations, in increasing order of confidence. The incorrect answers have been crossed out. Q: “Human Error” is the season finale of the third season of a TV show that aired on what network? A1A_1: USA Network. ⟨ -MiniLM-L6-v2, (1400, 2), mmr, xlm-roberta-large⟩ with confidence 0.000.00. A2A_2: House. ⟨ -MiniLM-L6-v2, (350, 8), mmr, xlm-roberta-large⟩ with confidence 0.050.05. A3A_3: ABC. ⟨ -MiniLM-L6-v2, (350, 8), similarity, xlm-roberta-large⟩ with confidence 0.300.30. A4A_4: Fox. ⟨ , (700, 4), mmr, xlm-roberta-large⟩ with confidence 0.760.76. The configuration corresponding to A3A_3 is actually the single best configuration overall (recall that it achieves an F1F_1 of 0.390.39). Yet, here it has low confidence in its answer, which also happens to be incorrect. Further, it is not the QA model that needs to be changed, but the embedding model and the VectorStore (use larger chunks)—see configuration corresponding to A4A_4. Contrary to intuition, it is not always the larger models that provide the correct answer or have the most confidence. For the question “The Oberoi family is part of a hotel company that has a head office in what city?” the heavier model in ⟨ -MiniLM-L6-v2, (350, 8), similarity, xlm-roberta-large⟩ does extract the correct answer (“Delhi”) but with a confidence of only 0.01. So FlexiFlow turns to the lighter model in ⟨ -MiniLM-L6-v2, (350, 8), similarity, tinyroberta⟩ , which gives the same answer but with a confidence of 0.980.98. Similarly for ICP, Figure 10(b) shows that the different ICP configurations can reach a maximum F1F_1 metric of 0.90, but when using operator choices with FlexiFlow, the F1F_1 metric reaches 0.94. The reason for this improvement is again uncorrelated failures between the configurations. We skip more qualitative analysis due to space constraints. Overall, we conclude that using operator choices using FlexiFlow could improve performance by up to 23%. 5.3. Runtime Performance of FlexiFlow Since existing systems lack native support for operator choices, assertions, and rollbacks, the accuracy advantage shown above could be achieved only by running the workflow configurations sequentially. We create this setup by writing Spark and Flink jobs for each of the 20 configurations in the reduced ICP workflow described above. All the cards are input to the workflow configuration with the highest F1F_1 score. The outputs are checked with the same assertion as in Figure 6(d). Cards that fail the assertion are input to the next configuration. This iterative process is continued until all the configurations are exhausted, effectively simulating rollback. FlexiFlow was evaluated using the same input cards to compare the total processing time. Each system is limited to 10 CPU cores. Each experiment was repeated 5 times. The variation in runtimes was negligible. Figure 12. Comparing runtime of FlexiFlow with simulating rollback in other dataflow systems. Figure 12 compares the total runtime for FlexiFlow with sequential execution of workflow configurations in Spark and Flink. Runtime of each execution in the sequence is shown in a different color for Spark and Flink. The figure shows that FlexiFlow completes the run 48% faster than sequential executions in Flink and 49% faster than those in Spark. The reason for this improvement is that FlexiFlow can reuse intermediate results during rollbacks and can better leverage data parallelism as it can run different configurations for different inputs simultaneously. We also compare the general performance of FlexiFlow with Spark and Flink by running just the best ICP configuration while disabling all the operator choices and assertions from FlexiFlow on identical inputs and setup. Figure 12 shows this as FlexiFlow (1). FlexiFlow can finish this run 15% faster than Flink and 22% faster than Spark88 8 This performance difference might be because our implementations are written in Python. This may cause Python-JVM interaction overheads for Spark and Flink.. Overall, we conclude that the general performance of FlexiFlow is comparable to state-of-the-art systems. The native support for operator choices, assertions, and rollbacks in FlexiFlow can improve runtimes by 48% compared to sequentially executing workflow configurations. 6. Discussion FlexiFlow reduces programmer burden by letting them specify operator choices instead of maintaining multiple workflow instances. FlexiFlow allows a diversity of choices. FlexiFlow reduces management burden of prioritizing between multiple workflow instances using bandit-based strategies. This paper only takes the first step in letting programmers rely on FlexiFlow strategies to pick the right choice at each operator. In future work, the proposed strategies can also take input data characteristics into account to potentially learn different optimal sequences of configurations for data with different characteristics (e.g., cards from different countries in ICP). FlexiFlow is most effective when failure modes of configurations are uncorrelated. But, failures may not be completely uncorrelated. For example, when a heavy model fails to predict correctly, it might be more likely that the light model will also fail. FlexiFlow strategies do not separately learn conditional probability of passing assertions which may be crucial for such cases. Modeling conditional probabilities without managing large amounts of state is also part of our future work. FlexiFlow strategies currently focus on lowering runtimes. But many batch workflows are more dollar-cost sensitive than they are runtime sensitive, such as call commercial LLMs less often. We believe a similar problem formulation may suffice for reducing dollar costs. 7. Related Work Workflow and DataFlow Systems. FlexiFlow is a dataflow system where the operator order within DAG defines the data dependencies. The operators follow pipeline execution in that operators execute as soon as their input data is available. This is similar to Dataflow systems such as MapReduce (Dean and Ghemawat, 2004) and others (Isard et al., 2007; Zhou et al., 2012). However these systems do not have support for alternative operator choices, assertions, and rollbacks to improve on overall workflow accuracy. Timely dataflow (Murray et al., 2013) allows for some form of rollback, but without alternative operator choices. The rollback and incremental computation in FlexiFlow is similar to incremental updates in incremental view maintenance (Blakeley et al., 1986; Zhuge et al., 1995; Nikolic et al., 2016; Ahmad et al., 2012), streaming computation (Akidau et al., 2015; Budiu et al., 2023), and provenance-based selective replay (Ikeda et al., 2011). These systems typically handle updates at the input, such as base table updates in incremental view maintenance or stream appends in streaming computation. In contrast, FlexiFlow allows updates at intermediate outputs during rollbacks. Programming Models. Operator choices are similar to McCarthy’s ambiguous functions (Amb) (McCarthy, 1959) where the language runtime tries to pick an alternative to satisfy assertions. Operator choices are also similar to choice blocks in template programs (Solar-Lezama et al., 2006; Alur et al., 2013) where programmers just write a template and let an SMT solver choose the right block so that the overall program meets a specification. To the best of our knowledge, FlexiFlow is the first to apply the concept of operator choices within the context of a dataflow system. The target use case of FlexiFlow is different from previous work. DataFlow Debugging. Systems such as Amber (Kumar et al., 2020) and those based on Spark including Titian (Interlandi et al., 2015), BigSift (Gulzar et al., 2018), BigDebug (Gulzar et al., 2016), and TagSniff (Contreras-Rojas et al., 2019) support debugging primitives such as breakpoints (to pause and resume execution) and watchpoints (to inspect intermediate data). While these can be used for error detection, often they require manual changes for error correction. Similarly, model assertions perform runtime checks to detect model errors at runtime (Kang et al., 2020). However, model assertions work in the context of a single model whereas we detect errors on overall workflows and correct them by rolling back to alternate workflow configurations. Online Learning. FlexiFlow strategies build upon Thompson Sampling (Thompson, 1933; Sutton and Barto, 2018; Russo et al., 2020), which samples all actions that could plausibly be optimal, while shifting sampling away from those that are unlikely to be optimal. Theoretically, TS has been shown to achieve asymptotically optimal logarithmic regret in stochastic MAB settings (Russo et al., 2020). While a formal regret analysis for our approach is beyond the scope of the current paper, experimentally we benefit from learning the right choices. We also note that, while there are situations that do not require exploitation or exploration, or situations where it is better to exploit a high performing suboptimal action than to invest resources exploring actions, often modeling such situations depends upon availability of some high-quality historical data. We currently do not assume access to any such historical data and we learn in an online setting. Confidence Estimation. When the assertion in FlexiFlow is a confidence threshold (as in IC, SCQA, LCQA), our system is related to the work done in confidence estimation which is applied to, e.g., out-of-distribution detection and selective prediction. Examples of such work include Bayesian neural networks, ensemble of neural networks, and temperature scaling (see, e.g., the surveys (Gawlikowski et al., 2023; Geng et al., 2023). Unlike our work, these do not attempt to mitigate the runtime cost of the ensemble by learning an optimal sequence of models to try. 8. Conclusion We showed that in many ML pipelines, relying on a single model to perform inference across all data batches often fails to maximize accuracy and thus overall performance. Our FlexiFlow system, incorporates operator choices, assertions, and rollbacks and uses a novel multi-armed bandit approach to perform automatic model switching and thus improves overall accuracy and efficiency of various ML pipelines such as document processing, image classification, and question-answering tasks. Moving forward, our approach opens new avenues for AutoML-based systems that autonomously adapt to evolving data distributions. Future work will explore incorporating intermediate data characteristics, such as the country of an ID card, into the learning strategies. Ultimately, FlexiFlow represents a step toward more intelligent, data-aware workflow systems that reduce manual intervention and enhance the reliability of ML-driven automation. References Ahmad et al. (2012) Y. Ahmad, O. Kennedy, C. Koch, and M. Nikolic DBToaster: higher-order delta processing for dynamic, frequently fresh views. Proc. VLDB Endow. 5 (10), p. 968–979. External Links: ISSN 2150-8097, Link, Document Cited by: §7. Akidau et al. (2015) T. Akidau, R. Bradshaw, C. Chambers, S. Chernyak, R. J. Fernández-Moctezuma, R. Lax, S. McVeety, D. Mills, F. Perry, E. Schmidt, and S. Whittle The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proc. VLDB Endow. 8 (12), p. 1792–1803. External Links: ISSN 2150-8097, Link, Document Cited by: §7. Alur et al. (2013) R. Alur, R. Bodik, G. Juniwal, M. M. K. Martin, M. Raghothaman, S. A. Seshia, R. Singh, A. Solar-Lezama, E. Torlak, and A. Udupa Syntax-guided synthesis. In 2013 Formal Methods in Computer-Aided Design, p. 1–8. External Links: Document Cited by: §7. Armbrust et al. (2015) M. Armbrust, R. S. Xin, C. Lian, Y. Huai, D. Liu, J. K. Bradley, X. Meng, T. Kaftan, M. J. Franklin, A. Ghodsi, and M. Zaharia Spark SQL: Relational Data Processing in Spark. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (SIGMOD), Melbourne, Australia, p. 1383–1394. External Links: Document, Link Cited by: §2. Berger and Berger (1985) J. O. Berger and J. O. Berger Statistical decision theory and bayesian analysis. Springer-Verlag, New York. External Links: ISBN 0387960988 9780387960982 3540960988 9783540960980, Link Cited by: §4.2. Blakeley et al. (1986) J. A. Blakeley, P. Larson, and F. W. Tompa Efficiently updating materialized views. SIGMOD Rec. 15 (2), p. 61–71. External Links: ISSN 0163-5808, Link, Document Cited by: §7. Budiu et al. (2023) M. Budiu, T. Chajed, F. McSherry, L. Ryzhyk, and V. Tannen DBSP: automatic incremental view maintenance for rich query languages. Proc. VLDB Endow. 16 (7), p. 1601–1614. External Links: ISSN 2150-8097, Link, Document Cited by: §7. Bui (2021) D. T. Bui Applications of machine learning in ekyc’s identity document recognition. Cited by: §5. Bulatov et al. (2021) K. Bulatov, E. Emelianova, D. Tropin, N. Skoryukina, Y. Chernyshova, A. Sheshkus, S. Usilin, Z. Ming, J. Burie, M. Luqman, and V. Arlazarov MIDV-2020: a comprehensive benchmark dataset for identity document analysis. External Links: Document Cited by: 6c, §5. Carbone et al. (2017) P. Carbone, S. Ewen, G. Fóra, S. Haridi, S. Richter, and K. Tzoumas State management in apache flink®: consistent stateful distributed stream processing. Proceedings of the VLDB Endowment 10 (12), p. 1718–1729. Cited by: item 3. Contreras-Rojas et al. (2019) B. Contreras-Rojas, J. Quiané-Ruiz, Z. Kaoudi, and S. Thirumuruganathan Tagsniff: simplified big data debugging for dataflow jobs. In Proceedings of the ACM Symposium on Cloud Computing, p. 453–464. Cited by: §7. Dean and Ghemawat (2004) J. Dean and S. Ghemawat MapReduce: simplified data processing on large clusters. OSDI’04, p. 10. Cited by: §7. DeGroot and Schervish (2011) M. H. DeGroot and M. J. Schervish Probability and statistics. 4th edition, Pearson Education, Upper Saddle River, NJ. External Links: ISBN 9780321500465 Cited by: §4.1. Deng et al. (2009) J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei ImageNet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, p. 248–255. Cited by: 6a, §5. Gao et al. (2008) J. Gao, B. Ding, W. Fan, J. Han, and S. Y. Philip Classifying data streams with skewed class distributions and concept drifts. IEEE Internet Computing 12 (6), p. 37–49. Cited by: §1. Gawlikowski et al. (2023) J. Gawlikowski, C. R. N. Tassi, M. Ali, et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review 56 (Suppl 1), p. 1513–1589. External Links: Document, Link Cited by: §7. Geng et al. (2023) J. Geng, F. Cai, Y. Wang, H. Koeppl, P. Nakov, and I. Gurevych A survey of confidence estimation and calibration in large language models. Note: arXiv preprint arXiv:2311.08298v2Last revised March 25, 2024 External Links: Link Cited by: §7. Gulzar et al. (2016) M. A. Gulzar, M. Interlandi, S. Yoo, S. D. Tetali, T. Condie, T. Millstein, and M. Kim Bigdebug: debugging primitives for interactive big data processing in spark. In Proceedings of the 38th International Conference on Software Engineering, p. 784–795. Cited by: §7. Gulzar et al. (2018) M. A. Gulzar, S. Wang, and M. Kim Bigsift: automated debugging of big data analytics in data-intensive scalable computing. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, p. 863–866. Cited by: §7. He et al. (2015) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. CoRR abs/1512.03385. External Links: Link, 1512.03385 Cited by: §5. Hennessy and Patterson (2017) J. L. Hennessy and D. A. Patterson Computer architecture: a quantitative approach. 6th edition, Morgan Kaufmann. External Links: ISBN 9780128119051 Cited by: §2. Ikeda et al. (2011) R. Ikeda, S. Salihoglu, and J. Widom Provenance-based refresh in data-oriented workflows. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM ’11, p. 1659–1668. External Links: ISBN 9781450307178, Link, Document Cited by: §7. Interlandi et al. (2015) M. Interlandi, K. Shah, S. D. Tetali, M. A. Gulzar, S. Yoo, M. Kim, T. Millstein, and T. Condie Titian: data provenance support in spark. Proc. VLDB Endow. 9 (3), p. 216–227. External Links: ISSN 2150-8097, Link, Document Cited by: §7. Isard et al. (2007) M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly Dryad: distributed data-parallel programs from sequential building blocks. In Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, p. 59–72. Cited by: §7. Jiao et al. (2019) X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu TinyBERT: distilling BERT for natural language understanding. CoRR abs/1909.10351. External Links: Link, 1909.10351 Cited by: 6a, §5. Jindal et al. (2024) A. Jindal, K. Beedkar, V. Singh, J. N. Mohammed, T. Singla, A. Gupta, and K. Choudhary Reactive dataflow for inflight error handling in ml workflows. In Proceedings of the Eighth Workshop on Data Management for End-to-End Machine Learning, p. 51–61. Cited by: 1st item, §5. Kang et al. (2020) D. Kang, D. Raghavan, P. Bailis, and M. Zaharia Model assertions for monitoring and improving ml models. Proceedings of Machine Learning and Systems 2, p. 481–496. Cited by: §1, §7. Kumar et al. (2020) A. Kumar, Z. Wang, S. Ni, and C. Li Amber: a debuggable dataflow system based on the actor model. Proc. VLDB Endow. 13 (5), p. 740–753. External Links: ISSN 2150-8097, Link, Document Cited by: §7. [29] S. Kunwar Github: ocr id card parser. Note: https://github.com/sumn2u/ocr-id-card-parserAccessed: 2025-06-27 Cited by: §5. LangChain Development Team (2025) LangChain Development Team Build a retrieval augmented generation (rag) app: part 1. LangChain. Note: Accessed: 2025-06-27 External Links: Link Cited by: §5. Liu et al. (2016) W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Fu, and A. C. Berg SSD: Single Shot MultiBox Detector. In Proceedings of the European Conference on Computer Vision (ECCV), p. 21–37. External Links: Document Cited by: §1. Liu et al. (2019) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov RoBERTa: A robustly optimized BERT pretraining approach. CoRR abs/1907.11692. External Links: Link, 1907.11692 Cited by: 6a, §5. McCarthy (1959) J. McCarthy A basis for a mathematical theory of computation. In Studies in Logic and the Foundations of Mathematics, Vol. 26, p. 33–70. Cited by: §7. Murray et al. (2013) D. G. Murray, F. McSherry, R. Isaacs, M. Isard, P. Barham, and M. Abadi Naiad: a timely dataflow system. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles (SOSP 13), p. 439–455. Cited by: §7. Nikolic et al. (2016) M. Nikolic, M. Dashti, and C. Koch How to win a hot dog eating contest: distributed incremental view maintenance with batch updates. In Proceedings of the 2016 International Conference on Management of Data, p. 511–526 (en). External Links: ISBN 978-1-4503-3531-7, Link, Document Cited by: §7. Rajpurkar et al. (2018) P. Rajpurkar, R. Jia, and P. Liang Know what you don’t know: unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), I. Gurevych and Y. Miyao (Eds.), Melbourne, Australia, p. 784–789. External Links: Link, Document Cited by: 6a, §5. Redmon et al. (2016) J. Redmon, 绍. Divvala, R. Girshick, and A. Farhadi You only look once: unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 779–788. Cited by: §1. Ren et al. (2015) S. Ren, K. He, R. Girshick, and J. Sun Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Advances in Neural Information Processing Systems, Vol. 28. Cited by: §1. Russo et al. (2020) D. Russo, B. V. Roy, A. Kazerouni, I. Osband, and Z. Wen A tutorial on thompson sampling. External Links: 1707.02038, Link Cited by: §4, §7. Solar-Lezama et al. (2006) A. Solar-Lezama, L. Tancau, R. Bodik, S. Seshia, and V. Saraswat Combinatorial sketching for finite programs. In Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XII, p. 404–415. External Links: ISBN 1595934510, Link, Document Cited by: §7. [41] Source code and experiments. Note: https://anonymous.4open.science/r/flexiflow-5BC6/ Cited by: §5. Sutton and Barto (2018) R. S. Sutton and A. G. Barto Reinforcement learning: an introduction. Second edition, The MIT Press. External Links: Link Cited by: §1, §4.1, §4, §7. Thompson (1933) W. R. Thompson On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika 25 (3/4), p. 285–294. External Links: Document, Link Cited by: §7. Touvron et al. (2021) H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou Training data-efficient image transformers and distillation through attention. In International Conference on Machine Learning, Vol. 139, p. 10347–10357. Cited by: 6a, §5. [45] M. Usman Github: id card ocr. Note: https://github.com/MuhamaadUsman/ID-Card-OCRAccessed: 2025-06-27 Cited by: §5. Wikipedia contributors (2025) Wikipedia contributors Precision and recall. Note: https://en.wikipedia.org/wiki/Precision_and_recallAccessed: 2025-06-27 Cited by: footnote 2. Wu et al. (2020) B. Wu, C. Xu, X. Dai, A. Wan, P. Zhang, Z. Yan, M. Tomizuka, J. Gonzalez, K. Keutzer, and P. Vajda Visual transformers: token-based image representation and processing for computer vision. External Links: 2006.03677 Cited by: 6a, §5. Yang et al. (2018) Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: 6b, §5. Zaharia et al. (2012) M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauly, M. J. Franklin, S. Shenker, and I. Stoica Resilient distributed datasets: a \fault-tolerant\ abstraction for \in-memory\ cluster computing. In 9th USENIX symposium on networked systems design and implementation (NSDI 12), p. 15–28. Cited by: item 3. Zhou et al. (2012) J. Zhou, N. Bruno, M. Wu, P. Larson, R. Chaiken, and D. Shakib SCOPE: parallel databases meet MapReduce. The VLDB Journal 21 (5), p. 611–636. External Links: ISSN 1066-8888, 0949-877X, Document Cited by: §2, §5, §7. Zhuge et al. (1995) Y. Zhuge, H. García-Molina, J. Hammer, and J. Widom View maintenance in a warehousing environment. SIGMOD Rec. 24 (2), p. 316–327. External Links: ISSN 0163-5808, Link, Document Cited by: §7.

Paper	Published	Area	Tags	Intel	Citations