Paper deep dive
Less Is More: Tuning Configurable Systems with Imperfect Fidelity
Yulong Ye, Miqing Li, Tao Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/4/2026, 5:03:40 AM
Summary
The paper introduces MFTune, a configuration tuning tool that leverages multi-fidelity optimization to improve budget utilization. It proposes a conceptual framework defining fidelity factors (budget, workload, dataset-related) for configurable systems, challenging the monotonic cost-perfection assumption found in hyperparameter optimization. MFTune proactively explores a large space of imperfect-fidelity settings to find high-quality seeds that accelerate tuning under the target perfect-fidelity environment, demonstrating superior performance over state-of-the-art tuners.
Entities (8)
Relation Signals (9)
Imperfect-Fidelity → ischeaperthan → Perfect-Fidelity
confidence 95% · imperfect-fidelity—an environment that is similar, but cheaper to measure, compared with the concerned perfect-fidelity
PostgreSQL → isexampleof → Configurable System
confidence 95% · For instance, we found that when tuning the database system PostgreSQL
MFTune → usesmethodology → Multi-Fidelity Optimization
confidence 95% · MFTune is a fidelity-aware configuration tuning tool... proactively exploits/extracts the multi-dimensional fidelity space
MFTune → exploresspaceof → Imperfect-Fidelity Settings
confidence 92% · proactively explores in the space of >10^4 possible imperfect-fidelity settings
MFTune → generatesseedsfor → Perfect-Fidelity
confidence 90% · This creates high-quality seeds for the perfect-fidelity, which in turn ensures the tuning depth.
MFTune → outperforms → State-of-the-Art Tuners
confidence 90% · MFTune performs considerably better on 83.33% cases with up to 19.34% improvement
Sysbench → isusedby → PostgreSQL
confidence 85% · Figure 1 illustrates the idea where the fidelity of environment is set by the limits of execution time and table size in Sysbench
Configurable Systems → violatesassumptionof → Multi-Fidelity Optimization
confidence 85% · While such an assumption is reasonable in HPO, it does not necessarily hold for configurable systems.
Multi-Fidelity Optimization → assumesmonotonicrelation → Cost and Perfection
confidence 80% · Most existing multi-fidelity tuners assume a monotonic relationship between fidelity perfection and cost
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Configuration tuning is essential for optimizing the performance of highly configurable systems, e.g., throughput or runtime, under a given environment. Yet, this is a challenging process as there can be many options to tune, and configuration measurement is often highly expensive. In this paper, we demonstrate the phenomenon of ``less can be more'': system configuration tuning can be greatly improved with much superior budget utilization by partially tuning under the imperfect-fidelity---an environment that is similar, but cheaper to measure, compared with the concerned perfect-fidelity of environment under which the system should be tuned. We codify a conceptual framework of fidelity for configurable systems, drawing on which allows us to propose MFTune, a tuner that proactively explores in the space of $>10^4$ possible imperfect-fidelity settings to approximate a useful one, which strikes for the wideness of tuning. This creates high-quality seeds for the perfect-fidelity, which in turn ensures the tuning depth. Experiment results against $10$ state-of-the-art tuners, obtained from running diverse real-world systems for $19$ months $24 \times 7$, show that MFTune performs considerably better on $83.33$\% cases with up to $19.34\%$ improvement while achieving hours of budget saving in general.
Tags
Links
- Source: https://arxiv.org/abs/2608.00759v1
- Canonical: https://arxiv.org/abs/2608.00759v1
Trouble viewing inline? Open PDF directly →
Full Text
87,601 characters extracted from source content.
Expand or collapse full text
by Less Is More: Tuning Configurable Systems with Imperfect Fidelity Yulong Ye 0009-0007-4856-8577 IDEAS Lab University of BirminghamBirminghamUnited Kingdom YXY382@student.bham.ac.uk , Miqing Li 0000-0002-8607-9607 University of BirminghamBirminghamUnited Kingdom m.li.8@bham.ac.uk and Tao Chen 0000-0001-5025-5472 IDEAS Lab University of BirminghamBirminghamUnited Kingdom t.chen@bham.ac.uk (2026-06-18) Abstract. Configuration tuning is essential for optimizing the performance of highly configurable systems, e.g., throughput or runtime, under a given environment. Yet, this is a challenging process as there can be many options to tune, and configuration measurement is often highly expensive. In this paper, we demonstrate the phenomenon of “less can be more”: system configuration tuning can be greatly improved with much superior budget utilization by partially tuning under the imperfect-fidelity—an environment that is similar, but cheaper to measure, compared with the concerned perfect-fidelity of environment under which the system should be tuned. We codify a conceptual framework of fidelity for configurable systems, drawing on which allows us to propose MFTune, a tuner that proactively explores in the space of >104>10^4 possible imperfect-fidelity settings to approximate a useful one, which strikes for the wideness of tuning. This creates high-quality seeds for the perfect-fidelity, which in turn ensures the tuning depth. Experiment results against 1010 state-of-the-art tuners, obtained from running diverse real-world systems for 1919 months 24×724× 7, show that MFTune performs considerably better on 83.3383.33% cases with up to 19.34%19.34\% improvement while achieving hours of budget saving in general. Configurable systems, configuration performance tuning, hyperparameter optimization, multi-fidelity optimization †copyright: c†doi: 10.1145/3832783.3834376†journalyear: 2026†isbn: 979-8-4007-2882-2/2026/10†conference: Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering; October 12–16, 2026; Munich, Germany†booktitle: Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE ’26), October 12–16, 2026, Munich, Germany†submissionid: ase26main-p988-p†ccs: Software and its engineering Software performance†ccs: Software and its engineering Search-based software engineering 1. Introduction Modern software systems are often highly configurable, exposing numerous tunable configuration options (Liang et al., 2025; Gong et al., 2025). The goal thereof is intuitive: by allowing flexible configuration, the system can achieve greater applicability across diverse domains and cater to varying performance requirements, e.g., runtime and throughput (Chen and Bahsoon, 2015; Chen et al., 2018a; Xiang et al., 2026). Yet, excessive configurability comes with its own costs: it has been shown that globally 59% of the performance issues, where performance requirements were severely violated, are attributed to poorly chosen configurations rather than code (Han and Yu, 2016). With proper configuration, systems could unlock their full performance potential (Jamshidi and Casale, 2016). Theoretically, the optimal configuration could be identified by exhaustively profiling the system across all possible configurations under a certain environment, e.g., a setting of anticipated workload or job. This, however, is impractical because measuring even a single configuration can be rather costly, taking considerable time and computational resources (e.g., minutes to hours) (Chen and Li, 2021, 2024; Chen et al., 2025). The tuning efficiency is further exacerbated by the fact that the configuration space is typically high-dimensional and grows exponentially with the number of configuration options (Ye et al., 2026; Xiong and Chen, 2025; Chen et al., 2018b). For example, MySQL—a database system—offers dozens of tunable configuration options, making exhaustive profiling infeasible (Zhang et al., 2022; Xu et al., 2015; Ma et al., 2025). Existing system tuners often focus on advanced algorithm designs with smart heuristics (e.g, BestConfig (Zhu et al., 2017)), hoping that the tuner would find the promising configurations soon; or leverage a surrogate model to predict configuration performance (e.g., PromiseTune (Chen and Chen, 2026b)), which can be unreliable at times. The key unaddressed question is: how to efficiently utilize the budget in tuning? In this paper, we take a different perspective to explicitly tackle this challenge: we view tuning a configurable system under different environments as exhibiting a certain degree of exactness to each other, resembling the notion of fidelity. Therefore, we hypothesize that proactively exploring good configurations found from an imperfect, cheaper environment under which the system is tuned could be beneficial to tuning it for the perfect, more expensive target environment. For example, configurations found by tuning PostgreSQL under a workload of 100100 requests per second might expedite and benefit the tuning under that with 10,00010,000 requests per second—a concept we borrowed from the paradigm known as multi-fidelity optimization (Klein et al., 2017; Kandasamy et al., 2017; Hu et al., 2019). Noteworthily, while similar in spirit, multi-fidelity optimization differs from prior work on knowledge transfer of environments for configuration tuning (Zhang et al., 2021; Aken et al., 2017; Zhang et al., 2023): the former proactively explores and exploits unknown fidelity settings of environments for tuning, whereas the latter relies on reusing what is available from historically explored and structurally similar environments, limiting their ability to adapt to a completely unforeseen environment. Figure 1 illustrates the idea where the fidelity of environment is set by the limits of execution time and table size in Sysbench (58). Suppose our “target perfection” is to tune configuration under fidelity-A (hence we are only interested in testing therein), Figure 1(a) shows that tuning under fidelity-B produces almost similarly-performing configurations (solid cycle line), when being tested in fidelity-A, to those produced by tuning/testing under fidelity-A directly (solid square line), but is ≈36≈ 36 hours faster to complete—it might well exceed the result of tuning/testing under fidelity-A if given the same wall-clock time. However, this is not straightforward as not all fidelity settings can be useful for fidelity-A, especially given the complex dimensions related to environments and fidelity settings for configurable systems: in Figure 1(b), even though the time saving remains significant, the tuning guided in fidelity-C has severely misled the results when the configurations found therein are tested under fidelity-A (solid cycle line vs. solid square line). This is a core issue that has yet been well-addressed in current multi-fidelity optimization. [width=]figures/pre-exp1 (a) Helpful fidelity [width=]figures/pre-exp2 (b) Harmful fidelity Figure 1. Tuning PostgreSQL under different environments/fidelity settings via a random search tuner. fidelity-A (time=180s; size=5000k) is the “target perfection”; fidelity-B (time=30s; size=5000k) and fidelity-C (time=60s; size=1050k) are two imperfect fidelity settings. “Tuning in fidelity-A test for fidelity-B” means that the tuning is guided by configurations measured under fidelity-A and the best one found therein is progressively tested under fidelity-B. Indeed, multi-fidelity optimization is an established paradigm (Li et al., 2017; Falkner et al., 2018; Awad et al., 2021; Klein et al., 2017; Kandasamy et al., 2017), particularly for hyperparameter optimization (HPO) (Hutter et al., 2011; Snoek et al., 2012; Hu et al., 2023). For example, when tuning the hyperparameters for deep neural network training, one can reduce the number of epochs in the training to achieve a low-fidelity measurement, which provides a cheaper yet informative approximation of the final model performance under a perfect, often much higher, epoch setting of fidelity (Li et al., 2021). Yet, despite those advances, adopting this paradigm for general system configuration tuning is not easy, because: 1) Mismatched Problem Formulation: Unlike HPO, system configuration tuning lacks a general multi-fidelity problem formulation that defines how fidelity should be modeled and controlled: in HPO, since machine learning models follow similar pipeline, it is well-known that fidelity is related to factors that are highly influential to training time (i.e., the cost in this context), such as training epochs (Domhan et al., 2015) and training sample size (Hu et al., 2019; Klein et al., 2017). Yet, given the high variety of system domains, there is no known general definition of fidelity for system configuration tuning. 2) Incompatible Tuner Assumption: Most existing multi-fidelity tuners assume a monotonic relationship between fidelity perfection and cost, i.e., more costly measurements always yield a higher exactness approximation to the target perfect environment (Echevarrieta et al., 2025; Carstensen et al., 2025). While such an assumption is reasonable in HPO, it does not necessarily hold for configurable systems. For instance, training with more epochs generally yields measurements that more closely approximate the performance under the perfect full epochs; in contrast, we found that when tuning the database system PostgreSQL, extending the benchmarking duration does not always yield measurements that better approximate those obtained under a higher, perfect duration runs (see §3.3). Moreover, the multi-fidelity tuners in HPO often rely on a restricted single factor to determine fidelity setting (e.g., the epoch) (Li et al., 2017; Falkner et al., 2018; Li et al., 2021; Hu et al., 2019). Such a design overlooks the possibility that more cost-effective fidelity settings can exist in the multi-dimensional fidelity spaces exhibited in configurable systems (see §3.3). To fill the above gap and fully exploit the benefits of fidelity for system configuration tuning, we first present a tailored conceptual framework to codify the definition of fidelity for configurable systems. Drawing on this, we propose MFTune, a fidelity-aware configuration tuning tool specifically for systems. What makes MFTune unique is that it proactively exploits/extracts the multi-dimensional fidelity space of more than ten thousands fidelity settings, finding a fair one (i.e., a conceptual “knee point” that represents a desirable trade-off between cost and approximation of the perfect-fidelity111We use the term perfect-fidelity to denote the target fidelity setting of environment under which one prefers to tune a configurable system.) that can accelerate the tuning under the perfect-fidelity setting via a shared archive, hence striking for both the “wideness” and “depth” of configuration tuning with improved budget utilization. In a nutshell, our contributions are: • We propose a unified and extensible conceptual framework to define the notions of fidelity for configurable systems, leading to a new problem formulation (§3). • Deriving from the above framework, we present an active fidelity discovery strategy in MFTune to quantify and extract cheaper, yet useful fidelity settings with respect to the perfect-fidelity setting (§4.1). • MFTune embeds a mechanism that synergizes tuning between a selected imperfect- and the perfect-fidelity settings with diversity preservation: tuning under the imperfect-fidelity setting, which is less costly yet with a good level of perfection, aims to cover a wide, promising area of the configuration space with good diversity, finding high-quality archived seeds for the perfect-fidelity setting. In contrast, tuning under the perfect-fidelity setting seeks to fully leverage the rich seeds discovered under the imperfect-fidelity setting, encouraging a deep investigation of the tuning directions implied (§4.2–§4.3). • We assess MFTune on six real-world configurable systems against 10 state-of-the-art tuners scaling up to 11,20011,200 fidelity settings (§5–§6). The results suggest that MFTune considerably outperforms the others, ranking the best on the majority of the systems (5/6) with up to 19.3419.34% improvement while generally saving hours of budget. All source code and data are available at: https://github.com/ideas-labo/mftune. 2. Preliminaries 2.1. System Configuration Tuning A configurable system often comprises a set of configuration options, each taking categorical or numerical values. The objective is to identify a configuration that optimizes a specific performance metric (e.g., minimizing runtime or maximizing throughput) of the system under a target environment. This can be formulated as: (1) argminf() or argmaxf(), f( x) or f( x), s.t. ∑∈τ()≤ℬ, s.t. _ x∈ Xτ( x) , where x = (x1,x2,…,xdx_1,x_2,…,x_d) is a configuration with the values of d options in configuration space X. f is the concerned performance metric. τ()τ( x) denotes the cost of measuring a configuration x under the given environment. ℬB is the tuning budget, e.g., wall-clock time. 2.2. Multi-Fidelity Optimization Typically, multi-fidelity optimization reduces the cost of expensive measurements by exploiting cheaper approximates of the objective. Here, fidelity is modeled and controlled via a single factor, such as the number of training epochs or the proportion of training data, while extensions to multiple factors remain rare (Kandasamy et al., 2017). Formally, the objective of a solution ∈ x∈ X measured with budget r can be denoted by f(,)f( x, r), where larger budgets incur higher cost c()c( r) but are assumed to provide more accurate approximations of the measurements in the perfect-fidelity setting at ∗ r^*. This one-dimensional and cost-oriented view of fidelity underpins most multi-fidelity optimization algorithms in, e.g., HPO (Li et al., 2017; Falkner et al., 2018; Awad et al., 2021; Li et al., 2021). 3. Codifying Fidelity for Configurable Systems 3.1. Fidelity Factors Fidelity factors are control variables for the fidelity settings of environments. In HPO (Eggensperger et al., 2021), the fidelity factor is commonly related to “resource types”, such as training epochs, which are general across any models. However, in configuration tuning, the fidelity factors are typically implicit, environment/system-specific, and lack standardized identification criteria. To formalize this notion, in this work, we define a fidelity factor for configurable systems as: “A variable that modulates the environment under which a configuration is measured but is exogenous to the configurable system.” To specify this, we provide a framework of categories grounded in the above definition. Instead of an exhaustive classification, our goal is to offer a structured lens through which potential fidelity factors can be defined, compared, and applied across systems. As illustrated in Figure 2, the fidelity factors can be assigned into three broad types based on their role and impact on the tuning: • Budget-related: Factors that control resource allocation, e.g., the duration, which controls how long each load-test runs, for tuning Tomcat. • Workload-related: Factors that shape workload characteristics, e.g., the max-func of programs, which sets the maximum number of functions, for tuning Gcc. • Dataset-related: Factors that define the dataset composition, e.g., table-size in the benchmark that tunes MySQL. In general, a fidelity factor can be categorical, numerical or ordinal, depending on the system/environment. Those fidelity factors can often be found in the benchmark used to tune a configurable system, e.g., Sysbench (58) and Wrk (60). Figure 2. Taxonomy of fidelity factors. 3.2. Problem Formulation Drawing on the definition of fidelity factors for configurable systems, we extend the problem from Equation 1 as a fidelity-aware one. Suppose ∈ z∈ Z denotes a fidelity setting, where z = (z1,z2,…,zkz_1,z_2,…,z_k) is a vector defined by k (k≥1k≥ 1) fidelity factor(s), and Z stands for the entire fidelity space of settings. The objective function under the fidelity setting z is f()f_ z( x), i.e., the performance of configuration x measured at z. We denote the target, perfect-fidelity setting by ∗ z^* (∗∈ z^*∈ Z), and its objective function by f∗()f_ z^*( x) (i.e., f∗()=f()f_ z^*( x)=f( x)). We denote the cost under the fidelity setting z by τ() _ z( x). Now, the multi-fidelity configuration tuning problem becomes: (2) argminf∗() or argmaxf∗(), f_ z^*( x) or f_ z^*( x), s.t. ∑∈; ‡∈τ()≤ℬ. s.t. _ x∈ X; z∈ Z _ z( x) . The goal is to find a configuration whose performance value under the perfect-fidelity setting is optimized, through exploring the measurements across the entire fidelity space, including both imperfect-fidelity settings (/‡∗ Z/ z^*) and the perfect-fidelity setting (∗ z^*), subject to a budget ℬB. From the above, the iith fidelity setting i z_i for configurable systems might influence two aspects: • Cost: The cost of an (imperfect-)fidelity setting refers to the average wall-clock time consumed for profiling the system on u given configurations under that setting to obtain their performance measurements222The time taken for the deployment of the environment is also included.: (3) τi=1u∑j=1uτi(j). _ z_i= 1u _j=1^u _ z_i( x_j). • Fidelity Perfection: We use fidelity perfection to denote the extent to which an (imperfect-)fidelity setting can approximate the target perfect-fidelity setting, under which the configurable system should be tuned. To quantify such, in this work we use Spearman correlation (Hauke and Kossowski, 2011), which captures the relative nonlinear ranking of the configurations’ performance measured under the iith (imperfect-)fidelity setting (i z_i) and the perfect-fidelity setting (∗ z^*): (4) ρi=cov((‡⟩),(‡∗))σ(‡⟩)σ(‡∗), _ z_i= cov ( R( F_ z_i), R( F_ z^*) ) _ R( F_ z_i)\, _ R( F_ z^*), whereby S = 1,…,u\ x_1,…, x_u\ means a set of u configurations. ‡⟩ F_ z_i = [fi(1)f_ z_i( x_1), …, fi(u)f_ z_i( x_u)] and ‡∗ F_ z^* = [f∗(1)f_ z^*( x_1), …, f∗(u)f_ z^*( x_u)] denote the performance of all configurations ∈ x∈ S measured under the fidelity setting i z_i and the perfect-fidelity setting ∗ z^*, respectively. (⋅) R(·) returns performance ranks; cov(⋅,⋅)cov(·,·) is their covariance while σ(‡⟩) _ R( F_ z_i) and σ(‡∗) _ R( F_ z^*) are the corresponding standard deviations. A higher ρi _ z_i means the setting is closer to the perfect-fidelity setting. With the cost and fidelity perfection, we anticipate that most of the time, if not all, the tuning is guided by configurations measured under a selected imperfect-fidelity setting fi()f_ z_i( x), which should often be cheaper yet provides a good perfection. 3.3. Characteristics of Systems Fidelity To better showcase the characteristics of fidelity in configurable systems, we conduct an exploratory study on PostgreSQL using the Sysbench (58) benchmark as an example. We tune 20 widely used configuration options and examine 5 commonly mentioned fidelity factors, leading to a configuration space of 8.70×101228.70× 10^122 and fidelity space of 11,200, respectively333The details can be found at Tables 1 and 2.. Here, we aim to tune for better throughput while reducing the time taken for profiling configurations (the cost). Specifically, we generate 1,000 configurations using Latin Hypercube Sampling (LHS) (McKay, 1992) and measure each under a given perfect-fidelity setting and 10 randomly selected imperfect-fidelity settings. We obtain several observations: Observation 1: Top-performing configurations in the imperfect- and perfect-fidelity settings are never completely identical; they can, however, be quite similar or discrepant. Figures LABEL:fig:ma and LABEL:fig:mb visualize the top-50 high-performing configurations under the perfect-fidelity setting, which requires about 250250s per measurement, together with those under two representative imperfect-fidelity settings, requiring roughly 6060s and 7575s per measurement, respectively. We see that the top-performing configurations measured under different fidelity settings can rarely be identical, but the level of perfection differs: as shown in Figure LABEL:fig:ma, there is a strong overlap of the promising configurations between imperfect- and perfect-fidelity settings (the connected points), i.e., the configurations that perform very well under both fidelity settings are identical, evidencing the feasibility of leveraging imperfect-fidelity proxies to guide efficient tuning. However, as illustrated in Figure LABEL:fig:mb, an opposite pattern emerges, where the top-performing configurations under imperfect- and perfect-fidelity settings differ significantly. This highlights the importance of discovering reliable imperfect-fidelity settings. (a) (b) [width=0.48]figures/motivation_c (c) [width=0.44]figures/motivation_d (d) Figure 3. (a) and (b) visualize the landscapes (processed by MDS (Cox and Cox, 2000)) of top-50 performing configurations under perfect-fidelity and two distinct imperfect-fidelity settings; (c) presents the cost and fidelity perfection of 10 imperfect-fidelity settings; and (d) shows the selected imperfect-fidelity settings with and without considering factor interactions. As mentioned, HPO assumes a monotonic relation between fidelity perfection and cost (more cost means better perfection), but what we found for configurable systems is that: Observation 2: Cost and fidelity perfection are not always monotonically correlated, making cost variation an unreliable/uncertain indicator of the change to perfection level. Figure LABEL:fig:mc plots the fidelity perfection (via Equation 4) of each imperfect-fidelity setting against its measurement cost. The results show that cost and fidelity perfection are not always monotonically related, e.g., some highly costly settings yield a lower perfection than the cheaper ones. This challenges the common assumption in many multi-fidelity optimization cases that higher cost inherently brings better exactness to the perfect-fidelity (Li et al., 2017; Falkner et al., 2018; Li et al., 2021; Hu et al., 2019; Carstensen et al., 2025). To further understand the unique complexity of fidelity space for configurable systems beyond what was assumed for HPO, within the 10 imperfect-fidelity settings, we randomly choose five that differ only on a single factor, e.g., (time=180s) vs. (time=30s), against the other five that differ on multiple factors, e.g., (time=180s; tables=50) vs. (time=30s;tables=20), based on all of which we measure the performance of 1,0001,000 configurations. We found that: Observation 3: Considering interaction between multiple fidelity factors can yield “fair” imperfect-fidelity settings444Not perfect, but fair enough to be useful given its cost/perfection level. that would otherwise be difficult to find. As in Figure LABEL:fig:md, the imperfect-fidelity settings where multiple factors are varied reflect much better cost and fidelity perfection than their single-factor varying counterparts. This implies that assuming imperfect-fidelity settings with only a single factor, as in the classic HPO, would leave their full potential untapped. The above, together with the proposed conceptual framework of fidelity, motivates our idea on proactively exploiting imperfect-fidelity settings to accelerate tuning under the perfect-fidelity setting. Yet, this raises three key challenges: • Challenge 1: How can we identify the fair imperfect-fidelity settings in multi-dimensional fidelity space? • Challenge 2: How to explore useful information under the imperfect-fidelity setting(s)? • Challenge 3: How to enable effective exploitation of the imperfect-fidelity setting to benefit tuning under the perfect-fidelity setting? Those are what we address via MFTune. 4. The MFTune Framework Figure 4 and Algorithm 1 illustrate the workflow of MFTune. Given a perfect-fidelity setting, the key idea is to first explore the fidelity space to identify a “fair” imperfect-fidelity setting—one that offers good fidelity perfection while incurring a much lower cost, hence improving budget utilization. We then conduct a specifically designed diversity-preserving tuning under the selected fair imperfect-fidelity setting to achieve “wide” tuning, after which the resulting archive seeds the subsequent “deep” tuning under the perfect-fidelity setting. To that end, MFTune comprises three phases: Imperfect-Fidelity Discovering (line 1) formulates an active, multi-objective fidelity discovery problem (maximizing level of perfection while minimizing cost) and applies NSGA-I (Deb et al., 2002)—a common multi-objective genetic algorithm—to evolve a “fair” imperfect-fidelity setting, together with some initially archived seeds of configurations (Challenge 1). Imperfect-Fidelity Seeding (line 2) embeds the discovered imperfect-fidelity setting into the tuning, together with a two-stage tuning strategy, which covers a wide area of the space and generates high-quality configurations to improve archived seeds for tuning under the perfect-fidelity setting (Challenges 2 and 3). Perfect-Fidelity Assuring (line 3) initializes the tuning by filtering the archive of seeded configurations, hence fully exploiting the benefits from the tuning thereof for deeply exploring along the tuning directions implied and returns the best configuration found under the perfect-fidelity setting (Challenge 3). We equally split the budget into the above: tuning under the fair imperfect-fidelity setting for seeds has half the budget (since it has two-stage tuning), while the other two share a quarter of the budget each. All phases are connected by an archive A, which contains configurations progressively measured under the perfect-fidelity setting; hence, the tuning can stop whenever needed. 1 Input: Budget ℬB; perfect-fidelity setting ∗ z^*; parameter α 2 Output: The best configuration found under perfect-fidelity: best x_best 3 4 A , fair← z_fair (ℬB, ∗ z^*) 5 6 A ← ImperfectFidelitySeeding(ℬB, fair z_fair, ∗ z^*, A, α) 7 8best← x_best← SeededPerfectFidelityTuning(ℬB, A, ∗ z^*) return best x_best Algorithm 1 MFTune Figure 4. Workflow and architecture overview of MFTune. 4.1. Discovering Fair Imperfect-Fidelity At the beginning, MFTune aims to identify a fair imperfect-fidelity that has lower cost while exhibiting a good level of fidelity perfection to the target perfect-fidelity. As such, this forms a typical multi-objective search problem, for which we adopt the NSGA-I (Deb et al., 2002) algorithm to explore the interactions between fidelity factors, together with their nonlinear implications to cost and fidelity perfection (Observations 2 and 3). Specifically, it runs in the following step with a quarter budget (also shown in Figure 5 and Algorithm 2): (1) Generate an archive A = 1,…,l\ x_1,…, x_l\ with l diverse configurations via Latin Hypercube Sampling (LHS) (McKay, 1992) to provide the basis for fidelity perfection ( 11 ). (2) Initialize a fidelity population Q = 1,…,m z_1,…, z_m by randomly sampling m imperfect-fidelity settings ( 22 ). (3) Measure the configurations in A under every fidelity setting in Q and the perfect-fidelity setting ∗ z^*. These allow us to measure the cost (using Equation 3) and fidelity perfection (using Equation 4) for each of the imperfect-fidelity settings in Q ( 11 – 22 ). (4) Reproduce m new imperfect-fidelity settings based on Q via boundary mutation and uniformed crossover operators555Boundary mutation perturbs a decision variable by resampling it within its feasible range, while uniform crossover exchanges variables between two parent solutions. in NSGA-I and measure the cost and fidelity perfection under newly explored imperfect-fidelity settings, if any, using the configurations in A ( 33 – 55 ). (5) Keep the top m imperfect-fidelity settings among the old and new ones as the new Q via the non-dominated sorting666Non-dominated sorting ranks solutions into Pareto fronts based on dominance relations, where a solution is non-dominated if no other solution is better in all objectives, followed by crowding distance (Deb et al., 2002). in NSGA-I on both the cost and fidelity perfection ( 66 ). (6) Repeat from Step 4 when there are still budgets left; otherwise, return the fair imperfect-fidelity setting with the biggest ρziτzi _z_i _z_i from the final population Q ( 77 ). The produced archive A and the fair imperfect-fidelity setting, denoted as fair z_fair, would be used for archiving more high-quality configuration seeds to improve the “wideness” of configuration tuning thereafter. Input: Budget ℬB; perfect-fidelity setting ∗ z^*; consumed budget τ=0τ=0 Output: The archive A and the fair imperfect-fidelity setting fair z_fair 1 2← A← generate l configurations using LHS 3 ← Q← randomly generate m imperfect-fidelity settings and measure their cost and fidelity perfection using A (via Equations 3 and 4) 4 τ←τ← update total time taken 5 6while τ<ℬ4τ B4 do 7 ′← Q ← reproduce m new imperfect-fidelity settings from Q with the boundary mutation and uniformed crossover (Chen and Li, 2021), and measure their cost and fidelity perfection using A (via Equations 3 and 4) 8 τ←τ← update total time taken 9 Q ← nondominated sorting on Q ∪ ′ Q as in NSGA-I 10 11 end while 12 return A , fair← z_fair← the imperfect-fidelity setting from Q with the biggest ρziτzi _z_i _z_i Algorithm 2 ImperfectFidelityDiscovering Figure 5. Discovering the fair imperfect-fidelity setting. 4.2. Finding Seeds under Imperfect-Fidelity Here, MFTune explores and archives high-quality configuration seeds under the fair imperfect-fidelity setting fair z_fair (with a low measurement cost). Indeed, directly tuning under fair z_fair might reflect the tuning under the perfect-fidelity setting ∗ z^*, but at the same time, it might also mislead it into unwanted local optima, since it remains exhibiting “imperfection” with respect to ∗ z^* (Observation 1). As such, to improve the likelihood of archiving more high-quality seeds for the perfect-fidelity tuning later on, we equip a diversity-preserved, “wideness-oriented” two-stage tuning under fair z_fair in MFTune, where the former stage globally samples for diverse configurations progressively while the latter aims for stable convergence of configurations with local diversity preservation that can benefit the tuning under ∗ z^*. Each of the above stages owns a quarter of the budget. In this way, MFTune ensures that it archives the best configurations found under fair z_fair, together with some diverse configurations that are of sub-optimal performance at fair z_fair, but might still perform well under ∗ z^*. Input: Budget ℬB; fair imperfect-fidelity setting fair z_fair; perfect-fidelity setting ∗ z^*; the archive A; the probability α; consumed budget τ=0τ=0 Output: The updated archive A 1 /* Progressive Diversity Sampling */ 2 31,…,p←\ x_1,…, x_p\← sample p=ℬ4×τfairp= B4× _ z_fair configurations via LHS 4 while τ<ℬ4τ< B4 do 5 foreach x ∈ 1,…,p\ x_1,…, x_p\ do 6 Measure x under fair z_fair and update τ 7 if x is currently best under fair z_fair and x ∉ A and rand()<αrand()<α then 8 Measure x under ∗ z^* and update τ 9 A ← A ∪ x 10 11 end if 12 13 end foreach 14 15 end while 16 P ← top n configurations from 1,…,p\ x_1,…, x_p\ under fair z_fair 17 18←∪ A← A∪ P without redundancy 19 /* Diversity-Preserved Tuning under Imperfect-Fidelity */ 20 21Reset consumed budget τ = 0 22 23Measure any unmeasured configurations in A under ∗ z^*; return if all the next budget of ℬ4 B4 is exhausted, otherwise let the consumed budget be τ′τ 24 25τ = τ′τ 26 27while τ<ℬ4τ B4 do 28 29 ′ P ← reproduce n new configurations from P with the boundary mutation and uniformed crossover (Chen and Li, 2021), and measure their performance under fair z_fair 30 τ ← update total time taken 31 P ← top n performing configurations from ′∪ P ∪ P under fair z_fair 32 33 ← x← the best configuration from P under fair z_fair 34 35 if x ∉ A then 36 Measure x under ∗ z^* and update τ 37 A ← A ∪ x 38 39 end if 40 41 end while 42 43←∪ A← A∪ P without redundancy 44 45return The archive A 46 Algorithm 3 ImperfectFidelitySeeding 4.2.1. Progressive Diversity Sampling To mitigate the imperfection in fair z_fair, MFTune adopts a “top-down” strategy: globally, an initial set of configurations is measured, filtered, and archived progressively within a quarter budget ( 11 – 33 in Figure 6 and Algorithm 3): (1) Diversely sample ℬ4×τfair B4× _ z_fair configurations using LHS. In this way, the sampled configurations can be proportional to the budget and cost of fair z_fair (line 1). (2) Measure a randomly chosen configuration from the above set under fair z_fair. If, during the process, a configuration has the best performance for fair z_fair so far, then place it into the archive A with a probability of α (and measure its performance under ∗ z^* if that is the case). This controllable parameter, together with the order of configurations, is the key to creating global diversity of preserved configurations in the archive that will seed the tuning under ∗ z^* (lines 2-10). (3) Repeat from 2 until all sampled configurations are measured or the quarter budget is exhausted. Finally, those top n configurations are ready for the subsequent stage of tuning under fair z_fair. To prevent discarding well-performing configurations under fair z_fair, we also archive any previously unarchived top n configurations for fair z_fair (over all measured ones). Figure 6. Finding seeds under fair imperfect-fidelity setting. 4.2.2. Diversity-Preserved Tuning under Imperfect-Fidelity As from 33 – 99 in Figure 6 and Algorithm 3, to tune under the fair imperfect-fidelity setting fair z_fair, we follow the “bottom-up” process in Genetic Algorithm (GA) (Bäck and Schwefel, 1993), where a population of configurations are reproduced by boundary mutation and uniformed crossover, after which the top n measured configurations under fair z_fair are preserved into the next iteration. Here, we not only archive those best configurations found under fair z_fair, but also those sub-optimal ones explored under fair z_fair during the tuning, hence enforcing local diversity. The budget for this is again capped as a quarter. Notably, we make several extensions to GA for MFTune: • The initial set of configurations are the top n ones obtained (under fair z_fair) from the progressive diversity sampling. We also ensure that all configurations in A so far have their performance values under ∗ z^* (lines 11-15). • At each iteration, the best configuration measured under fair z_fair so far is also pushed into the archive A and measured under ∗ z^* (lines 16-25). This maintains local diversity by tracking the progress of tuning, hence further mitigating the possible “imperfection” of fair z_fair to ∗ z^*. • At the end, all the persevered but non-archived configurations would be pushed to A and measured under ∗ z^* (line 26)—this seeks to align with Observation 1 that well-performing configurations under different fidelity settings might also be consistent. The above consolidates the archive for more high-quality seeds that are either sub-optimal or optimal as measured under fair z_fair, since they could both be beneficial for tuning under ∗ z^*. 4.3. Assuring under Perfect-Fidelity with Seeds Tuning under the perfect-fidelity setting ∗ z^* resembles a single-fidelity tuning (Figure 7 and Algorithm 4); we again use GA with boundary mutation/uniform crossover ( 33 – 77 ). Unlike classic GA, the archive A accumulated so far serve as good starting points. MFTune firstly measures all configurations in A, which have not been measured, under ∗ z^*. The configurations in A would be filtered, such that only the top n therein can be used as seeds, enabling a warm start ( 11 – 22 ). The tuning terminates and returns the best configuration found under ∗ z^* when the quarter budget runs out. This final phase plays a pivotal role for the “depth” of tuning in MFTune, offering extra assurance to the imperfection/uncertainty of the fair imperfect-fidelity setting. Input: Budget ℬB; the archive A; perfect-fidelity setting ∗ z^*; τ=0τ=0 Output: The best configuration best x_best found under ∗ z^* 1 2Measure any unmeasured configurations in A under ∗ z^*; return if all the budget of ℬ4 B4 is exhausted, otherwise let the consumed budget be τ′τ 3 τ=τ′τ=τ 4 P ← top n configurations from A under ∗ z^* 5 6while τ<ℬ4τ B4 do 7 8 ′ P ← reproduce n new configurations from P with the boundary mutation and uniformed crossover (Chen and Li, 2021), and measure their performance under ∗ z^* 9 τ ← update total time taken 10 P ← top n high-performing configurations from ′∪ P ∪ P under ∗ z^* 11 12 end while 13 14return The best configuration from P measured under ∗ z^* 15 Algorithm 4 SeededPerfectFidelityTuning Figure 7. Seeded tuning under perfect-fidelity setting. 5. Experimental Setup To evaluate MFTune , we ask four research questions (RQs): • RQ1: How beneficial are the imperfect-fidelity settings? • RQ2: How does MFTune perform compared with the state-of-the-art tuners? • RQ3: How does the imperfect-fidelity discovery and its two-stage tuning individually contribute? • RQ4: What is the sensitivity of MFTune to α? All experiments are independently performed on Ubuntu 22.04.4 LTS running on a cluster of 20 servers, each of which is an Intel NUC with 16 CPU cores and 64GB RAM. In total, the full experiment takes over 19 months of CPU time 24×7. 5.1. Systems, Options, and Fidelity Systems: As in Table 1, we select six widely used systems spanning diverse application domains and performance objectives while covering configuration space sizes between 101610^16 and 1072410^724. Options: Each system features a distinct set of options that may influence performance, including categorical options (e.g., enumerated and binary) and numerical options (e.g., integers), as have been used in prior work (Zhang et al., 2022, 2021; Kanellis et al., 2022; He et al., 2022; Chen et al., 2021). Fidelity: To generate fidelity settings, we use: • Sysbench (58): A transactional database benchmarking tool, generating stressed workloads controlled by various fidelity factors (e.g., table-size and thread). • Wrk (60): A high-performance HTTP load generator for web servers, controlling request load and traffic intensity via fidelity factors such as connections and duration. • Csmith (21): A C program generator that produces programs of diverse scale and structural complexity for compilers, controlled by, e.g., max-funcs and max-block-size. The fidelity used is summarized in Table 2. In practice, the perfect-fidelity setting is typically designated by developers based on business needs (often expensive), serving as the ground truth for assessing configuration quality. To stress test the tuners, we set the setting that incurs the highest costs as the perfect-fidelity setting under which we seek to tune the configuration. Table 1. Subject systems studied. X is the size of configuration space; (||/||| C|/| N|) are the number of categorical/numerical options. tps/rps means transactions/requests per second. System Domain Performance ||/||| C|/| N| X Cost (z∗ z^*) Cost (zi z_i) MySQL (53) Database Throughput (tps) 2/18 4.24 ×10724× 10^724 ≈ 250s 39s-90s PostgreSQL (55) Database Throughput (tps) 0/20 8.70 ×10122× 10^122 ≈ 250s 40s-76s Tomcat (59) Web server Throughput (rps) 3/17 4.47 ×10118× 10^118 ≈ 180s 10s-60s Httpd (37) Web server Throughput (rps) 7/13 4.76 ×1078× 10^78 ≈ 180s 10s-101s Gcc (28) Compiler Runtime (ms) 20/0 7.21 ×1016× 10^16 ≈ 43s 2s-6s Clang (18) Compiler Runtime (ms) 20/0 7.21 ×1017× 10^17 ≈ 40s 2s-4s Table 2. Details of fidelity generators. “Type” refers to the type of fidelity factors from §3.1. ∗ z^* and Z denote the perfect-fidelity setting used and fidelity space, respectively. Benchmark Fidelity Values Z ∗ z^* Type Factors Sysbench Budget time 30,35,…,180\30,35,...,180\ 1120011200 180180 Dataset tables 20,25,…,50\20,25,...,50\ 5050 Budget threads 4,6,…,10\4,6,...,10\ 44 Workload r/w ratio 0.5,0.6,…,0.9\0.5,0.6,...,0.9\ 0.50.5 Dataset table-size 10K,15K,…,100K\10K,15K,...,100K\ 100K100K Wrk Workload post true,false\true,false\ 12961296 truetrue Budget threads 1,2,…,10\1,2,...,10\ 88 Budget duration 10,15,…,180\10,15,...,180\ 180180 Workload connections 10,15,…,50\10,15,...,50\ 5050 Csmith Workload max-funcs 5,10,…,50\5,10,...,50\ 58805880 5050 Dataset max-block-size 1,2,3,4\1,2,3,4\ 44 Workload max-block-depth 1,2,3,6\1,2,3,6\ 66 Workload inline-function-prob 10,50,…,90\10,50,...,90\ 1010 Workload max-array-len-per-dim 5,10,…,50\5,10,...,50\ 5050 5.2. State-of-the-Art Tuners We compare MFTune against diverse types of tuners777FLASH uses exhaustive sampling at each iteration for acquisition evaluation, which does not work on systems with an intractable space; we extend this, namely FLASH+, by randomly sampling 10001000 configurations., spanning across different domains, as summarized in Table 3. • Single-fidelity tuners only conduct tuning under the perfect-fidelity setting. • Multi-fidelity tuners exploit resource-allocation strategies across different fidelity settings. Since they work on a single fidelity factor and assume a monotonic relationship between cost and fidelity perfection, we preserve their original design by choosing the factor that is the most influential on cost. Regardless of the fidelity, a tuner can be either model-based or model-free: the former learns surrogate models (e.g., Gaussian Process) to approximate the configuration–performance landscape, steering tuning for the promising regions; the latter relies solely on system measurements to guide the tuning. 5.3. Metrics The evaluation metrics would clearly be the performance metric that a system is of concerned, i.e., throughput or runtime. We also measure the relative efficiency and budget utilization of MFTune via Δℬ : the budget saving (or overspend) for MFTune to reach the best performance of a counterpart tuner, if applicable, as Δℬ=ℬc−ℬm =B_c-B_m, where ℬcB_c is the earliest time the counterpart tuner reaches its best performance (average over runs) while ℬmB_m is the time when MFTune achieves the same. As such, Δℬ>0 >0 indicates the amount of budget saved by MFTune; or otherwise it means MFTune has no budget saving (Δℬ=0 =0) or could even overspend (Δℬ<0 <0). 5.4. Tuning Budget In this work, we set the tuning budget ℬB as the wall-clock time allowed to tune the real system. To ensure realism of our experiment and consider the diverse measurement costs of different systems domains, we set ℬ=24B=24 hours for MySQL/PostgreSQL, ℬ=12B=12 hours for Tomcat/Httpd, and ℬ=4B=4 hours for Gcc/Clang. These time budgets correspond to ≈250≈ 250–350350 measurements under the perfect-fidelity setting, which aligns with most of the prior studies (Chen et al., 2025). Note that we also examine the trajectories of tuning therein. Table 3. Summary of the compared tuners. Tuner Fidelity Strategy Domain Year PromiseTune (Chen and Chen, 2026b) Single-fidelity Model-based Configuration 2026 HEBO (Cowen-Rivers et al., 2022) Single-fidelity Model-based General 2022 GA (Shahbazian et al., 2020) Single-fidelity Model-free General 2020 FLASH+ (Nair et al., 2020) Single-fidelity Model-based Configuration 2018 BestConfig (Zhu et al., 2017) Single-fidelity Model-free Configuration 2017 SMAC (Hutter et al., 2011) Single-fidelity Model-free General 2011 PriorBand (Mallik et al., 2023) Multi-fidelity Model-free HPO 2023 DEHB (Awad et al., 2021) Multi-fidelity Model-free HPO 2021 BOHB (Falkner et al., 2018) Multi-fidelity Model-based HPO 2018 Hyperband (Li et al., 2017) Multi-fidelity Model-free HPO 2018 5.5. Parameter Settings The initial sample size for model-based tuners (e.g., SMAC, FLASH+, and HEBO) is set to 30, following common practice in prior studies (Nair et al., 2020; Chen et al., 2024). For other parameters, such as the configuration population size (n=20n=20) of GA, we adopt either the default setting or those reported in the literature to ensure fair comparison (Li et al., 2017; Falkner et al., 2018; Awad et al., 2021; Mallik et al., 2023; Chen and Chen, 2026b). For MFTune, the initial configuration sample size for discovering imperfect-fidelity settings l and fidelity population size m are both set to 10. We use a smaller fidelity population (m<nm n) because quantifying a fidelity setting is more expensive, which requires measuring a batch of l configurations to estimate its perfection level. In particular, we set α=0.5α=0.5—a generally best value (see §6.4). To avoid bias, we repeat 10 runs for each experiment. Table 4. Comparing MFTune with state-of-the-art tuners over 10 runs. ↑ and ↓ denote maximized (throughput) and minimized (runtime) performance, respectively. r denotes Scott-Knott ESD rank on performance. Δℬ≥0 ≥ 0 and Δℬ<0 <0 refers to the budget saving and overspend of MFTune compared with a counterpart, respectively. ✗ indicates that MFTune fails to achieve the best of a counterpart when budget runs out. The best ranked tuner(s) for each system is highlighted in green. MySQL (↑ ) PostgreSQL (↑ ) Httpd (↑ ) Tomcat (↑ ) Gcc (↓ ) Clang (↓ ) Tuner r Mean (Std) Δℬ r Mean (Std) Δℬ r Mean (Std) Δℬ r Mean (Std) Δℬ r Mean (Std) Δℬ r Mean (Std) Δℬ Hyperband 3 360.12 (20.61) 9.7 hours 7 498.71 (33.27) 13.1 hours 2 3028.27 (303.15) 3.4 hours 4 2890.92 (67.34) 7.4 hours 3 68.01 (1.98) 0.6 hours 5 69.09 (3.03) 0.1 hours BOHB 3 358.35 (20.24) 9.9 hours 7 503.46 (34.03) 23.6 hours 6 2826.15 (62.30) 6.7 hours 5 2880.84 (87.21) 6.6 hours 5 69.85 (2.50) 0.2 hours 1 67.41 (2.83) ✗ DEHB 2 365.50 (30.26) 7.5 hours 6 514.26 (68.39) 21.7 hours 5 2849.06 (80.17) 6.7 hours 7 2784.64 (50.90) 6.9 hours 4 69.65 (2.65) 0.8 hours 3 68.04 (3.40) ✗ PriorBand 1 378.02 (22.10) 0.1 hours 6 516.47 (37.93) 17.9 hours 2 3049.98 (578.41) 3.0 hours 4 2899.83 (60.98) 6.8 hours 4 69.62 (2.82) 0.2 hours 6 71.27 (2.76) 0.4 hours SMAC 3 361.56 (31.48) 9.5 hours 2 556.77 (34.14) 4.2 hours 2 2969.58 (297.48) 5.0 hours 3 2926.93 (133.76) 6.0 hours 2 67.26 (1.84) −-0.9 hours 2 67.63 (2.60) ✗ BestConfig 2 365.06 (19.58) 7.1 hours 6 519.42 (35.35) 5.2 hours 6 2817.72 (55.43) 7.2 hours 6 2820.93 (69.37) 6.4 hours 4 69.29 (3.10) 0.8 hours 3 68.32 (2.23) ✗ FLASH+ 1 374.72 (14.79) −-0.5 hours 3 541.14 (30.30) 16.6 hours 2 3010.90 (268.23) 3.6 hours 1 3309.26 (730.02) 1.9 hours 2 67.92 (2.30) 0.6 hours 4 69.02 (1.90) 0.0 hours GA 4 350.90 (30.00) 6.3 hours 3 542.72 (42.63) 16.7 hours 4 2877.19 (73.24) 5.2 hours 4 2898.54 (87.09) 6.6 hours 2 67.78 (2.91) 0.5 hours 4 68.69 (2.41) −-1.4 hours HEBO 1 378.03 (18.34) 0.0 hours 4 537.17 (22.20) 22.8 hours 3 2920.09 (118.56) 4.9 hours 2 3072.72 (613.00) 5.7 hours 5 70.14 (2.34) 0.7 hours 4 69.07 (2.97) −-0.1 hours PromiseTune 2 365.24 (23.10) 5.0 hours 5 532.60 (14.34) 22.8 hours 1 3259.97 (532.48) 2.6 hours 2 3145.03 (659.50) 4.2 hours 3 69.12 (1.84) 0.9 hours 6 70.26 (1.40) 0.3 hours MFTune 1 378.14 (21.75) 0.0 hours 1 570.54 (37.20) 0.0 hours 1 3359.75 (710.07) 0.0 hours 1 3323.13 (843.83) 0.0 hours 1 67.01 (2.93) 0.0 hours 3 68.65 (4.43) 0.0 hours 5.6. Test for Statistical Significance To ensure statistical significance when comparing the performance of multiple tuners, we employ the Scott-Knott ESD test (Herbold, 2017). In a nutshell, it begins by ranking tuners according to their mean performance, then recursively partitions this ordered list into statistically different subgroups. For instance, given three tuners A, B, and C, Scott-Knott ESD might partition them into two subgroups: A,B\A,B\ at rank r=1r=1 and C\C\ at rank r=2r=2. This indicates that A and B cannot be distinguished statistically, yet both significantly outperform C. Scott-Knott ESD is chosen because, unlike the Kruskal-Wallis test, it overcomes the confounding factor of overlapping groups and eliminates the dependence on post-hoc correlation (McHugh, 2011). 6. Results and Analysis 6.1. RQ1: Benefits of Imperfect-Fidelity [width=0.32]figures/rq1_mysql (a) [width=0.32]figures/rq1_postgresql (b) [width=0.34]figures/rq1_tomcat (c) [width=0.34]figures/rq1_httpd (d) [width=0.32]figures/rq1_gcc (e) [width=0.32]figures/rq1_clang (f) Figure 8. Tuning trajectories with and without multi-fidelity over 10 runs. 6.1.1. Method. For RQ1, we compare MFTune with a classic GA, which is essentially the single fidelity version of MFTune that tunes under the perfect-fidelity setting only; all other designs are identical. We plot the trajectory of the best configuration found under the perfect-fidelity setting (the ones in archive for MFTune). 6.1.2. Results. As shown in Figure 8, MFTune consistently outperforms GA across all six systems, demonstrating superior performance and competitive convergence speed. In efficacy, MFTune achieves up to 16.77% performance improvement (on Httpd), suggesting that integrating the imperfect-fidelity setting enables more effective exploration of high-quality configurations for the perfect-fidelity setting. Notably, at the early stage, MFTune shows slightly slower convergence since part of its budget is devoted to discover the fair imperfect-fidelity setting. However, this initial investment quickly pays off as the subsequent tuning proceeds more efficiently. Once an appropriate fair imperfect-fidelity setting is found, the tuners benefits from it to accelerate convergence toward promising configurations under the perfect-fidelity setting. As a result, MFTune generally leads to significant budget utilization, saving up to Δℬ=24−7.3=16.7 =24-7.3=16.7 hours (on PostgreSQL). Thus, we conclude that: RQ1: Considering imperfect fidelity settings in MFTune is greatly beneficial, yielding up to 16.77%16.77\% performance improvement and 16.716.7 hours budget saving. 6.2. RQ2: Effectiveness and Efficiency 6.2.1. Method. For RQ2, we compare MFTune with 10 state-of-the-art tuners on all systems. To ensure statistical significance, we use the Scott-Knott ESD test to rank the tuners over 10 runs. 6.2.2. Results. As from Table 4, MFTune achieves remarkable results: it is ranked the best in 83.33% (5/6) of the cases. This considerably outperforms the generally second-best tuner, FLASH+, which is ranked the best for 33.33% (2/6) cases only. Among single-fidelity tuners, model-based approaches (e.g., SMAC, FLASH+, HEBO, and PromiseTune) generally outperform model-free ones (e.g., GA and BestConfig), as their surrogate models better balance exploration and exploitation, steering the tuning towards higher-quality configurations. Notably, multi-fidelity HPO tuners (e.g., DEHB) perform poorly. This is because they assume a monotonic relationship between evaluation cost and fidelity perfection, which is commonly not held for configurable systems (as in §3.3). In contrast, MFTune’s handling of the interaction between multiple fidelity factors, together with the tuning for “wideness” and “depth”, has resulted in up to 19.34% performance improvement (on Tomcat). MFTune also shows clear practical advantages in budget utilization: it saves more budget than its counterparts in 50 out of 60 comparisons, with up to 23.6 hours. Therefore: RQ2: MFTune leads to considerably better tuning quality than others: it ranks first in 83.33% of the cases with up to 19.34% performance improvement, achieves better budget utilization in 50 comparisons, and saves up to 23.6 hours. 6.3. RQ3: Ablation Study 6.3.1. Method. For RQ3, we ablate MFTune with two variants: • MFTune-I removes the first phase of imperfect-fidelity discovering and randomly select the fair imperfect-fidelity. • MFTune-I simplifies the two-stage tuning under the fair imperfect-fidelity setting, leaving only one stage without the progressive diversity sampling. Again, we apply the Scott-Knott test on all comparisons. 6.3.2. Results. As summarized in Table 5, MFTune obtains the best rank in the majority of the cases (5/6), suggesting the necessity of both. Indeed, omitting imperfect-fidelity discovery might lead to a rather harmful imperfect-fidelity setting being used; simplifying two-stage tuning under imperfect-fidelity might cause the tuning to be prone to local-optimal, since without the global diversity encouragement, the “wideness” of tuning would be restricted. In light of the above, we can conclude: RQ3: Discovering imperfect-fidelity and its two-stage tuning are both essential to the superiority of MFTune. Table 5. Comparing MFTune with its two variants over 10 runs. The format is the same as Table 4. MFTune-I MFTune-I MFTune System r Mean (Std) Δℬ r Mean (Std) Δℬ r Mean (Std) MySQL 1 375.53 (17.12) 1.7 hours 2 338.17 (34.69) 13.1 hours 1 378.14 (21.75) PostgreSQL 1 601.17 (30.57) ✗ 3 552.67 (44.61) 13.3 hours 2 570.54 (37.20) Tomcat 2 2977.38 (271.63) 7.3 hours 3 2810.84 (63.48) 6.6 hours 1 3323.13 (843.83) Httpd 2 2761.12 (73.68) 6.4 hours 3 2730.95 (58.04) 7.2 hours 1 3359.75 (710.07) Gcc 3 70.79 (2.48) 1.7 hours 2 70.17 (2.78) 0.4 hours 1 67.01 (2.93) Clang 2 68.91 (3.58) 0.0 hours 2 69.57 (2.20) 0.2 hours 1 68.65 (4.43) 6.4. RQ4: Sensitivity to α 6.4.1. Method. To study the sensitivity of MFTune to the α—the probability of including diverse configurations under the fair imperfect-fidelity—in RQ4, we study five values: 0.1,0.3,0.5,0.7,0.9\0.1,0.3,0.5,0.7,0.9\. Scott Knott ESD is used to ensure statistical significance over 10 runs. 6.4.2. Results. As shown in Figure 9, the performance of MFTune is sensitive to α, but generally α=0.5α=0.5 achieves the best outcomes on both rank and average performance—neither too small nor too large α is ideal. This makes sense, because a too small α would cause too few diverse configurations to be archived, which may lose some configurations that perform well under the perfect-fidelity setting but are sub-optimal under the fair imperfect-fidelity setting. In contrast, when α is too large, too many diverse configurations might be archived, hence missing the chance to find those that perform well under both settings. Overall, we show that: [width=0.48]figures/find_alpha (a) [width=0.395]figures/alpha_perf (b) Figure 9. The sensitivity of MFTune to α values over all systems/runs (smaller normalized performance is preferred). RQ4: MFTune is sensitives to α, but α=0.5α=0.5 is generally the best. 7. Discussion 7.1. How Imperfect-Fidelity Discovery Helps? To understand how MFTune benefits from the imperfect-fidelity discovery, Figure 10 illustrates how the explored imperfect-fidelity settings evolve across iterations when tuning MySQL. We can see that MFTune progressively reveals candidate imperfect-fidelity settings that yield better fidelity perfection at lower costs, enabling MFTune with a good start for both “wideness” and “depth” of tuning. [width=0.34]figures/fidelity_evo_1 (a) [width=0.34]figures/fidelity_evo_2 (b) [width=0.34]figures/fidelity_evo_3 (c) Figure 10. The changes within imperfect-fidelity discovering on MySQL. “⋆ ” and “∘ ” denote the imperfect-fidelity settings and the nondominated imperfect-fidelity settings, respectively, where the perfect-fidelity setting has a cost ≈250≈ 250s. 7.2. Why Tuning for Imperfect-Fidelity Work? To disclose why tuning under a fair imperfect-fidelity setting fair z_fair works, Figure 11 plots an example of the tuning trajectory under the fair imperfect-fidelity setting fair z_fair and the changing top n configurations in the archive A when tested under the perfect-fidelity setting ∗ z^*. We see that the progressive diversity sampling increases the performance measured under both fair z_fair and ∗ z^* at a slow but steady pace. At around the 2525th iteration, the large improvement of the archive under ∗ z^* is due to the top configurations under fair z_fair being pushed in. Both trace increases more steeply afterwards, especially when the overall top-performing configurations for fair z_fair are pushed to the archive again at the 4040th iteration. [width=0.9]figures/diss3 Figure 11. Tuning trajectory of MySQL under the fair imperfect-fidelity and testing the corresponding configurations found thereof under the perfect-fidelity setting. Notably, the wall-clock time required for tuning/exploring the configuration under fair z_fair is only ≈10≈ 10 hours, compared with the ≈55≈ 55 hours if we were to tune/explore the same configurations under ∗ z^*. This means that, if given the same wall-clock time (e.g., 1010 hours), the number of configurations that can be explored under fair z_fair would be much larger than that under ∗ z^* (782782 vs. 142142), leading to superior results measured under ∗ z^* even though fair z_fair is imperfect. The above evidence supports the rationale of MFTune’s superiority: finding the right imperfect-fidelity setting and properly tuning configurations can greatly help the tuning under the perfect-fidelity setting—a reflection of our hypothesis mentioned in §1. 7.3. Threats to Validity Internal threats to validity could be raised from the parameter setting of α, which controls the probability of triggering full-fidelity evaluations during low-fidelity guided tuning. In our study, we set α=0.5α=0.5, following empirical evidence from our sensitivity analysis (RQ3), where this value offered a favorable balance between exploration cost and evaluation reliability. Similarly, parameter l controls the number of paired configurations for estimating fidelity perfection. We set l=10l=10 to strike a trade-off between estimation reliability and discovery cost: a large l may provide a more reliable estimate but also incurs higher cost for fidelity discovery. Nevertheless, we recognize that the best value of these parameters may vary across systems, and exploring alternative settings could further enhance robustness. 8. Related Work Tuning with or without models. Model-free tuners address configuration tuning by guiding the search process exclusively through direct system measurements without models (Behzad et al., 2013; Chen et al., 2019; Ye et al., 2025). For example, GA has inspired many tuners that work on a population of configurations (Behzad et al., 2013; Shahbazian et al., 2020). In contrast, model-based tuners learn a surrogate model from measured configuration-performance, paired with other heuristics, to accelerate the tuning (Gong and Chen, 2023, 2024; Aken et al., 2017; Chen et al., 2021; Zhu and Hao, 2023; Chen et al., 2018c; Chen and Chen, 2026a). However, all of the above are single-fidelity tuners, where each configuration is measured at the perfect-fidelity setting, guaranteeing accurate measurements at the expense of high costs. MFTune, in contrast, explicitly leverages the imperfect-fidelity setting to improve the tuning and budget utilization. Multi-workloads tuners. Recent works have transferred knowledge and reused information from past tuning tasks/environments/workloads, which might be similar to the fidelity: • Workload mapping (e.g., OtterTune (Aken et al., 2017)) identifies the most similar historical environment/workload and reuses the data to build a surrogate model. • Model ensemble (e.g., ResTune (Zhang et al., 2021)) combines multiple surrogate models trained on prior environments/workloads and generalizes such to a new workload for tuning. • Option pruning (e.g., OpAdviser (Zhang et al., 2023)) learns prior data under certain environments/workloads to identify key options/ranges for tuning under a new workload. MFTune fundamentally differs from them in several aspects: • They assume a reactive situation where a tuner can only reuse historically explored and structurally similar environments/workloads, limiting their ability to adapt to a completely unforeseen environment; while MFTune is a proactive tuner that actively explores unknown fidelity settings. • They assume dozens of environments/workloads; MFTune scales to ten thousand environments/fidelity settings. • They typically reuse the knowledge of other environments/workloads for updating the surrogate model or dimensionality reduction, while MFTune leverages them for consolidating the tuning process directly. Multi-fidelity optimization. Multiple fidelity settings have been considered primarily in the context of HPO (Hu et al., 2019; Forrester et al., 2007; Klein et al., 2017). A representative solution is Hyperband (Li et al., 2017), which samples a set of random solutions and progressively allocates more resources to promising ones via successive halving (Jamieson and Talwalkar, 2016). Variants such as BOHB (Falkner et al., 2018) and DEHB (Awad et al., 2021) extend Hyperband by incorporating Bayesian Optimization and Differential Evolution, respectively, to enhance optimization efficiency. However, as mentioned, they mostly consider a limited dimension of the fidelity factors and have also assumed a monotonic correlation between fidelity perfection and cost (Hossen et al., 2026; Echevarrieta et al., 2025; Carstensen et al., 2025), which, as we have shown, is not the case for configurable systems. This work formulates a conceptual framework specifically for system configuration tuning while presenting MFTune as the tuner that works effectively without the above constraints/assumptions. 9. Conclusion This paper presents MFTune, a tuner that tackles system configuration tuning by proactively exploring and exploiting the imperfect-fidelity setting. Drawing on a newly formulated conceptual framework of multi-fidelity for configurable systems, MFTune explore in the scale of more than ten thousand imperfect-fidelity settings, finding the most fair one for tuning, which then seeds the tuning under the perfect-fidelity setting for improved budget utilization and superior results. Experiments against 10 state-of-the-art tuners and under six real-world systems show that MFTune achieves better results on 83.3383.33% cases with up to 19.3419.34% improvements, while doing so by generally saving hours’ budget. Looking forward, we envision that this work can inspire new research directions that integrate multi-fidelity with diverse optimization paradigms, paving the way for more advanced configuration tuning for configurable systems. Acknowledgements.This work was supported by an NSFC Grant (62372084) and a UKRI Grant (10054084). Data Availability Statement All source code and data are available at: https://github.com/ideas-labo/mftune and https://zenodo.org/records/21364961. References D. V. Aken, A. Pavlo, G. J. Gordon, and B. Zhang (2017) Automatic database management system tuning through large-scale machine learning. In Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD Conference 2017, Chicago, IL, USA, May 14-19, 2017, S. Salihoglu, W. Zhou, R. Chirkova, J. Yang, and D. Suciu (Eds.), p. 1009–1024. External Links: Link, Document Cited by: §1, 1st item, §8. N. H. Awad, N. Mallik, and F. Hutter (2021) DEHB: evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, Z. Zhou (Ed.), p. 2147–2153. External Links: Link, Document Cited by: §1, §2.2, §5.5, Table 3, §8. T. Bäck and H. Schwefel (1993) An overview of evolutionary algorithms for parameter optimization. Evol. Comput. 1 (1), p. 1–23. External Links: Link, Document Cited by: §4.2.2. B. Behzad, H. V. T. Luu, J. Huchette, S. Byna, Prabhat, R. A. Aydt, Q. Koziol, and M. Snir (2013) Taming parallel I/O complexity with auto-tuning. In International Conference for High Performance Computing, Networking, Storage and Analysis, SC’13, Denver, CO, USA - November 17 - 21, 2013, W. Gropp and S. Matsuoka (Eds.), p. 68:1–68:12. External Links: Link, Document Cited by: §8. T. Carstensen, N. Mallik, F. Hutter, and M. Rapp (2025) Frozen layers: memory-efficient many-fidelity hyperparameter optimization. In Proceedings of the Fourth International Conference on Automated Machine Learning, L. Akoglu, C. Doerr, J. N. van Rijn, R. Garnett, and J. R. Gardner (Eds.), Proceedings of Machine Learning Research, Vol. 293, p. 4/1–24. External Links: Link Cited by: §1, §3.3, §8. J. Chen, V. Nair, R. Krishna, and T. Menzies (2019) ”Sampling” as a baseline optimizer for search-based software engineering. IEEE Trans. Software Eng. 45 (6), p. 597–614. External Links: Link, Document Cited by: §8. J. Chen, N. Xu, P. Chen, and H. Zhang (2021) Efficient compiler autotuning via bayesian optimization. In 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, p. 1198–1209. External Links: Link, Document Cited by: §5.1, §8. P. Chen, T. Chen, and M. Li (2024) MMO: meta multi-objectivization for software configuration tuning. IEEE Trans. Software Eng. 50 (6), p. 1478–1504. External Links: Link, Document Cited by: §5.5. P. Chen and T. Chen (2026a) CDS4RAG: cyclic dual-sequential hyperparameter optimization for rag. In Proceedings of the 35th International Joint Conference on Artificial Intelligence, IJCAI 2026, Bremen, Germany, 15-21 August 2026, Cited by: §8. P. Chen and T. Chen (2026b) PromiseTune: unveiling causally promising and explainable configuration tuning. In 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE), Cited by: §1, §5.5, Table 3. P. Chen, J. Gong, and T. Chen (2025) Accuracy can lie: on the impact of surrogate model in configuration tuning. IEEE Trans. Software Eng. 51 (2), p. 548–580. External Links: Link, Document Cited by: §1, §5.4. T. Chen, R. Bahsoon, and X. Yao (2018a) A survey and taxonomy of self-aware and self-adaptive cloud autoscaling systems. ACM Comput. Surv. 51 (3), p. 61:1–61:40. External Links: Link, Document Cited by: §1. T. Chen and R. Bahsoon (2015) Toward a smarter cloud: self-aware autoscaling of cloud configurations and resources. Computer 48 (9), p. 93–96. External Links: Link, Document Cited by: §1. T. Chen, K. Li, R. Bahsoon, and X. Yao (2018b) FEMOSAA: feature-guided and knee-driven multi-objective optimization for self-adaptive software. ACM Transactions on Software Engineering and Methodology (TOSEM) 27 (2), p. 1–50. Cited by: §1. T. Chen and M. Li (2021) Multi-objectivizing software configuration tuning. In ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021, D. Spinellis, G. Gousios, M. Chechik, and M. D. Penta (Eds.), p. 453–465. External Links: Link, Document Cited by: §1, 7, 29, 8. T. Chen and M. Li (2024) Adapting multi-objectivized software configuration tuning. Proc. ACM Softw. Eng. 1 (FSE), p. 539–561. External Links: Link, Document Cited by: §1. T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Q. Yan, H. Shen, M. Cowan, L. Wang, Y. Hu, L. Ceze, C. Guestrin, and A. Krishnamurthy (2018c) TVM: an automated end-to-end optimizing compiler for deep learning. In 13th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2018, Carlsbad, CA, USA, October 8-10, 2018, A. C. Arpaci-Dusseau and G. Voelker (Eds.), p. 578–594. External Links: Link Cited by: §8. [18] (2024) Clang. Note: https://releases.llvm.org/download.htmlVersion: 17.0.6. Accessed: 2024-02-20 Cited by: Table 1. A. I. Cowen-Rivers, W. Lyu, R. Tutunov, Z. Wang, A. Grosnit, R. R. Griffiths, A. M. Maraval, H. Jianye, J. Wang, J. Peters, et al. (2022) Hebo: pushing the limits of sample-efficient hyper-parameter optimisation. Journal of Artificial Intelligence Research 74, p. 1269–1349. Cited by: Table 3. T. F. Cox and M. A. Cox (2000) Multidimensional scaling. CRC press. Cited by: Figure 3, Figure 3. [21] (2024) Csmith. Note: https://github.com/csmith-project/csmithAccessed: 2024-02-20 Cited by: 3rd item. K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan (2002) A fast and elitist multiobjective genetic algorithm: nsga-i. IEEE Trans. Evol. Comput. 6 (2), p. 182–197. External Links: Document Cited by: §4.1, §4, footnote 6. T. Domhan, J. T. Springenberg, and F. Hutter (2015) Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, Q. Yang and M. J. Wooldridge (Eds.), p. 3460–3468. External Links: Link Cited by: §1. J. Echevarrieta, E. Arza, and A. Pérez (2025) Speeding-up evolutionary algorithms to solve black-box optimization problems. IEEE Trans. Evol. Comput. 29 (1), p. 117–131. External Links: Link, Document Cited by: §1, §8. K. Eggensperger, P. Müller, N. Mallik, M. Feurer, R. Sass, A. Klein, N. H. Awad, M. Lindauer, and F. Hutter (2021) HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, J. Vanschoren and S. Yeung (Eds.), External Links: Link Cited by: §3.1. S. Falkner, A. Klein, and F. Hutter (2018) BOHB: robust and efficient hyperparameter optimization at scale. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, J. G. Dy and A. Krause (Eds.), Proceedings of Machine Learning Research, Vol. 80, p. 1436–1445. External Links: Link Cited by: §1, §1, §2.2, §3.3, §5.5, Table 3, §8. A. I. Forrester, A. Sóbester, and A. J. Keane (2007) Multi-fidelity optimization via surrogate modelling. Proceedings of the royal society a: mathematical, physical and engineering sciences 463 (2088), p. 3251–3269. Cited by: §8. [28] (2024) Gcc. Note: https://gcc.gnu.org/pub/gcc/releases/gcc-14.2.0/Version: 14.2. Accessed: 2024-02-20 Cited by: Table 1. J. Gong, T. Chen, and R. Bahsoon (2025) Dividable configuration performance learning. IEEE Trans. Software Eng. 51 (1), p. 106–134. External Links: Link, Document Cited by: §1. J. Gong and T. Chen (2023) Predicting software performance with divide-and-learn. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA, December 3-9, 2023, S. Chandra, K. Blincoe, and P. Tonella (Eds.), p. 858–870. External Links: Link, Document Cited by: §8. J. Gong and T. Chen (2024) Predicting configuration performance in multiple environments with sequential meta-learning. Proc. ACM Softw. Eng. 1 (FSE), p. 359–382. External Links: Link, Document Cited by: §8. X. Han and T. Yu (2016) An empirical study on performance bugs for highly configurable software systems. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2016, Ciudad Real, Spain, September 8-9, 2016, p. 23:1–23:10. External Links: Link, Document Cited by: §1. J. Hauke and T. Kossowski (2011) Comparison of values of pearson’s and spearman’s correlation coefficients on the same sets of data. Quaestiones geographicae 30 (2), p. 87–93. Cited by: 2nd item. H. He, Z. Jia, S. Li, Y. Yu, C. Zhou, Q. Liao, J. Wang, and X. Liao (2022) Multi-intention-aware configuration selection for performance tuning. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, p. 1431–1442. External Links: Link, Document Cited by: §5.1. S. Herbold (2017) Comments on scottknottesd in response to ”an empirical comparison of model validation techniques for defect prediction models”. IEEE Trans. Software Eng. 43 (11), p. 1091–1094. External Links: Link, Document Cited by: §5.6. Md. A. Hossen, M. A. Javidian, V. Narayanan, J. M. O’Kane, and P. Jamshidi (2026) Multi-objective multi-fidelity bayesian optimization with causal priors. CoRR abs/2602.00788. External Links: Link, Document, 2602.00788 Cited by: §8. [37] (2024) Httpd. Note: https://httpd.apache.org/download.cgiVersion: 2.4.57. Accessed: 2024-02-20 Cited by: Table 1. Q. Hu, Z. Ye, M. Zhang, Q. Chen, P. Sun, Y. Wen, and T. Zhang (2023) Hydro: surrogate-based hyperparameter tuning service in datacenters. In 17th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2023, Boston, MA, USA, July 10-12, 2023, R. Geambasu and E. Nightingale (Eds.), p. 757–777. External Links: Link Cited by: §1. Y. Hu, Y. Yu, W. Tu, Q. Yang, Y. Chen, and W. Dai (2019) Multi-fidelity automatic hyper-parameter tuning via transfer series expansion. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, p. 3846–3853. External Links: Link, Document Cited by: §1, §1, §1, §3.3, §8. F. Hutter, H. H. Hoos, and K. Leyton-Brown (2011) Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization - 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers, C. A. C. Coello (Ed.), Lecture Notes in Computer Science, Vol. 6683, p. 507–523. External Links: Link, Document Cited by: §1, Table 3. K. Jamieson and A. Talwalkar (2016) Non-stochastic best arm identification and hyperparameter optimization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016, A. Gretton and C. C. Robert (Eds.), JMLR Workshop and Conference Proceedings, Vol. 51, p. 240–248. External Links: Link Cited by: §8. P. Jamshidi and G. Casale (2016) An uncertainty-aware approach to optimal configuration of stream processing systems. In 24th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2016, London, United Kingdom, September 19-21, 2016, p. 39–48. External Links: Link, Document Cited by: §1. K. Kandasamy, G. Dasarathy, J. G. Schneider, and B. Póczos (2017) Multi-fidelity bayesian optimisation with continuous approximations. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, D. Precup and Y. W. Teh (Eds.), Proceedings of Machine Learning Research, Vol. 70, p. 1799–1808. External Links: Link Cited by: §1, §1, §2.2. K. Kanellis, C. Ding, B. Kroth, A. Müller, C. Curino, and S. Venkataraman (2022) LlamaTune: sample-efficient DBMS configuration tuning. Proc. VLDB Endow. 15 (11), p. 2953–2965. External Links: Link, Document Cited by: §5.1. A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter (2017) Fast bayesian optimization of machine learning hyperparameters on large datasets. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA, A. Singh and X. (. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 54, p. 528–536. External Links: Link Cited by: §1, §1, §1, §8. L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar (2017) Hyperband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res. 18, p. 185:1–185:52. External Links: Link Cited by: §1, §1, §2.2, §3.3, §5.5, Table 3, §8. Y. Li, Y. Shen, J. Jiang, J. Gao, C. Zhang, and B. Cui (2021) MFES-HB: efficient hyperband with multi-fidelity quality measurements. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, p. 8491–8500. External Links: Link, Document Cited by: §1, §1, §2.2, §3.3. H. Liang, Y. Huang, and T. Chen (2025) The same only different: on information modality for configuration performance analysis. In 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025, p. 2522–2534. External Links: Link, Document Cited by: §1. Y. Ma, T. Chen, and K. Li (2025) Faster configuration performance bug testing with neural dual-level prioritization. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering, p. 988–1000. External Links: Link Cited by: §1. N. Mallik, E. Bergman, C. Hvarfner, D. Stoll, M. Janowski, M. Lindauer, L. Nardi, and F. Hutter (2023) PriorBand: practical hyperparameter optimization in the age of deep learning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), External Links: Link Cited by: §5.5, Table 3. M. L. McHugh (2011) Multiple comparison analysis testing in anova. Biochemia medica 21 (3), p. 203–209. Cited by: §5.6. M. D. McKay (1992) Latin hypercube sampling as a tool in uncertainty analysis of computer models. In Proceedings of the 24th Winter Simulation Conference, Arlington, VA, USA, December 13-16, 1992, R. C. Crain (Ed.), p. 557–564. External Links: Link, Document Cited by: §3.3, item 1. [53] (2024) MySQL. Note: https://dev.mysql.com/doc/relnotes/mysql/5.7/en/Version: 5.7.19. Accessed: 2024-02-20 Cited by: Table 1. V. Nair, Z. Yu, T. Menzies, N. Siegmund, and S. Apel (2020) Finding faster configurations using FLASH. IEEE Trans. Software Eng. 46 (7), p. 794–811. External Links: Link, Document Cited by: §5.5, Table 3. [55] (2024) PostgreSQL. Note: https://w.postgresql.org/docs/release/12.7/Version: 12.7. Accessed: 2024-02-20 Cited by: Table 1. A. Shahbazian, S. Karthik, Y. Brun, and N. Medvidovic (2020) EQual: informing early design decisions. In ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, P. Devanbu, M. B. Cohen, and T. Zimmermann (Eds.), p. 1039–1051. External Links: Link, Document Cited by: Table 3, §8. J. Snoek, H. Larochelle, and R. P. Adams (2012) Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, P. L. Bartlett, F. C. N. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.), p. 2960–2968. External Links: Link Cited by: §1. [58] (2024) Sysbench. Note: https://github.com/akopytov/sysbenchAccessed: 2024-02-20 Cited by: §1, §3.1, §3.3, 1st item. [59] (2024) Tomcat. Note: https://tomcat.apache.org/download-10.cgiVersion: 10.1.34. Accessed: 2024-02-20 Cited by: Table 1. [60] (2024) Wrk. Note: https://github.com/wg/wrkAccessed: 2024-02-20 Cited by: §3.1, 2nd item. Z. Xiang, J. Gong, and T. Chen (2026) Dually hierarchical drift adaptation for online configuration performance learning. In 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE), Cited by: §1. G. Xiong and T. Chen (2025) CoTune: co-evolutionary configuration tuning. In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025, p. 1490–1502. External Links: Link, Document Cited by: §1. T. Xu, L. Jin, X. Fan, Y. Zhou, S. Pasupathy, and R. Talwadker (2015) Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015, E. D. Nitto, M. Harman, and P. Heymans (Eds.), p. 307–319. External Links: Link, Document Cited by: §1. Y. Ye, T. Chen, and M. Li (2025) Distilled lifelong self-adaptation for configurable systems. In 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025, p. 1333–1345. External Links: Link, Document Cited by: §8. Y. Ye, H. Liang, C. Jiang, M. Li, and T. Chen (2026) Revealing domain-spatiality patterns for configuration tuning: domain knowledge meets fitness landscapes. ACM Trans. Softw. Eng. Methodol.. Note: Just Accepted External Links: ISSN 1049-331X, Link, Document Cited by: §1. X. Zhang, Z. Chang, Y. Li, H. Wu, J. Tan, F. Li, and B. Cui (2022) Facilitating database tuning with hyper-parameter optimization: A comprehensive experimental evaluation. Proc. VLDB Endow. 15 (9), p. 1808–1821. External Links: Link, Document Cited by: §1, §5.1. X. Zhang, H. Wu, Z. Chang, S. Jin, J. Tan, F. Li, T. Zhang, and B. Cui (2021) ResTune: resource oriented tuning boosted by meta-learning for cloud databases. In SIGMOD ’21: International Conference on Management of Data, Virtual Event, China, June 20-25, 2021, G. Li, Z. Li, S. Idreos, and D. Srivastava (Eds.), p. 2102–2114. External Links: Link, Document Cited by: §1, §5.1, 2nd item. X. Zhang, H. Wu, Y. Li, Z. Tang, J. Tan, F. Li, and B. Cui (2023) An efficient transfer learning based configuration adviser for database tuning. Proc. VLDB Endow. 17 (3), p. 539–552. External Links: Link, Document Cited by: §1, 3rd item. M. Zhu and D. Hao (2023) Compiler auto-tuning via critical flag selection. In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023, p. 1000–1011. External Links: Link, Document Cited by: §8. Y. Zhu, J. Liu, M. Guo, Y. Bao, W. Ma, Z. Liu, K. Song, and Y. Yang (2017) BestConfig: tapping the performance potential of systems via automatic configuration tuning. In Proceedings of the 2017 Symposium on Cloud Computing, SoCC 2017, Santa Clara, CA, USA, September 24-27, 2017, p. 338–350. External Links: Link, Document Cited by: §1, Table 3.