Paper deep dive
EffiPair: Improving the Efficiency of LLM-generated Code with Relative Contrastive Feedback
Samira Hajizadeh, Suman Jana
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/10/2026, 3:02:22 AM
Summary
EffiPair is an inference-time iterative refinement framework that improves the efficiency of LLM-generated code by using Relative Contrastive Feedback (RCF). Instead of relying on absolute scalar metrics, EffiPair compares structurally similar program pairs with large efficiency gaps to generate targeted, lightweight feedback, significantly reducing token overhead and profiling costs while improving runtime performance.
Entities (5)
Relation Signals (3)
EffiPair → utilizes → Relative Contrastive Feedback
confidence 100% · EffiPair, an inference-time iterative refinement framework that operates... by... summarizing their execution differences into lightweight feedback
EffiPair → improves → Code Efficiency
confidence 95% · EffiPair consistently improves efficiency while preserving correctness.
EffiPair → uses → Scalene
confidence 95% · We use Scalene (Berger et al. (2023)) as our profiler
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) often generate code that is functionally correct but inefficient in runtime and memory. Prior approaches to improving code efficiency typically rely on absolute execution feedback, such as profiling a single program's runtime or memory usage, which is costly and provides weak guidance for refinement. We propose Relative Contrastive Feedback (RCF), an inference-time feedback mechanism that requires no model fine-tuning or parameter updates. RCF compares two structurally similar programs for the same task and highlights the differences associated with better efficiency. Building on this idea, we introduce EffiPair, an inference-time iterative refinement framework that operates entirely at test time by generating multiple candidate solutions, identifying informative program pairs with large efficiency gaps, summarizing their execution differences into lightweight feedback, and using this signal to produce more efficient solutions. By replacing isolated scalar feedback with pairwise contrastive comparisons, EffiPair provides more direct guidance while reducing profiling and prompting overhead. Experiments on code-efficiency benchmarks show that EffiPair consistently improves efficiency while preserving correctness. For instance, with DeepSeek-Chat V3.2, EffiPair achieves up to 1.5x speedup over generation without performance feedback, while reducing token usage by more than 90% compared to prior work.
Tags
Links
- Source: https://arxiv.org/abs/2604.05137v1
- Canonical: https://arxiv.org/abs/2604.05137v1
Trouble viewing inline? Open PDF directly →
Full Text
49,103 characters extracted from source content.
Expand or collapse full text
Preprint. Under review. EFFIPAIR: Improving the Efficiency of LLM-generated Code with Relative Contrastive Feedback Samira Hajizadeh Department of Electrical Engineering Columbia University New York, NY, USA samira.hajizadeh@columbia.edu Suman Jana Department of Computer Science Columbia University New York, NY, USA suman@cs.columbia.edu Abstract Large language models (LLMs) often generate code that is functionally correct but inefficient in runtime and memory. Prior approaches to improv- ing code efficiency typically rely on absolute execution feedback, such as profiling a single program’s runtime or memory usage, which is costly and provides weak guidance for refinement. We propose Relative Contrastive Feedback (RCF), an inference-time feedback mechanism that requires no model fine-tuning or parameter updates. RCF compares two structurally similar programs for the same task and highlights the differences associated with better efficiency. Building on this idea, we introduce EFFIPAIR, an inference-time iterative refinement framework that operates entirely at test time by generating multiple candidate solutions, identifying informative program pairs with large efficiency gaps, summarizing their execution dif- ferences into lightweight feedback, and using this signal to produce more efficient solutions. By replacing isolated scalar feedback with pairwise contrastive comparisons, EFFIPAIR provides more direct guidance while re- ducing profiling and prompting overhead. Experiments on code-efficiency benchmarks show that EFFIPAIR consistently improves efficiency while preserving correctness. For instance, with DeepSeek-Chat V3.2, EFFIPAIR achieves up to 1.5x speedup over generation without performance feed- back, while reducing token usage by more than 90% compared to prior work. 1 Introduction Large language models (LLMs) are increasingly used for code generation in settings ranging from interactive “vibe coding” to production software pipelines. In many of these workflows, code is generated and refined at inference time with minimal or no additional training, placing the burden of optimization directly on the generation process. As a result, models often produce programs that are unnecessarily verbose, inefficient, or suboptimal in their use of memory and compute. When such code is reused across services or deployed at scale, even modest inefficiencies can compound into significant execution cost. Improving the efficiency of model-generated programs has therefore emerged as an important problem alongside ensuring functional correctness. Recent work has begun treating program efficiency as an explicit optimization objective rather than a fixed outcome of generation. Benchmarks such as EffiBench (Huang et al., 2025b) measure execution time and memory usage of LLM-generated programs and show that model outputs are often substantially less efficient than canonical human solutions. To address this, several approaches focus on inference-time refinement, where the model iteratively generates candidate programs, executes them, and uses feedback (e.g., runtime, memory usage, or execution traces) to guide subsequent revisions. These methods treat efficiency optimization as a feedback-driven process, analogous to learning from reward signals or human preferences, but operate entirely at inference time. 1 arXiv:2604.05137v1 [cs.PL] 6 Apr 2026 Preprint. Under review. Figure 1: Overview of EFFIPAIR. Given a coding task, the LLM samplesNcandidate programs. Candidates are checked for correctness and profiled for efficiency, then stored in a candidate pool. EFFIPAIR selects a pair consisting of an efficient reference programp + and a similar but less efficient candidatep − , summarizes their relative execution differences into compact Relative Contrastive Feedback (RCF), and uses this signal to refine the program. The refined program is added back to the pool for subsequent rounds. However, existing approaches rely primarily on absolute execution feedback, where each can- didate program is evaluated independently and assigned scalar measurements such as runtime or memory usage. While informative, this form of feedback is fundamentally lim- ited in its information efficiency: scalar measurements provide only absolute performance values without revealing the relative structure among candidate programs. As a result, models must infer optimization directions indirectly, often requiring multiple iterations, extensive profiling, and high token overhead to converge toward efficient solutions. In contrast, we argue that relative performance comparisons between candidate programs provide a richer and more actionable signal for efficiency optimization. When two programs solve the same task, their relative efficiency can directly reveal which structural choices lead to better performance. Rather than requiring the model to interpret absolute profiling values in isolation, such comparisons expose the optimization-relevant differences between candidates and provide clearer guidance on how the program should be revised. Motivated by this insight, we introduce Relative Contrastive Feedback (RCF), a new opti- mization signal for LLM-based program synthesis. RCF is defined as structured feedback derived from pairwise comparisons between candidate programs based on their execution behavior. Rather than treating candidate programs as isolated trajectories, RCF leverages relational information between programs to guide refinement. This transforms efficiency optimization from scalar reward maximization into a contrastive learning problem over program executions. RCF offers several fundamental advantages over absolute profiling feedback. First, it provides stronger directional guidance by explicitly identifying which program variants are more efficient. Second, it is more information-efficient, as each comparison conveys optimization-relevant structure without requiring precise scalar measurements. Third, it enables cooperative optimization across multiple candidate generations, allowing models to learn from the relationships between programs rather than from individual executions alone. These properties make RCF particularly well-suited for iterative self-refinement settings. To operationalize RCF, we introduce EFFIPAIR, a framework that integrates RCF into the program generation and refinement loop. EFFIPAIR executes multiple candidate programs in parallel, compares their efficiency using lightweight summarized profiling signals, and uses these comparisons to guide subsequent refinements. This pairing mechanism enables efficient information sharing across candidate programs while minimizing token and execu- tion overhead. Across multiple code-efficiency benchmarks and strong language models, EFFIPAIR consistently improves runtime efficiency while preserving, and in some cases improving correctness. Relative to EffiLearner (Huang et al., 2025a), EFFIPAIR achieves comparable or better efficiency gains, higher correctness, and much lower token overhead, 2 Preprint. Under review. reducing prompt tokens from EffiLearner ’s 23.6M to 200k and completion tokens from 350k to 77k. In summary, our work makes the following contributions: (1) We introduce Relative Con- trastive Feedback (RCF), a new supervision signal for LLM program synthesis that leverages pairwise execution comparisons to guide optimization. (2) We present EFFIPAIR, a refine- ment framework that operationalizes RCF and integrates contrastive execution feedback into the program generation loop. (3) We show that RCF enables more efficient optimization of generated programs compared to methods based solely on absolute profiling feedback. (4) We demonstrate that summarized execution statistics are sufficient to provide effective contrastive feedback, reducing profiling overhead while preserving optimization effec- tiveness. Together, these results establish RCF as a principled and effective signal for improving the efficiency of LLM-generated programs and suggest a broader paradigm for execution-grounded learning based on relational feedback between candidate solutions. 2 Related Work Large language models (LLMs) are increasingly used for code generation, spanning interac- tive prototyping to production pipelines (Jiang et al., 2026; Fakhoury et al., 2024; Jimenez et al., 2024). However, generated programs are often inefficient in runtime and memory (Huang et al., 2025b), leading to nontrivial cost and sustainability concerns at scale. Recent work therefore treats efficiency as an explicit optimization objective, considered alongside (Peng et al., 2024), or even prior to, correctness (Ye et al., 2025). A growing set of benchmarks has been proposed to evaluate this problem (Waghjale et al., 2024; Huang et al., 2025b; Du et al., 2024; Liu et al., 2024; Qiu et al., 2025), consistently showing that LLM-generated code lags behind expert implementations in efficiency. To address this gap, prior work explores both training-time and inference-time strategies. Training-time approaches curate performance-improving edits or incorporate efficiency- aware fine-tuning (Shypula et al., 2024; Huang et al., 2025c). Inference-time methods instead rely on iterative refinement using execution feedback. EffiLearner (Huang et al., 2025a) feeds runtime and memory profiles back to the model, while PerfCodeGen (Peng et al., 2024) uses execution feedback from test runs to guide refinement. LLM4EFFI (Ye et al., 2025) further structures this process by separating algorithm selection, implementation optimization, and correctness refinement. Existing methods mainly rely on absolute execution feedback, evaluating each candidate independently with scalar metrics that provide limited guidance and require repeated profiling, increasing compute and token overhead. Our approach instead compares struc- turally similar candidates with different efficiency profiles to directly identify useful design changes, yielding more informative and efficient refinement. It also exploits candidate diversity through similarity-based pairing, allowing cooperative refinement across parallel trajectories, and jointly optimizes correctness and efficiency within a unified loop, rather than decomposing them into separate stages as in prior approaches (Ye et al., 2025; Peng et al., 2024; Huang et al., 2025a). 3 Methodology 3.1 Core Idea EFFIPAIR improves the efficiency of LLM-generated code during generation by leveraging Relative Contrastive Feedback (RCF): it explicitly contrasts pairs of candidate solutions that are highly similar (above a similarity threshold) yet exhibit substantially different efficiency. Instead of returning verbose profiler traces or relying on absolute scalar metrics alone, EFFIPAIR distills execution measurements into a compact pairwise feedback signal that identifies which candidate is more efficient and why, enabling targeted refinements with minimal token overhead. 3 Preprint. Under review. Figure 1 illustrates the end-to-end workflow of EFFIPAIR. Given a coding task, the LLM samplesNcandidate programs, checks them for correctness, and profiles their efficiency before storing them in a candidate pool. To guide refinement, EFFIPAIR constructs a Relative Contrastive Feedback (RCF) instance by selecting a pair of correct programs: an efficient reference candidatep + (typically the fastest correct solution currently available) and a sufficiently similar but less efficient candidatep − , chosen as the worst-performing among similar correct candidates. From their executions, EFFIPAIR summarizes the key relative per- formance differences into compact RCF, highlighting which operations dominate runtime or memory in the slower program and which corresponding choices in the faster program avoid them. The model is then prompted with both programs and this RCF summary to produce edits that preserve correctness while targeting the inefficient design choices identi- fied by the contrast. The refined program is added back to the pool for subsequent rounds; if no correct candidates exist, the system instead selects from the available candidates to continue exploration until a correct solution is found. 3.2 Problem setup Given a certain programming taskT (d,S)with natural-language descriptiondand correct- ness testsS =s i |S| i=0 , our objective is to produce a programpthat is functionally correct while minimizing efficiency cost. Lets i ( p)∈0, 1indicate whetherppasses the ith test in S , and let e( p) denote an efficiency cost (e.g., runtime). Our end goal therefore is: p ∗ = arg min p e( p)s.t. ∀is i ( p) = 1. We pursue this goal by iteratively refining a candidate pool via Relative Contrastive Feed- back: we present the LLM with pairs of candidates and a distilled relative performance comparison, and prompt it to produce a more efficient correct solution. 3.3 Generation and pairing strategy To achieve our goal, we implement an iterative pipeline that maps each description to a program through repeated propose–evaluate–refine rounds. First, we generate an initial pool of candidates,C 0 = p (j) 0 n j=1 , by prompting the LLM with the task descriptiond without any additional guiding signals and requesting a code solution. We run the LLMN times per task to produce diverse candidate variants. For roundt ≥1, given the candidate setC i t = p (i,j) t n j=1 for thei-th taskT i (d,S), we first run correctness checks to computes i ( p), and profilers to estimate efficiencye( p). We then derive, for each candidate, an embeddingE (i,j) and an abstract syntax tree (AST) representationA (i,j) , corresponding to thej-th generated program for taski. Using these measurements, we select a reference solution p + as the most efficient correct program: p + = arg min p e( p)s.t. ∀is i ( p) = 1, Next, we construct a paired negative (inefficient) samplep − . Letsim( p,q)∈ [0, 1]denote a program similarity score which is a weighted sum of the cosine similarity of the embeddings and the cosine similarity of the AST of p + and p − : sim( p, q) = α· cos E( p), E(q) + (1− α)· cos A( p), A(q) ,α∈ [0, 1]. We define the candidate setN ( p + )consisting of programs that are sufficiently similar top + , and choose the paired inefficient program p − as the worst-performing element in this set. p − = arg max q∈N ( p + ) e(q), N ( p + ) =q :∀is i (q) = 1,sim( p + , q)≥ τ, 4 Preprint. Under review. ModelMethodPass@1TimingDPSDPS norm DeepSeek-Chat V3.2 Baseline90.68%0.12879.7379.97 Paired (No Profiling)91.53%0.12685.0183.29 Solo (Summary Profiling) 92.37%0.04385.8983.21 EFFIPAIR92.37% 0.023 (↓0.020) 87.46 (↑1.57) 84.30 (↑1.09) Claude Sonnet 4.6 Baseline92.37%0.2182.7478.92 Paired (No Profiling)94.07%0.04989.2286.09 Solo (Summary Profiling)94.07%0.03791.7389.05 EFFIPAIR94.07% 0.021 (↓0.016) 92.97 (↑1.24) 89.06 (↑0.01) GPT-5.4 Baseline91.53%0.05780.1079.73 Paired (No Profiling)94.92%0.05285.7184.33 Solo (Summary Profiling)94.92%0.03485.7884.87 EFFIPAIR94.92% 0.029 (↓0.005) 86.89 (↑1.11) 85.08 (↑0.21) GPT-5.4 mini Baseline90.68%0.07783.7481.60 Paired (No Profiling)93.22%0.06984.6784.76 Solo (Summary Profiling) 94.07%0.06484.4586.44 EFFIPAIR94.07% 0.028 (↓0.036) 89.20 (↑4.75) 87.65 (↑1.21) Table 1: Model-level comparison of baseline generation, paired prompting without profiling, single-candidate with summarized profiling, and EFFIPAIR. Green annotations in the EFFIPAIR row indicate improvements relative to the corresponding Solo baseline. 3.4 Iterative refinement After we choose our program pair( p + ,p − ), we obtain the profiling signal or the execution feedback for the incorrect code, and feed this information to the LLM. For execution feedback we run the codes with the input/output pairs provided in the datasets for correctness verification, and feed the resulting assertion error as an execution feedback signal. For the performance analytics signal we, for the first time, use a sampling profiler. A sampling profiler probes the target program’s call stack at regular intervals using operating system interrupts. Instead of feeding raw profiler traces, we summarize profiling outputs into a compact signal containing lightweight cues (e.g., hottest functions/lines, relative time shares, peak-memory hotspots). With the pairing ofp + andp − and their execution and performance feedback, we construct a paired contrast signal∆( p + ,p − )that highlights what to change to arrive at a more efficient solution. Given task descriptiond, the pair( p + ,p − ), and the contrast signal∆, the refinement step produces an improved program ̃ p = R(d, p + , p − ,∆( p + , p − )) which is added back to the candidate set/pool for subsequent rounds. The key idea of EFFIPAIR is that conditioning on both a strong efficient referencep + and an inefficient-but- similar target( p − ), plus compact contrastive feedback, encourages targeted edits with low token overhead. 4 Experiments 4.1 Setup We evaluate EFFIPAIR on a suite of code-generation benchmarks designed to assess not only functional correctness but also the runtime and memory efficiency of generated programs. Our analysis emphasizes relative improvements, namely speedup and memory reduction, rather than absolute execution times, since raw latency is sensitive to hardware, operating system scheduling, and library versions. Specifically, we study the effect of EFFIPAIR on three code-efficiency benchmarks: Mercury (Du et al. (2024)), ENAMEL (Qiu et al. (2025)), and EvalPerf (Liu et al. (2024)). EvalPerf targets performance-challenging tasks and reports 5 Preprint. Under review. DPS which measures a solution’s efficiency relative to the nearest slower reference solution. DPS norm is the normalized DPS by the total number of solutions. Mercury uses the Beyond metric, which normalizes runtime percentiles over each task’s runtime distribution to enable hardware-robust comparisons. ENAMEL reports eff@1, a weighted score based on the worst execution time across test cases of different difficulty levels; higher values indicate better efficiency, and values above 1 mean the generated code outperforms the expert solution. We also report Pass@1 and Speedup (average runtime of the baseline divided by average runtime of the method) for all runs. We use a set of state-of-the-art language models to examine whether EFFIPAIR can improve systems that already have strong efficiency and correctness performance. Specifically, our evaluation includes GPT-5.4, GPT-5.4 mini (OpenAI (2026)), Claude Sonnet 4.6 (Anthropic (2026)), and DeepSeek-Chat V3.2 (DeepSeek-AI et al. (2025)). This setting provides a strin- gent test of whether EFFIPAIR remains effective even when the underlying model is already highly capable. All experiments were run on a dedicated machine with dual Intel Xeon E5-2640 @ 2.50 GHz CPUs (2 sockets × 6 cores/socket, 2 threads/core; 24 logical cores) and 125 GB RAM, running Ubuntu 18.04.6 LTS with Python 3.10.8 (GCC 7.5.0 / Clang 12.0.0 toolchain). We pin package versions and provide the full environment specification in the supplemental material to support reproducibility. 4.2EFFIPAIR configuration EFFIPAIR usesN =3 initial candidates andT =3 refinement rounds by default, which provides a practical balance between candidate diversity and inference cost. In each round, we pair the current best candidate with the slowest sufficiently similar alternative; when such a pair is unavailable, we fall back to the closest available incorrect candidate or to single- candidate refinement. To reduce test-case overfitting, benchmark tests are not included in the prompt; instead, the model receives only compact execution and profiling feedback. Detailed similarity and profiling settings are given in 4.2.1 and 4.2.2. 4.2.1 Similarity We compute similarity using the embedding-AST mixture described in subsection 3.3, with mixing weightα =0.8 and thresholdτ =0.85 for the main experiments, and defer discussion of these parameter choices and sensitivity analyses to Appendix subsections A.1.1 and A.1.2. We combine semantic and structural similarity because either signal alone is incomplete: embeddings help identify programs that solve the task in a similar semantic way, while AST features help preserve local structural comparability. We use Jina Embeddings v2 Base Code model (G ̈ unther et al. (2024); Jina AI) for the embedding component because it is a code-oriented encoder model, making it a practical choice for computing semantic similarity between candidate programs at scale. Appendix subsection A.2 discusses the implementation of AST in more detail. 4.2.2 Profiling We use Scalene (Berger et al. (2023)) as our profiler because it offers line-level CPU and memory measurements at relatively low overhead, which is particularly important in an iterative refinement pipeline. Since Scalene is sampling-based, it reports relative CPU usage rather than exact execution costs. For our purposes, this level of granularity is sufficient: EFFIPAIR does not require perfectly precise profiling, but only a reliable signal about which parts of the program dominate execution and therefore indicate promising directions for optimization and RCF generation. We discuss this design choice in more detail in Appendix subsection A.4.1. Additionally, we summarize the profiling output into a compact signal containing only lines responsible for more than 1% of CPU time or at least 100 allocations. This compression removes verbose and often uninformative details while preserving the principal hotspots most relevant to program improvement. To improve measurement reliability, we repeatedly run the target function until total execution time reaches at least 1.0 second, since very short runs can produce noisy samples and unstable hotspot estimates. 6 Preprint. Under review. LLMsMethods EvalPerfMercuryENAMEL DPS norm Pass@1Beyond@1Pass@1eff@1Pass@1 GPT-4o mini EffiLearner Generation80.0485.5969.5982.8148.2680.28 ECCO75.1844.0772.2986.3330.7557.75 EffiLearner Optimization79.8081.3673.4588.6745.6977.46 LLM4EFFI83.7888.1474.9489.4549.8980.99 Baseline78.8086.4471.5983.9840.6775.00 EFFIPAIR (Ours)84.74 (↑5.94)92.37 (↑5.93)77.39 (↑5.80)87.50 (↑3.52)50.37 (↑9.70)78.46 (↑3.46) Table 2: Cross-benchmark comparison with prior methods. MethodSpeedup Pass@1DPS DPS norm ∼Token Usage PromptCompletion EffiLearner Generation1x89.83%80.6679.424,733,71564,529 EffiLearner Optimization1.49x88.14%75.9181.6723,651,451362,683 EFFIPAIR (Ours)1.50x92.37%87.4684.30218,05377,009 Table 3: Method-level comparison between EffiLearner and EFFIPAIR. We additionally set the CPU sampling interval to 0.001 to further stabilize the profiling signal across runs. 4.3 Evaluation We evaluate candidates with task-specific harnesses in isolated subprocesses and count solutions as correct only when they pass the original benchmark tests without modification. Efficiency is measured with repeated wall-clock runs under fixed timeouts and determin- istic seeds to improve robustness and reproducibility. Full implementation details of the evaluation setup are provided in Appendix A.3. Table 1 row definitions.Baselinedenotes direct generation without iterative refinement or performance feedback.Paired (No Profiling)refines using a pair of candidates but without profiler feedback.Solo (Summary Profiling)refines a single candidate using only summarized profiling feedback; code/AST similarity and candidate pairing are not used in this setting. The solo candidate is chosen as the fastest correct active entry, or the fastest incorrect active entry when no correct candidate is available. EFFIPAIR denotes the full method, combining similarity-based pairing with summarized contrastive profiling feedback. 4.4 Main results Table 1 shows a consistent pattern across all four evaluated models: EFFIPAIR preserves the strongest correctness while delivering the best efficiency metrics. For DeepSeek-Chat V3.2, EFFIPAIR matches the best Pass@1 of 92.37% while reducing timing from 0.043 under solo summarized refinement to 0.023 and improving DPS/DPS norm from 85.89/83.21 to 87.46/84.30. For Claude Sonnet 4.6, Pass@1 remains unchanged at 94.07% across the strongest refinement settings, but EFFIPAIR achieves the best timing and the highest DPS- based scores, reaching 0.021 timing, 92.97 DPS, and 89.06DPS norm . The same trend holds for GPT-5.4 and GPT-5.4 mini: EFFIPAIR preserves the top Pass@1 attained by refinement while further lowering timing and improving both DPS andDPS norm relative to paired prompting without profiling and solo summarized refinement. Overall, these results indicate that contrastive paired refinement provides a robust efficiency benefit across strong code- generation models without sacrificing functional correctness. Table 2 presents a cross-benchmark comparison of EFFIPAIR against prior methods on GPT- 4o mini across EvalPerf, Mercury, and ENAMEL. The main pattern is that EFFIPAIR achieves the strongest efficiency metric on all three benchmarks, reaching 84.74DPS norm on EvalPerf, 7 Preprint. Under review. 77.39 Beyond@1 on Mercury, and 50.37 eff@1 on ENAMEL. These results improve over both the direct Baseline (78.80 / 71.59 / 40.67) and the strongest prior competitor, LLM4EFFI (83.78 / 74.94 / 49.89), indicating that EFFIPAIR delivers more consistent efficiency gains across diverse evaluation settings. In terms of correctness, EFFIPAIR also attains the highest Pass@1 on EvalPerf at 92.37%, outperforming Baseline (86.44%) and LLM4EFFI (88.14%). On Mercury and ENAMEL, the best Pass@1 remains with prior methods, LLM4EFFI reaches 89.45% and 80.99%, respectively, compared with 87.50% and 78.46% for EFFIPAIR. Overall, Table 2 suggests that the primary advantage of EFFIPAIR is its robust, benchmark-wide improvement in efficiency, while maintaining competitive correctness and even substantially improving correctness on EvalPerf. The results above the horizontal line are taken directly from the LLM4EFFI paper for reference, whereas the EFFIPAIR and baseline results are obtained in our experimental setup. Table 3 shows that EFFIPAIR performs favorably relative to both the baseline generation setting and EffiLearner. It reaches the highest Pass@1 (92.37%) and slightly improves speedup (the average runtime of EffiLearner Generation divided by method) while also obtaining stronger DPS andDPS norm scores. In addition, EFFIPAIR uses substantially fewer prompt tokens than EffiLearner, requiring only∼200k prompt tokens and∼77k completion tokens, compared with EffiLearner ’s∼23.6M prompt tokens and∼360k completion tokens. These results suggest that EFFIPAIR offers a more efficient refinement process without sacrificing correctness, providing a promising balance among accuracy, efficiency, and token cost. 4.5 Iterative Refinement Figure 2 shows the iterative refinement behavior of EFFIPAIR on EvalPerf across five models: GPT-5.4 mini, DeepSeek-Chat V3.2, GPT-4o mini, GPT-5.4, and Claude Sonnet 4.6. The figure reports execution time, DPS,DPS norm , and Pass@1 over successive generation and efficiency rounds. A consistent trend emerges across models: after the efficiency-refinement rounds, execution time decreases, while DPS andDPS norm are generally maintained or improved. At the same time, Pass@1 remains stable or increases, indicating that the refinement process improves efficiency without sacrificing correctness. Among the models tested, Claude Sonnet 4.6 shows the strongest overall efficiency improvements in timing, DPS, andDPS norm , despite not starting from the strongest initial baseline. This suggests that Claude Sonnet 4.6 benefits particularly strongly from EFFIPAIR’s refinement procedure. While the largest efficiency gains are observed for Claude Sonnet 4.6, GPT-5.4 achieves the highest Pass@1 across the evaluated models. Another clear pattern in Figure 2 is that the largest efficiency and Pass@1 gains typically occur in the first efficiency round. This suggests that even a single round of EFFIPAIR can deliver substantial improvements, making it a practical operating point when optimization cost matters. Additional rounds still provide further gains, but the figure indicates that much of the benefit is already realized early in the refinement process. Overall, the results suggest that EFFIPAIR produces targeted optimization gains during iterative refinement, with the clearest benefits appearing in reduced runtime alongside strong task performance, even for models that already start from a strong initial baseline. 4.6 Correctness in EFFIPAIR In contrast to prior efficiency-focused refinement methods that typically optimize a single candidate trajectory, EFFIPAIR combines a persistent candidate pool with similarity-based pairing, making refinement more robust to correctness regression. Once a correct program enters the pool, it is retained as a valid solution, so later efficiency-oriented refinements cannot erase previously achieved correctness. Additionally, pairing an efficient reference with a structurally similar but weaker candidate provides a localized refinement signal, which makes it easier to transfer efficient design choices without requiring large rewrites that might break functionality. Figure 2 is consistent with this behavior: Pass@1 does not decline as refinement proceeds, and in several cases it improves across rounds. This suggests that the multi-generation setup of EFFIPAIR increases the chance of finding correct solutions early, while the combination of pool retention and contrastive pairing supports efficiency improvement without sacrificing correctness. 8 Preprint. Under review. Figure 2: Iterative refinement behavior of EFFIPAIR on EvalPerf across evaluated models. 5 Discussion and Limitations The main advantage of EFFIPAIR is not more profiling data, but better feedback. By contrast- ing an efficient candidate with a similar slower one, it gives the model a clear, actionable signal about what to change, making refinement more targeted than single-candidate prompting. Our results also suggest that compact hotspot summaries capture most of the useful optimization signal while sharply reducing token and profiling overhead, making test-time refinement practical when optimized code will be reused. EFFIPAIR also has clear limitations. Its gains vary across benchmarks, suggesting that relative contrastive feedback works best when inefficiencies are stable and localizable across similar candidates. The method further depends on informative candidate pairs: limited diversity, poor similarity thresholds, or noisy sampling-based profiles can weaken the signal and increase regression risk. More broadly, our evaluation focuses on correctness and efficiency rather than software qualities such as readability or maintainability. Although EFFIPAIR is training-free, it still adds generation, execution, and profiling cost, so it is most suitable when downstream performance improvements justify that overhead. Future work includes better pair selection, adaptive similarity criteria, and broader optimization objectives. 6 Conclusion We introduced EFFIPAIR, a training-free inference-time framework for improving the effi- ciency of LLM-generated code through Relative Contrastive Feedback (RCF). Instead of relying on isolated scalar profiling signals, EFFIPAIR compares structurally similar candi- dates with different efficiency profiles and distills their differences into compact guidance for refinement. Across multiple benchmarks and strong language models, this approach improves runtime efficiency while preserving, and sometimes improving, functional cor- rectness. We also find that the lightweight summarized profiling used to instantiate RCF is sufficient for effective refinement, substantially reducing overhead. Overall, our results suggest that pairwise execution feedback is a practical and effective alternative to scalar-only supervision for test-time code optimization. 9 Preprint. Under review. References Anthropic.Introducing claude sonnet 4.6.https://w.anthropic.com/news/ claude-sonnet-4-6, February 2026. Accessed: 2026-03-22. Emery D. Berger, Sam Stern, and Juan Altmayer Pizzorno. Triangulating python perfor- mance issues with Scalene. In 17th USENIX Symposium on Operating Systems Design and Im- plementation (OSDI 23), p. 51–64, Boston, MA, July 2023. USENIX Association. ISBN 978-1- 939133-34-2. URL https://w.usenix.org/conference/osdi23/presentation/berger. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Erhang Li, Fangqi Zhou, Fangyun Lin, Fucong Dai, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Li, Haofen Liang, Haoran Wei, Haowei Zhang, Haowen Luo, Haozhe Ji, Honghui Ding, Hongxuan Tang, Huanqi Cao, Huazuo Gao, Hui Qu, Hui Zeng, Jialiang Huang, Jiashi Li, Jiaxin Xu, Jiewen Hu, Jingchang Chen, Jingting Xiang, Jingyang Yuan, Jingyuan Cheng, Jinhua Zhu, Jun Ran, Junguang Jiang, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Kexin Huang, Kexing Zhou, Kezhao Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Wang, Liang Zhao, Liangsheng Yin, Lihua Guo, Lingxiao Luo, Linwang Ma, Litong Wang, Liyue Zhang, M. S. Di, M. Y Xu, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Panpan Huang, Peixin Cong, Peiyi Wang, Qiancheng Wang, Qihao Zhu, Qingyang Li, Qinyu Chen, Qiushi Du, Ruiling Xu, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runqiu Yin, Runxin Xu, Ruomeng Shen, Ruoyu Zhang, S. H. Liu, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaofei Cai, Shaoyuan Chen, Shengding Hu, Shengyu Liu, Shiqiang Hu, Shirong Ma, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, Songyang Zhou, Tao Ni, Tao Yun, Tian Pei, Tian Ye, Tianyuan Yue, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjie Pang, Wenjing Luo, Wenjun Gao, Wentao Zhang, Xi Gao, Xiangwen Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xingyou Li, Xinyu Yang, Xinyuan Li, Xu Chen, Xuecheng Su, Xuehai Pan, Xuheng Lin, Xuwei Fu, Y. Q. Wang, Yang Zhang, Yanhong Xu, Yanru Ma, Yao Li, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Qian, Yi Yu, Yichao Zhang, Yifan Ding, Yifan Shi, Yiliang Xiong, Ying He, Ying Zhou, Yinmin Zhong, Yishi Piao, Yisong Wang, Yixiao Chen, Yixuan Tan, Yixuan Wei, Yiyang Ma, Yiyuan Liu, Yonglun Yang, Yongqiang Guo, Yongtong Wu, Yu Wu, Yuan Cheng, Yuan Ou, Yuanfan Xu, Yuduan Wang, Yue Gong, Yuhan Wu, Yuheng Zou, Yukun Li, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehua Zhao, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhixian Huang, Zhiyu Wu, Zhuoshu Li, Zhuping Zhang, Zian Xu, Zihao Wang, Zihui Gu, Zijia Zhu, Zilin Li, Zipeng Zhang, Ziwei Xie, Ziyi Gao, Zizheng Pan, Zongqing Yao, Bei Feng, Hui Li, J. L. Cai, Jiaqi Ni, Lei Xu, Meng Li, Ning Tian, R. J. Chen, R. L. Jin, S. S. Li, Shuang Zhou, Tianyu Sun, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xinnan Song, Xinyi Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Dongjie Ji, Jian Liang, Jianzhong Guo, Jin Chen, Leyi Xia, Miaojun Wang, Mingming Li, Peng Zhang, Ruyi Chen, Shangmian Sun, Shaoqing Wu, Shengfeng Ye, T. Wang, W. L. Xiao, Wei An, Xianzu Wang, Xiaowen Sun, Xiaoxiang Wang, Ying Tang, Yukun Zha, Zekai Zhang, Zhe Ju, Zhen Zhang, and Zihua Qu. Deepseek-v3.2: Pushing the frontier of open large language models, 2025. URL https://arxiv.org/abs/2512.02556. Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, and See-Kiong Ng. Mercury: A code efficiency benchmark for code large language models, 2024. URLhttps://arxiv.org/abs/2402. 07844. Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K. Lahiri. Llm-based test-driven interactive code generation: User study and empirical evaluation. IEEE Transactions on Software Engineering, 50(9):2254–2268, September 2024. ISSN 2326-3881. doi: 10.1109/tse.2024.3428972. URLhttp://dx.doi.org/10.1109/TSE. 2024.3428972. 10 Preprint. Under review. Michael G ̈ unther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mo- hammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, and Han Xiao. Jina embeddings 2: 8192-token general- purpose text embeddings for long documents, 2024. URLhttps://arxiv.org/abs/2310. 19923. Dong Huang, Jianbo Dai, Han Weng, Puzhen Wu, Yuhao Qing, Heming Cui, Zhijiang Guo, and Jie M. Zhang. Effilearner: Enhancing efficiency of generated code via self- optimization, 2025a. URL https://arxiv.org/abs/2405.15189. Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M. Zhang. Effibench: Benchmarking the efficiency of automatically generated code, 2025b.URLhttps: //arxiv.org/abs/2402.02037. Dong Huang, Guangtao Zeng, Jianbo Dai, Meng Luo, Han Weng, Yuhao Qing, Heming Cui, Zhijiang Guo, and Jie M. Zhang. Efficoder: Enhancing code generation in large language models through efficiency-aware fine-tuning, 2025c. URLhttps://arxiv.org/abs/2410. 10209. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology, 35(2):1–72, January 2026. ISSN 1557-7392. doi: 10.1145/3747588. URL http://dx.doi.org/10.1145/3747588. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL https://arxiv.org/abs/2310.06770. Jina AI.jina-embeddings-v2-base-code.Hugging Face model card.URLhttps:// huggingface.co/jinaai/jina-embeddings-v2-base-code. Accessed: 2026-03-22. Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. Evaluating language models for efficient code generation, 2024. URLhttps://arxiv.org/ abs/2408.06450. OpenAI.Introducing gpt-5.4 mini and nano.https://openai.com/index/ introducing-gpt-5-4-mini-and-nano/, March 2026. Accessed: 2026-03-22. Yun Peng, Akhilesh Deepak Gotmare, Michael Lyu, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Perfcodegen: Improving performance of llm generated code with execution feedback, 2024. URL https://arxiv.org/abs/2412.03578. Ruizhong Qiu, Weiliang Will Zeng, James Ezick, Christopher Lott, and Hanghang Tong. How efficient is llm-generated code? a rigorous & high-standard benchmark, 2025. URL https://arxiv.org/abs/2406.06647. Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. Learning performance-improving code edits, 2024. URLhttps://arxiv.org/abs/2302. 07867. Siddhant Waghjale, Vishruth Veerendranath, Zora Zhiruo Wang, and Daniel Fried. Ecco: Can we improve model-generated code efficiency without sacrificing functional correctness?, 2024. URL https://arxiv.org/abs/2407.14044. Tong Ye, Weigang Huang, Xuhong Zhang, Tengfei Ma, Peiyu Liu, Jianwei Yin, and Wenhai Wang. Llm4effi: Leveraging large language models to enhance code efficiency and correctness, 2025. URL https://arxiv.org/abs/2502.18489. 11 Preprint. Under review. A Appendix A.1 Parameter Analysis We study the sensitivity of EFFIPAIR to key design parameters in the pairing and refinement pipeline. Our goal is to understand how these choices affect the quality of the contrastive signal and, consequently, the efficiency and correctness of the refined programs. We focus in particular on the similarity function and the similarity threshold used for selecting candidate pairs, and fix dataset and model to EvalPerf and GPT-5.4 mini. In these experiment an efficiency round is skipped when no viable pair exists, unlike the runs in the main body of the paper, to isolate the impact of the parameters strictly on pairing. A.1.1 AST-Embedding Score We run EFFIPAIR on EvalPerf for 3 generation rounds and 1 efficiency round with GPT-5.4 mini. Figure 3 shows the DPS,DPS norm , and Pass@1 values for various embedding weights. Our default value, τ = 0.80, shows competitive performance across metrics. A.1.2 Similarity Threshold In this experiment EFFIPAIR was run on EvalPerf for 3 generation rounds and 1 efficiency round with GPT-5.4 mini. Figure 4 shows the DPS,DPS norm , and Pass@1 values for various similarity thresholds. Our default value,τ =0.85, because it delivers the best Pass@1 and, at the same time, yields DPS andDPS norm values that are close to the best observed, representing the most balanced setting in this experiment. A.2 AST Configuration We use AST for the structural component of our similarity score, representing each program as a Python AST bag-of-node-types vector obtained after stripping comments, docstrings, and top-level asserts, parsing the code, counting occurrences of each ast.AST node type, and L2-normalizing the resulting vector. This representation is simple, inexpensive, and sufficient for capturing broad structural overlap without introducing heavy preprocessing. The same AST component is also used for pool deduplication, where programs with sim- ilarity 1.00 are treated as duplicates, because repeated near-identical candidates add cost without increasing useful diversity. A.3 Evaluation Configuration For each dataset, we construct a task-specific executable harness (reference/expected-output checks for EvalPerf, Mercury, and ENAMEL) and execute each candidate in a separate subprocess. This isolation prevents failed or pathological programs from affecting other runs and allows us to capture grounded failure signals for refinement. We treat a solution as correct only if it passes the benchmark harness without modifying the benchmark tests. On failure, we record timeout/exit status, error type, stderr excerpt/tail, and (when available) a structured mismatch tuple (input, expected, actual); this feedback is used in refinement prompts. Each candidate run is isolated in its own process with fixed timeouts (30s for correctness runs, 120s for profiling runs). We use deterministic seeding for model sampling with base seed of 1 (pass-specific and per-sample offsets are derived from this base seed). For stochastic benchmark harnesses (e.g., ENAMEL), fixed RNG states are used inside the harness (random.Random(0)) to keep inputs reproducible. Correctness evaluation and profiling use 16 workers by default. At startup, workers are pinned to low-utilization cores selected with a utilization threshold of<0.02. API generation/optimization requests are batched when provider support exists. Each prompt contains exactly one task (and, in paired mode, two candidates from that same task), avoiding cross-task information leakage. For determining the most efficient candidate, we use code wall-time. Each candidate is timed in a fresh subprocess using the same benchmark-specific harnesses. For correctness timing, we use the wall-clock time from three measured runs. We report the arithmetic mean over successful runs to reduce sensitivity to run-to-run variance. For runtime measurement, 12 Preprint. Under review. Figure 3: DPS, DPS norm , and Pass@1 for various embedding weights. Figure 4: DPS, DPS norm , and Pass@1 for various similarity thresholds. our current configuration uses the wall-clock time on 3 measured runs. The reported elapsed time is the arithmetic mean over successful measured runs. Round-one correctness is computed over the outputs from an initial generation pass. After each iteration, the system updates the candidate pool and writes per-round artifacts, including round statistics and best-elapsed snapshots. Each candidate is executed in an isolated subprocess with fixed timeouts of 30 s for correctness evaluation and 120 s for profiling. A.4 Profiling Method Analysis A.4.1 Rationale for Sampling-Based Profiling Our goal in profiling is not to recover perfectly precise line-by-line execution costs. Rather, we need a lightweight and reliable signal that identifies where most of the runtime is spent, so that refinement can focus on the most promising optimization opportunities. This is why a sampling profiler such as Scalene is sufficient for our setting. 13 Preprint. Under review. ModelMethodPromptCompletionTotal DeepSeek-Chat V3.2 Paired (No Profiling)160k60k220k Full Profile Solo10.1M150k10.25M Solo (Summary Profiling)90k30k120k EFFIPAIR210k80k290k Claude Sonnet 4.6 Paired (No Profiling)190k75k265k Full Profile Solo12.3M100k12.4M Solo (Summary Profiling)180k90k270k EFFIPAIR250k90k340k GPT-5.4 Paired (No Profiling)150k40k190k Full Profile Solo8.1M50k8.15M Solo (Summary Profiling)140k50k190k EFFIPAIR200k40k240k GPT-5.4 mini Paired (No Profiling)150k50k200k Full Profile Solo8.4M60k8.46M Solo (Summary Profiling)140k50k190k EFFIPAIR200k50k250k Table 4: Token usage by model and refinement setting. Total tokens are the sum of prompt and completion tokens. Full-profile variants incur substantially higher prompt costs than summarized-profile and paired methods. Amdahl’s Law. If a fractionpof total runtime is spent in one component, and that component is accelerated by a factor of k, then the overall speedup is S total = 1 (1− p) + p k . Amdahl’s law shows that the end-to-end benefit of an optimization depends primarily on how much of the total runtime is covered by the optimized component. In particular, larger values ofpyield larger global gains. This formalizes the intuition to optimize the hottest code first: improving a small bottleneck cannot produce a large overall speedup, no matter how aggressively it is optimized. This observation is especially relevant for our use case. Since the refinement loop only needs to know which functions or lines dominate execution, approximate hotspot attribution is often enough. A sampling profiler does not need to report exact costs for every line; it only needs to recover a reliable ranking of the major bottlenecks. We can also view each refinement step as a greedy optimization move. LetJdenote the total execution cost. Choosing the hotspot with the largest estimated contribution toJis analogous to taking a local step in the direction of steepest decrease. Even when profiling provides only relative percentages rather than exact measurements, these percentages still supply a useful ordering over candidate bottlenecks. This is the main reason we use Scalene. Compared with heavier profiling approaches, Scalene provides low-overhead, line-level estimates of CPU and memory hotspots, which are well matched to an iterative test-time refinement loop. In our setting, the objective is not exhaustive offline diagnosis, but fast identification of the code regions most likely to produce meaningful end-to-end gains when optimized. A.4.2 Token Usage Analysis Table 4 shows that full-profile refinement is substantially more expensive than summarized- profile methods, with prompt usage in the multi-million-token range, whereas EFFIPAIR remains in the few-hundred-thousand-token regime across all models. 14 Preprint. Under review. A.5 Prompts A.5.1EFFIPAIR Generation EFFIPAIR Generation (ENAMEL) Complete the following function. prompt_stub Requirements: - Keep the required function signature (top -level , no class). - Return only a Python code block. EFFIPAIR Generation (EvalPerf) prompt_text Requirements: - Implement the top -level function named`entry_point` (no class wrapper). - Return only a Python code block. EFFIPAIR Generation (Mercury) Task: pretty_problem_content Code Stub: stub Requirements: - Provide class Solution with the required method`method_name`. - Return only a Python code block. A.5.2EFFIPAIR Efficiency Improve the following Python code with correctness as the top priority and efficiency as secondary. Guidance: guidance task_block Candidate A: code_a [optional] Profile A: profile_a [optional] Candidate B: code_b [optional] Profile B: profile_b Requirements: requirements A.5.3 EffiLearner Generation HumanEval. 15 Preprint. Under review. Please complete Python code based on the task description. # Task description: prompt #Solution: MBPP. Please complete Python code based on the task description and test cases. # Task description: prompt tests #Solution: A.5.4 EffiLearner Efficiency Optimize the efficiency of the following Python code based on the task , test case , and overhead analysis provided. Ensure the optimized code can pass the given test case. Task Description: task_description Test Case: test_case Original Code: completion Overhead Analysis: overhead_prompt Optimization Rules: - Encapsulate the optimized code within a Python code block (i.e., python [Your Code Here] ). - Do not include the test case within the code block. - Focus solely on code optimization; test cases are already provided. - Ensure the provided test case passes with your optimized solution. 16