Paper deep dive
Can Large Language Models Recover Semantic Optimization Opportunities That Compilers Miss?
Hailong Jiang, Feng Yu, Emran Hossain, Jianfeng Zhu, Mengfei Ren, Qiang Guan, Chunwei Xia
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Optimizing compilers miss profitable transformations when their enabling semantics are absent from the analyzed program representation. We ask whether large language models (LLMs) can recover such semantics from heterogeneous C/C++ context and realize them as validated, contract-preserving artifacts. We introduce SeGaBench, an executable benchmark containing 100 synthetic and 20 source-backed cases spanning low-level assumptions, data-structure invariants, and high-level semantic lifting. Each case includes hidden enabling semantics, an oracle artifact, correctness and semantic validators, and a reproducible performance protocol. We evaluate five LLMs using five independent responses per case. The strongest model produces correct artifacts in 94.8% of responses, achieves at least 1.05x speedup in 83.3%, and obtains a performance success on 93.3% of cases. Nevertheless, correct artifacts often close only part of the oracle gap. These results show that LLMs can complement compiler analysis as speculative semantic proposers, provided that their artifacts are validated and evaluated.
Tags
Links
- Source: https://arxiv.org/abs/2608.03983v1
- Canonical: https://arxiv.org/abs/2608.03983v1
Trouble viewing inline? Open PDF directly →
Full Text
47,639 characters extracted from source content.
Expand or collapse full text
Can Large Language Models Recover Semantic Optimization Opportunities That Compilers Miss? Hailong Jiang1 , Feng Yu1, Emran Hossain1, Jianfeng Zhu2, Mengfei Ren3, Qiang Guan2, Chunwei Xia4 Abstract Optimizing compilers miss profitable transformations when their enabling semantics are absent from the analyzed program representation. We ask whether large language models (LLMs) can recover such semantics from heterogeneous C/C++ context and realize them as validated, contract-preserving artifacts. We introduce SeGaBench, an executable benchmark containing 100 synthetic and 20 source-backed cases spanning low-level assumptions, data-structure invariants, and high-level semantic lifting. Each case includes hidden enabling semantics, an oracle artifact, correctness and semantic validators, and a reproducible performance protocol. We evaluate five LLMs using five independent responses per case. The strongest model produces correct artifacts in 94.8% of responses, achieves at least 1.05×1.05× speedup in 83.3%, and obtains a performance success on 93.3% of cases. Nevertheless, correct artifacts often close only part of the oracle gap. These results show that LLMs can complement compiler analysis as speculative semantic proposers, provided that their artifacts are validated and evaluated. Introduction Motivation. C/C++ compilers optimize programs using the semantic information, analyses, and transformations available in the compilation pipeline (Burow 2018; Vafeiadis et al. 2015; Lossing et al. 2015). Even aggressive compiler configurations such as -O3 may leave profitable transformations unexploited when their enabling semantics cannot be established from the analyzed representation and scope (Barany 2018; D’Silva et al. 2015). Representative gaps include (i) missing context information. For example, by analyzing how data is used across multiple functions, an LLM may determine that two memory regions do not overlap and expose this information to the compiler, enabling vectorization (Sheth 2026; Anand et al. 2025); (i) data-structure invariants such as sortedness or diagonal structure; and (i) program equivalences such as loops implementing scans or aggregations (Gao et al. 2023b; Blelloch 1990; Steuwer et al. 2017). Such evidence may be scattered across callers, tests, documentation, profiles, and configuration, requiring reasoning beyond an isolated function (Paraskevopoulou 2020; Tang et al. 2026). Recent LLM-for-compilation work has focused on IR reasoning, low-level kernel generation, and iterative refinement with compilation and performance feedback (Jiang et al. 2025; Chen et al. 2025; Grubisic et al. 2024). These works show that LLMs can reason over compiler artifacts and refine optimization candidates using compilation and performance feedback. This motivates our hypothesis that LLMs may recover such missing optimization-enabling semantics from long program context and express them as high-level semantic artifacts. Research Questions. For LLMs to serve as semantic bridges for compiler optimization, they must succeed in three steps: identify the semantic fact or better program equivalence that compilers cannot establish, realize it as a compiler-usable artifact, and translate that artifact into performance improvement. We formulate these steps as three research questions. RQ1 (Semantic Identification) asks whether LLMs can identify the semantic fact and its evidence; RQ2 (Artifact Realization) asks whether LLMs can realize the fact as a valid artifact; RQ3 (Performance Realization) asks whether valid artifacts deliver measurable speedups and how much of the oracle opportunity they recover. Together, these three questions ask: “ Can LLMs recover semantic optimization opportunities missed by a strong C/C++ compiler baseline, and realize their enabling semantics as validated artifacts that deliver measurable speedups? Approach. To answer these RQs, we introduce SeGaBench (semantic gap benchmark), an executable benchmark for evaluating semantic opportunity realization. Each case includes the original C/C++ program, relevant context, hidden target semantics and a reference implementation for evaluation, correctness checks, and a fixed performance-testing procedure. The oracle artifact serves as a validated reference: it passes all validators and achieves measurable speedup over the strongest compiled baseline. SeGaBench contains two suites: a Synthetic Suite with 100 carefully designed cases covering 50 common optimization patterns, and a Real-world Suite with 20 cases collected from six widely used HPC projects. Together, they cover three semantic types: low-level assumptions, data-structure invariants, and high-level semantic lifting. Evaluation Protocol. To evaluate LLMs, we use a blind, single-turn setup in which the model gives its answer without receiving any feedback from the evaluator. For each case, the model sees the original program and selected context, while the target semantics, reference solution, validation checks, and performance results are kept hidden. The model can either decline to answer or identify a useful semantic fact or equivalent program behavior, explain the supporting evidence, and produce a source-level change. After the response is fixed, the evaluator turns the proposed change into a patch, compiles the modified program, checks its correctness, and measures the performance against the strongest compiled baseline. The model receives no feedback from the compiler, validators, reference solution, or profiler. Results. Across five LLMs and 3,000 responses, artifact realization is strongly model-dependent. The strongest model produces correct, contract-preserving artifacts in 94.8% of responses, achieves at least 1.05× speedup in 83.3%, and, with five independent responses per case, finds a speedup on 93.3% of the benchmark. Yet correctness alone does not realize the full oracle opportunity: many correct artifacts close only part of the oracle gap, and performance success drops from 86.6% on synthetic cases to 67.0% on real-world cases. These results highlight both the promise of LLM-guided semantic optimization and the need for validation and measurement. Contributions. This paper makes three contributions: ■ We formalize semantic optimization opportunities missed by compiler baselines and formulate their recovery as an evidence-grounded LLM task. We organize their enabling semantics into three types and 50 recurring archetypes. ■ We introduce SeGaBench, an executable benchmark comprising 100 synthetic and 20 source-backed HPC cases. Each case provides hidden target semantics, an oracle artifact, functional and semantic validators, and fixed build and performance protocols. ■ We design an evaluation protocol that separately measures semantic recovery, artifact correctness, and performance realization across models, suites, and semantic types. Task Definition Semantic Optimization Opportunity. Let P be a C/C++ program with a defined input domain and behavioral contract D, and let B denote the strongest evaluated compiler baseline for P. An enabling semantics S is a program property or equivalent computation that enables a more efficient implementation while producing the same observable output for every valid input, as illustrated in Figure 1. We say that (S⋆,A⋆)(S ,A ) establishes a semantic optimization opportunity for P when three conditions hold. First, S⋆S is valid for D, or its use is protected by a runtime guard with a correct fallback implementation. Second, the oracle artifact A⋆A realizes S⋆S while preserving the program contract: ∀x∈,(P⊕A⋆)(x)≃P(x),∀ x , (P A )(x) P(x), where ≃ denotes observational equivalence under the benchmark contract. Third, the transformed program achieves a statistically significant speedup over the baseline: Speedup(A⋆)=T(P,B)T(P⊕A⋆,B)≥padmit,Speedup(A )= T(P,B)T(P A ,B)≥ p_admit, under the benchmark admission protocol. Throughout this paper, a compiler misses an opportunity when the strongest evaluated compilation of the original program leaves this admitted opportunity unexploited. Semantic Artifacts. A semantic artifact is a concrete source-level change that makes an optimization-enabling semantic fact or program equivalence explicit. Applying an artifact A to a program P yields P⊕AP A. Artifacts may include qualifiers, assumptions, alignment annotations, guarded specializations, contracts, library substitutions, or rewrites. For example, non-aliasing may be expressed as restrict or a no-overlap guard, while an aggregation equivalence may be realized as an algorithmic rewrite. Each benchmark case contains evaluator-only target semantics S⋆S and an oracle artifact A⋆A , both hidden from the model. The oracle artifact is validated to preserve the benchmark contract and improve over the strongest baseline, witnessing a realizable opportunity rather than a global optimum. The model produces a candidate artifact A A, which may differ from A⋆A but must realize a supported semantic claim and pass correctness and performance evaluation. LLM Task. For each case, the LLM receives the original program P and an available context view C, while S⋆S and A⋆A remain hidden. The model may abstain or return a proposal Y^=(S^,E^,A^), Y=( S, E, A), where S S is the proposed enabling semantics; E E identifies supporting evidence and scope; A A is the corresponding artifact. Taxonomy of Semantic Optimization Opportunity SeGaBench organizes semantic optimization opportunities into three types: low-level assumptions, data-structure invariants, and high-level semantic lifting. Within these types, we derive 50 archetypes by grouping recurring compiler limitations and performance transformations according to their enabling semantics, required evidence, and realizable artifact. Each archetype therefore, represents a distinct semantic reason that an optimization remains unexploited and a corresponding way to realize it. These archetypes guide the construction of the synthetic suite, while the same three types categorize the source-backed real-world cases. Table 1 summarizes the taxonomy. Low-Level Assumptions. Low-level assumptions refine a general C/C++ interface with a more precise execution domain (Avans et al. 2025; Radtke et al. 2024). They describe memory accesses, scalar values, or iteration spaces, including non-aliasing pointers, aligned buffers, bounded values, fixed strides, and fixed loop trip counts (Lattner 2005). Supporting evidence commonly appears in callers, allocators, validation logic, or workload configuration. A property guaranteed by the program contract can be realized as a scoped qualifier or assumption; a dynamically established property requires a guard, a specialized path, and a correct fallback. These artifacts can enable vectorization, unrolling, address simplification, and check elimination (Lattner and Adve 2004; Zheng et al. 2025). Data-Structure Invariants. Data-structure invariants describe semantic relationships among elements that give a concrete representation a stronger abstract meaning (Liskov 1987). Examples include sorted or unique sequences, permutations, diagonal matrices, canonical sparse layouts, injective index maps, and disjoint intervals (Di Felice et al. 1989). Such relations typically span the complete, data-dependent structure and are not expressed by ordinary C/C++ element types or storage layouts (Kawaguchi et al. 2009). A trusted constructor or complete invariant validator can establish the stronger representation, after which a specialized algorithm operates on it; a generic implementation handles inputs for which the invariant does not hold (Gao et al. 2023a). High-Level Semantic Lifting. High-level semantic lifting recovers an equivalence between a low-level implementation and a higher-level operation, algorithm, or fused computation (Paraskevopoulou and others 2020). Representative examples include recognizing loop nests as group-by aggregation, prefix scan, sliding-window aggregation, stable compaction, symmetric pairwise interaction, or a producer–consumer pipeline (Blelloch 1990; Steuwer et al. 2017; Lattner et al. 2020). These equivalences span loops, calls, and intermediate data structures, and their realization often requires whole-computation restructuring (Bondhugula et al. 2008). Typical artifacts include equivalent source rewrites, fused implementations, and specialized traversals. Semantic and differential validation check equivalence with the original program, including its ordering, boundary, and numerical behavior (Necula 2000). Table 1: Taxonomy of semantic optimization opportunities. (A “miss” is an opportunity left unexploited by the strongest compiler baseline.) Type Opportunity Why baseline misses it Artifact / optimization Low-level assumptions (LLA) Memory, value, and iteration facts: non-aliasing, alignment, fixed strides/bounds, bounded indices. The interface permits more executions than the caller, allocator, or configuration actually allows. Assumptions or guarded specialization; enables vectorization, unrolling, address simplification, and check elimination. Data-structure invariants (DSI) Global data relations: sortedness, uniqueness, diagonal structure, canonical sparsity, injective maps, disjoint intervals. Types and layouts expose storage, but not relations across elements. Invariant contract or validator with specialization; enables simpler access and specialized algorithms. High-level semantic lifting (HSL) Program equivalences: scans, aggregation, sliding windows, compaction, pairwise interaction, producer–consumer fusion. Recognition spans loops, calls, and intermediates; realization requires restructuring the computation. Equivalent rewrite, fusion, or specialized traversal; reduces passes, intermediates, or asymptotic work. Benchmark Design We turn the task in Section I into SeGaBench. It tests whether LLMs can recover optimization-enabling semantics missed by strong C/C++ compiler baselines and realize them as validated, contract-preserving artifacts. The benchmark includes a synthetic suite for controlled analysis and a real-world suite of source-backed application cases. Figure 1: Structure and admission flow of a SeGaBench case. Case Representation and Admission. As shown in Figure 1, each case is represented as ℬi=(Pi,Ci,Si⋆,Ai⋆,Vi,Mi),B_i=(P_i,C_i,S_i ,A_i ,V_i,M_i), where PiP_i is the original C/C++ program; CiC_i is its context, such as callers, tests, documentation, profiles, or compiler reports; Si⋆S_i is the hidden enabling property or program equivalence; Ai⋆A_i is the hidden oracle artifact; ViV_i contains functional and property-specific validators; and MiM_i fixes the build, workload, and timing protocol. Runtime guards and fallback paths, when needed, are part of the artifact. A case is admitted only if the oracle artifact applies, compiles, preserves the program contract, passes all validators, and exposes the intended optimization. The strong baseline is the fastest original program compiled with -O3, LTO, or PGO (LLVM Project 2025). The oracle-realized program must achieve at least 1.05×1.05× speedup, with the lower bound of its 95% confidence interval above 1.01.0, in two independent measurement sessions. Any guard or validation cost executed within the workload is included. Synthetic Suite. The synthetic suite contains 100 purpose-built cases organized into 50 semantic archetypes, with two distinct program instances per archetype. Each case isolates one enabling property or equivalence while fixing its program contract, context sources, oracle artifact, semantic validator, and executable workload. The paired construction supports controlled analysis across semantic categories, archetypes, and context views. Real-World Suite. The real-world suite contains 20 source-backed cases drawn from six established HPC projects: HPCG, LAMMPS, LULESH, miniFE, RAJAPerf, and XSBench (Heroux et al. 2013; Thompson et al. 2022; Karlin et al. 2013; Lin et al. 2015; Pearce et al. 2024; Tramm et al. 2014). Each case is derived from a source-locked application hotspot and retains its relevant program context, build boundary, and workload. The cases represent 20 distinct archetypes and pass source and provenance verification, blinded human semantic review, oracle isolation, functional and semantic validation, compiler-evidence checks, and the common admission protocol. The synthetic suite is distributed as 40 LLA, 30 DSI, and 30 HSL cases, while the real-world suite adds 20 source-backed cases, distributed as 8 LLA, 6 DSI, and 6 HSL cases. Evaluation We evaluate SeGaBench along the three RQs: semantic identification, artifact realization, and performance realization. This section describes the blinded protocol and the metrics used for each stage. Blinded Evaluation Protocol. For each case i, SeGaBench creates an anonymized request containing the original program PiP_i and a selected context view CivC_i^v. The target semantics Si⋆S_i , oracle artifact Ai⋆A_i , validators, and performance data are hidden from the model. The model may abstain or return one proposal consisting of a semantic claim, supporting evidence, and either a transformed original.cpp or an explicit no_artifact response. Each response is frozen before evaluation. The importer checks schema conformance and converts any returned source into an original-file-only patch. The evaluator applies the patch in an isolated workspace, compiles it, runs functional and property-specific semantic validators, and profiles every correctness-passing candidate. No compiler, validator, oracle, or profiler feedback is returned to the model. Models and Run Configurations. We evaluate five LLM configurations: GPT-5.4 mini (gpt-5.4-mini-2026-03-17) (OpenAI 2026a), GPT-5.6 Sol (gpt-5.6-sol)(OpenAI 2026b), DeepSeek-V4 Pro (deepseek-ai/DeepSeek-V4-Pro)(DeepSeek-AI 2026), Llama 3.3 70B Instruct Turbo (meta-llama/Llama-3.3-70B-Instruct-Turbo)(Meta AI 2024), and Ternary Bonsai 27B (Prism-ML/Ternary-Bonsai-27B) (Prism ML 2026). All models are evaluated using the same prompt, response schema, maximum output length, and frozen run plans. For each model, we draw five independent full-context responses for each of the 120 cases at temperature 0.70.7, yielding 600 requests per model and 3,000 requests overall. Each request is stateless. We use a maximum output length of 16,384 tokens. Responses are preserved and evaluated as returned, without repair, retry, or iterative optimization feedback. All compilation, validation, and performance measurements are conducted on a Mac mini with a 10-core Apple M4 processor (four performance and six efficiency cores) and 16 GB of memory, running macOS 26.5.2. We use Apple Clang 17.0.0 targeting ARM64. Benchmark cases and variants are evaluated serially. We do not apply explicit CPU-frequency locking or core-affinity controls. Baselines and Measurement. For case i, the baseline runtime TibT_i^b is the minimum runtime of the original program compiled with -O3, LTO, and PGO. The oracle-enabled runtime TioT_i^o represents the admitted semantic opportunity, and TircT_ir^c is the runtime of the candidate produced by request r. All variants use the same compiler, workload, source boundary, and measurement configuration. Oracle admission uses two independent measurement sessions and a confidence-interval criterion, as described in Section Benchmark Design. Candidate results use the point estimate from the fixed candidate measurement protocol unless otherwise stated. End-to-End Success. We evaluate each proposal as a three-stage pipeline: semantic identification, artifact realization, and performance realization. A proposal is an end-to-end (E2E) success only if it identifies the target semantics with supporting evidence, realizes it as a valid artifact, and achieves the fixed performance criterion over the strongest compiler baseline. RQ1: Semantic Identification. Two reviewers, blinded to model identity and candidate performance, judge whether a proposal states Si⋆S_i or an equivalent claim and whether its cited evidence supports the stated scope. Disagreements are resolved through adjudication to obtain the final RQ1 labels. Schema conformance and citation bounds are checked mechanically. Let srs_r indicate that request r passes this review. Over the set of all requests ℛR, the primary RQ1 metric is RecoveryRate=1|ℛ|∑r∈ℛsr.RecoveryRate= 1|R| _r s_r. RQ2: Artifact Realization. Reviewers first check that the artifact realizes the same claim assessed in RQ1. The evaluator then checks that the patch applies, compiles, and passes both functional and property-specific semantic validation. Let ara_r indicate that these conditions hold. We report the unconditional correct-artifact rate ArtifactRateall=∑r∈ℛsrar|ℛ|,ArtifactRate_all= _r s_ra_r|R|, and the rate conditioned on successful semantic recovery, ArtifactRatecond=∑r∈ℛsrar∑r∈ℛsr.ArtifactRate_cond= _r s_ra_r _r s_r. The first measures E2E artifact production, while the second isolates artifact construction from semantic recovery. RQ3: Performance Realization. RQ3 evaluates whether semantically valid and correctness-preserving artifacts translate into runtime improvements. We first define =r∈ℛ:srar=1,C=\r :s_ra_r=1\, the set of requests that pass both RQ1 and RQ2. Let mr=1m_r=1 when candidate timing completes and the quantities required for performance classification are available. We define perf=r∈:mr=1.C_perf=\r :m_r=1\. For each r∈perfr _perf, we compute Speedupir=TibTircSpeedup_ir= T_i^bT_ir^c and GapClosedir=Tib−TircTib−Tio.GapClosed_ir= T_i^b-T_ir^cT_i^b-T_i^o. Because the oracle is a validated reference rather than a global optimum, GapClosedir>1GapClosed_ir>1 is permitted. Using p=1.05p=1.05, we partition perfC_perf into: ■ Oracle-level realization: GapClosedir≥1GapClosed_ir≥ 1; ■ Partial realization: Speedupir≥pSpeedup_ir≥ p & GapClosedir<1GapClosed_ir<1; ■ No meaningful realization: Speedupir<pSpeedup_ir<p. These outcome rates use |perf||C_perf| as their denominator and sum to one. We report |perf|/|||C_perf|/|C| as performance-measurement coverage. Candidates without a valid performance classification remain failures in request-level E2E@p. Aggregation and Sampling. We report RQ1, RQ2, and E2E results overall and by benchmark suite and semantic type. RQ1, RQ2, and E2E rates use all requests, whereas the three RQ3 outcome shares use the performance-classifiable correct-artifact set perfC_perf. For repeated sampling, Success@kSuccess@k counts a case as successful if at least one of its first k responses passes RQ1 and RQ2 and achieves Speedup≥pSpeedup≥ p. We report k=1,…,5k=1,…,5 and use Success@5Success@5 as the summary case-level metric. Abstention, schema compliance, artifact generation, compilation, functional validation, and semantic validation are reported as diagnostic outcomes. Experimental Results Each of the five models produces five independent responses for each of the 120 cases, yielding 600 requests per model and 3,000 in total. Recovery and ArtifactRateallArtifactRate_all use all requests as the denominator, including abstentions and malformed responses; ArtifactRatecondArtifactRate_cond is conditioned on RQ1 recovery. RQ3 is evaluated on =r:srar=1C=\r:s_ra_r=1\. E2E@1.051.05 additionally requires at least 1.05×1.05× speedup, while Success@k counts a case when any of its first k responses succeeds E2E. Table 2 shows a wide capability gap. GPT-5.6 Sol is strongest at all three stages, with 95.0% recovery, 94.8% correct artifacts, and 83.3% E2E performance success. DeepSeek-V4-Pro and GPT-5.4-mini reach 63.8% and 41.7% E2E@1.051.05, whereas Llama-3.3-70B-Turbo and Ternary-Bonsai-27B reach only 5.8% and 5.0%. The benchmark therefore distinguishes not only semantic reasoning but also the ability to turn recovered semantics into executable artifacts. Table 2: Overall results over 600 requests per model. Model RecoveryRate allArtifactRate_all condArtifactRate_cond E2E@1.051.05 Success@55 GPT-5.4-mini 86.0% 54.8% 63.8% 41.7% 76.7% GPT-5.6 Sol 95.0% 94.8% 99.8% 83.3% 93.3% DeepSeek-V4-Pro 84.0% 81.3% 96.8% 63.8% 91.7% Ternary-Bonsai-27B 14.8% 7.7% 51.7% 5.0% 20.8% Llama-3.3-70B-Turbo 21.0% 10.0% 47.6% 5.8% 12.5% Repeated sampling improves case coverage for every model. From k=1k=1 to k=5k=5, E2E coverage rises from 39.2% to 76.7% for GPT-5.4-mini, from 82.5% to 93.3% for GPT-5.6 Sol, and from 67.5% to 91.7% for DeepSeek. At k=5k=5, DeepSeek produces at least one RQ1- and RQ2-passing artifact for 119 of 120 cases, but GPT-5.6 Sol retains slightly higher performance coverage (112 versus 110 cases). Figure 2: Five-model results: (a) per-request RQ1, RQ2, and E2E@1.051.05 rates; (b) case coverage versus samples per case; and (c) RQ3 outcomes among performance-classifiable correct artifacts perfC_perf. RQ1: Semantic Opportunity Identification Blinded reviewers judge whether a response identifies the target enabling semantics, or an equivalent claim, and grounds it in evidence supporting its scope. GPT-5.6 recovers 570/600 opportunities (95.0%), followed by GPT-5.4-mini at 516/600 (86.0%) and DeepSeek at 504/600 (84.0%). Llama recovers 126/600 (21.0%) and Ternary-Bonsai 89/600 (14.8%). Their low rates primarily reflect missing reviewable claims: Llama abstains in 474 responses, while Ternary-Bonsai has 511 non-recoveries, including 71 schema-invalid responses. Semantic identification is thus reliable for the strongest models but is not a general property of all evaluated LLMs. RQ2: Artifact Realization GPT-5.6 realizes 569/570 recovered opportunities as correct artifacts, giving ArtifactRatecond=99.8%ArtifactRate_cond=99.8\%. DeepSeek realizes 488/504 (96.8%), GPT-5.4-mini 329/516 (63.8%), Ternary-Bonsai 46/89 (51.7%), and Llama 60/126 (47.6%). Consequently, the correct-artifact rate ranges from 7.7% to 94.8%. Execution failures further separate the models. All 570 GPT-5.6 artifacts compile and only one fails validation. Ternary-Bonsai has 20 compilation, six functional, and two semantic-validation failures; Llama has 47 compilation, 18 functional, and one semantic-validation failure. Structured output compliance alone therefore does not imply artifact correctness. RQ3: Performance Realization The RQ3 correctness sets C contain 329 GPT-5.4-mini, 569 GPT-5.6, 488 DeepSeek, 46 Ternary-Bonsai, and 60 Llama responses. Of these, respectively, 76.0%, 87.9%, 78.5%, 65.2%, and 58.3% achieve at least 1.05×1.05×. Thus, even the two weaker models occasionally produce useful artifacts, but their low RQ1 and RQ2 coverage reduces request-level performance success to 5–6%. Figure 2 shows the fraction of the oracle opportunity realized. GPT-5.6 has the smallest no-meaningful share (11.7%). This share increases to 21.4% for DeepSeek, 22.8% for GPT-5.4-mini, 34.8% for Ternary-Bonsai, and 40.7% for Llama. Median speedups among measured correct artifacts are 2.10×2.10×, 2.02×2.02×, 1.77×1.77×, 1.63×1.63×, and 1.11×1.11× for GPT-5.6, DeepSeek, GPT-5.4-mini, Ternary-Bonsai, and Llama, respectively. Conditional speedup must therefore be interpreted together with E2E coverage. Synthetic versus real-world behavior. For the three strongest models, request-level E2E performance decreases on the real-world suite. Ternary-Bonsai and Llama show higher request-level rates on real-world cases because they return artifacts more frequently. Nevertheless, median speedup among measured correct artifacts decreases from synthetic to real-world cases for all five models, from 1.95–7.29× to 1.04–1.29×. Suite comparison must therefore separate artifact coverage from the speedup of correct artifacts. Speedup across Semantic Types. Figure 3 compares the runtime speedups of correct, measured artifacts across the three semantic types and models. Among the three strongest models, low-level assumptions produce the largest median speedups: 2.03× for GPT-5.4-mini, 3.10× for GPT-5.6, and 2.51× for DeepSeek-V4-Pro. Ternary-Bonsai instead peaks on data-structure invariants at 2.72×, although this estimate contains only 12 measured artifacts. Llama remains close to baseline parity across all three types, with medians of 1.19×, 0.99×, and 1.07×. Thus, performance realization depends jointly on semantic type and model capability. GPT-5.6 Sol achieves the highest median speedup on low-level assumptions and high-level semantic lifting. DeepSeek-V4-Pro has the highest median on data-structure invariants, although GPT-5.6 Sol has the higher geometric mean, indicating a more right-skewed speedup distribution. Overall, low-level semantic information produces the largest absolute runtime benefits, whereas the performance of higher-level opportunities depends more strongly on the model and individual benchmark case. Figure 3: Speedup of measured correct artifacts by semantic type and model. Boxes show IQRs, center lines medians, and whiskers the 10th–90th percentiles; the dashed line marks baseline parity. “ Answer to the central research question: Within the admitted opportunities in SeGaBench, capable LLMs can recover and realize optimizations missed by compiler baselines, but the capability is strongly model-dependent. GPT-5.6 Sol achieves 95.0% semantic recovery, 94.8% correct artifacts, and 83.3% E2E@1.051.05. Performance is lower on source-backed cases for the strongest models, and reliable use still requires validation and measurement. Discussion and Limitations Implications. Our results support a division of labor in which an LLM searches heterogeneous program context for enabling semantics, validators establish whether the proposed claim and artifact preserve the benchmark contract, and the compiler performs downstream optimization and code generation. This approach retains existing compiler infrastructure while extending the semantic information available to it. However, the large differences across models show that semantic opportunity realization is not yet a uniform LLM capability. Repeated sampling improves case coverage, but it also increases the need for reliable validation and candidate selection. The lower speedups on real-world cases further indicate a transfer gap between isolated opportunities and application-level performance. Limitations. SeGaBench currently covers C/C++, three semantic categories, and source-backed cases from six HPC projects. Its real-world cases are selected for reproducibility and measurable oracle speedup and may not represent the full distribution of optimization opportunities in production software. Performance is measured in one hardware and compiler environment, so absolute speedups and model rankings may change across platforms. Oracle artifacts are validated witnesses rather than global performance upper bounds, and correctness guarantees apply only within each case’s defined contract and workload boundary. Finally, RQ1 relies on blinded human judgments of semantic equivalence and evidence sufficiency. Future work should extend the benchmark to additional domains, languages, compilers, and hardware platforms, and investigate machine-checkable contracts and automated candidate selection. Candidate success uses point estimates, whereas oracle admission additionally requires two independent sessions and a confidence-interval criterion; candidates near the 1.05×1.05× threshold may therefore be sensitive to measurement noise. Related Work Learning-guided compiler optimization. Prior work applies machine learning to decisions within existing compiler pipelines. MLGO integrates learned optimization policies into LLVM for decisions such as inlining, while CompilerGym provides an environment for learning-based exploration of compiler optimization sequences (Trofin et al. 2021; Cummins et al. 2022). Recent language-model-based approaches predict optimization passes or optimized LLVM IR directly (Grubisic et al. 2024; Cummins et al. 2025). HintPilot instead synthesizes compiler hints to steer existing compiler optimizations (Jiang et al. 2026). These approaches operate over compiler-visible representations and available optimization mechanisms, either by improving optimization decisions or directly generating optimized code. Our work instead studies whether LLMs can recover enabling semantics that are not available to the original compilation in a usable form. LLMs for performance-oriented code generation. KernelBench evaluates LLM-generated GPU kernels using functional correctness and speedup over PyTorch baselines, while CUDA-LLM iteratively improves CUDA kernels using compilation, correctness, and performance feedback (Ouyang et al. 2025; Chen et al. 2025). These and related approaches similarly treat the LLM as a generator or tuner of a replacement implementation. In contrast, SeGaBench begins with an existing C/C++ program and asks the model to recover an optimization enabling semantic property or program equivalence and realize it as a semantic artifact. The existing compiler then remains responsible for downstream optimization and code generation. Invariant discovery and verified transformation. Daikon dynamically infers likely program invariants from execution traces (Ernst et al. 2001). Superoptimizers search or synthesize equivalent program rewrites that improve a target cost metric (Sasnauskas et al. 2017; Liu et al. 2023), while translation validation systems such as Alive2 verify whether LLVM transformations preserve program semantics (Lopes et al. 2021). These techniques respectively discover trace-supported properties, search a predefined transformation space, or validate a supplied rewrite. SeGaBench connects these concerns by evaluating evidence-grounded semantic recovery, executable artifact construction, contract validation, and realized performance within a common benchmark. Conclusion This paper asks whether LLMs can realize semantic optimization opportunities missed by strong C/C++ compilers. We introduce SeGaBench, comprising 100 synthetic and 20 source-backed cases with hidden semantics, oracle artifacts, validators, and fixed performance protocols. Across five LLMs, the strongest produces correct artifacts in 94.8% of responses, achieves E2E@1.051.05 in 83.3%, and succeeds on 93.3% of cases with five samples. However, many artifacts close only part of the oracle gap, and performance is lower on source-backed cases for the strongest models. The large differences among models further show that semantic opportunity realization is not yet a uniform LLM capability. Overall, the results support a complementary workflow in which LLMs propose semantic artifacts, validators establish correctness, and compilers perform downstream optimization. Reliable use still requires contract-aware fallbacks and platform-specific performance measurement. References V. Anand, D. Garg, and A. Kaufmann (2025) Iridescent: a framework enabling online system implementation specialization. arXiv preprint arXiv:2508.16690. Cited by: Introduction. C. N. Avans, A. A. Correa, S. Ghosh, M. Schimek, J. Schuchart, A. Skjellum, E. D. Suggs, and T. N. Uhl (2025) Concepts for designing modern c++ interfaces for mpi. In European MPI Users’ Group Meeting, p. 165–183. Cited by: Taxonomy of Semantic Optimization Opportunity. G. Barany (2018) Finding missed compiler optimizations by differential testing. In Proceedings of the 27th international conference on compiler construction, p. 82–92. Cited by: Introduction. G. E. Blelloch (1990) Prefix sums and their applications. Technical report Technical Report CMU-CS-90-190, School of Computer Science, Carnegie Mellon University. Cited by: Introduction, Taxonomy of Semantic Optimization Opportunity. U. Bondhugula, A. Hartono, J. Ramanujam, and P. Sadayappan (2008) Pluto: a practical and fully automatic polyhedral program optimization system. In Proceedings of the ACM SIGPLAN 2008 Conference on Programming Language Design and Implementation (PLDI 08), Tucson, AZ (June 2008). Citeseer, Vol. 146. Cited by: Taxonomy of Semantic Optimization Opportunity. N. H. Burow (2018) Taking back control: closing the gap between c/c++ and machine semantics. Ph.D. Thesis, Purdue University. Cited by: Introduction. W. Chen, J. Zhu, Q. Fan, Y. Ma, and A. Zou (2025) CUDA-LLM: LLMs can write efficient CUDA kernels. arXiv preprint arXiv:2506.09092. External Links: Document, Link Cited by: Introduction, Related Work. C. Cummins, V. Seeker, D. Grubisic, B. Rozière, J. Gehring, G. Synnaeve, and H. Leather (2025) LLM compiler: foundation language models for compiler optimization. In Proceedings of the 34th ACM SIGPLAN International Conference on Compiler Construction, p. 141–153. Cited by: Related Work. C. Cummins, B. Wasti, J. Guo, B. Cui, J. Ansel, S. Gomez, S. Jain, J. Liu, O. Teytaud, B. Steiner, et al. (2022) Compilergym: robust, performant compiler optimization environments for ai research. In 2022 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), p. 92–105. Cited by: Related Work. V. D’Silva, M. Payer, and D. Song (2015) The correctness-security gap in compiler optimization. In 2015 IEEE Security and Privacy Workshops, p. 73–87. Cited by: Introduction. DeepSeek-AI (2026) DeepSeek-v4 pro model card. Note: https://huggingface.co/deepseek-ai/DeepSeek-V4-ProHugging Face Model Card. Accessed July 2026 Cited by: Evaluation. P. Di Felice, A. Agnifili, and E. Clementini (1989) Data structures for compact sparse matrices representation. Advances in Engineering Software (1978) 11 (2), p. 75–83. Cited by: Taxonomy of Semantic Optimization Opportunity. M. D. Ernst, J. Cockrell, W. G. Griswold, and D. Notkin (2001) Dynamically discovering likely program invariants to support program evolution. IEEE transactions on software engineering 27 (2), p. 99–123. Cited by: Related Work. J. Gao, W. Ji, F. Chang, S. Han, B. Wei, Z. Liu, and Y. Wang (2023a) A systematic survey of general sparse matrix-matrix multiplication. ACM Computing Surveys 55 (12). External Links: ISSN 0360-0300, Document, Link Cited by: Taxonomy of Semantic Optimization Opportunity. J. Gao, W. Ji, F. Chang, S. Han, B. Wei, Z. Liu, and Y. Wang (2023b) A systematic survey of general sparse matrix-matrix multiplication. ACM Computing Surveys 55 (12), p. 1–36. Cited by: Introduction. D. Grubisic, C. Cummins, V. Seeker, and H. Leather (2024) Compiler generated feedback for large language models. arXiv preprint arXiv:2403.14714. Cited by: Introduction, Related Work. M. A. Heroux, J. Dongarra, and P. Luszczek (2013) HPCG benchmark technical specification. Technical report Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States). Cited by: Benchmark Design. H. Jiang, J. Zhu, Y. Wan, B. Fang, H. Zhang, R. Jin, and Q. Guan (2025) Can large language models understand intermediate representations in compilers?. arXiv preprint arXiv:2502.06854. Cited by: Introduction. H. Jiang, P. Yao, K. Li, T. Lin, C. Wang, and K. Ren (2026) HintPilot: llm-based compiler hint synthesis for code optimization. In Findings of the Association for Computational Linguistics: ACL 2026, p. 24970–24986. Cited by: Related Work. I. Karlin, J. Keasler, and J. R. Neely (2013) Lulesh 2.0 updates and changes. Technical report Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States). Cited by: Benchmark Design. M. Kawaguchi, P. Rondon, and R. Jhala (2009) Type-based data structure verification. In Proceedings of the 30th ACM SIGPLAN Conference on Programming Language Design and Implementation, p. 304–315. Cited by: Taxonomy of Semantic Optimization Opportunity. C. Lattner and V. Adve (2004) LLVM: a compilation framework for lifelong program analysis & transformation. In International Symposium on Code Generation and Optimization, 2004. CGO 2004., Vol. , p. 75–86. External Links: Document Cited by: Taxonomy of Semantic Optimization Opportunity. C. Lattner, M. Amini, U. Bondhugula, A. Cohen, A. Davis, J. Pienaar, R. Riddle, T. Shpeisman, N. Vasilache, and O. Zinenko (2020) MLIR: a compiler infrastructure for the end of moore’s law. arXiv preprint arXiv:2002.11054. Cited by: Taxonomy of Semantic Optimization Opportunity. C. Lattner (2005) Macroscopic data structure analysis and optimization. University of Illinois at Urbana-Champaign. Cited by: Taxonomy of Semantic Optimization Opportunity. P. T. Lin, M. A. Heroux, R. F. Barrett, and A. B. Williams (2015) Assessing a mini-application as a performance proxy for a finite element method engineering application. Concurrency and Computation: Practice and Experience 27 (17), p. 5374–5389. Cited by: Benchmark Design. B. Liskov (1987) Keynote address-data abstraction and hierarchy. In Addendum to the proceedings on Object-oriented programming systems, languages and applications (Addendum), p. 17–34. Cited by: Taxonomy of Semantic Optimization Opportunity. Z. Liu, S. Mada, and J. Regehr (2023) Minotaur: a simd-oriented synthesizing superoptimizer. arXiv preprint arXiv:2306.00229. Cited by: Related Work. LLVM Project (2025) Clang users manual. LLVM Project. Note: Accessed July 27, 2026 External Links: Link Cited by: Benchmark Design. N. P. Lopes, J. Lee, C. Hur, Z. Liu, and J. Regehr (2021) Alive2: bounded translation validation for llvm. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, p. 65–79. Cited by: Related Work. N. Lossing, P. Guillou, M. Amini, and F. Irigoin (2015) From data to effects dependence graphs: source-to-source transformations for c. In The 18th international workshop on compilers for parallel computing (CPC’15), Cited by: Introduction. Meta AI (2024) Llama 3.3 70b instruct model card. Note: https://huggingface.co/meta-llama/Llama-3.3-70B-InstructHugging Face Model Card. Accessed July 2026 Cited by: Evaluation. G. C. Necula (2000) Translation validation for an optimizing compiler. In Proceedings of the ACM SIGPLAN 2000 conference on Programming language design and implementation, p. 83–94. Cited by: Taxonomy of Semantic Optimization Opportunity. OpenAI (2026a) GPT-5.4 mini. Note: https://developers.openai.com/api/docs/models/gpt-5.4-miniOpenAI API Documentation. Accessed July 2026 Cited by: Evaluation. OpenAI (2026b) GPT-5.6 sol. Note: https://developers.openai.com/api/docs/models/gpt-5.6-solOpenAI API Documentation. Accessed July 2026 Cited by: Evaluation. A. Ouyang, S. Guo, S. Arora, A. L. Zhang, W. Hu, C. Ré, and A. Mirhoseini (2025) KernelBench: can LLMs write efficient GPU kernels?. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 47356–47415. External Links: Link Cited by: Related Work. P. Paraskevopoulou et al. (2020) Semantic program alignment for equivalence checking. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, Cited by: Taxonomy of Semantic Optimization Opportunity. Z. Paraskevopoulou (2020) Verified optimizations for functional languages. Princeton University. Cited by: Introduction. O. Pearce, J. Burmark, R. Hornung, B. Bogale, I. Lumsden, M. McKinsey, D. Yokelson, D. Boehme, S. Brink, M. Taufer, and T. Scogland (2024) RAJA performance suite: performance portability analysis with caliper and thicket. In Proceedings of the 2024 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC), Note: Held in conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis (SC-W 2024) Cited by: Benchmark Design. Prism ML (2026) Ternary bonsai 27b. Note: Hugging Face Model Card. Accessed July 2026 External Links: Link Cited by: Evaluation. P. K. Radtke, C. G. Barrera-Hinojosa, M. Ivkovic, and T. Weinzierl (2024) An extension of c++ with memory-centric specifications for hpc to reduce memory footprints and streamline mpi development. arXiv preprint arXiv:2406.06095. Cited by: Taxonomy of Semantic Optimization Opportunity. R. Sasnauskas, Y. Chen, P. Collingbourne, J. Ketema, G. Lup, J. Taneja, and J. Regehr (2017) Souper: a synthesizing superoptimizer. arXiv preprint arXiv:1711.04422. Cited by: Related Work. S. Sheth (2026) Preserving structural alias information across the mlir-to-llvm lowering boundary. Master’s thesis, California Polytechnic State University, San Luis Obispo. Cited by: Introduction. M. Steuwer, T. Remmelg, and C. Dubach (2017) Lift: a functional data-parallel ir for high-performance gpu code generation. In 2017 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), p. 74–85. Cited by: Introduction, Taxonomy of Semantic Optimization Opportunity. A. S. Tang, C. Priebe, R. Mahapatra, L. Qin, and H. Esmaeilzadeh (2026) Reasoning compiler: llm-guided optimizations for efficient model serving. Advances in Neural Information Processing Systems 38, p. 106904–106930. Cited by: Introduction. A. P. Thompson, H. M. Aktulga, R. Berger, D. S. Bolintineanu, W. M. Brown, P. S. Crozier, P. J. In’t Veld, A. Kohlmeyer, S. G. Moore, T. D. Nguyen, et al. (2022) LAMMPS-a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales. Computer physics communications 271, p. 108171. Cited by: Benchmark Design. J. R. Tramm, A. R. Siegel, T. Islam, and M. Schulz (2014) XSBench-the development and verification of a performance abstraction for monte carlo reactor analysis. The Role of Reactor Physics toward a Sustainable Future (PHYSOR). Cited by: Benchmark Design. M. Trofin, Y. Qian, E. Brevdo, Z. Lin, K. Choromanski, and D. Li (2021) Mlgo: a machine learning guided compiler optimizations framework. arXiv preprint arXiv:2101.04808. Cited by: Related Work. V. Vafeiadis, T. Balabonski, S. Chakraborty, R. Morisset, and F. Zappa Nardelli (2015) Common compiler optimisations are invalid in the c11 memory model and what we can do about it. In Proceedings of the 42Nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, p. 209–220. Cited by: Introduction. Z. Zheng, K. Wu, L. Cheng, L. Li, R. C. Rocha, T. Liu, W. Wei, J. Zeng, X. Zhang, and Y. Gao (2025) Vectrans: enhancing compiler auto-vectorization through llm-assisted code transformations. arXiv preprint arXiv:2503.19449. Cited by: Taxonomy of Semantic Optimization Opportunity.