Paper deep dive
ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval
Aaryan Kapoor, Md Abdullah Al Hafiz Khan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/4/2026, 2:48:31 AM
Summary
The paper introduces ExecRetrieval, a benchmark designed to measure the functional-correctness gap in code-embedding retrieval. It consists of 939 Python tasks, each with a canonical implementation and up to four execution-verified buggy distractors generated by single-targeted edits. The study evaluates 23 dense embedding models and BM25, finding that while top-10 retrieval is strong (exec@10=1.00 for the best system), rank-1 retrieval is weak (exec@1=0.331), with rank-1 misses often being near-identical buggy variants.
Entities (10)
Relation Signals (9)
ExecRetrieval → contains → 939 Python tasks
confidence 100% · We introduce ExecRetrieval, 939 Python tasks each paired with one execution-verified canonical implementation
Gemini-Embedding-2 → achieves → exec@10=1.00
confidence 95% · the top hosted system reaches exec@10 = 1.00... on Gemini Embedding 2
Gemini-Embedding-2 → achieves → exec@1=0.331
confidence 95% · the top hosted system reaches exec@10 = 1.00 but only exec@1 = 0.331
ExecRetrieval → measures → functional-correctness gap
confidence 95% · Measuring the Functional-Correctness Gap in Code-Embedding Retrieval
ExecRetrieval → uses → mechanical mutation
confidence 95% · each generated by a mechanical mutation making a single targeted edit
ExecRetrieval → evaluates → 23 dense embedding configurations
confidence 90% · evaluate 23 dense embedding configurations plus BM25
ExecRetrieval → evaluates → BM25
confidence 90% · evaluate 23 dense embedding configurations plus BM25
CodeSearchNet → is → prior work
confidence 80% · Public code-retrieval benchmarks... CodeSearchNet... all operate above this axis.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Embedding-based code retrieval is a core component of coding agents and retrieval-augmented code generation, where retrieving correct code matters more than retrieving lexically similar code. Existing code-retrieval benchmarks do not plant controlled, execution-verified single-edit variants of each query's canonical implementation in the search pool, leaving the question of whether embeddings can functionally discriminate correct from near-clone-but-incorrect code unanswered in a retrieval setting. Resolving this requires a benchmark whose search pool itself contains the relevant counterfactuals -- execution-verified buggy variants near-identical to each canonical -- so that a retriever's rank ordering can be directly tested for functional discrimination rather than topical or identity overlap. We introduce ExecRetrieval, 939 Python tasks each paired with one execution-verified canonical implementation and up to four execution-verified buggy distractors, each generated by a mechanical mutation making a single targeted edit, and evaluate 23 dense embedding configurations plus BM25 under provider-native invocation with paired McNemar tests and query-level bootstrap intervals. With near-clone counterfactuals in the pool, the top hosted system reaches exec@10 = 1.00 but only exec@1 = 0.331; rank-1 misses are paired buggy variants 91.5-99.4% of the time across the four leading systems, and the canonical scores below at least one of its four paired distractors in 67-78% of queries on the leading systems. The full dataset, execution oracle, embedding matrices, environment snapshot, and pairwise statistical tests are released at the URL in Appendix D.
Tags
Links
- Source: https://arxiv.org/abs/2609.01865v1
- Canonical: https://arxiv.org/abs/2609.01865v1
Trouble viewing inline? Open PDF directly →
Full Text
71,505 characters extracted from source content.
Expand or collapse full text
EXECRETRIEVAL: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval Aaryan Kapoor Kennesaw State University aaryankapoor2006@gmail.com Md Abdullah Al Hafiz Khan Kennesaw State University mkhan74@kennesaw.edu Abstract Embedding-based code retrieval is a core component of coding agents and retrieval- augmented code generation, where retrieving correct code matters more than retrieving lex- ically similar code. Existing code-retrieval benchmarks do not plant controlled, execution- verified single-edit variants of each query’s canonical implementation in the search pool, leaving the question of whether embeddings can functionally discriminate correct from near- clone-but-incorrect code unanswered in a re- trieval setting. Resolving this requires a bench- mark whose search pool itself contains the relevant counterfactuals — execution-verified buggy variants near-identical to each canon- ical — so that a retriever’s rank ordering can be directly tested for functional discrim- ination rather than topical or identity over- lap.We introduce EXECRETRIEVAL, 939 Python tasks each paired with one execution- verified canonical implementation and up to four execution-verified buggy distractors, each generated by a mechanical mutation making a single targeted edit, and evaluate 23 dense embedding configurations plus BM25 under provider-native invocation with paired McNe- mar tests and query-level bootstrap intervals. With near-clone counterfactuals in the pool, the top hosted system reachesexec@10=1.00 but onlyexec@1=0.331; rank-1 misses are paired buggy variants 91.5–99.4% of the time across the four leading systems, and the canonical scores below at least one of its four paired dis- tractors in 67–78% of queries on the leading systems. The full dataset, execution oracle, em- bedding matrices, environment snapshot, and pairwise statistical tests are released at the URL in Appendix D. 1 Introduction Embedding-based retrieval is becoming central to practical software engineering: coding assistants re- trieve snippets from local repositories and the wider open-source corpus to ground their suggestions (Zhang et al., 2023; Wang et al., 2025); IDEs use embeddings to locate prior occurrences of a func- tion; and retrieval-augmented generation pipelines for code (Lewis et al., 2020; Zhang et al., 2023) rely on embeddings to surface relevant context. In all of these settings, the embedding model serves as the candidate-recall stage of a longer pipeline: correct- ness is ultimately checked downstream by rerank- ing, execution, tests, or agentic post-processing, but the first-stage ranking determines which can- didates those later stages ever see, and in what order. What has not been measured is how much of the correctness-discrimination work this first stage passes on to the rest of the system when a near-identical buggy variant of the correct imple- mentation sits in the search pool. There is a measurement gap. The axis we care about — whether the retriever surfaces an implementation that actually passes a test suite when a near-identical buggy variant is one cosine- similarity step away — is not isolated as a con- trolled measurement anywhere.Public code- retrieval benchmarks score against canonical iden- tity and topical similarity on found code: Code- SearchNet (Husain et al., 2019), CodeXGLUE (Lu et al., 2021), CoSQA (Huang et al., 2021), CoIR (Li et al., 2025a), and CodeRAG-Bench (Wang et al., 2025) all operate above this axis. None of them pair every canonical with controlled single-edit buggy variants in the search pool, so a perfect topical retriever and a perfect functional retriever score identically on them, even though the two notions disagree on the cases that matter for downstream code use. This paper isolates the functional-correctness gap. Figure 1 shows a concrete instance: for the querylowest_set_bit, the strongest hosted em- bedder ranks three execution-failing buggy vari- ants above the canonical implementation, which passes all of its tests yet lands only at rank 4. Con- arXiv:2609.01865v1 [cs.SE] 1 Sep 2026 1 QUERY q_0348 Return the value of the lowest (least significant) set bit of a positive integer. Use the trick n & (-n). domain: bit_manipulation 2 RANK BY COSINE TO QUERY TOP-5 OF 4,694 rank cosine similaritymutation #1 0.836 no guard #2 0.817 &→| #3 0.812 off-by-1 #4 0.805 canonical #5 0.794 wrong cmp 0.780.800.820.84 embedding:google/gemini-embedding-2 3 EXEC@k: FOR THIS QUERY k= 1 × 2 × 3 × 4 ✓ 5 ✓ 10 ✓ first hit across 939 queries best of 24 systems exec@10 1.000 exec@1 0.331 rank-4 (canonical) vs rank-2 (buggy)•one-token edit, opposite semantics ✓ RANK 4•COS 0.805•canonical•PASSES 9/9 TESTS × RANK 2•COS 0.817•wrong_operator•FAILS 7/9 TESTS def lowest_set_bit(n): ifn < 1: return0 returnn&(-n) def lowest_set_bit(n): ifn < 1: return0 returnn|(-n) Uses the bit trickn & (-n)to isolate the lowest set bit; the guard returns0for non-positive inputs. n | (-n)sets every bit from the lowest set bit upward, so the result is negative (−4forn= 12) instead of being the lowest set bit alone. Figure 1: EXECRETRIEVAL in one figure (queryq_0348,lowest_set_bit). The five paired snippets occupy the top-5 retrieval slots under Gemini Embedding 2 (best of 24 systems); the canonical lands at rank 4 because three buggy variants embed closer to the query. Across all 939 queries, the best embedder reachesexec@10=1.000 yet exec@1=0.331; rank-1 misses are paired buggy variants 91.5–99.4% of the time across the four leading systems. cretely, we ask: when each natural-language query in a code-retrieval benchmark has both a canonical implementation and several near-identical buggy variants in the search pool, can modern embedding models still place a passing implementation at the top of the ranking? Every magnitude we report is conditional on that premise — a near-clone can- didate present in the pool — not an estimate of how often deployed corpora pose this choice; Sec- tion 5.3 measures how the gap scales with density, and Limitations states the scope. Closing this gap requires a benchmark with four properties: (i) for each query, a search pool con- taining both a known-correct implementation and a small set of near-identical buggy variants; (i) correctness verified by execution against a deter- ministic test suite, not by human labelling; (i) coverage across enough algorithmic domains and mutation archetypes that failure modes cannot be attributed to a single domain or bug pattern; and (iv) per-query paired statistics, so that retrievers are compared on the same inputs rather than on aggregate scores. We refer to a benchmark satisfy- ing these as functionally grounded code retrieval; EXECRETRIEVAL is one instantiation. Contributions. 1. Dataset. We construct EXECRETRIEVAL, a Python code-retrieval benchmark of 939 tasks across ten algorithmic domains. Each task ships with one canonical implementation, up to four mechanically mutated buggy distrac- tors (one query carries 3, the rest 4), and a 7–10-test execution oracle. All canonicals pass all of their own tests; all distractors fail at least one. Total corpus: 4,694 snippets. 2.Construction pipeline. We describe a high- yield reasoning-LLM pipeline (GPT-5.4 with high reasoning effort) with a 91% first-attempt validation rate (91 of 100 validation-batch en- tries). The pipeline includes a five-stage vali- dation gate (schema, AST semantics, canoni- cal execution, distractor execution, corpus in- tegrity) and an isolated-subprocess execution runner. 3. Provider-native evaluation. We evaluate 23 dense embedding configurations plus BM25 under each model’s documented best-fair-shot invocation (task types, query and passage prefixes, dtype, normalization, batch size, and similarity metric, all sourced from pri- mary provider documentation). We report exec@kandexecution_precision@kalong- side canonical-ID nDCG (Normalized Dis- counted Cumulative Gain), with paired McNe- mar tests and query-level bootstrap intervals. 4. Empirical findings. Top-kembedding re- trieval is surprisingly strong (top hosted sys- tem reachesexec@10=1.00 across all 939 queries), but rank-1 retrieval is weak: 33.1% at best. The same canonical implementa- tion scores lower in query-cosine than at least one of its four paired buggy variants in 66.8% of queries on Gemini Embedding 2 and 78.4% on Qwen3-Embedding-8B. When rank-1 is wrong, it is almost always a paired mutation (91.5–99.4% across the four lead- ing systems). The deception rate is broadly distributed across the six mechanical muta- tion types: 44.3% overall, spanning a 39.3%– 48.0% band, withremove_edge_case_check the least deceptive andwrong_comparison the most. 2 Related Work Code-retrieval benchmarks.The shared gap rel- evant to this work is that no prior code-retrieval benchmark pairs every canonical with controlled, execution-verified single-edit variants in the search pool, so the effect of a specific near-clone on rank ordering cannot be isolated from topical or iden- tity overlap. This applies whether the benchmark is human-annotated (CodeSearchNet’s challenge split (Husain et al., 2019) with∼4,000 expert rel- evance labels over 99 queries; CoSQA (Huang et al., 2021) with 20,604 human-annotated query- code labels), an aggregation suite (CodeXGLUE (Lu et al., 2021) bundles 14 datasets across 10 tasks; CoIR (Li et al., 2025a) aggregates 10 datasets across 8 retrieval tasks), or downstream-of-retrieval (CodeRAG-Bench (Wang et al., 2025) measures generation quality after retrieval). CoQuIR (Geng et al., 2026) is the closest prior work on the qual- ity axis, with 42,725 queries over 134,907 snip- pets annotated for correctness, efficiency, security, and maintainability; its correctness labels come from online-judge verdicts and Defects4J bug/fix pairs over naturally occurring code, whereas EXEC- RETRIEVAL plants controlled single-edit mutants of each query’s own canonical, so every wrong retrieval is one attributable, test-checkable edit. xCodeEval (Khan et al., 2024) scores NL-to-code retrieval by execution, with wrong-answer submis- sions as negatives, over naturally occurring solu- tions rather than controlled edits of each query’s canonical. Closest in spirit on the correctness axis, Li et al. (2025b) also study syntactically similar but functionally divergent code: an LLM- driven synthesis framework evolves Type-IV vari- ants (similar syntax, different function) validated against generated test suites, and fine-tuning em- bedding models on these pairs improves pairwise functional-consistency classification and code-to- code retrieval over a standard pool (xCodeEval; Khan et al., 2024). The distinction is construction- level: their execution-validated variants are eval- uated pairwise and serve as fine-tuning data, and retrieval is measured code-to-code, whereas EXEC- RETRIEVAL plants each execution-verified coun- terfactual inside the natural-language retrieval pool itself, so what is tested is a retriever’s rank order- ing over query-to-code search, with every failure attributable to a single targeted edit. Execution-based evaluation. On the genera- tion side, HumanEval andpass@k(Chen et al., 2021) established the pattern of testing whether sampled code passes a held-out test suite; Hu- manEval+/EvalPlus (Liu et al., 2023) showed strengthening tests materially reduces reported pass@k. MBPP (Austin et al., 2021), DS-1000 (Lai et al., 2023), CRUXEval (Gu et al., 2024), Live- CodeBench (Jain et al., 2025), and SWE-bench (Jimenez et al., 2024) extend this idea up to whole- repository issue resolution. We adopt the same execution-grounded discipline for the retrieval step that precedes generation. Embedding models.We treat the current genera- tion of code-aware and general-purpose embedding models as black-box services. Code-trained models include CodeBERT (Feng et al., 2020), GraphCode- BERT (Guo et al., 2021), CodeT5+ (Wang et al., 2023), ContraCode (Jain et al., 2021), and CodeRe- triever (Li et al., 2022). General-purpose dense retrievers we evaluate trace to Sentence-BERT (Reimers and Gurevych, 2019), DPR (Karpukhin et al., 2020), and the OpenAI (Neelakantan et al., 2022), E5 (Wang et al., 2022), GTE (Li et al., 2023), BGE (Xiao et al., 2024; Chen et al., 2024), Qwen3 Embedding (Zhang et al., 2025), and Gemini Em- bedding (Lee et al., 2025) families. MTEB (Muen- nighoff et al., 2023) established large-scale embed- ding evaluation; domain-specific MTEB-style work shows general-domain rankings transfer poorly to specialized retrieval (Tang and Yang, 2025). 3 The EXECRETRIEVAL Dataset 3.1 Task and structure Informally, an EXECRETRIEVAL instance gives a model a natural-language description of a function and asks it to rank one execution-verified canonical implementation above four near-identical buggy variants of that canonical, where “near-identical” means the variants differ from the canonical by a single targeted edit. Formally, an instance is a tuple (q, f, c, T, D 1 ,...,D n ), whereqis a natural-language description of a func- tion,fis the target function name,cis a canonical Python implementation,T = (t 1 ,...,t m )with 7≤ m≤ 10is an executable test suite ofassert statements, andD 1 ,...,D n withn ∈ 3, 4are single-mutation buggy distractors ofc. We require cto pass everyt i and eachD j to fail at least onet i . The retrieval corpus is the union of all canonicals and distractors across the benchmark: 4,694 snip- pets (939canonicals plus 3,755 distractors; 938 queries contribute 4 distractors each and 1 query contributes 3, as one pilot-era distractor was retired in the validation audit). Domain# queriesDomain# queries bit_manip.97math_num.90 collections98sort_search94 data_xform91state_mach.84 date_time99string_proc.94 geometry94validation98 Total: 939 Table 1: Domain composition of EXECRETRIEVAL. Each entry contributes 1 canonical and up to 4 distrac- tors to the retrieval corpus (one entry contributes 3). Tasks span ten algorithmic domains:bit- manipulation, collections, data-transformation, date-time,geometry,math/numerical,sort- ing/searching, state-machines, string-processing, and validation (Table 1). Function names are globally unique across the registry, and each canonical defines exactly its target function plus optional locally scoped helpers. 3.2 Generation pipeline Generation uses a two-phase, registry-driven pipeline. In the first phase, we prompt Claude Sonnet 4.6 (Anthropic, 2026) with high rea- soning effort to produce 1,000 candidate (function_name, query, domain)triples — 100 per domain — in a single completion, then manually deduplicate across domains in two passes (962 unique triples after the first pass, 954 after a second pass removed near-duplicates surfaced during pipeline iteration), shipping the 954-entry registry from which 939 validated entries were retained. Each registry entry specifies the target NL query and function name but contains no implementation; the LLM that later writes implementations cannot collude with itself on task choice. In the second phase, for each registry triple, we issue one API call per attempt to GPT-5.4 (OpenAI, 2026) withreasoning_effort=high through the OpenAI API, asking the model to return JSON with a canonical implementation, a 7–10-statementassert-only test suite, and fourmechanical-mutationdistractors,each making a single targeted edit.The locked prompt forbidswrong_semantics(“write an alternative algorithm”) and constrains distractors to six bug types:off_by_one,wrong_operator, swap_arguments,remove_edge_case_check, wrong_comparison,off_by_one_boundary. These correspond closely to classical mutation- testing operators — constant replacement, relational/arithmetic operator replacement, and statement deletion (Jia and Harman, 2011) — a family shown to produce faults that couple to real ones for testing purposes (Just et al., 2014). Each generated distractor also carries a free-textbug_descriptionfield explaining the mutation; this field is part of the released dataset and provides a per-distractor explanation that downstream users can audit. The system prompt and the key user-prompt clauses are reproduced verbatim, along with a stylized excerpt of one validated entry, in Appendix A; the complete prompts and per-entry records are in the released queries.jsonl/corpus.jsonl. Aggregate token counts and the cost of phase 2 are in Appendix B; per-entry counts ship in batch_usage. The choice of GPT-5.4 with high reasoning ef- fort is decisive. Earlier iterations using Claude Sonnet 4 (Anthropic, 2025) and Claude Sonnet 4.6 (Sonnet 4 for the original pilot, Sonnet 4.6 for the first scaled run) produced accidentally correct dis- tractors in 127 of 400 cases (31.8%): the model often emitted a correct alternative implementation when asked to write incorrect code, with 40 of those 127 cases self-admitting “actually correct” in thebug_descriptionfield. The locked prompt forbidswrong_semanticslargely because of this failure mode — almost every Sonnet correctness- compulsion case was logged under that bug type, and removing it from the allowed set forces the model to issue a single targeted mutation rather than rewriting the algorithm. GPT-5.4 with high reasoning all but eliminated the failure (3 of 4,112 first-attempt distractors, 0.07%, passed their tests); 935 of the 939 released entries (99.6%) come from GPT-5.4, the rest pilot-era Sonnet leftovers that passed all gates. Two further prompt-level refine- ments lifted the first-attempt validation rate to 91% (91 of 100 validation-batch entries; failures were regenerated): an assert-only test format directive eliminated 28% of semantic rejects, and a must pro- duce wrong output, not crash clause cut crash-type distractor rejects from 24% to 3%. Each released entry carries ametadatafield with model identifier, endpoint, and generation timestamp; 926 of 939 also carry per-entry token counts inbatch_usage, with latency on real-time entries only. 3.3 Validation oracle and integrity audit Generated entries pass five sequential gates and are retained only if all five clear: (1) schema (required fields, exactly 4 distractors, 7–10 tests); (2) AST semantics (canonical and every distractor define the target function name; almost every test is a singleAssertthat calls it, with 15 of the 8,499 released tests using a multi-statement form); (3) canonical execution (canonical passes every test); (4) distractor execution (each distractor fails at least one test, 938/939 quads do not all fail on identical test subsets, no distractor is a pure syntax error, and 3,750/3,755 distractors produce at least one bare assertion failure so the dominant rejection mode is wrong output rather than crashing on ev- ery input); and (5) corpus integrity (every query’s correct_corpus_idsreference exists; no correct canonical is unreferenced). The validator drives execution through a process- isolated runner. Each(code, test_suite)pair runs in a single fresh Python subprocess (-I, minimal env) that iterates the suite with a fresh namespace per test, with a 5-second per-suite timeout. Per-test outcomes are categorized into pass,FAIL,FAIL:<ExceptionType>,TIMEOUT, andERROR:<ExceptionType>.The runner in- jects a minimalpytest.raisesshim so exception- raising tasks are evaluable without a full pytest dependency.Final state: 939/939 canonicals pass their own tests; 3,755/3,755 paired distrac- tors fail at least one test; the execution cache contains 46,458 rows, one per distinct(code, test_suite)pair encountered across validation, the cross-canonical integrity sweep below, and top- 10 scoring for every retrieval system; each row stores a per-test outcome list. One pilot-era dis- tractor (bug typeboundary_error, queryq_0001) was retired during a late audit because it crashed on every input withNameError(malformed code) rather than failing on output; 3,755 paired distrac- tors remain. Bug type (locked prompt)count% wrong_operator85722.8 off_by_one_boundary65017.3 swap_arguments65017.3 off_by_one64217.1 wrong_comparison61716.4 remove_edge_case_check3298.8 Pre-lock legacy (4 types, combined)100.3 Total3,755 100.0 Table 2:Distractor distribution by bug type in the released corpus.The six mechanical muta- tion types of the locked prompt account for 99.7% of distractors; the remaining 10 entries are pilot- era legacy bug types that survived all validation gates —ignores_constraint,missing_edge_case, wrong_semantics (3 each); wrong_algorithm (1). Because mechanically mutated distractors are similar to their canonical, test-suite ambiguity could corrupt the deception measurement. We ran two integrity sweeps: a distractor-sanity sweep, in which every one of the 3,755 paired distractors is executed against its own query’s tests, and a cross- canonical sweep, which exhaustively searches ev- ery corpus item’s AST for a function definition whose name matches any other query’s target func- tion and, when found, runs that item against the other query’s tests. The first sweep finds 0 dis- tractors that pass all of their own tests; the second sweep finds 0 module-level cross-entry name col- lisions in the released corpus, so no canonical or distractor can accidentally satisfy another query’s tests. The 939 target function names are globally unique across the registry. Distractor bug-type com- position is given in Table 2. 4 Experimental Setup 4.1 Metrics For a queryq, letˆr 1 ,..., ˆr |C| be the corpus re- trieval ranking and letPASS(c,T q )∈0, 1denote whether snippet c passes all of q’s tests. Define: exec@k(q) = 1[∃i≤ k : PASS(ˆr i ,T q ) = 1 ], execp@k(q) = 1 k k X i=1 1[PASS(ˆr i ,T q ) = 1]. We reportexec@k(at least one passing in top-k), execution_precision@k(execp@k; fraction of top-kthat pass), and canonical-ID nDCG (Järvelin and Kekäläinen, 2002). Aggregates are taken as the unweighted mean over the 939 queries. All three families are reported for k ∈1, 3, 5, 10. 4.2 Confidence intervals and paired tests Confidence intervals on each metric are computed by bootstrap resampling (Efron, 1979) of the 939 queries with 5,000 replicates and fixed seed. For pairwise comparisons between models on a binary exec@k, we report the exact McNemar test (Mc- Nemar, 1947; Dietterich, 1998) on the discordant- pair counts, alongside the paired-bootstrap differ- ence and its query-level 95% interval. For contin- uous metrics (execution_precision, nDCG) we report only the paired bootstrap interval. All raw test outputs are released alongside the dataset. 4.3 Models and provider-native invocation We evaluate 23 dense embedding configura- tions across Google Gemini, Mistral, OpenAI, Qwen3, BGE, BGE-M3, E5, GTE, and Sentence- Transformers families, plus a BM25 baseline (Robertson and Zaragoza, 2009) (k1=1.5,b=0.75). Each model is invoked with its documented best- fair-shot setup, taken from the provider’s pri- mary API docs or model card. Gemini Embed- ding 001 uses task-typeCODE_RETRIEVAL_QUERY on queries andRETRIEVAL_DOCUMENTon corpus snippets, per Google’s documented code-retrieval recipe (Google, 2025); Gemini 2’s API does not expose task types, so we adopt the textual in- struction conventions from the same documen- tation (“task: code retrieval” for queries, neu- tral framing for corpus items).Qwen3 mod- els prepend“Instruct: ⟨task⟩ : ”to queries per the official model cards; E5 prepends “query: ”/“passage: ”; BGE prepends the official retrieval prefix to queries; OpenAI, Mistral, GTE, and Sentence-Transformers models receive raw text (OpenAI, 2024; Mistral AI, 2025). All mod- els use cosine similarity overL 2 -normalized em- beddings exceptmulti-qa-mpnet-base-dot-v1, which uses unnormalized dot product per its card. Qwen3-8B uses BF16 because FP16 pro- duced non-finite similarities on our A40 hard- ware; we add Qwen3-4B-BF16 as a dtype con- trol row. BGE-M3 is run in dense-only mode; hy- brid dense+sparse+ColBERT would be a separate model row. Operationally critical settings appear in Appendix E, Table 6; the machine-readable invoca- tion table (model IDs, dimensions, dtypes, similar- ity, batch sizes, conditioning, provider doc URLs) is released asresults/invocation_table.json; SDK versions are in pip_freeze.txt. 4.4 Reproducibility The frozen evaluation consists of: the 4,694- snippet corpus and 939-query files (deterministic IDs, SHA-256 hashed); 23 saved.npzembedding matrices (each passes shape, finite-value, and zero- norm checks across queries and corpus); a per-pair execution-result cache; apip freeze; and a SHA- 256 manifest over all the above. The validator and runner depend only on the Python standard library plus a minimalpytest.raisesshim. Provider API calls use retry/backoff and fail fast on missing API keys. The full bundle is released at the URL listed in Appendix D. 5 Results 5.1 Top-k saturates while rank-1 fails Figure 2 shows the 23-model dense leaderboard plus BM25, sorted byexec@1; the full per-model table withexec@k,execution_precision@k, and nDCG@kfork ∈ 1, 3, 5, 10is reproduced in Appendix F. Three observations dominate. Rank-1 retrieval is hard. The topexec@1 system, Gemini Embedding 2, reaches only exec@1=0.331,withbootstrap95%CI [0.299, 0.362]. Gemini Embedding 001 reaches 0.329, and the two are statistically indistinguish- able atexec@1(paired McNemar exactp=0.958; difference+0.002, 95% CI[−0.038, +0.042]). Mistral Embed ranks third (0.224), followed by the open-weight Qwen3 family. Top-kretrieval is easy.exec@10is essentially saturated for hosted systems: Gemini Embedding 2 reachesexec@10=1.00 over all 939 queries, and Gemini Embedding 001 and Codestral Embed 2505 also achieve 1.00. Even mid-tier embed- ding models like BGE-base and E5-large-v2 reach exec@10∈ [0.94, 0.95]. Lexical retrieval is a poor proxy.BM25 reachesexec@1=0.058 andexec@10=0.422. Lex- Gemini Emb. 2 Gemini Emb. 001 Mistral Embed Qwen3-8B Qwen3-4B BF16 Qwen3-4B FP16 Codestral OpenAI 3-large OpenAI ada-002 GTE-large MiniLM-L12 BGE-M3 mE5-large BGE-base OpenAI 3-small BGE-large E5-large MPNet-base MQ-MPNet GTE-base E5-base MiniLM-L6 BM25 Para-MiniLM 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 exec@1 (bars, 95% bootstrap CI) exec@5 ( ) Google nativeMistral nativeOpenAI nativeLocal HFBM25 (lexical)exec@5 0.331 0.329 0.224 0.214 0.204 0.196 0.185 0.113 0.1040.104 0.086 0.0850.085 0.0830.083 0.078 0.0770.076 0.073 0.071 0.0690.069 0.058 0.050 Gemini Emb. 2: rank-1 floor 0.33, rank-5 ceiling 1.00 Figure 2: EXECRETRIEVAL leaderboard. Bars areexec@1with 95% bootstrap intervals; diamonds markexec@5. Hosted Google Gemini Embedding models lead at rank 1; open-weight Qwen3 dominates at the top of the open- weight tier; and twelve of the thirteen embedding models below 10% at rank 1 reach exec@5 of 0.80–0.89. ical overlap occasionally identifies the canonical solution, but its top-10 success is below the worst dense embedding model (paraphrase-MiniLMat 0.671). BM25 is included as a calibration baseline, not as a competitive system. Figure 3 plotsexec@kcurves for representative systems and shows the structural pattern across the leaderboard: for hosted systems and Qwen3, the slope fromk=1 tok=3 is steep, the curve has effectively saturated byk=5, and the rank-1 floor is much lower than top-k headroom suggests. 13510 k 0.0 0.2 0.4 0.6 0.8 1.0 exec@k exec@5/@10 saturation band 0.331 0.329 0.224 0.214 0.185 0.113 0.104 0.083 0.058 Gemini Emb. 2 Gemini Emb. 001 Codestral Qwen3 8B OpenAI 3-large Mistral Embed GTE-large BGE-base BM25 (lexical) Figure 3:exec@kcurves for representative systems. Most strong systems saturateexec@10but plateau far below 1 at exec@1. 5.2 When the model misses, what does it retrieve? To understand the structure of rank-1 failure, we ask, for each query where the rank-1 retrieved snip- pet does not pass the tests, which corpus item was placed at rank 1? Two possibilities exist: it could be one of the four paired buggy variants of that query’s canonical, or it could be any other snippet in the 4,694-snippet corpus (e.g., a canonical or distractor from a different entry). Because function Modelmiss@1paired dist. rank-1 Gemini Embedding 2628624 (99.4%) Gemini Embedding 001630618 (98.1%) Mistral Embed729667 (91.5%) Qwen3-Embedding-8B738718 (97.3%) Table 3: When rank-1 is wrong, it is almost always a paired buggy variant. “miss@1” is the count of queries whose top-1 fails the tests; the last column is how many of those are paired single-mutation distractors of the same query’s canonical. names are globally unique and our integrity audit (Section 3.3) shows no cross-canonical test-suite leakage, only the paired distractors are functional near-clones of the canonical; everything else is un- related code. Table 3 shows the result. For all four lead- ing models, between 91.5% and 99.4% of rank-1 misses are paired buggy variants. Failure of embed- ding retrieval at rank 1 is therefore not an arbitrary lookup error; it is a specific, structural inability to separate a canonical from its single-mutation buggy near-clones. 5.3 One near-clone is enough: pool-density ablation Is the measured gap an artifact of the deliberately dense pool (up to four planted near-clones per query)? We ablate density directly: for each query we remove a subset of its own paired distractors, holding the rest of the corpus fixed, and compute the exact expectedexec@1over all retained subsets of sized∈0,..., 4(the 3-distractor query con- tributes its shipped configuration atd=4). Ranking follows the scoring engine’s realized sort order, so the full-pool column reproduces the published leaderboard exactly for all 23 configurations. Table 4 shows the result. With zero near-clones 0.7000.7250.7500.7750.8000.8250.8500.875 query cosine (higher = better) 0 5 10 15 queries (density) Gemini Embedding 2: per-query cosine best paired distractor canonical 0.030.020.010.000.010.020.030.040.05 canonical best-distractor gap (< 0: canonical loses) 0 50 100 queries Gemini Embedding 2: per-query gap 0.550.600.650.700.750.800.850.900.95 query cosine (higher = better) 0 2 4 6 8 queries (density) Qwen3-Embedding 8B: per-query cosine best paired distractor canonical 0.040.020.000.020.040.060.080.100.12 canonical best-distractor gap (< 0: canonical loses) 0 100 200 300 queries Qwen3-Embedding 8B: per-query gap Figure 4: Per-query cosine of the canonical (green) vs. highest-cosine paired distractor (red), for Gemini 2 (top) and Qwen3-8B (bottom); right column: per-query difference. The canonical lies below at least one paired distractor in 66.8% and 78.4% of queries. 0.00.10.20.30.40.50.6 Deception rate (distractor cosine canonical cosine) wrong comparison swap arguments off by one wrong operator off by one boundary remove edge case check 48.0% (n=6,812 / 14,191) 47.0% (n=7,024 / 14,950) 43.7% (n=6,455 / 14,766) 43.1% (n=8,490 / 19,711) 42.8% (n=6,399 / 14,950) 39.3% (n=2,972 / 7,567) overall = 44.3%50% (random) Figure 5: Per-(query, paired distractor, model) deception rate by mechanical mutation type. A triple is “deceived” when the distractor’s similarity to the query is at least as high as the canonical’s. Aggregated over the 23 dense embedding configurations and the six mechanical mutation types defined in Section 3.2. exec@1 by # paired distractors retained Model0123all Gemini Emb. 20.993 0.678 0.507 0.4020.331 Gemini Emb. 001 0.984 0.718 0.541 0.4180.329 Mistral Embed0.928 0.615 0.425 0.3040.224 Qwen3-Emb.-8B0.976 0.613 0.409 0.2900.214 Table 4: Pool-density ablation: expectedexec@1when each query retainsdof its own paired distractors (ex- act expectation over all retained subsets; the rest of the 4,694-snippet corpus is unchanged). The “all” column is the shipped configuration and reproduces the published leaderboard exactly for every one of the 23 configura- tions. One near-clone drops the strongest system from 0.993 to 0.678. retained, rank-1 retrieval is essentially solved (0.93– 0.99 for the four leading systems; 0.532–0.993 across all 23, 6 at or above 0.95) — for the leading systems, topical retrieval is not the hard part. Re- taining a single near-clone, averaged over which one, drops the strongest system from 0.993 to 0.678; the relative one-clone drop is 27–49% across all 23. Density scales the effect but does not create it: the benchmark measures what happens given a near-clone candidate in the pool, already at the lowest density. 5.4 Canonical-vs-best-distractor similarity gap Figure 4 quantifies the mechanism. For each query, we compute (a) the cosine similarity between the query and the canonical, and (b) the maximum cosine over its paired mechanically mutated buggy variants. The right column of Figure 4 plots the per- query difference (canonical−best distractor); over the 938 queries with at least one mechanical-type paired distractor, the canonical is below at least one of those distractors in 66.8% of queries on Gemini Embedding 2 (median gap−0.002, mean−0.002) and 78.4% on Qwen3-Embedding-8B (median gap −0.002, mean−0.002). The mean per-query gap is small in absolute terms (the canonical and its best mutated variant are nearly indistinguishable to the embedding), but the sign is wrong on the majority of queries. The two systems use different invocation recipes (Gemini uses a task-prefix instruction; Qwen uses an instruction-style prompt), yet both show the same qualitative pattern: overlapping cosine dis- tributions for canonical vs. best paired distractor, with the per-query difference concentrated tightly around zero but biased toward the negative side. At exec@1 paired tests Comparison (A vs B)A-only B-only ∆exec@1 McNemar exact p∆exec@3 ∆exec@5 Gemini Emb. 2 vs Gemini Emb. 001179177+0.0020.958 −0.066 (p=2.7×10 −5 )+0.009 Gemini Emb. 2 vs Mistral Embed224123+0.1086.5×10 −8 +0.028+0.060 Gemini Emb. 2 vs Qwen3-8B228118+0.1173.4×10 −9 +0.039+0.018 Qwen3-8B vs Qwen3-4B BF16136127+0.0100.622−0.002 −0.004 Qwen3-4B BF16 vs Qwen3-4B FP164840+0.0090.456−0.005+0.001 Table 5: Selected paired comparisons. “A-only”/“B-only” are McNemar discordant counts atk = 1out of 939. The full pairwise table (every pair, all k, all three metrics) is released as pairwise_tests.json. least one of a query’s near-clones typically embeds closer to the NL query than the canonical, though an individual distractor does so in 44.3% of triples. 5.5 Deception is broadly distributed across bug types We define a (query, paired distractor, model) triple as deceived when the distractor’s similar- ity score to the query is at least as high as the canonical’s (cosine over L 2 -normalized em- beddings for 22 of 23 models; raw dot prod- uct formulti-qa-mpnet-base-dot-v1per its model card). Across 23 dense configurations and the six mechanical mutation types of the locked prompt, 86,135 paired triples are evaluable, of which 44.3% are deceived (Figure 5). Per-bug- type rates span a narrow 39.3%–48.0% band: remove_edge_case_checkis the least deceptive (39.3%) andwrong_comparisonthe most (48.0%); the other four cluster between 42.8% and 47.0%. No mechanical mutation is systematically near ran- dom; all six fool an embedding roughly two times in five. This argues that the failure mode is not bound to a single mutation pattern (e.g. off-by-one); it is a property of embedding near-identical code rather than of any particular bug. Nor is the failure mode bound to micro-edits (Ap- pendix G): deception shows no meaningful associ- ation with edit size (39.5–47.5% across character- span buckets, 1 to 21+ characters; tie-aware Spear- man|ρ| ≤ 0.07), and stacking 2–4 execution- verified, line-disjoint mutations into the same func- tion leaves deception within 1.7 points of the single- mutation rate, with no drop statistically distinguish- able from zero (matched 371-query comparison, four locally run embedders). 5.6 Between-family gaps are detectable; within-family scaling is not Table 5 reports the headline paired tests. Gemini 2 and Gemini 001 are indistinguishable at rank 1 (paired McNemar exactp = 0.958) but Gemini 001 is significantly stronger atexec@3(p≈ 2.7×10 −5 ) — which is the operating regime where the choice between the two Google models matters. Gemini 2 is a clear winner over Mistral Embed and Qwen3- 8B atexec@1(bothp < 10 −7 ); the absolute gap on Qwen3-8B is+0.117, i.e. Gemini 2 succeeds at rank 1 on∼110 more queries than Qwen3-8B out of 939. Within the Qwen3 family we detect no scaling: 8B is not significantly better than 4B- BF16 atexec@1(p = 0.622), nor does 4B-BF16 outperform 4B-FP16 (p = 0.456). We therefore re- port the family-level claim (Qwen3 leads the open- weight tier) rather than a specific scaling claim. 5.7 Execution and identity ranking disagree Execution and identity-based retrieval are not inter- changeable. Gemini 001 leads on nDCG@5 and onexec@3, but Gemini 2 has the highestexec@1; Codestral Embed 2505 reaches the highestexec@5 among non-Gemini systems with lower nDCG@5. The median query is solved at rank 1 by only 3 of 24 systems and 88 queries by no system (Ap- pendix C); the per-domain breakdown shows date- time as the hardest category (meanexec@1 =0.091 across all 24 systems, vs. 0.107–0.151 elsewhere), bottom-three for every top-five model. 6 Discussion and Conclusion With up to four near-clones planted per query, top-kis strong (exec@10≥0.98 for the four lead- ing systems) but rank-1 is brittle: misses return a paired buggy variant 91.5–99.4% of the time, and the canonical scores below at least one paired distractor in 67–78% of queries on the leading systems. Current code embeddings function as candidate-recall components; both metric fami- lies matter. EXECRETRIEVAL thus measures the correctness-discrimination burden that candidate re- trieval passes to downstream verification — rerank- ing, execution, or an LLM examining the for- warded candidates — not the failure of complete coding-agent systems. Limitations Python only. All canonicals, distractors, and tests are Python. Cross-language generalization re- mains open; the generation and validation pipeline is language-agnostic in design but each new lan- guage requires its own runner. Mechanical, not human-written, distractors. Our distractors are single-targeted-edit LLM muta- tions rather than natural human bugs (8.5% of me- chanical pairs span multiple lines, up to 15, predom- inantly edge-case-guard removals); correctness is relative to each entry’s generated tests, which may exceed the query’s stated domain. By design, me- chanical mutations always differ from the canon- ical, the mutation family corresponds to classical mutation-testing operators (Jia and Harman, 2011; Just et al., 2014), and the locked prompt forbids LLM-style rewrites that often turn out to be func- tionally correct. Real-world bugs include subtler patterns and stylistic confounds this benchmark does not exhaust; Appendix G finds no detectable dependence of deception on edit size (23 configu- rations) or stacked-mutation count (four local em- bedders). Closed-world corpus.The retrieval corpus is ex- actly the 4,694 snippets generated for the bench- mark. We do not add open-domain code or dis- tractors from other corpora. This isolates the functional-correctness signal but makesexec@kin- comparable to open-domain code-search numbers. Every reported magnitude is likewise conditional on a near-clone candidate in the pool (Section 5.3); how often deployed corpora pose this choice is unmeasured — doing so requires an execution or- acle over open-domain code. Near-clones plau- sibly co-occur with their targets in agentic loops caching failed attempts, retrieval over several sam- pled implementations of a task, monorepos with copy-pasted diverged utilities, and commit-history indices holding pre-fix versions. Clone studies make this plausible: 70% of GitHub code consists of file-level clones of earlier files (Lopes et al., 2017), about half of clone groups contain incon- sistently modified members (52%; Juergens et al., 2009), replicated on three further industrial sys- tems (Wagner et al., 2016), and Python commit his- tories alone contain nearly a million deduplicated single-statement pre-fix variants of later-fixed code (Richter and Wehrheim, 2022; Prenner and Robbes, 2025) — though these establish plausibility, not frequency in retrieval results. Execution runner is not a hardened sandbox. The Python subprocess runner provides process isolation and a 5-second timeout, but does not con- strain filesystem or network access. We do not recommend using it on untrusted code outside this benchmark. Embedding-only first-stage retrieval.EXEC- RETRIEVAL measures the first retrieval stage. The natural next question is how cross-encoder rerankers (or LLM rerankers) behave on these dis- tractors; see Future work below. Provider API drift. Hosted-API embeddings are not permanently reproducible across provider model updates. We mitigate this by releasing the resulting embedding matrices and the execution cache; re-running scoring is fully deterministic from the saved.npzmatrices, even if the live provider endpoint changes. Scale. 939 queries is small compared to topical retrieval corpora that contain millions of (query, function) pairs. The binding cost is per-query exe- cution verification of every distractor against every test, and is intentional. The tasks are deliberately self-contained functions — the level at which a uniform oracle and attributable single-edit failures are tractable; repository-scale code with cross-file context is beyond them. The pipeline scales with generation budget (Appendix B). Future work.Three follow-ups are natural. First, cross-encoder and LLM-based rerankers operating on the embedding-retrieved top-kmay close the rank-1 gap without retraining the first-stage embed- ding models; the released oracle and frozen matri- ces make measuring that rank-1 lift deterministic. Second, porting the pipeline to JavaScript, Rust, or Go would test whether the near-clone deception we report is Python-specific or general across lan- guages. Third, free-form LLM-generated buggy implementations, kept only when they fail the tests, would probe a broader, more naturalistic bug distri- bution; this complements rather than replaces me- chanical mutation — a free-form rewrite loses the single-attributable-edit control, and execution filter- ing is not optional (127 of 400 pilot buggy-rewrite requests were accidentally correct; Section 3.2). Acknowledgments We thank the three anonymous ARR reviewers and the area chair, whose requests and suggestions shaped this version: the pool-density, edit-size, and multi-mutation analyses were added during the re- view period at their prompting. This work received no institutional or grant funding; the first author thanks Manish Kapoor for funding this project and for his encouragement and thoughtful discussions throughout. References Anthropic. 2025. Introducing Claude 4.https://w. anthropic.com/news/claude-4. Anthropic. 2026. Claude Sonnet 4.6.https://w. anthropic.com/claude/sonnet. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024.M3- Embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self- knowledge distillation. In Findings of the Associa- tion for Computational Linguistics: ACL 2024, pages 2318–2335. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluat- ing large language models trained on code. arXiv preprint arXiv:2107.03374. Thomas G. Dietterich. 1998. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7):1895–1923. Bradley Efron. 1979. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1):1–26. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code- BERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020. Jiahui Geng, Fengyu Cai, Shaobo Cui, Qing Li, Liang- wei Chen, Chenyang Lyu, Haonan Li, Derui Zhu, Alexander Pretschner, Heinz Koeppl, and Fakhri Kar- ray. 2026. CoQuIR: A comprehensive benchmark for code quality-aware information retrieval. In Proceed- ings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL), pages 11169– 11185. Google. 2025. Gemini API: Embeddings.https://ai. google.dev/gemini-api/docs/embeddings. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. 2024. CRUXEval: A benchmark for code reasoning, understanding and execution. In Proceedings of the 41st International Conference on Machine Learning (ICML), pages 16568–16621. Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svy- atkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. Graph- CodeBERT: Pre-training code representations with data flow. In International Conference on Learning Representations (ICLR). Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. 2021. CoSQA: 20,000+ web queries for code search and question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL). Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Code- SearchNet challenge: Evaluating the state of seman- tic code search. arXiv preprint arXiv:1909.09436. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. 2025. Live- CodeBench: Holistic and contamination free evalu- ation of large language models for code. In Inter- national Conference on Learning Representations (ICLR). Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph E. Gonzalez, and Ion Stoica. 2021. Con- trastive code representation learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). Kalervo Järvelin and Jaana Kekäläinen. 2002. Cu- mulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446. Yue Jia and Mark Harman. 2011. An analysis and sur- vey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5):649– 678. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can language mod- els resolve real-world GitHub issues?In Inter- national Conference on Learning Representations (ICLR). Elmar Juergens, Florian Deissenboeck, Benjamin Hum- mel, and Stefan Wagner. 2009. Do code clones mat- ter? In Proceedings of the 31st International Con- ference on Software Engineering (ICSE), pages 485– 495. René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), pages 654–665. Vladimir Karpukhin, Barlas O ̆ guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2024. XCodeEval: An execution- based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (ACL), pages 6766–6805. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. DS-1000: A natural and reliable benchmark for data science code generation. In Proceedings of the 40th International Conference on Machine Learning (ICML). Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Hen- rique Schechter Vera, Xiaoqi Ren, Shanfeng Zhang, Daniel Salz, Michael Boratko, Jay Han, Blair Chen, Shuo Huang, Vikram Rao, Paul Suganthan, Feng Han, Andreas Doumanoglou, Nithi Gupta, Fedor Moiseev, Cathy Yip, Aashi Jain, Simon Baumgart- ner, Shahrokh Shahi, Frank Palma Gomez, Sandeep Mariserla, Min Choi, Parashar Shah, Sonam Goenka, Ke Chen, Ye Xia, Koert Chen, Sai Meher Karthik Duddu, Yichang Chen, Trevor Walker, Wenlei Zhou, Rakesh Ghiya, Zach Gleicher, Karan Gill, Zhe Dong, Mojtaba Seyedhosseini, Yunhsuan Sung, Raphael Hoffmann, and Tom Duerig. 2025. Gemini embed- ding: Generalizable embeddings from Gemini. arXiv preprint arXiv:2503.07891. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge- intensive NLP tasks. In Advances in Neural Informa- tion Processing Systems (NeurIPS). Xiangyang Li, Kuicai Dong, Yi Quan Lee, Wei Xia, Hao Zhang, Xinyi Dai, Yasheng Wang, and Ruiming Tang. 2025a. CoIR: A comprehensive benchmark for code information retrieval models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL). Xiaonan Li, Yeyun Gong, Yelong Shen, Xipeng Qiu, Hang Zhang, Bolun Yao, Weizhen Qi, Daxin Jiang, Weizhu Chen, and Nan Duan. 2022. CodeRetriever: A large scale contrastive pre-training method for code search. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281. Zhuohao Li, Wenqing Chen, Jianxing Yu, and Zhichao Lu. 2025b. Functional consistency of LLM code em- beddings: A self-evolving data synthesis framework for benchmarking. arXiv preprint arXiv:2508.19558. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Ling- ming Zhang. 2023. Is your code generated by Chat- GPT really correct? Rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems (NeurIPS). Cristina V. Lopes, Petr Maj, Pedro Martins, Vaibhav Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan Vitek. 2017. DéjàVu: A map of code duplicates on GitHub. Proceedings of the ACM on Programming Languages, 1(OOPSLA):84:1–84:28. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundare- san, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. Mistral AI. 2025. Mistral embeddings API reference. https://docs.mistral.ai/api/endpoint/ embeddings. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics (EACL). Arvind Neelakantan, Tao Xu, Raul Puri, Alec Rad- ford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hal- lacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welin- der, and Lilian Weng. 2022. Text and code em- beddings by contrastive pre-training. arXiv preprint arXiv:2201.10005. OpenAI. 2024.OpenAI embeddings.https: //platform.openai.com/docs/guides/ embeddings. OpenAI. 2026. GPT-5.4 Model Documentation.https: //platform.openai.com/docs/models/gpt-5.4. Julian Aron Prenner and Romain Robbes. 2025. Bogus bugs, duplicates, and revealing comments: Data qual- ity issues in NPR. In Proceedings of the IEEE/ACM International Workshop on Automated Program Re- pair (APR), pages 43–47. Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). Cedric Richter and Heike Wehrheim. 2022. TSSB-3M: Mining single statement bugs at massive scale. In Proceedings of the 19th International Conference on Mining Software Repositories (MSR), pages 418–422. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and be- yond. Foundations and Trends in Information Re- trieval, 3(4):333–389. Yixuan Tang and Yi Yang. 2025. FinMTEB: Finance massive text embedding benchmark. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP). Stefan Wagner, Asim Abdulkhaleq, Kamer Kaya, and Alexander Paar. 2016. On the relationship of in- consistent software clones and faults: An empirical study. In Proceedings of the 23rd IEEE Interna- tional Conference on Software Analysis, Evolution, and Reengineering (SANER), pages 79–89. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly- supervised contrastive pre-training. arXiv preprint arXiv:2212.03533. Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023. CodeT5+: Open code large language models for code understanding and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yiqing Xie, Graham Neubig, and Daniel Fried. 2025. CodeRAG-Bench: Can retrieval aug- ment code generation? In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3199–3214. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muen- nighoff, Defu Lian, and Jian-Yun Nie. 2024. C-Pack: Packed resources for general Chinese embeddings. In Proceedings of the 47th International ACM SI- GIR Conference on Research and Development in Information Retrieval (SIGIR). Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. RepoCoder: Repository-level code completion through iterative retrieval and gen- eration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176. A Generation Prompts and Example Entry Listings 1–2 reproduce the verbatim system prompt and user-prompt clauses used by the generation pipeline of Section 3.2.Listing 3 shows a stylized excerpt of one validated entry (next_power_of_two_bitwise, queryq_0134, canonicalc_0666, distractorsc_0667–c_0670); the full record (with all 9 tests and verba- timbug_descriptionstrings) is in the released JSONL files. Listing 1: System prompt for generation. You are an expert Python programmer generating benchmark data for a code retrieval evaluation system. You must return ONLY valid JSON -- no markdown fences, no commentary, no explanation. Just a JSON array. Listing 2: User-prompt clauses (verbatim). For each query, produce: 1. canonical: correct stdlib-only Python. 2. test_suite: 7-10 assert statements covering normal, edge, and distinguishing cases. STRICT FORMAT: each test MUST be a single `assert function_name(...) == ...`. No helpers, no setup, no comments. 3. distractors: 4 plausible-but-wrong functions, each a MECHANICAL MUTATION of the canonical (single targeted change). Must produce WRONG OUTPUT, not crash. Allowed bug types: off_by_one, wrong_operator, swap_arguments, remove_edge_case_check, wrong_comparison, off_by_one_boundary. DO NOT use "wrong_semantics" -- writing an alternative correct implementation is not a valid distractor. Listing 3: Stylized excerpt of one validated entry; full record in released JSONL files. "function_name": "next_power_of_two_bitwise", "query": "Return the smallest power of two >= a given positive integer n. Use bit manipulation, not math.log.", "canonical": "def next_power_of_two_bitwise(n): if n < 2: return 1 n -= 1 shift = 1 while (n >> shift) > 0: n |= n >> shift shift <<= 1 return n + 1", "test_suite": [ "assert next_power_of_two_bitwise(0)==1", "assert next_power_of_two_bitwise(1)==1", ... ], "distractors": [ "code": "...", "bug_type": "remove_edge_case_check", "bug_description": "Deleted the initial`if n < 2:`.", "code": "...", "bug_type": "wrong_comparison", "bug_description": "Changed`n < 2` to`n <= 2`.", "code": "...", "bug_type": "off_by_one_boundary", "bug_description": "Changed`n -= 1` to`n -= 2`.", "code": "...", "bug_type": "wrong_operator", "bug_description": "Final return`n + 1` ->`n - 1`." ] B Generation Economics We aggregate the per-entrybatch_usagemetadata of the 926 released entries with available token counts. GPT-5.4 withreasoning_effort=high consumed, in total, 915,178 prompt tokens and 7,083,062 completion tokens across these 926 API calls (5,871,712 of which are hidden reason- ing tokens, reported by OpenAI as a subset of completion_tokens; mean per entry: 988 prompt, 7,649 completion of which 6,341 reasoning; me- dian wall-clock 99 s for real-time-endpoint entries; batch-endpoint entries do not record per-request latency). At GPT-5.4 list pricing of $2.50 per 1M input tokens and $15.00 per 1M output tokens (out- put equalscompletion_tokens, which already in- cludes reasoning per OpenAI’s billing convention; pricing and billing conventions per the provider documentation (OpenAI, 2026), as of dataset con- struction), the generation cost at native real-time list pricing is $108.53. Approximately 57% of these API calls were routed through the OpenAI Batch API, which carries a 50% discount on both rates and a 24-hour SLA; we report the list-price figure as the headline cost since it is a lower bound on the budget required to reproduce the dataset without depending on batch-endpoint availability. With either prompt fix from Section 3.2 missing, end-to-end validation rates fall to 62–70% even with high-effort reasoning, so the cost reported here would scale roughly inversely with the validation rate. C Per-Query Difficulty and Per-Domain exec@1 05101520 # models (out of 24) with exec@1 = 1 on the query 0 25 50 75 100 125 150 175 queries 88 queries solved by no model 1 queries solved by every model median 3 of 24 models correct mean 3.05 models correct per query (a) per-query difficulty bit manipulation collectionsdata transformation date time geometrymath numerical sorting searching state machines string processing validation Gemini 2 Gemini 001 Mistral Embed Qwen3-8B Qwen3-4B BF16 Codestral OpenAI 3-large GTE-large BM25 0.410.390.440.200.320.340.300.300.290.33 0.330.360.420.200.380.360.240.330.330.35 0.230.260.300.170.280.190.230.180.150.26 0.240.230.200.180.290.180.190.200.200.22 0.210.240.240.130.200.200.160.200.230.22 0.140.180.160.210.340.190.130.110.120.26 0.150.130.070.120.120.120.060.120.100.13 0.090.060.140.110.090.140.110.100.070.13 0.090.060.030.040.070.060.020.140.040.02 (b) per-domain exec@1, top-9 models 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 exec@1 Figure 6: (a) Distribution of how many of the 24 eval- uated systems achieveexec@1=1 on each query. The median query is solved by exactly 3 of 24 systems; 88 queries are solved by no system, 1 by every system. (b) Per-domain exec@1 for the top-9 systems. ModelProviderdimsim batchdtypeQuery / passage conditioning Gemini Emb. 001Google nat. 3072cos 32–task type CODE_RETRIEVAL_QUERY / RETRIEVAL_DOCUMENT Gemini Emb. 2Google nat. 3072cos 32–“task: code retrieval| query: ” / “title: none| text: ” Mistral EmbedMistral nat. 1024cos 64–raw text Codestral Embed 2505 Mistral nat. 1536cos 64–raw text OpenAI 3-L / 3-S / ada-002 OpenAI nat. 3072/1536/1536 cos 128–raw text Qwen3 4B / 4B BF16 / 8B Local HF2560/2560/4096 cos 8/8/4fp16/bf16/bf16 “Instruct: ⟨task⟩ : ” on queries BGE-base / -large v1.5 Local HF768/1024cos 64/32–“Represent this sentence for searching relevant passages: ” BGE-M3 (dense only)Local HF1024cos 32– native FlagEmbedding dense path E5-base/-large-v2, mE5-large Local HF768/1024/1024cos 64/32/32–“query: ” / “passage: ” GTE-base / -largeLocal HF768/1024cos 64/32–raw text S-T MiniLM-L6 / MiniLM-L12 / MPNet / Para-MiniLM Local HF384/384/768/384 cos 128/128/64/128 –native sentence-transformers Multi-QA MPNetLocal HF768dot 64–native; no L2 normalization BM25lexical– k1=1.5, b=0.75 Table 6: Provider-native and local-best invocation settings used in EXECRETRIEVAL. D Released Artifacts The full release lives at a Hugging Face dataset:https://huggingface.co/datasets/ AaryanK/ExecRetrieval. All files are released under C BY 4.0. The bundle contains: 1. the corpus (corpus.jsonl, 4,694 rows) and the query file (queries.jsonl, 939 rows), with deterministic IDs and per-entry metadata(generator model id, endpoint, generation timestamp) plusbatch_usage (prompt/completion/reasoning tokens and la- tency) for the 926 instrumented entries of Ap- pendix B; 2. the per-distractor metadata file with the bug_descriptionexplanation field for each of the 3,755 paired distractors; 3. theper-pairexecutioncache(46,458 outcome rows keyed by(code_sha256, tests_sha256)); 4.the 23 saved.npzembedding matrices (one per dense embedding configuration); 5.the full leaderboard table (Table 7) and the full raw pairwise-tests JSON (every model-pair comparison fork ∈ 1, 3, 5, 10onexec@k, execution_precision@k, and nDCG@k); 6. the runtime pip freeze; 7.a SHA-256 manifest covering all of the above; 8.the evaluation harness (eval/: subprocess executor, scoring engine, BM25 baseline, embedder scripts, pairwise tests), including eval/reproduce_leaderboard.sh, a single command that recomputes the entire leader- board offline from the frozen matrices and diffs it against the released results, and the construction pipeline (generation/: 954- entry registry, LLM generation scripts for both real-time and OpenAI Batch endpoints, five- gate validator, corpus builder) so that both the scoring step of Section 4 and the construction procedure of Sections 3.2–3.3 are indepen- dently re-runnable; 9.the analysis scripts and outputs for Sec- tion 5.3 and Appendix G (rebuttal/: the pool-density ablation, edit-size, and mutation- stacking analyses). Metrics recompute deterministically from the frozen matrices without provider API calls. E Per-Model Invocation Table Table 6 reproduces the operationally critical invocation settings for every evaluated system. exec@k execution_precision@knDCG@k Modelk=1k=3k=5 k=10k=1k=3k=5 k=10k=1k=3k=5 k=10 Gemini Emb. 20.331 0.823 0.997 1.000 0.331 0.274 0.199 0.100 0.331 0.614 0.687 0.688 Gemini Emb. 0010.329 0.889 0.988 1.000 0.329 0.296 0.198 0.100 0.329 0.656 0.698 0.702 Mistral Embed0.224 0.795 0.937 0.984 0.224 0.265 0.187 0.098 0.224 0.552 0.612 0.627 Qwen3-8B0.214 0.784 0.979 0.997 0.214 0.261 0.196 0.100 0.214 0.539 0.621 0.627 Qwen3-4B BF160.204 0.786 0.983 0.998 0.204 0.262 0.197 0.100 0.204 0.535 0.618 0.623 Qwen3-4B FP160.196 0.791 0.982 0.998 0.196 0.264 0.196 0.100 0.196 0.535 0.616 0.621 Codestral Embed 2505 0.185 0.781 0.988 1.000 0.185 0.260 0.198 0.100 0.185 0.521 0.609 0.613 OpenAI 3-large0.113 0.701 0.939 0.985 0.113 0.234 0.188 0.099 0.113 0.443 0.544 0.558 OpenAI ada-0020.104 0.662 0.904 0.965 0.104 0.221 0.181 0.097 0.104 0.420 0.521 0.540 GTE-large0.104 0.661 0.897 0.959 0.104 0.220 0.179 0.096 0.104 0.415 0.514 0.534 MiniLM-L120.086 0.564 0.832 0.909 0.086 0.188 0.166 0.091 0.086 0.354 0.466 0.491 mE5-large0.085 0.622 0.840 0.932 0.085 0.207 0.168 0.093 0.085 0.385 0.476 0.506 BGE-M3 (dense)0.085 0.571 0.811 0.888 0.085 0.190 0.162 0.089 0.085 0.356 0.457 0.481 OpenAI 3-small0.083 0.624 0.863 0.941 0.083 0.208 0.172 0.094 0.083 0.383 0.483 0.508 BGE-base0.083 0.626 0.862 0.943 0.083 0.209 0.172 0.094 0.083 0.387 0.486 0.512 BGE-large0.078 0.630 0.859 0.941 0.078 0.210 0.172 0.094 0.078 0.385 0.481 0.507 E5-large0.077 0.636 0.858 0.950 0.077 0.212 0.172 0.095 0.077 0.389 0.483 0.512 MPNet-base0.076 0.603 0.851 0.939 0.076 0.201 0.170 0.094 0.076 0.370 0.474 0.502 Multi-QA MPNet0.073 0.535 0.800 0.897 0.073 0.178 0.160 0.090 0.073 0.330 0.441 0.472 GTE-base0.071 0.627 0.887 0.952 0.071 0.209 0.177 0.095 0.071 0.382 0.491 0.512 MiniLM-L60.069 0.571 0.800 0.892 0.069 0.190 0.160 0.089 0.069 0.348 0.444 0.474 E5-base0.069 0.609 0.863 0.935 0.069 0.203 0.172 0.093 0.069 0.371 0.477 0.500 BM25 (lexical)0.058 0.234 0.328 0.422 0.058 0.078 0.066 0.042 0.058 0.159 0.198 0.228 Para-MiniLM0.050 0.383 0.556 0.671 0.050 0.128 0.111 0.067 0.050 0.235 0.308 0.344 Table 7: Full per-model leaderboard. exec@kis the fraction of queries with at least one passing snippet in the topk; execution_precision@kis the fraction of top-ksnippets that pass; nDCG@kscores rank quality against canonical-ID matches. Sorted by exec@1 descending. At k = 1 the three metrics coincide. Provider documentation we used to fix invoca- tion conventions: Google Gemini Embedding (https://ai.google.dev/gemini-api/docs/ embeddings(Google, 2025)); OpenAI Embed- dings(https://platform.openai.com/docs/ guides/embeddings(OpenAI, 2024)); Mistral Embeddings (https://docs.mistral.ai/api/ endpoint/embeddings(Mistral AI, 2025)); Qwen3 Embedding model cards on Hugging Face (Qwen/Qwen3-Embedding-4B,8B); BAAI BGE model cards (BAAI/bge-base,large-en-v1.5, BAAI/bge-m3);E5modelcards (intfloat/e5-base,large-v2, intfloat/multilingual-e5-large);GTE model cards (thenlper/gte-base,large); Sentence-Transformers model cards on Hugging Face for the remaining models. F Full Per-Model Leaderboard Table 7 reportsexec@kfor every system andk ∈ 1, 3, 5, 10 so that every figure-level claim is veri- fiable from the paper alone. G Robustness Analyses: Edit Size and Stacked Mutations Two further analyses test whether the deception measurement of Section 5.5 is an artifact of micro- edits. Both are derived from the released artifacts; the analysis scripts ship with the release. G.1 Deception vs. edit size For each of the 3,745 released mechanical (canon- ical, distractor) pairs (the six locked-prompt mu- tation types; the 10 pre-lock legacy distractors are excluded), we measure the number of characters spanned by the non-equal regions of a character- level diff. Edit sizes range from 1 to 540 characters; the median mutation changes a single character, so we report fixed buckets rather than quantiles. Ag- gregated over the 23 dense configurations (86,135 triples; aggregate deception 44.3%, matching Sec- tion 5.5): Edit sizepairsdeception 1 character1,89443.4% 2–5 characters97746.4% 6–20 characters51347.5% 21+ characters36139.5% Table 8: Deception by edit size over the 3,745 mechani- cal pairs× 23 dense configurations. Deception is flat across edit sizes. The tie-aware Spearman correlation between edit size and per- pair deception is|ρ| ≤ 0.07(relative edit size: ρ = −0.069; absolute:ρ = +0.035) — statis- tically significant atn = 3,745but negligible in magnitude. The dip in the largest bucket is consis- tent with its composition: multi-line edits (8.5% of mechanical pairs, up to 15 lines) are predomi- nantlyremove_edge_case_checkmutations, the least deceptive type overall (39.3%, Figure 5). G.2 Deception vs. number of stacked mutations Distractors whose mutations occupy disjoint line regions of the same canonical can be composed. We take every mechanical distractor whose diff against its canonical is a single contiguous line block, stack 2–4 blocks with pairwise-disjoint line ranges (deterministic per-query subsampling), and execute every composed variant against its query’s full test suite in the same isolated-subprocess run- ner as the release. Compositions that accidentally pass all tests are discarded (13 of 4,646), leaving 4,633 execution-verified multi-mutation variants covering 915 of 939 queries. We embed the composed variants with four of the locally run leaderboard embedders (all-mpnet-base-v2,bge-base-en-v1.5, e5-base-v2,gte-large) under the release’s exact passage-side conventions — verified by re-embedding released corpus rows and requiring cosine≥ 0.999against the frozen matrices (observed≥ 0.999999) — while query and canonical similarities come from the frozen matrices themselves.The remaining systems are not re-embedded; the analysis was scoped to these four local embedders. Queries admitting four disjoint edits are not a random subset (their single-mutation deception runs 1.5–1.6 points above the corpus-wide rate pooled over these four models), so we compare on the 371 queries with validated variants at every mutation count, aggregating per query first and then averaging over the four models (Table 9): Mutationsdeception∆ vs. 1 (95% CI) 149.0%— 248.2% −0.7 [−2.1, +0.6] 347.3% −1.6 [−3.5, +0.3] 447.6% −1.4 [−3.7, +0.9] Table 9: Deception vs. number of stacked mutations on the matched 371-query subset (per-query means aver- aged over four local embedders);∆in percentage points with query-level bootstrap 95% CIs.∆is computed be- fore rounding. Deception stays within 1.7 points of the single- mutation baseline, and no drop is statistically distin- guishable from zero (query-level bootstrap, 5,000 replicates;∆in percentage points); even the most pessimistic interval edge leaves deception near 45%. The 49.0% baseline here differs from Sec- tion 5.5’s 44.3% for two reasons, both composi- tional: the matched 371-query subset is biased up- ward (49.0% vs. 47.4% for the same four models unrestricted), and the four local embedders differ from the 23-model pool (47.4% vs. 44.3%). Stack- ing up to four verified bugs into one function does not restore detectability for these embedders.