Paper deep dive
Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark
Makar Ulesov, Vladislav Smirnov, Omar Ibrahim, Arsenii Bobovnikov
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/26/2026, 3:30:30 AM
Summary
This paper introduces a clean-control calibration benchmark to evaluate Large Language Models (LLMs) as auditors for quantitative finance backtests. The benchmark consists of 96 paired items (48 flawed, 48 clean controls) designed to isolate flaw detection from false positives. The study finds that while models like DeepSeek V4 Flash achieve 100% recall for flaws, open prompts cause high false positive rates on clean controls. A 'clean-aware' prompt significantly reduces false positives without sacrificing recall, demonstrating that calibration is as critical as flaw detection. The research highlights that reporting only recall is misleading and that prompt engineering and model choice drastically impact audit utility.
Entities (15)
Relation Signals (10)
DeepSeek-v4-Flash → achieves → 100.0% recall
confidence 98% · the primary DeepSeek auditor reaches 100.0% closed and clean-aware code recall
Clean-aware prompt → reduces → False Positive Rate
confidence 97% · A clean-aware warning drops DeepSeek code false positives from 20.8%... to 0.0%... while preserving 100.0% recall.
Open prompt → causes → High False Positives
confidence 96% · open prompts over-flag 93.8% of clean code controls
Arsenii Bobovnikov → affiliatedwith → University of Pittsburgh
confidence 95% · Arsenii Bobovnikov 2 ... 2 School of Computing and Information, University of Pittsburgh
Makar Ulesov → affiliatedwith → MBZUAI
confidence 95% · Makar Ulesov 1 ... 1 Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
Clean-Control Calibration Benchmark → evaluates → LLMs
confidence 95% · We build a 96-item paired benchmark... Over 1440 cached audits from four text endpoints... the primary DeepSeek auditor reaches 100.0% closed and clean-aware code recall
DeepSeek-v4-Flash → hasfalsepositiverate → 0.0%
confidence 95% · A clean-aware warning drops DeepSeek code false positives from 20.8%... to 0.0%
GPT-4o-mini → hasfalsepositiverate →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Backtest auditing is a calibration problem: high flaw recall is not useful when the model falsely flags matched clean strategies. We build a 96-item paired benchmark in which every flawed backtest has a clean control that holds strategy, dates, code style, labels, and reporting scaffold fixed while changing one methodology detail. A deterministic scorer separates flaw recall, clean-control false positives, evidence localization, and fix relevance. Over 1440 cached audits from four text endpoints, the primary DeepSeek auditor reaches 100.0\% closed and clean-aware code recall, but open prompts over-flag 93.8\% of clean code controls, and clean-aware all-three specificity is 87.5\% even where recall saturates. A clean-aware warning drops DeepSeek code false positives from 20.8\% (95\% CI 11.7--34.3) to 0.0\% (0.0--7.4) at unchanged recall, while the budget anchor still flags 38/48 clean controls under the same prompt. Reporting recall alone would rank three of these four models identically; reporting the clean-control rate separates them by 79 points.
Tags
Links
- Source: https://arxiv.org/abs/2609.28090v1
- Canonical: https://arxiv.org/abs/2609.28090v1
Trouble viewing inline? Open PDF directly →
Full Text
17,229 characters extracted from source content.
Expand or collapse full text
Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark Makar Ulesov 1 Vladislav Smirnov 1 Omar Ibrahim 1 Arsenii Bobovnikov 2 1 Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) 2 School of Computing and Information, University of Pittsburgh Abstract Backtest auditing is a calibration problem: high flaw recall is not useful when the model falsely flags matched clean strategies. We build a 96- item paired benchmark in which every flawed backtest has a clean control that holds strat- egy, dates, code style, labels, and reporting scaffold fixed while changing one methodol- ogy detail. A deterministic scorer separates flaw recall, clean-control false positives, ev- idence localization, and fix relevance. Over 1440 cached audits from four text endpoints, the primary DeepSeek auditor reaches 100.0% closed and clean-aware code recall, but open prompts over-flag 93.8% of clean code con- trols, and clean-aware all-three specificity is 87.5% even where recall saturates. A clean- aware warning drops DeepSeek code false pos- itives from 20.8% (95% CI 11.7–34.3) to 0.0% (0.0–7.4) at unchanged recall, while the budget anchor still flags 38/48 clean controls under the same prompt. Reporting recall alone would rank three of these four models identically; re- porting the clean-control rate separates them by 79 points. 1 Calibration as Much as Flaw-Finding Backtest review is a calibration problem as much as a flaw-finding problem. A useful LLM auditor must detect look-ahead, survivorship, leakage, p- hacking, and execution errors while also letting a clean strategy pass; closed-set recall alone is mis- leading because false positives add review threads and can obscure the one backtest that actually needs repair. We therefore make calibration the object of study with a controlled paired benchmark. Each flawed item has a matched clean control that keeps strat- egy, dates, code style, labels, and reporting scaf- fold fixed while changing one methodology detail, which separates flaw recognition from a high sus- picion prior (Figure 1). The benchmark does not certify trading systems; it isolates one review skill. Prior quantitative-finance work owns backtest overfitting, data snooping, and survivorship bias (White, 2000; Harvey et al., 2016; Bailey et al., 2017; Lopez de Prado, 2018). Prior LLM-finance and LLM-review benchmarks own backtest gener- ation, look-ahead mitigation, issue-level critique, and LLM-assisted review (Wang et al., 2026; Li et al., 2026; Chaudhuri et al., 2026; Ho et al., 2026; Zurawicki et al., 2026). This work differs by jointly evaluating matched clean controls, prose and code surfaces, canonical flaw labels, and deterministic evidence and fix scoring. 2 Benchmark and Validator Items.The 96-item set has 48 clean/flawed pairs (48 flawed, 48 clean controls), 6 flawed examples per flaw class, three difficulty strata, and two sur- faces per item (prose and code) over|K| = 8canon- ical flaws: look-ahead, survivorship, data snoop- ing, full-sample and overlapping-label leakage, un- realistic fills/costs, regime overfit, and p-hacked stop/rule variants, plus a clean label. Clean controls are not merely the absence of an edit. They con- tain the protective constructs a careful quantitative researcher would write—purged and embargoed splits, explicit one-way costs, half-spread slippage, non-overlapping labels—so a model that pattern- matches on the presence of those constructs will misfire in both directions. Conditions. The prompt grid crosses open, closed, clean-aware with prose, code. Open omits the taxonomy, closed includes it, and clean- aware adds that clean controls are common. The primary auditor is DeepSeek V4 Flash at tempera- ture 0; the code-slice ladder addsgpt-4o-mini (budget anchor),gemini-2.5-flash-lite, and gpt-4.1-mini.All 1440 item-condition calls parse after a token-cap rerun, at a parse rate of 100.0%. arXiv:2609.28090v1 [cs.CL] 23 Sep 2026 Matched clean control L09 labels = forward_labels(non_overlap=True) L10 split = PurgedSplit(embargo=21d) L12 signals = build_signal(...).shift(1) L13 fills = next_bar(open, costs, slippage) legitimate split, lag, and cost lines Same strategy with planted edit # planted edit L18 future_ret = pct_change().shift(-1) L19 signals = (future_ret > 0) L20 fills = execute_at_timestamp(..., close) target span: future-return signal Open-prompt DeepSeek code responses FALSE POSITIVEFalse positive on clean control: predicts data leakage from forward labels in split definition; q=0.85; cites L10/L09 purged-label code. CORRECTCorrect detection on planted edit: predicts lookahead bias; q=1.00; cites L18-L19 future return. Figure 1: Paired clean-control design. Left: legitimate split, lag, and cost lines in a clean control. Right: the same strategy with a planted look-ahead edit. The same open code audit correctly flags the flawed edit but falsely accuses the matched clean control; without a matched control, the false accusation would look like a plausible audit insight. Table 1: Primary DeepSeek prompt-by-surface con- fusion matrix. Flawed cells are true positives/false negatives; clean cells are false positives/true negatives. Threads are expected unnecessary reviews per 100 clean submissions. ConditionTP/FNFP/TNRFPRThreads open prose39/918/3081.237.537.5 open code35/1345/372.993.893.8 closed prose48/01/47100.02.12.1 closed code48/010/38100.020.820.8 clean-warn prose48/00/48100.00.00.0 clean-warn code48/00/48100.00.00.0 Scoring. The deterministic validator scores five quantities separately: clean-control false-positive rateF c , with review burdenC c = 100F c unneces- sary threads per 100 clean submissions; evidence credit, where the cited span must hit the planted target lines; fix credit, where a class-specific key- word earns the point and generic advice earns noth- ing; all-three specificity, meaning simultaneously class-, evidence-, and fix-correct; and a utility U c (λ) = ̄ R c − λF c withλ ∈ 0.25, 1, 2. Bi- nomial intervals are Wilson score intervals at 95%. The dataset seed is 20260612. 3 Results: Recall Is Easy, Calibration Is Not Table 1 is the primary prompt-by-surface confusion matrix. Open code audits detect 35/48 flawed items but create 45 unnecessary review threads per 48 clean controls, i.e. 93.8 per 100 clean submissions. Closed and clean-aware prompts both reach 100.0% code recall, but only the clean-aware prompt also removes DeepSeek clean false positives on this paired set, dropping code FPR from 20.8% to 0.0% and prose FPR from 2.1% to 0.0% while preserving 100.0% recall. The clean-control drop is larger than sampling noise. On code the closed prompt flags 10/48 clean controls, a rate of 20.8% with a 95% Wilson inter- val of 11.7–34.3%; the clean-aware prompt flags 0/48, an interval of 0.0–7.4%. Those intervals do not overlap. The corresponding recall does not move: both prompts sit at 48/48. Perfect recall is not complete audit quality. Clean-aware code all-three specificity is only 87.5% (95% CI 75.3–94.1), decomposing into 100.0% evidence localization and 87.5% fix rel- evance. The model points at the right lines in every flagged item and still proposes an off-class remedy in six of them. Prose is the easier surface here, at 100.0% all-three specificity under the same prompt. A single audit score would hide the fact that the residual failure sits entirely in the fix column. Calibration varies by model.Table 2 separates the four endpoints on a column that recall cannot see. DeepSeek and Gemini have 0/48 clean false positives, GPT-4.1 mini has 8/48, and the budget anchor GPT-4o mini has 38/48, despite all four keeping high recall. UnderU c (λ), DeepSeek clean- aware code hasU 0.25 = U 1 = U 2 = 1.00, while GPT-4o mini clean-aware code falls fromU 0.25 = Table 2: Clean-aware code slice, four endpoints, 48 flawed and 48 clean items each. FPR intervals are 95% Wilson. A3 is all-three specificity; U λ is ̄ R− λF . ModelRFPR (95% CI)A3 U 1 U 2 DeepSeek V4 Flash100.00.0 (0.0–7.4) 87.5 1.001.00 Gemini 2.5 Flash Lite87.50.0 (0.0–7.4) 85.4 0.880.88 GPT-4.1 mini 100.016.7 (8.7–29.6) 97.9 0.830.67 GPT-4o mini 100.0 79.2 (65.7–88.3) 75.0 0.21−0.58 0.80toU 1 = 0.21toU 2 = −0.58despite perfect recall. Three of the four models tie at 100.0% recall and span a 79-point range on the clean-control rate. The rule baseline is a sanity anchor only, at 25.0% code recall and 100.0% code clean FPR, since clean controls contain protective phrases such as purges and realistic slippage.False positives are not parser noise. Most name an incorrect flaw class and point at a legitimate safeguard: one flagsunrealistic_fills_costs while citing a line that sets a 10 bp one-way cost with half-spread slippage, and another flags overlapping_label_leakagewhile citing non- overlapping labels combined with a purged walk- forward split at a 21-day embargo. Self-reported confidence does not rescue the ranking either. Among the 281 audits in the top confidence bin (above 0.85, mean 0.939), 35.0% of the clean- control judgments are still false positives. Where open prompts fail. The open-prompt deficit is concentrated rather than spread. On code, the primary auditor recovers 6/6 look-ahead, sur- vivorship, and data-snooping items but only 1/6 full-sample preprocessing and 2/6 regime overfit; on prose the same two classes fall to 0/6 and 4/6. Difficulty strata do not explain it, since open code recall is 12/16 obvious, 11/16 moderate, and 12/16 subtle. Without the taxonomy the model finds the flaws that have familiar names and misses the ones that live inside a preprocessing step. Appendix A gives the full per-class breakdown. What the rates cost a reviewer. The false- positive column converts directly into review load, which is why we reportC c = 100F c alongsideF c . A team that runs the open code prompt over a sub- mission queue of 100 clean strategies opens 93.8 unnecessary threads. The same team running the clean-aware prompt with the primary auditor opens none, and running it with the budget anchor opens 79.2. Recall is identical for the second and third of Table 3: Reviewer persona on the clean-aware code condition, primary model, 48 flawed and 48 clean items each. Only the assigned role changes. PersonaRFPR (95% CI)A3Conf. Auditor100.00.0 (0.0–7.4)87.50.953 Quant reviewer100.02.1 (0.4–10.9)93.80.952 Skeptical reviewer 100.010.4 (4.5–22.2)93.80.928 Risk manager95.812.5 (5.9–24.7)91.70.911 these, so the entire difference in operating cost is invisible to a recall-only report. The asymmetry also runs the other way. Under the open prompt the primary auditor misses 13 of 48 flawed code items while raising 45 unsupported flags, so the flagged pile contains more clean strate- gies than rigged ones. An auditor in that regime is worse than uninformative for triage, because the re- viewer who works the queue in order spends most of the budget confirming that legitimate code is legitimate. Persona moves calibration, not recall. Hold- ing the model, the prompt mode, and the surface fixed, we vary only the reviewer role the system prompt assigns. Table 3 reports the four roles on the clean-aware code condition. Recall is flat at 100.0% for three of them and 95.8% for the fourth, while the clean-control rate moves from 0/48 to 6/48. The two roles that carry an adversarial conno- tation,risk_managerandskeptical_reviewer, account for 11 of the 12 false positives across the four roles, andrisk_manageris also the only role that misses a flawed item. Mean self-reported confidence tracks the false- positive rate in the wrong direction: the roles that flag more clean controls report lower confidence (0.911 and 0.928 against 0.953 and 0.952), so a con- fidence filter would discard the flags of the better- calibrated roles first. Surface matters, and not uniformly.Prose and code are not interchangeable inputs. Under the open prompt, prose recall exceeds code recall (81.2% against 72.9%) while prose false positives are less than half the code rate (37.5% against 93.8%). Under the closed prompt the recall gap closes to zero and the false-positive gap persists (2.1% against 20.8%). A natural-language descrip- tion of a methodology therefore elicits a better- calibrated audit than the code that implements it, at least at these sample sizes, and the gap survives being told what to look for. 0.00.20.40.60.81.0 Clean-control false-positive rate 0.85 0.90 0.95 1.00 Flawed-item macro recall DeepSeek V4 FlashDeepSeek V4 FlashGPT-4o mini GPT-4o mini Gemini Flash-Lite Gemini Flash-Lite GPT-4.1 miniGPT-4.1 mini marker closed taxonomy clean-aware Figure 2: Clean-control false-positive rate by model and prompt mode on the code surface. The vertical spread at the clean-aware prompt is the quantity that recall-only reporting discards: every model in that column is at or near ceiling recall. 4 Discussion Under deterministic paired edits, prompt and model choice separate high flaw recall from clean-control calibration; this does not show that an LLM can validate a trading system in production. The take- away for benchmark design is narrow and prac- tical: backtest-auditor benchmarks should report flaw recall, clean false positives, evidence, fix, and confidence separately, because a single aggregate would have rated all four endpoints in Table 2 as near-solved. Limitations The sample is small, at 6 flawed examples per class, so per-class rates move in steps of 16.7 points and the Wilson intervals in Table 2 are correspond- ingly wide. Many clean-aware results saturate at 100% recall, which we treat as a finding rather than a solved task, since GPT-4o mini over-flags clean code controls despite perfect recall. The strict lexical scorer may under-credit unusual fixes; alias-only matches account for 56 of the correct responses, or 4.6%, so the scorer’s synonym table is a live sensitivity. Items are synthetic paired edits over a fixed set of strategy skeletons, not externally collected backtest notebooks, and every audit is a single temperature-0 call rather than a multi-turn review. A human baseline packet of 16 items was prepared but not completed, so no human reference rate is reported. A next validator should add se- mantic entailment for the fix column and externally sourced notebooks for the item pool. Ethics Statement All backtests in the benchmark are synthetic, and no market data, broker record, or proprietary strat- egy is redistributed. The benchmark measures whether a model recognizes planted methodology flaws under matched controls. It does not certify any trading system, and a high score here is not evidence that a model can validate a real strategy or that any strategy is sound. We report clean-control false positives alongside recall because an auditing tool that over-flags imposes review cost and can crowd out the one submission that genuinely needs repair. The paper makes no claim about profitabil- ity and offers no investment advice. References H. White. A reality check for data snooping. Economet- rica, 68(5):1097–1126, 2000. C. R. Harvey, Y. Liu, and H. Zhu. . . . and the cross- section of expected returns. Review of Financial Studies, 29(1):5–68, 2016. D. H. Bailey et al. The probability of backtest overfit- ting. Journal of Computational Finance, 20(4):39– 69, 2017. M. Lopez de Prado. Advances in Financial Machine Learning. Wiley, 2018. Z. Wang et al. BacktestBench. arXiv:2605.17937, 2026. W. W. Li et al. Mitigating look-ahead bias in financial backtesting with LLMs. arXiv:2605.24564, 2026. Y. Chaudhuri et al. E3 issue-level backtesting for auto- mated research critique. arXiv:2605.27072, 2026. S.-T. Ho et al. SoundnessBench. arXiv:2605.30329, 2026. K. Zurawicki et al. PRAIB: LLM-assisted peer review benchmark. arXiv:2605.29815, 2026. A Per-Class Recall Without the Taxonomy Table 4 gives the primary auditor’s open-prompt re- call per flaw class on both surfaces, with the closed- prompt column for reference. The closed prompt recovers every class on both surfaces; the open prompt does not, and the shortfall concentrates in full-sample preprocessing and regime overfit. Table 4: Primary-auditor recall by flaw class, 6 flawed items per class. Open prompts omit the taxonomy; the closed column reaches 6/6 on every class and both sur- faces. Flaw class open prose open code closed Look-ahead6/66/66/6 Survivorship6/66/66/6 Data snooping6/66/66/6 Overlapping-label leakage6/65/66/6 Unrealistic fills/costs5/65/66/6 P-hacked stop/rule6/64/66/6 Regime overfit4/62/66/6 Full-sample preprocessing0/61/66/6 Total39/48 35/48 48/48 B Recall Against False Positives by Surface Figure 3 plots the primary auditor’s recall against its clean-control false-positive rate for each prompt mode and surface. The three prompt modes trace a path that moves up and to the left rather than along a single frontier: the clean-aware prompt gains recall over the open prompt and loses false positives at the same time, so the two are not being traded against each other. 0.0 0.2 0.4 0.6 0.8 1.0 open prose open code closed prose closed code clean-aware prose clean-aware code Rate macro recallclean FPRall-three Figure 3: Primary auditor recall against clean-control false positives, by surface and prompt mode.