Paper deep dive
FIDES: A Concordance Protocol for LLM-Generated Trading Strategies
Arther Tian, Alex Ding, Simon Wu, Aaron Chan
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:An LLM asked for a trading strategy returns three artifacts at once: a natural-language rationale, an executable implementation, and once run, a track record. Whether these are the same object is rarely checked. We present FIDES, a measurement protocol that treats them as three views to be reconciled rather than one deliverable to be graded. Through dual delivery, a single model call returns both a natural-language strategy with an explicit claimed edge and a self-contained strategy(df) function. FIDES executes the code in a sandbox against a lag-one out-of-sample backtest and scores three concordance gaps: say to do, do to real, and say to result. On 8 liquid US ETFs across four models plus a two-stage elicitation arm, 40 strategies, 2023 to 2024 out-of-sample, three findings stand out. First, concordance does not predict profit: only 2 of 40 strategies beat buy-and-hold, and a plain sma(50,200) rule outperforms every model's mean Sharpe. Second, self-assessment is badly calibrated: 32 of 40 strategies claim to beat buy-and-hold and exactly one does. Third, swapping the language-code judge for a second model flips say to do on more than half of items. Injecting this http URL(-1) drops do to real by 0.33 on average, while our runtime future-information probe fired on neither clean nor injected code. We frame FIDES as a protocol for measurement fidelity, not a claim about market performance.
Tags
Links
- Source: https://arxiv.org/abs/2608.23308v1
- Canonical: https://arxiv.org/abs/2608.23308v1
Trouble viewing inline? Open PDF directly →
Full Text
40,759 characters extracted from source content.
Expand or collapse full text
FIDES: A Concordance Protocol for LLM-Generated Trading Strategies Arther Tian Affiliation: DGrid AI Alex Ding Affiliation: DGrid AI Simon Wu Affiliation: DGrid AI Aaron Chan Affiliation: DGrid AI [0.5em] Corresponding author: alex.ding@dgrid.ai Abstract An LLM asked for a trading strategy returns three artifacts at once: a natural-language rationale, an executable implementation, and—once run—a track record. Whether these are the same object is rarely checked. We present FIDES, a measurement protocol that treats them as three views to be reconciled rather than one deliverable to be graded. Through dual delivery—a single model call that returns both a natural-language strategy (with an explicit claimed edge) and a self-contained strategy(df) function—FIDES executes the code in a sandbox against a lag-one out-of-sample backtest the model cannot game, and scores three concordance gaps in [0,1][0,1]: say→ (does the code implement the stated rules?), do→ (does it execute as written, without look-ahead?), and say→ (does the claimed edge survive the numbers?). On 8 liquid US ETFs across four models plus a two-stage elicitation arm (40 strategies; 2023–24 out-of-sample), three findings stand out, and none is flattering. First, concordance does not predict profit: the most speech–code consistent model (say→ =1.00=1.00) still loses to buy-and-hold, only 22 of 4040 strategies beat it, and a plain sma(50,200) rule outperforms every model’s mean Sharpe. Second, self-assessment is badly calibrated: 3232 of 4040 strategies claim to beat buy-and-hold and exactly one does—a 3.1%3.1\% hit rate. Third, the language–code judge is not an oracle: swapping it for a second model flips say→ on more than half of items (exact agreement 16/4016/40, mean |Δ|=0.15| |=0.15). A look-ahead dose–response validates the execution gap—injecting Close.shift(-1) drops do→ by 0.330.33 on average—while we disclose, as a negative result, that our runtime future-information probe fired on neither clean nor injected code. We frame FIDES as a protocol for measurement fidelity, not a claim about market performance, and delimit external validity—including a model roster constrained to what our inference catalog served—as future work. 1 Introduction Large language models are increasingly asked not merely to discuss markets but to produce trading strategies outright—as natural-language rationales, as runnable code, and, through agent frameworks, as systems that place orders [11, 25, 29]. A single such request returns three artifacts at once: the strategy the model states in prose, the strategy its code actually implements, and the track record that strategy realizes when run. It is natural to treat these as one deliverable and grade it—did the code run, was the return positive—but that skips the prior question of whether the three artifacts even describe the same strategy. A fluent rationale can be paired with code that quietly does something else; code that runs cleanly can be silently peeking at future bars; and a confident claim to beat the market can sit atop a track record that does not. In finance these gaps are not cosmetic: acting on a stated rationale when the code diverges, or trusting a backtest that looked ahead, loses real money. No existing evaluation reconciles the three views jointly. Code-generation benchmarks measure functional correctness against hidden unit tests [7, 1], but there is no unit test for did the code implement the English, and a generated trading strategy has no reference implementation to diff against. Financial LLM benchmarks grade domain knowledge and question answering [24, 27, 26, 28], not the internal consistency of a produced strategy. Return-prediction and trading-agent studies pursue profit directly [11, 25, 29] and thereby inherit the well-documented hazards of that target—backtest overfitting, data snooping, and look-ahead bias [2, 3, 23, 13]—hazards acute enough that beating a passive benchmark out of sample is the exception, not the rule [8, 4], and severe enough that look-ahead leakage is now studied specifically for LLMs [5]. Closest in spirit, work on chain-of-thought faithfulness shows that a model’s stated reasoning need not reflect the computation that produced its answer [19, 9]; but it operates on multiple-choice reasoning, where the “answer” is a label, rather than on an artifact whose correctness is settled by a non-gameable outcome. We introduce FIDES, a measurement protocol that reconciles the three artifacts rather than grading one. FIDES uses dual delivery: a single model call returns both a natural-language strategy (with an explicit claimed edge) and a self-contained strategy(df) function, so the language and the code come from one act of reasoning and can be compared without a translation step. It then executes the code in a sandbox and scores three concordance gaps, each in [0,1][0,1] with 11 fully concordant: say→ asks whether the code implements the stated rules, scored by a fixed, non-candidate language–code judge; do→ asks whether the code executes as written, without static look-ahead; and say→ asks whether the claimed edge survives the out-of-sample numbers. The load-bearing design choice is the anchor: outcomes come from a lag-one out-of-sample backtest the model cannot game—positions act on the prior bar, so the code cannot trade on information it should not have—and do→ and say→ are thereby grounded in what actually happened rather than in another model’s opinion (Figure 1). Profit is a side metric, and we deliberately do not repair or search over strategies to raise it: doing so would optimize the very quantity whose relationship to the stated logic we are trying to measure. Across 8 liquid US ETFs and four models (40 strategies; 2023–24 out-of-sample), the protocol turns up three results, none of them flattering to the models. Concordance does not predict profit: the most speech–code consistent model scores a perfect say→ of 1.001.00 yet loses to buy-and-hold, only 22 of 4040 strategies beat it, and a plain sma(50,200) rule outperforms every model’s mean Sharpe. Self-assessment is badly calibrated: 3232 of 4040 strategies claim to beat buy-and-hold and exactly one does, a 3.1%3.1\% hit rate. And the language–code judge is not an oracle—swapping it for a second model flips say→ on more than half of items (exact agreement 16/4016/40, mean |Δ|=0.15| |=0.15), so a single judge’s scores should be read as one measurement, not ground truth. We validate the execution gap with a look-ahead dose–response (injecting Close.shift(-1) drops do→ by 0.330.33 on average) and, in the same spirit, disclose a negative result: our runtime future-information probe fired on neither clean nor injected code—a limitation of that probe rather than evidence of clean execution. This cheap-first, measure-what-you-can, audit-the-judge stance is shared with a broader line of the authors’ work on trustworthy evaluation [14, 15, 17, 18]; here the object—an LLM’s own trading strategy—and the mechanism—three gaps anchored by a non-gameable backtest—are new. We present FIDES as a protocol for measurement fidelity, not as a claim about market performance, and we are explicit that the model roster is constrained by what our inference catalog actually served (Section 4). What the model saysnatural-language strategy++ claimed edgeWhat it doesgenerated strategy(df)→ signals ∈−1,0,1∈\-1,0,1\What really happenslag-1 out-of-sample backtestnon-gameable ground truthsay→ vs. code (fixed judge)say→ edge vs. OOS numbersdo→ as written, no look-ahead Figure 1: The three artifacts of an LLM trading strategy and the three gaps FIDES measures. A single model call (dual delivery) yields a natural-language strategy and its code; executing the code on a lag-one out-of-sample backtest yields a realized track record. Only the backtest is non-gameable ground truth. Each gap is scored in [0,1][0,1] (1=1= fully concordant); profit is treated as a side metric, not the objective. FIDES makes three contributions: • A concordance protocol for generated strategies. Dual delivery plus three interval-valued gaps—say→ , do→ , say→ —that reconcile an LLM’s stated logic, its code, and its realized track record, anchored by a lag-one out-of-sample backtest the model cannot game (Section 3). • An empirical study with uncomfortable findings. On 8 ETFs across four models we show that concordance does not predict profit, that models’ self-reported edge is badly calibrated, and that the language–code judge is not robust to being swapped—each quantified against passive, rule-based, and cost-adjusted baselines (Section 5). • Honest diagnostics and delimited scope. A validated look-ahead dose–response, a disclosed failed runtime probe, a zero-LLM rule baseline, and a two-stage elicitation control, framed throughout as measurement fidelity with external validity—including model coverage—delimited as future work (Sections 5 and 6). 2 Related Work LLMs for finance and trading. Domain models such as BloombergGPT [24] and open efforts such as FinGPT [28], together with benchmark suites [27, 26], establish that LLMs can be adapted to financial text; return-prediction and agentic-trading studies go further and ask models to forecast or to trade [11, 25, 29]. These efforts grade domain knowledge or chase realized profit. Neither asks our question—whether a single generated strategy’s prose, code, and outcome agree—which is prior to, and orthogonal to, whether the strategy makes money. Evaluating generated code. Functional-correctness benchmarks score generated programs against held-out unit tests [7, 1], treating execution as ground truth. We adopt the same execute-to-verify philosophy but face a harder object: a trading strategy has no reference implementation, so there is no test that says whether the code matches the stated rules. FIDES therefore pairs execution (for do→ and say→ ) with an explicit language–code comparison (for say→ ), and anchors the former in a non-gameable market backtest rather than a unit test. Faithfulness of stated reasoning. A model’s stated rationale need not reflect the computation that produced its output: chain-of-thought explanations can be plausible yet unfaithful [19, 9], even as chain-of-thought and self-consistency improve accuracy on reasoning [22, 21]. This literature motivates say→ , but it operates where the “answer” is a label; FIDES moves the question to an artifact—code—whose behaviour is settled by a backtest, so the say–do gap can be checked against something the model cannot argue with. LLM-as-judge and its biases. Strong models are now standard evaluators [30, 10], but their verdicts carry documented biases—order effects, verbosity, and self-preference—and shift with superficial factors [20, 6]. Because say→ is the one gap we score with an LLM, we treat the judge as a measurement to be audited: we fix a non-candidate judge and re-score every item with a second model (Section 5), reporting the disagreement as a first-class result rather than assuming the judge is an oracle. Backtest overfitting, data snooping, and look-ahead bias. A large finance literature shows that impressive backtests routinely fail out of sample—through overfitting, selection, and multiple testing [2, 3, 4], and through data snooping over trading rules [23, 13]—so that beating a passive benchmark is rare [8], and Sharpe ratios must be read with these hazards in mind [12]. Look-ahead leakage in particular is now studied for LLMs [5]. We take these as design constraints: our outcome anchor is a strict lag-one out-of-sample backtest, profit is never optimized inside the loop, and do→ includes an explicit look-ahead check whose sensitivity we validate by injection. Trustworthy, cost-aware evaluation. FIDES continues a line of the authors’ work on cheap-first, self-auditing evaluation [14, 16, 15, 17, 18]. The shared stance—measure what cheap signals can settle, escalate or distrust only where they cannot, and quantify the fallible component rather than hide it—carries over; the object (an LLM’s own trading strategy) and the mechanism (three gaps anchored by a non-gameable backtest) are new. 3 Protocol Design 3.1 Problem setup and notation A task fixes a ticker and two date windows: an in-sample window (shown to the model as context only—no in-sample fitting is performed) and an out-of-sample (OOS) window on which the strategy is scored. A single model call on a task yields three things: a natural-language strategy s, a self-contained function c=strategy(df)c= strategy(df), and a one-sentence claimed edge e. Executing c produces a signal series σ∈−1,0,1Tσ∈\-1,0,1\^T (short, flat, long), which a fixed backtester turns into an OOS Sharpe S and return R, alongside buy-and-hold references Sbh,RbhS_bh,R_bh. FIDES reports three gaps, each in [0,1][0,1] with 11 concordant: gsay→do(s,c)g_ say→ do(s,c), gdo→real(c,σ)g_ do→ real(c,σ), and gsay→result(e,S,R)g_ say→ result(e,S,R). Profit (S,RS,R) is retained as a side metric; the gaps, not the profit, are the object of study. 3.2 Dual delivery The central elicitation choice is to obtain the prose and the code from one completion. A single prompt requests a rule-based daily strategy (inputs, entry/exit, risk controls, and a line Claimed edge: ...) and a function strategy(df) returning a signal series, under the explicit instruction to use only information available at the close of day t to trade t+1t+1. Because both artifacts come from one act of reasoning, the say–do comparison is not confounded by a separate translation step. As a control we also run a two-stage variant that elicits the prose and then, in a second call, asks for code implementing that prose; we treat it as an elicitation ablation, not as a distinct model (Section 5). 3.3 The three gaps say→ : language vs. code. A fixed judge model J0J_0—deliberately not one of the models under test—reads the prose and the code and returns four binary checks: whether the long/short/flat direction rules match (d), whether the indicators and lookbacks named in the text appear in the code (i), whether exits and risk rules match or are jointly absent (x), and whether the code adds no material extra rule (m). The gap averages them, gsay→do=14(d+i+x+m),d,i,x,m∈0,1,g_ say→ do\;=\; 14\,(d+i+x+m), d,i,x,m∈\0,1\, (1) scored at temperature 00. As a zero-LLM ceiling we also compute a rule variant gsay→doruleg^rule_ say→ do: the fraction of lookback numbers named in the prose that also appear literally in the code. do→ : code vs. execution. Executed against real OHLCV in a sandbox, the code earns three binary credits—it runs to a valid signal series (ρ); the signals are not constant (α, i.e. not a degenerate all-long/short/flat series); and the code contains no static look-ahead (ℓ ), where ℓ=0 =0 if any of a small set of leakage patterns (e.g. .shift(-n), .pct_change(-n), center=True, .iloc[...+1]) match the source: gdo→real=13(ρ+α+ℓ),ρ,α,ℓ∈0,1.g_ do→ real\;=\; 13\,(ρ+α+ ), ρ,α, ∈\0,1\. (2) say→ : claim vs. outcome. The claimed edge is parsed for testable assertions and each is checked against the OOS numbers: a claim to beat buy-and-hold requires S>Sbh∧R>RbhS>S_bh R>R_bh; a claim about Sharpe requires S>0S>0; a claim of profit requires R>0R>0. The gap is 11 if all triggered checks pass, 00 if any fails, and 0.50.5 if no testable claim is detected: gsay→result=1all triggered checks pass,0some triggered check fails,0.5no testable claim detected.g_ say→ result= cases1&all triggered checks pass,\\ 0&some triggered check fails,\\ 0.5&no testable claim detected. cases (3) 3.4 The non-gameable backtest Outcomes come from a strict lag-one engine: the position on day t is the signal from day t−1t-1, so a strategy cannot act on the same-day close it is reacting to, post=σt−1,rt=post(CtCt−1−1)−b104|post−post−1|,pos_t= _t-1, r_t=pos_t\! ( C_tC_t-1-1 )- b10^4\, _t-pos_t-1 , (4) with annualized Sharpe S=252r¯/std(r)S= 252\, r/std(r). The main run uses zero cost (b=0b=0); a robustness ablation uses b=10b=10 bp one-way on turnover. Buy-and-hold sets post≡1pos_t≡ 1 on the same OOS slice; a long-only sma(50,200) crossover on the same engine is a non-LLM rule baseline. Strategies are never repaired or searched to raise S, which would break the identification the protocol provides. 3.5 Sandbox Code runs in a child process with a wall-clock timeout (55 s) and a whitelist of imports (pandas, numpy, math, statistics); returned signals are coerced, clipped to −1,0,1\-1,0,1\, and reindexed to the bar index. A failed run is recorded as a row (with ρ=0ρ=0), never a crash. Figure 2 shows the pipeline. Task ticker ++ IS (context) ++ OOS windowsDual delivery (one model call)NL strategy ++ claimed edge ++ strategy(df)Sandbox execution subprocess, 55 s, pandas, numpy, mathLag-1 OOS backtest post=σt−1pos_t= _t-1 → Sharpe, return vs. buy-and-holdThree concordance gapssay→ (fixed judge) ⋅\;·\; do→ (runs / not-constant / no look-ahead) ⋅\;·\; say→ (claim vs. numbers)Report 3 gaps ++ OOS metrics ++ beat-buy-and-hold ++ calibrationjudge swapre-score say→ (second model)controlsSMA, rule,10 bp, alt window Figure 2: End-to-end pipeline. One model call yields prose and code; the code is executed and backtested on a lag-one OOS engine; the three gaps (red hub) reconcile prose, code, and realized outcome. The judge for say→ is audited by re-scoring with a second model; SMA, zero-LLM rule, cost, and alternate-window controls feed the report. 4 Experimental Setup Universe and windows. The universe is eight liquid US ETFs spanning equities, size, international, bonds, gold, and sectors (SPY, Q, IWM, EFA, TLT, GLD, XLF, XLE). The in-sample window 2018–2022 is provided to the model as context only; the primary OOS window is 2023–2024, and an alternate 2020–2021 window is used for regime robustness (Table 7). Data are daily OHLCV on real tickers. Models and coverage. Four models are run under dual delivery—gpt-4o-mini, gpt-4o, gpt-4.1-mini, and qwen-2.5-7b-instruct—giving 3232 strategies (4×84× 8), plus a two-stage arm on gpt-4o-mini (88 more), for 4040 routing units at seed 4242 and temperature 0.20.2. Coverage is deliberately disclosed as a limitation: Claude, Gemini, DeepSeek, Llama-70B, and Qwen-72B all returned model-not-found on our inference catalog, so the roster is OpenAI-heavy with one open model. This is an infrastructure constraint, not a design choice, and we do not present the leaderboard as a broad model comparison. Judge and swap. say→ is scored by gpt-4o-mini; because that model is also on the leaderboard, we disclose the conflict and re-score every item with a second judge, qwen-2.5-7b-instruct (which is in turn circular on the qwen-7b rows). We report both judges and treat neither as ground truth. Controls and metrics. Table 1 lists the controls. We report the three gaps; OOS Sharpe and return against buy-and-hold; the beat-buy-and-hold rate; position density (fraction of non-flat days); and calibration of self-reported edge (claimed vs. actual beat-buy-and-hold). Table 1: Controls and baselines. FIDES measures gaps on the model strategies; these reference points isolate whether concordance tracks anything about performance, and stress the measurements. Control Mechanism Role Buy-and-hold always long on the OOS slice passive benchmark SMA(50,200) long when SMA50 >> SMA200, else flat non-LLM rule Rule say→ lookback-number overlap, zero LLM say→ ceiling Two-stage prose, then code in a second call translation control 10 bp cost one-way cost on |Δposition|| | friction robustness Alt window 2020–2021 OOS regime robustness Judge swap re-score say→ with a second model judge-robustness audit 5 Results 5.1 Leaderboard Table 2 reports per-model means. Two patterns appear before any ablation. The language–code gap is high across the board (say→ 0.810.81–1.001.00) and execution is largely clean (do→ 0.750.75–1.001.00), but say→ collapses—three of five arms score 0.000.00, meaning their stated edge never survives the numbers. Profit tracks say→ , not the other two gaps: every dual model’s mean OOS Sharpe is near or below zero, the best being a marginal gpt-4o at +0.07+0.07, and none is competitive with a passive rule (Section 5.2). The two-stage arm’s 0.0010.001 position density marks it as almost always flat (Section 5.6), not as a better strategy. Table 2: Leaderboard (per-model means, n=8n=8 tasks each). Gaps in [0,1][0,1]; Sharpe is OOS. “Run fails” counts strategies that did not execute to a valid signal series. The two-stage row is an elicitation arm, not a fifth model. Model say→ do→ say→ OOS Sharpe pos. dens. fails gpt-4.1-mini 1.000 0.917 0.000 −0.097-0.097 0.290 1 gpt-4o 0.938 1.000 0.125 +0.070+0.070 0.639 0 gpt-4o-mini 0.844 0.917 0.000 −0.341-0.341 0.307 1 qwen-2.5-7b 0.812 0.917 0.000 +0.030+0.030 0.776 1 gpt-4o-mini (two-stage) 0.812 0.750 0.500 −0.012-0.012 0.001 2 5.2 Concordance does not predict alpha Figure 3 plots each strategy’s say→ against its Sharpe excess over buy-and-hold. If speech–code concordance bought performance, high-say→ points would sit above the zero line; they do not. gpt-4.1-mini, the only model with a perfect say→ of 1.001.00 on every task, lies entirely below zero—its most “faithful” strategies still lose to holding the ETF. Across all 4040 strategies only 22 beat buy-and-hold on both Sharpe and return (Table 4), and the gate makes the non-relationship explicit: strategies with say→ =1.0=1.0 beat buy-and-hold once in 2222 (4.5%4.5\%), those with say→ <1.0<1.0 once in 1818 (5.6%5.6\%)—indistinguishable (Table 4). A plain sma(50,200) rule on the same engine averages 0.5830.583 OOS Sharpe, above every model’s mean, and only 88 of 4040 LLM strategies clear even that rule’s Sharpe. The result survives perturbation: the beat-buy-and-hold count is 22 in the 2020–21 window and 22 after 1010 bp costs (Table 4), and the two 2023–24 winners are not the same tasks as the 2020–21 winners. Both 2023–24 winners are on TLT, and one is degenerate—a two-stage strategy that stayed flat while TLT fell, “beating” a declining benchmark without taking a position (Appendix B). Concordance is thus necessary bookkeeping, not evidence of edge. 0.50.50.750.7511−3-3−2-2−1-10011beats buy-and-holdsay→ (language–code concordance)OOS Sharpe −- buy-and-hold Sharpegpt-4o-minigpt-4oqwen-2.5-7bgpt-4o-mini (2-stage)gpt-4.1-mini Figure 3: Concordance does not predict alpha. Each point is one strategy: x is its say→ score, y its OOS Sharpe minus buy-and-hold Sharpe (above the dashed line beats buy-and-hold on Sharpe). The perfect-say→ gpt-4.1-mini (red) is entirely below the line; only two of forty strategies clear it, both on TLT and one of them a flat degenerate. Table 3: Gate: does higher say→ beat buy-and-hold more often? It does not. say→ bucket n beat buy-and-hold =1.0=1.0 (high) 22 1 <1.0<1.0 (low) 18 1 Table 4: Beating buy-and-hold is rare and fragile (/ 40/\,40). Criterion count beat buy-and-hold, 2023–24 2 beat buy-and-hold, 2020–21 2 beat buy-and-hold, 2023–24 after 10 bp 2 OOS Sharpe >> SMA(50,200) 8 5.3 Self-assessment is badly calibrated Models are confident and wrong. Of the 4040 strategies, 3232 explicitly claim to beat buy-and-hold; exactly 11 does—a 3.1%3.1\% hit rate. This is why say→ is 0.000.00 for most arms (Table 2): the gap records claimed outperformance that the OOS numbers refuse. The lone two-stage say→ of 0.500.50 is not a counterexample—it is mostly the “no testable claim detected” default (Equation 3) firing on terse two-stage rationales, not a detected-and-confirmed edge. A user acting on these models’ stated confidence would have been wrong 3131 times out of 3232. 5.4 The language–code judge is not an oracle say→ is the one gap scored by an LLM, so we audit it by swapping the judge. Under the original judge (gpt-4o-mini) the five arms span 0.810.81–1.001.00; under a Qwen-7B judge they compress to a flat 0.720.72–0.750.75 (Table 5, Figure 4). The two judges agree exactly on 1616 of 4040 items, with mean absolute difference 0.150.15, and the model ranking collapses. Two caveats sharpen the reading: the original judge is itself on the leaderboard, and the swap judge shares a family with the qwen-7b rows it scores. A zero-LLM rule—counting lookback numbers common to prose and code—averages 0.9750.975; the surface tokens almost always match, so this is a weak ceiling, not a substitute for the judge, and its near-saturation is itself evidence that say→ ’s discriminating signal lives where a keyword rule cannot see. The lesson is not that one judge is right, but that a single judge’s score is one measurement: we report both and neither as ground truth. Table 5: say→ under two judges. Rankings collapse; the swap judge compresses to ≈0.75≈ 0.75. Model orig Qwen-7B gpt-4.1-mini 1.000 0.750 gpt-4o 0.938 0.750 gpt-4o-mini 0.844 0.750 two-stage 0.812 0.750 qwen-2.5-7b 0.812 0.719 0.50.50.750.75110.50.50.750.75112216original judge say→ -7B say→ say→ (n=40n=40) Figure 4: Bubble area ∝ count. Points sit on a flat y≈0.75y≈0.75 band, far from the y=xy=x line for high-say→ items. 5.5 Look-ahead dose–response, and an honest negative Two of the three gaps are anchored by execution, so we validate the anchor rather than assume it. We inject a one-line look-ahead— Close.shift(-1) at the top of each strategy—and re-score. Over the 3535 strategies that still run, mean do→ drops by exactly 0.3330.333 (Figure 5, Table 6): the static look-ahead check catches the injected leakage in every case, the intended dose–response. We also report a negative result plainly. An independent runtime probe—shuffle post-cut prices and test whether in-sample-era signals move—fired on neither clean nor injected code (00 of 4040 both times). This is a limitation of that probe, not evidence of clean execution: the shuffle boundary (2022-06-01) and the observation window (through 2021-12-31) do not overlap, so a short-lookback strategy is unaffected by construction. We keep the static check, drop the runtime claim, and flag the fix as future work (Section 7). Table 6: Look-ahead diagnostics. The static check responds to injected leakage; the runtime probe does not fire either way (disclosed limitation). Check value strategies injected (ran OK) 35 mean do→ drop after shift(-1) 0.333 runtime flag on clean code 0 runtime flag on injected code 0 cleaninjected000.50.5110.980.980.650.65mean do→ → under injection Figure 5: do→ before and after injecting Close.shift(-1) (35 running strategies). 5.6 Dual vs. two-stage elicitation Dual delivery is the default; two-stage (prose, then a separate code call) controls for the translation step. On the shared gpt-4o-mini base, dual is modestly higher on both language–code gaps (say→ 0.8440.844 vs. 0.8120.812; do→ 0.9170.917 vs. 0.7500.750). Two-stage looks less bad on Sharpe (−0.012-0.012 vs. −0.341-0.341) only because its strategies are almost always flat—position density 0.0010.001 versus 0.3070.307—so it neither loses nor wins. Staying out of the market is not a strategy; the concordance gaps, not the Sharpe, are the honest comparison. 6 Discussion Implications. The headline is a separation. An LLM can be highly self-consistent—its code faithfully implements its prose—and still produce nothing that beats holding the asset. Concordance and competence are different axes, and conflating them, as a fluent, well-coded, confidently-captioned strategy invites, is exactly the error a procurement or deployment decision cannot afford. Two of our three gaps can be measured cheaply and reliably from execution; the third—self-reported edge—is the one to distrust, and it is both the most confident and the least calibrated signal in the system. Practitioner guidance. Read the gaps separately, not as an average. do→ (does it run, without look-ahead) is a hard gate worth trusting. say→ (does the claim hold) should be treated as a claim to be checked against a non-gameable backtest, never taken at face value. say→ should be scored by a fixed judge that is not among the models under test, and reported alongside a swapped-judge number so readers can see its fragility. Profit should never be optimized inside the loop: repairing strategies to raise Sharpe destroys the identification the protocol provides. Limitations. Several bound our claims. The sample is small (4040 strategies, single seed at temperature 0.20.2), so we report effects, not confidence intervals. Model coverage is OpenAI-heavy with one open model, because Claude, Gemini, DeepSeek, Llama-70B, and Qwen-72B all returned not-found on our inference catalog—an infrastructure limitation, but a real one. The universe is eight liquid US ETFs and rule-based daily strategies only; we do not test intraday, multi-asset, portfolio, or learned strategies. The say→ judge is an LLM (audited but not resolved), the original judge sits on the leaderboard, and the swap judge is circular on its own family’s rows. say→ uses a keyword parser that returns “undetermined” when it cannot match a claim. Most importantly, FIDES measures internal concordance and execution fidelity, not tradability: a concordant, honestly-captioned strategy is not thereby a good one, and nothing here is investment advice or a claim that any model beats the market. 7 Conclusion and Future Work We presented FIDES, a protocol that treats an LLM trading strategy as three artifacts to reconcile—the stated logic, the generated code, and the realized track record—and scores three concordance gaps anchored by a backtest the model cannot game. Across 88 ETFs and four models, concordance did not predict profit, self-reported edge was badly calibrated (11 of 3232 claims held), and the one LLM-scored gap was not robust to swapping the judge; a look-ahead dose–response validated the execution gap while we disclosed a runtime probe that did not work. We frame these as measurement results and delimit external validity as future work. The clearest next steps follow the limitations: broaden the model roster once catalog access allows and add seeds for confidence intervals; replace the keyword say→ parser with a claim extractor; fix the runtime look-ahead probe so its shuffle boundary overlaps the observation window, or replace it with a stronger dynamic test; and extend beyond single-asset rule strategies toward portfolios and a genuine forward test, so that concordance can eventually be related to realized, out-of-sample tradability rather than to a closed backtest. References [1] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton (2021) Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: §1, §2. [2] D. H. Bailey, J. M. Borwein, M. López de Prado, and Q. J. Zhu (2014) Pseudo-mathematics and financial charlatanism: the effects of backtest overfitting on out-of-sample performance. Notices of the American Mathematical Society 61 (5), p. 458–471. Cited by: §1, §2. [3] D. H. Bailey, J. M. Borwein, M. López de Prado, and Q. J. Zhu (2017) The probability of backtest overfitting. Journal of Computational Finance 20 (4), p. 39–70. Cited by: §1, §2. [4] D. H. Bailey and M. López de Prado (2014) The deflated Sharpe ratio: correcting for selection bias, backtest overfitting, and non-normality. The Journal of Portfolio Management 40 (5), p. 94–107. Cited by: §1, §2. [5] M. Benhenda (2026) Look-ahead-bench: a standardized benchmark of look-ahead bias in point-in-time LLMs for finance. arXiv preprint arXiv:2601.13770. Cited by: §1, §2. [6] G. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang (2024) Humans or LLMs as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 8301–8327. Cited by: §2. [7] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §1, §2. [8] E. F. Fama (1970) Efficient capital markets: a review of theory and empirical work. The Journal of Finance 25 (2), p. 383–417. Cited by: §1, §2. [9] T. Lanham, A. Chen, A. Radhakrishnan, B. Steiner, C. Denison, D. Hernandez, D. Li, E. Durmus, E. Hubinger, J. Kernion, et al. (2023) Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702. Cited by: §1, §2. [10] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023) G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 2511–2522. Note: arXiv:2303.16634 Cited by: §2. [11] A. Lopez-Lira and Y. Tang (2023) Can ChatGPT forecast stock price movements? return predictability and large language models. arXiv preprint arXiv:2304.07619. Cited by: §1, §1, §2. [12] W. F. Sharpe (1994) The Sharpe ratio. The Journal of Portfolio Management 21 (1), p. 49–58. Cited by: §2. [13] R. Sullivan, A. Timmermann, and H. White (1999) Data-snooping, technical trading rule performance, and the bootstrap. The Journal of Finance 54 (5), p. 1647–1691. Cited by: §1, §2. [14] A. Tian, A. Ding, F. Chen, A. Wu, A. Chan, and B. Zhang (2025) Design and evaluation of cost-aware PoQ for decentralized LLM inference. arXiv preprint arXiv:2512.16317. Cited by: §1, §2. [15] A. Tian, A. Ding, F. Chen, S. Wu, and A. Chan (2026) A multi-dimensional quality scoring framework for decentralized LLM inference with proof of quality. arXiv preprint arXiv:2603.04028. Cited by: §1, §2. [16] A. Tian, A. Ding, F. Chen, S. Wu, and A. Chan (2026) Adaptive and robust cost-aware proof of quality for decentralized LLM inference networks. arXiv preprint arXiv:2601.21189. Cited by: §2. [17] A. Tian, A. Ding, F. Chen, S. Wu, and A. Chan (2026) PoQ-Judge: a multi-architecture evaluation framework for cost-aware proof-of-quality in decentralized LLM inference. arXiv preprint arXiv:2606.11196. Cited by: §1, §2. [18] A. Tian, A. Ding, F. Chen, S. Wu, and A. Chan (2026) SFGA: a statistics-first gating architecture with adjudicative escalation for trustworthy SFT data procurement. arXiv preprint arXiv:2607.18960. Cited by: §1, §2. [19] M. Turpin, J. Michael, E. Perez, and S. R. Bowman (2023) Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 36. Note: arXiv:2305.04388 Cited by: §1, §2. [20] P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, Q. Liu, T. Liu, and Z. Sui (2024) Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), p. 9440–9450. Note: arXiv:2305.17926 Cited by: §2. [21] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou (2023) Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), Note: arXiv:2203.11171 Cited by: §2. [22] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou (2022) Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurIPS) 35, p. 24824–24837. Note: arXiv:2201.11903 Cited by: §2. [23] H. White (2000) A reality check for data snooping. Econometrica 68 (5), p. 1097–1126. Cited by: §1, §2. [24] S. Wu, O. Irsoy, S. Lu, V. Dabravolski, M. Dredze, S. Gehrmann, P. Kambadur, D. Rosenberg, and G. Mann (2023) BloombergGPT: a large language model for finance. arXiv preprint arXiv:2303.17564. Cited by: §1, §2. [25] Y. Xiao, E. Sun, D. Luo, and W. Wang (2024) TradingAgents: multi-agents LLM financial trading framework. arXiv preprint arXiv:2412.20138. Cited by: §1, §1, §2. [26] Q. Xie, W. Han, Z. Chen, R. Xiang, X. Zhang, Y. He, M. Xiao, D. Li, Y. Dai, D. Feng, et al. (2024) The FinBen: a holistic financial benchmark for large language models. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 37. Note: arXiv:2402.12659 Cited by: §1, §2. [27] Q. Xie, W. Han, X. Zhang, Y. Lai, M. Peng, A. Lopez-Lira, and J. Huang (2023) PIXIU: a large language model, instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.05443. Cited by: §1, §2. [28] H. Yang, X. Liu, and C. D. Wang (2023) FinGPT: open-source financial large language models. arXiv preprint arXiv:2306.06031. Cited by: §1, §2. [29] Y. Yu, H. Li, Z. Chen, Y. Jiang, Y. Li, D. Zhang, R. Liu, J. W. Suchow, and K. Khashanah (2023) FinMem: a performance-enhanced LLM trading agent with layered memory and character design. arXiv preprint arXiv:2311.13743. Cited by: §1, §1, §2. [30] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 36. Note: arXiv:2306.05685 Cited by: §2. Appendix Appendix A Configuration Table 7: Frozen v0 configuration. Parameter Value universe SPY, Q, IWM, EFA, TLT, GLD, XLF, XLE in-sample (context only) 2018-01-01 – 2022-12-31 out-of-sample (primary) 2023-01-01 – 2024-12-31 alt window (ablation) 2020-01-01 – 2021-12-31 dual models gpt-4o-mini, gpt-4o, gpt-4.1-mini, qwen-2.5-7b-instruct two-stage arm gpt-4o-mini (elicitation ablation) routing units 40 (32 dual ++ 8 two-stage) seed / samples / temperature 42 / 1 / 0.2 backtest post=σt−1pos_t= _t-1; Sharpe ×252× 252 costs 0 bp (main); 10 bp one-way (ablation) SMA baseline 50 / 200, long-only sandbox subprocess, 5 s, pandas, numpy, math, statistics say→ judge gpt-4o-mini (orig); qwen-2.5-7b-instruct (swap) lookback set (rule say→ ) 5, 10, 14, 20, 21, 50, 100, 200 Appendix B The two “winners” Only two of the forty strategies beat buy-and-hold on the 2023–24 window, both on TLT, and they illustrate opposite failure modes of reading a raw beat-buy-and-hold count. Real-ish (gpt-4o / TLT). An RSI mean-reversion rule with a 2020-day moving-average filter and a volume liquidity gate; claimed edge: that TLT “exhibits short-term mean-reversion … which should outperform a buy-and-hold.” It realized OOS Sharpe 0.7590.759 vs. buy-and-hold −0.091-0.091 and return +0.084+0.084 vs. −0.055-0.055, at position density 0.0980.098—a genuine, if sparse, edge on this one ticker and window. Degenerate (gpt-4o-mini two-stage / TLT). A strategy that resolved to staying flat: zero position, zero return, and (correctly) no claim to beat buy-and-hold, so its say→ is undetermined rather than confirmed. It “beats” buy-and-hold only because TLT fell over the window and holding cash did not. Staying out of a falling asset is not a strategy; this row is why the calibration count of claimed-and-confirmed wins is 11, not 22.