Paper deep dive
Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact
Mateusz Kozłowski
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%
Last extracted: 8/3/2026, 10:47:30 AM
Summary
This paper presents a forensic reproducibility audit of a radiology vision-language model (VLM) benchmark, revealing significant divergences between the intended protocol and the released artifacts. The audit identified errors including prompt binding failures, incorrect DICOM rendering polarity, dataset split retention issues, and automated label extraction truncation. Consequently, the authors withdraw original performance claims and propose a machine-verifiable 'benchmark contract' to ensure integrity in future AI research.
Entities (9)
Relation Signals (9)
Cochran's Q → changesfrom → 154.73
confidence 90% · Reconstructing one common cohort of 369 complete case-finding blocks changed Cochran's Q from 154.73 to 182.29.
Cochran's Q → changesto → 182.29
confidence 90% · Reconstructing one common cohort of 369 complete case-finding blocks changed Cochran's Q from 154.73 to 182.29.
Forensic Reproducibility Audit → detectserrorsin → Vision-Language Model (VLM) Benchmark
confidence 90% · We performed a retrospective forensic reproducibility audit of a preserved chest-radiograph vision-language model (VLM) pilot
DICOM → hasrenderingissue → Polarity Inversion Error
confidence 90% · Four MONOCHROME1 images were rendered without required polarity inversion
MIMIC-CXR → usedin → VLM Benchmark
confidence 90% · using an enriched sample of 30 MIMIC-CXR studies
Automated Extractor → causes → Report Truncation
confidence 85% · the unvalidated extractor truncated five reports to 4000 characters
McNemar Test → hassignificantresults → 27 comparisons
confidence 85% · Of 45 McNemar comparisons, 27 had unadjusted p < 0.05
CheXpert → provideslabelsfor → Automated Extractor
confidence 85% · The reference side used the 14 CheXpert-style labels
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Medical-imaging AI benchmarks combine datasets, DICOM rendering, prompts, provider APIs, automated labels, statistical code, manuscripts, and repository releases. Agreement across these artifacts is usually assumed rather than tested. We performed a retrospective forensic reproducibility audit of a preserved chest-radiograph vision-language model (VLM) pilot; no model was called again and no image or report was newly annotated. We traced prompt bindings, DICOM metadata, output completeness, label extraction, matched analyses, and release propagation. Of 300 planned model-prompt calls, 297 yielded nonempty reports. Sixty Claude calls labeled A/B were executed with the same C prompt. The 30 studies represented 28 patients. Four MONOCHROME1 images were rendered without required polarity inversion, dataset split membership was not retained, and the unvalidated extractor truncated five reports to 4000 characters. Reconstructing one common cohort of 369 complete case-finding blocks changed Cochran's Q from 154.73 to 182.29. Of 45 McNemar comparisons, 27 had unadjusted p < 0.05 and 20 remained below 0.05 after Holm adjustment. These values describe only the archived automated-label matrix; they do not recover the intended prompt comparison or establish clinical performance. We withdraw the original performance, ranking, prompt-effect, and clinical claims and specify machine-verifiable controls for cohort, DICOM rendering, prompt and model identity, call status, annotation provenance, keyed analysis, and derived artifacts.
Tags
Links
- Source: https://arxiv.org/abs/2607.25589v1
- Canonical: https://arxiv.org/abs/2607.25589v1
Trouble viewing inline? Open PDF directly →
Full Text
76,501 characters extracted from source content.
Expand or collapse full text
Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact Mateusz Kozłowski ∗ Independent Researcher, Kraków, Poland Abstract Background: Medical-imaging AI benchmarks are often represented by a manuscript, analy- sis tables, executable code, provider APIs, and repository releases. Agreement between these artifacts is assumed but rarely tested directly. Purpose: To determine where an archived chest-radiograph vision-language model (VLM) pilot diverged from its reported protocol, how those divergences affected identifiability of its claims, and which machine-verifiable controls could have detected them. Materials and Methods: This was a forensic audit of an existing local project snapshot and its public repository record. No model was called again and no report or image was newly annotated. We traced executed prompt bindings, inventoried DICOM headers and patient aliases, reconciled planned versus nonempty outputs, inspected the automated label-extraction path, recomputed matched tests by explicit (case, finding) keys, and scanned manuscript, figure, submission, and repository artifacts for propagation failures. The audit unit was an artifact or scientific claim, not a patient outcome. Results: The nominal design contained 300 model-prompt calls; 297 yielded nonempty reports. Sixty Claude calls labeled A/B were executed with the same C prompt. The 30 studies rep- resented 28 patients and included 14 PA, 11 AP, one lateral, and four undocumented views; all four undocumented views were MONOCHROME1 and had been rendered without required polarity inversion. Dataset split was not retained. The automated extractor was unvalidated; for five reports it received only the first 4000 characters. Replacing positional truncation with one common cohort of 369 complete case-finding blocks changed the historical Cochran statistic from Q = 154.73 to 182.29. The number of McNemar comparisons with nominal unadjusted p < 0.05 changed from 26/45 to 27/45; after Holm adjustment, the count changed from 18/45 to 20/45. These arithmetic corrections did not restore the intended prompt estimand. Stale model names and results persisted in the released repository bundle after manuscript corrections. Conclusion: The corrected table was numerically reproducible, but its conditions differed from the reported experiment. The original performance and prompt-effect claims are withdrawn, and the legacy benchmark is invalid for outcome inference. A benchmark contract binding cohort membership, rendered-image and prompt hashes, provider-resolved model identity, call status, annotation provenance, keyed analysis, and derived-artifact checks is designed to expose these failure classes. The release implements selected guards with synthetic fixtures; the complete contract has not been prospectively validated. Keywords: medical imaging artificial intelligence; vision-language models; radiology; benchmark integrity; reproducibility audit; research integrity; DICOM rendering; prompt provenance; artifact traceability ∗ Corresponding author: mateusz.kozlowski@gmail.com 1 arXiv:2607.25589v1 [cs.CV] 28 Jul 2026 Forensic reproducibility audit2 1 Introduction Medical-imaging AI studies are often read as if the manuscript were the experiment. In practice, the reported result is the final product of a longer measurement and software chain. Cohort-selection code chooses records and sampling units; DICOM processing determines the model-visible pixels; a provider adapter binds the prompt, endpoint, model identifier, and decoding configuration; and a second system may convert free text into labels. Analysis code then selects paired and missing observations, after which build scripts and manual copy operations propagate values into tables, figures, submission packages, and public archives. Arithmetic may remain internally consistent even when the final result no longer describes the named condition. Reporting guidance recognizes many components of this problem. The 2024 update of the Checklist for Artificial Intelligence in Medical Imaging calls for explicit reporting of data selection, preprocessing, reference standards, partitions, missingness, software, and statistical analysis [1]. Radiology-specific guidance for large language models also emphasizes exact prompts, sampling settings, model versions, and expert evaluation [2]. General computational guidance recommends preserving raw observations, recording every transformation, avoiding manual data manipulation, and making analysis steps inspectable [3, 4]. Machine-learning reproducibility programs have like- wise shown the value of checklists, code release, and structured artifact review [5, 6]. These practices are necessary, but a list of disclosed components does not prove that the disclosed components were the ones that executed. Model cards and datasheets describe intended systems and datasets [7, 8]. FAIR principles improve the findability and reusability of digital objects [9], while the W3C PROV model provides a vocabulary for entities, activities, and agents [10]. Artifact-review frameworks ask whether released materials are documented, consistent, complete, and exercisable [11]. None of these properties alone establishes that a condition label equals the request bytes sent to a provider, that a radiograph was rendered with the intended polarity, or that the figure in a repository was rebuilt after the underlying result changed. Benchmarks that depend on remote vision-language model (VLM) services make these bound- aries difficult to inspect. Provider aliases can change, model snapshots may be unavailable later, and request metadata may not be returned in a form suitable for audit [2]. The model sees a rendered image rather than a DICOM header, so an intact source file can still become an incorrect model input. When another language model converts generated reports into labels, the endpoint combines at least three measurement processes: the clinical source report, the report-generating VLM, and the label extractor. A single binary “correct” cell can hide disagreement at any of these layers. The project examined here was initially presented as a pilot comparison of two reporting prompts across five VLM families using an enriched sample of 30 MIMIC-CXR studies [12]. A pre-submission review first appeared to require only numerical corrections. Deeper inspection showed that the preserved client-side call path bound both nominal Claude conditions to a third prompt. Four radiographs had been rendered with reversed polarity, the 30 studies belonged to 28 patients, and final split membership had not been retained. Three planned outputs failed to yield nonempty reports: two files were zero-byte and one was absent. Five reports were truncated before auto- mated extraction, and the matched analysis paired observations by position after condition-specific deletion. Corrected source text also failed to propagate into every public and submission artifact. The audit can correct reconstructable arithmetic, but it cannot recover an unexecuted prompt contrast or an absent clinical reference standard. It therefore does not produce a repaired leader- board and instead examines the divergence between the intended protocol and the preserved artifact system. This paper addresses three questions: Forensic reproducibility audit3 1. At which transitions did the intended, executed, observed, analyzed, reported, and released forms of the benchmark diverge? 2. Which historical quantities can still be reconstructed, and what is the narrowest interpretation that the preserved evidence permits? 3. Which machine-verifiable controls would have converted each silent divergence into a visible failure before publication? The analysis does not compare model performance. It reconstructs the archived arithmetic with an evidence hierarchy, explicit (case, finding) keys, and one stable matched cohort. Each claim is then adjudicated separately to distinguish recoverable arithmetic from unsupported scientific interpretation. The proposed fail-closed contract covers the cohort, pixels, prompts, model identity, outputs, annotation, analysis, and release. The accompanying aggregate-only package contains offline regression tests, figure-provenance records, privacy guards, and deterministic archive checks. 2 Conceptual framework 2.1 Six states of a computational result The audit treats a benchmark result as a sequence of state transitions rather than a single table. Table 1 defines the six states used throughout this paper. The states are related, but they are not interchangeable. For example, a prompt may exist in the intended protocol and in a source file, yet never become part of the executed request. A value may be correct in the current manuscript while an older value remains in the released archive. Table 1. States of a computational benchmark and the evidence needed to distinguish them. StateQuestionPreferred evidenceExample divergence in this audit IntendedWhat contrast and cohort were specified? Protocol, frozen plan, inclusion rules, expected prompt hashes Two prompt conditions and frontal test studies were intended. ExecutedWhat bytes, model endpoint, and image reached each call? Request-side payload, rendered-image hash, resolved model ID, execution path Two Claude labels received the same third prompt. ObservedWhat output and metadata were preserved? Atomic response record, byte count, request ID, finish state, raw response Two zero-byte files were logged as successful and one file was absent. AnalyzedWhich units, labels, joins, and denominators entered a statistic? Keyed analysis table, missingness map, code, multiplicity family Condition-specific deletion followed by truncation mispaired observations. ReportedWhat did the manuscript, table, and figure claim? Compiled source and derived objects linked to their inputs Headline values and model names changed across successive corrections. ReleasedWhat object could a reader retrieve? Repository version, archive bytes, manifest, DOI, publication timestamp The Zenodo bundle retained stale tables, figures, and names. Evidence had to match the state under review. Docstrings documented intent, runtime bindings established prompt execution, status fields summarized outputs, and file bytes established whether data existed. Current local sources documented the author’s corrected position; archived packages showed what readers had received. The audit therefore used runtime bindings for prompt identity, Forensic reproducibility audit4 DICOM attributes and rendering code for pixel identity, file bytes for completion, explicit scientific keys for pairing, and archived package contents for the public release. 2.2 Identity-preserving transitions Each comparison required a different identity check. Prompt identity was checked against exact request text, and image identity against the selected study, documented view, and rendered pixels. Matched analyses required a common (case, finding) key across columns. Public claims were linked to the code, input ledger, and release that generated them. Checksums verify identity at these transitions, including whether prompt strings differ or figures are byte-identical. Separate semantic checks are required to assess whether the prompts define a clinically appropriate contrast and whether a figure supports a justified claim. The proposed contract therefore combines identity checks with semantic invariants, such as allowed views, unique sampling units, explicit missingness, and a fixed multiplicity family. 2.3 What can be corrected without repeating the experiment The audit used a three-level repair boundary. A transcription or arithmetic error can be corrected when the authoritative inputs are preserved and the transformation is identified. A measurement or denominator problem can sometimes be reconstructed descriptively, but its interpretation must be restricted to the retained measurement system. A failure of treatment, input, or reference-standard identity cannot be repaired by reanalysis because the intended observation does not exist. The repair boundary explains why the historical Q and McNemar values are reported while the leaderboard is withdrawn. The keyed binary matrix permits a well-defined calculation about that archived matrix. It does not establish that the columns correspond to the intended prompts, that all images satisfy the inclusion rule, or that the automated labels represent clinical truth. This audit therefore does not claim to rescue the original experiment. It studies why that rescue is not identified and how the failure could have been detected earlier. 3 Materials and methods 3.1 Design, audit unit, and frozen scope This was a retrospective forensic audit of a project state preserved on 27 July 2026. The audit was initiated after discrepancies were found during pre-submission correction; it was not preregistered and did not test a prospectively specified clinical hypothesis. The unit of investigation was an artifact, transformation, or scientific claim. A patient outcome was not an audit unit. The examined materials included source code, prompt definitions, provider adapters, execution logs where available, a case manifest, DICOM headers, generated-report file metadata, cached four- state extractor labels, aggregate confusion-category cells, analysis code and outputs, manuscript versions, generated tables and figures, multiple submission packages, and the public Zenodo record [13]. The audit retained a local copy of the version-specific released archive. The relevant local files were read in place. Record-level clinical content was not copied into the release workspace. The scope was deliberately narrower than the original project. No provider was called again, no image was rendered again for outcome analysis, and no report or reference label was newly anno- tated. The audit did not infer unlogged provider snapshots, reconstruct raw extractor responses that had not been retained, or create a clinical reference standard. Private development cases previously called “Study 1” were excluded because their scoring rules changed during annotation, one source document was not retained, and the same small case set informed prompt develop- Forensic reproducibility audit5 ment and evaluation. Including them would have added non-reproducible clinical material without strengthening the artifact audit. 3.2 Evidence sources and rules of precedence Evidence was ranked separately for each audited state, as summarized in Table 2. Human-readable labels documented claims about objects; object identity required evidence from the corresponding execution or release boundary. Table 2. Evidence rules used to prevent a convenient proxy from replacing the object under audit. DomainInsufficient proxyPreferred evidenceFail-closed rule Prompt identity Condition name, source constant, or console label Request-side text or a binding traced to the API payload If payload identity cannot be established, no prompt-effect claim. Image identity Filename or assumption that every radiograph is conventional DICOM attributes, preprocessing path, and rendered-image bytes Unknown view or unresolved polarity fails eligibility. Model identity Family nickname or mutable provider alias Requested and provider-resolved identifiers with timestamp and endpoint If the resolved snapshot is absent, model-specific attribution is restricted. CompletionLogged ok value or file path Atomic record with nonzero bytes, request ID, and finish state Empty, missing, blocked, and truncated outputs remain distinct states. PairingEqual vector lengths after deletion Explicit (case, finding) keys and a documented cohort rule Unkeyed truncation is rejected even when dimensions match. Public releaseCorrected working source Bytes retrievable from the dated archive or submission package A correction is not public until every derived artifact is rebuilt. The rules distinguish missing identity metadata from a documented mismatch. An absent re- solved model identifier restricts model-specific verification without showing that the wrong model ran. Byte-identical prompt bindings for conditions intended to differ, by contrast, document that the intended contrast did not execute. 3.3 Audit workflow Execution trace. We traced imports, function arguments, mutable module state, adapter calls, and request construction from the batch runner to the provider boundary. Comments, docstrings, display labels, and nominal condition names were recorded as intent evidence but were not allowed to override the executed binding. Special attention was given to Python imports by value and to runtime monkey-patching, because changing a name in one module does not necessarily change a value already bound in another module. For adapters that accepted the prompt as a function argument, the argument path was followed into the request object. For the Claude path, the import order and module namespace used by the actual request function were inspected. Cohort and image audit. We matched the retained case manifest to study identity, a pseudony- mous patient alias, and the DICOM fields for view position and photometric interpretation. Counts were made at both study and patient level so that repeated patients could not be mistaken for in- dependent sampling units. The selector was inspected to determine which official MIMIC-CXR Forensic reproducibility audit6 partitions were eligible and whether final split membership survived in the manifest. The ren- dering path was checked for modality transformation, windowing, conversion to eight-bit pixels, JPEG creation, and conditional polarity handling. The DICOM standard defines the minimum MONOCHROME1 sample value as white and the minimum MONOCHROME2 value as black [14]; treating both encodings identically therefore changes the displayed polarity for MONOCHROME1 inputs. This was a bounded audit of retained headers and preserved code. Images were not re- rendered, and the full modality LUT, VOI LUT, window selection, bit-depth, compression, and display chain was not independently validated. A blank ViewPosition was classified as undocu- mented, not as proof of a nonfrontal acquisition. Output reconciliation. The frozen design implied 30× 5× 2 = 300 calls. We reconciled ev- ery expected combination with the corresponding path, byte count, and logged status. A report was counted as nonempty only if the expected file existed and contained at least one byte after whitespace handling. A missing file and a zero-byte file were not treated as interchangeable, and a success status did not override either condition. The separate states matter because downstream annotation code can silently convert both states into missing cells while an upstream dashboard still reports a successful call. Annotation and endpoint audit. We inspected the automated extraction code from input construction through JSON parsing and confusion-category assignment. The audit recorded the extractor model name present in code, maximum input length, retry and fallback behavior, allowed label states, uncertainty mapping, and false-positive severity rule. It also checked whether raw extractor responses, full decoding settings, and a human validation sample were retained. The four candidate states were POSITIVE, NEGATIVE, UNCERTAIN, and NOT_MENTIONED. The reference side used the 14 CheXpert-style labels, including No Finding [15]. The audit did not assume that either side constituted image-adjudicated truth. Statistical reconstruction. We assigned each retained observation a scientific key formed from its case and finding identifiers. The key, not row position, determined alignment across the ten nominal model and prompt conditions. The original analysis path was reproduced separately to identify the source of the published values, then replaced for the corrective calculation by one glob- ally complete matrix. Alternative routes were retained as named sensitivity calculations rather than silently mixed. No missing value was imputed, and no missing-completely-at-random assumption was made. Propagation audit. We searched for canonical model names, call counts, cohort counts, and headline statistics across manuscript sources, compiled tables, plotting inputs, figure labels, repos- itory README files, compressed archives, submission variants, and copy-ready metadata. We treated the dated Zenodo deposit as the released artifact and inspected a locally preserved copy of its version-specific archive. We did not infer its contents from the newest local folder. This tested whether a correct edit in one source had reached every object presented to readers. 3.4 Historical statistical reconstruction The reconstruction used the archived confusion-category matrix. For each of the 420 possible case- finding blocks, every nominal condition was reduced to a binary indicator: TP and TN were coded 1, while FN and every FP severity category were coded 0. A block entered the global matrix only when all ten conditions had a defined binary value. This produced an n×k matrix Y = (y ij ), where k = 10, i indexes a (case, finding) key, and j indexes a nominal condition. Equivalently, the base Forensic reproducibility audit7 index contained 30× 14 = 420 blocks and 4,200 condition-specific cells. The global rule retained 369 blocks and 3,690 condition-specific cells. Cochran’s Q was calculated for the globally complete matrix following the matched-sample for- mulation [16]. Let C j = P i y ij be the column totals, R i = P j y ij the row totals, and T = P j C j . The statistic was Q = (k− 1) k P j C 2 j − T 2 kT − P i R 2 i , with k− 1 = 9 degrees of freedom. The survival probability was evaluated from the chi-square distribution without rounding Q. All 10 2 = 45 post-hoc comparisons used the same rows as the omnibus test. For a pair of columns, b counted keys correct only in the first condition and c counted keys correct only in the second. The uncorrected McNemar statistic was (b − c) 2 /(b + c), and the continuity-corrected statistic was (|b− c|− 1) 2 /(b + c) when discordant observations were present [17]. Exact two-sided binomial and exact mid-p values were also calculated. These variants have different small-sample properties [18], so agreement of the threshold counts was checked rather than assumed. Holm adjustment was applied to the unrounded p-values within each complete 45-comparison family [19]. No comparison was removed after seeing its result. A separate pairwise available-case route was computed to check the reviewer’s 28/45 value. None of the four implemented McNemar variants reproduced that count: each yielded 27/45 unadjusted and 20/45 after Holm adjustment. This route nevertheless targets different observation sets because each pair may use a different denominator; it was therefore excluded from the fixed-cohort family reported here. The statistical target was the historical binary matrix, not clinical accuracy. CheXpert-derived report labels are a reference procedure rather than image-level adjudication [15, 1], and the can- didate labels were generated by an extractor that had not been validated against human readers. P-values were consequently used to reproduce the archived calculation, not to reinstate a clinical or comparative claim. The 369 retained blocks did not constitute 369 independent clinical sampling units. Fourteen finding labels were nested within each study, and two patients contributed two studies. The usual chi-square reference for Cochran’s Q and binomial reference for McNemar’s test therefore do not have defensible population-level calibration in this dataset. The statistic, nominal p-value, and threshold counts are preserved solely to reconstruct the legacy arithmetic and to explain the differ- ence among analysis routes. They are not interpreted as inferential evidence of a model, prompt, or clinical difference. 3.5 Aggregate ledger, figures, and offline test harness We stored publishable findings in a versioned aggregate JSON ledger governed by JSON Schema. A dependency-free validator checks the schema during tests and release preflight; additional tests enforce cross-field arithmetic, such as whether missing, empty, and nonempty outputs sum to the plan. Every numerical value shown in Figures 1–4 is read from this ledger rather than copied into plotting code. Figure 5 is explicitly a conceptual build diagram and contains no measured value. The ledger assigns each aggregate variable an explicit field [20]. A companion trace links the ledger and script hashes to each figure hash, caption claim, and stated limitation. The local harness contains synthetic record-level fixtures and public aggregate artifacts; it does not contain restricted clinical records. It tests request-side prompt hashes, collapsed nominal conditions, DICOM polarity and view rules, duplicate patients, split retention, terminal output validation, joins on explicit keys, Cochran’s Q, four McNemar variants, Holm adjustment, forbidden Forensic reproducibility audit8 claim text, identifier-like privacy patterns, unsafe archive paths, and manifest verification. Known small matrices and analytically known p-values support the statistical regression tests. The suite requires only Python’s standard library; Matplotlib is used to regenerate figures. These tests verify control logic, not the historical outcomes. Because the record-level historical matrix and MIMIC material cannot be redistributed, a third party can execute the statistical functions and failure guards but cannot independently regenerate the study-specific 27/45 and 20/45 counts from the release candidate alone. The candidate states this limit explicitly. 3.6 Claim adjudication Each headline claim was evaluated against the minimum evidence needed for its scientific meaning, then assigned one of four statuses: • retained as an artifact fact when preserved files directly established the statement without requiring a clinical interpretation; • restricted when a calculation was valid only for the archived automated-label matrix; • unverifiable when a necessary identity, configuration, or source was not retained; • withdrawn when the executed condition, input, or measurement did not identify the reported contrast. The status was not determined by statistical significance. A small p-value could coexist with withdrawal if the comparison was not the one reported. Conversely, an artifact fact could be retained without an outcome test. When several defects affected one claim, the strongest limitation governed its disposition. Claim adjudication was author-led and was not independently replicated. 3.7 Data governance and public-release boundary MIMIC-CXR is credentialed data. Current PhysioNet guidance for use with online services requires verified zero retention, no training, and no human review, and advises against a service whose handling cannot be confirmed [21]. This audit initiated no new transfer and cannot infer the historical provider-account configuration from source code. It therefore does not certify historical compliance. The public candidate contains no MIMIC image, source report, patient or study identifier, gen- erated report, private development case, or record-level derivative. It contains aggregate counts, audit code, schemas, offline tests with synthetic record-level fixtures and public aggregate or source artifacts, figure code, and publication documentation. A conservative text scanner is included to catch common identifier patterns. Passing that scanner does not establish de-identification or per- mission to redistribute credentialed data. Venue-specific ethics and data-use requirements must be assessed independently. 4 Results 4.1 Observed divergences Table 3 summarizes the observed breaks between reported intent and the preserved artifacts. Forensic reproducibility audit9 Table 3. Divergences between reported intent and the preserved executable or released artifact. LayerReported or intended Preserved evidenceConsequence Prompt routing Claude System A versus B The pipeline imported the default prompt by value before a later module monkey-patch. The preserved client-side call path bound both labels to byte-identical System C. The 60 labeled A/B calls identify C/C variability, not a prompt effect. Photometry Standard grayscale radiographs Four inputs were MONOCHROME1; the renderer did not invert them. Model-visible polarity was reversed for four studies. View selection Frontal radiographs Headers contained 14 PA, 11 AP, one L, and four blank view positions. The claimed frontal cohort was not executed. Sampling unit 30 independent cases The manifest contained 30 studies from 28 patients; two patients each contributed two studies. Independence and the claimed participant count were unsupported. PartitionOfficial test setThe selector admitted both test and validation studies; the retained manifest did not store split. Final partition composition is not recoverable from the release. Completeness Near-complete output set One Gemini B report was absent and two Qwen A files were zero bytes despite an ok status. The correct file-level count is 297/300. ExtractionComplete automated annotation The extractor truncated input at 4000 characters; five Claude reports exceeded that limit. Raw API responses and decoding settings were not retained. Candidate labels cannot be fully audited or independently reconstructed. Forensic reproducibility audit10 LayerReported or intended Preserved evidenceConsequence ScoringClinical FP severity and one binary convention Severity was assigned deterministically by finding name. No Finding was pooled with pathologies; generated uncertainty was silently non-positive. Severity was not clinical; endpoint directions depended on coding choices. Matched tests Equal-length paired vectors Missing entries were dropped by condition and remaining vectors truncated by position. Observations after the first gap were mispaired. ReleaseCorrected source propagated to all artifacts Stale model names and values remained in repository tables, figures, and parallel submission packages. The public bundle did not describe the corrected manuscript. Stage 1 — Protocol identity Stage 2 — Measurement and release 01 Cohort 30 studies · 28 patients split not retained 02 Pixels 4 MONOCHROME1 not inverted 03 Prompts nominal A/B executed C/C 04 Models resolved identity not retained 05 Outputs 297 / 300 nonempty 06 Labels unvalidated extractor 5 inputs truncated 07 Analysis positional mispairing 08 Release stale names and values in archive Figure 1. Observed audit findings along the archived artifact path. The upper row sum- marizes protocol identity; the lower row follows derived measurements through release. Arrows show artifact dependency, not estimated causal effects. Every count comes from the public aggregate ledger. Recalculation can repair arithmetic but cannot restore unrecorded prompt-condition or model iden- tity. 4.2 Condition identity and prompt routing The batch runner exposed two nominal prompt keys, A and B, and passed the selected string as an argument to four provider adapters. The Claude route was different. It first imported the report function from a separate pipeline module, then attempted to replace a prompt variable in the Forensic reproducibility audit11 original prompts module before calling that function. The pipeline module had already imported the default prompt by value. Reassigning the original module attribute did not update the name bound inside the pipeline module, whose request construction continued to use the default System C text. This control-flow path applied to 30 studies under two nominal Claude conditions, accounting for 60 affected calls. Filenames, progress labels, and analysis columns still recorded A and B, but both conditions were bound to prompt C. Repeated execution under one prompt can describe run-to-run output variation; it cannot identify an A-versus-B effect because the treatment did not vary. The preserved Python import and call path deterministically establishes what the function read under the recorded import order. Because provider-side request payloads were not retained, this is an inference from code rather than direct API-log evidence. The release candidate tests the corresponding guard with synthetic payloads and does not present historical request bytes. 4.3 Cohort, sampling unit, and pixel identity The four MONOCHROME1 images were exactly the four images with undocumented view posi- tion. Adding the single L image produced five studies that failed the reported requirement for a documented AP or PA input. The selector ranked PA ahead of AP and other values, but it did not enforce a frontal-only filter. Blank view metadata therefore survived selection. A blank value does not prove that an acquisition was nonfrontal; it shows that the released evidence cannot establish the reported frontal criterion. The same four blank-view studies were encoded as MONOCHROME1. The preserved renderer applied windowing and converted the result to eight-bit JPEG without the conditional polarity inversion required by that photometric interpretation. The model-visible grayscale polarity was therefore reversed for those four inputs. The audit did not attempt to estimate how this changed a particular output. The scientific problem occurs earlier: nominal calls using those four study inputs did not receive the image representation defined by the intended benchmark. Two patients each contributed two studies. One contributed a MONOCHROME1 study and the L study; the other contributed two otherwise frontal studies. Thus the most compact description is 30 studies from 28 patients, not 30 independent patients. Repeated studies need not be forbidden in every design, but they require a declared sampling unit, patient-level split policy, and clustered analysis. None was preserved here. The partition claim was likewise unresolved. The selection code admitted both the official test and validation partitions, while the retained manifest did not store final split membership. This establishes that “test only” was not enforced. The final mixture was not reconstructed in this audit and is excluded from the release candidate; controlled re-linkage to official split metadata may be possible if a credentialed review preserves the necessary restricted keys. 4.4 Output completeness and the post hoc boundary Of 300 planned report files, 297 were nonempty. Because two Qwen A files were zero bytes and one Gemini B file was absent, affecting three different studies, available denominators differed by condition. A count of 299 resulted from treating zero-byte files with logged ok status as completed. The file-level reconciliation distinguishes these states because an optimistic status can otherwise propagate into a claimed denominator even when no report is available for annotation. A mechanical post hoc eligibility subset would remove the four undocumented, non-inverted MONOCHROME1 studies and the L study, leaving 25 AP/PA MONOCHROME2 studies. After also removing the two non-identifiable Claude columns, eight nominal conditions would remain. Those 25 studies supplied 350 potential case-finding blocks. Eight blocks had an uncertain reference Forensic reproducibility audit12 label. A remaining study lacked an entire Qwen A output, affecting 14 blocks, two of which overlapped the uncertain-reference set. The union was therefore 8 + 14− 2 = 20 incomplete blocks, leaving 330 complete blocks from 24 studies. This calculation is a technical eligibility boundary, not a corrected benchmark. The exclusions were defined after inspecting the defects, the final partition was still unknown, and the candidate- label measurement remained unvalidated. Reporting a new leaderboard from the 330 blocks would create a post hoc experiment that was neither planned nor clinically adjudicated. No such table was produced. 051015 Studies (n) PA AP Undoc. L (a) View positions 14 11 4 1 (b) Output files 297 nonempty of 300 2 zero-byte 1 missing (c) Complete blocks 87.9% globally complete 369 complete 51 incomplete Figure 2. Aggregate accounting of views, report files, and matched case-finding blocks. Panel (a) shows the retained DICOM header counts. Panel (b) distinguishes a nonempty file from zero-byte and missing outputs. Panel (c) uses one 420-block frame for all ten nominal conditions. The visualization contains no record-level membership data. 4.5 Annotation semantics The extractor assigned each report one of four labels per finding: POSITIVE, NEGATIVE, UNCERTAIN, or NOT_MENTIONED. For five reports it received only the first 4000 characters. Per-report label caches survived, but raw extractor responses and the complete historical decoding configuration did not. The released aggregate retained only the derived TP, TN, FN, or severity-coded FP cell. Table 4 makes the historical reduction explicit. Candidate uncertainty was treated as non- positive rather than preserved as a third state in the binary analysis. Reference uncertainty excluded the block. These choices are analytic conventions, not clinical facts. Forensic reproducibility audit13 Table 4. Historical mapping from report-level label states to the archived binary endpoint. Severity suffixes were assigned by finding name, not by clinical adjudication. Reference state Candidate extractor state Archived cell Audit interpretation PositivePositiveTP, coded 1Report labels agree on presence. PositiveNegative, not mentioned, or uncertain FN, coded 0Historical convention treats candidate uncertainty as non-positive. NegativePositiveFP, coded 0Severity is a deterministic finding-name category. NegativeNegative, not mentioned, or uncertain TN, coded 1Absence and silence are pooled for this endpoint. UncertainAny stateExcludedNo binary reference value is assigned. No Finding supplied 10 of 38 positive reference cells in the pooled 14-label endpoint despite having opposite semantics to the 13 pathology labels. Removing it reversed the reported A/B direction for one nominal model family. Treating generated UNCERTAIN as positive instead of non- positive reversed the direction for another nominal model family. These sensitivity analyses show that the archived direction depended on endpoint conventions; they do not establish that either mapping is clinically correct. The non-identifying totals are frozen in the public aggregate ledger, while study-level verification requires the retained local caches. The severity labels were not adjudicated by a radiologist. Of 551 historical FP cells, 550 were classified minor and one moderate by a deterministic finding-name rule; none was severe. All clin- ical interpretations of this variable were therefore withdrawn. The endpoint measures agreement between two automated report-labeling paths. It cannot determine whether the image supports a finding, whether the source report omitted a visible abnormality, or whether the extractor misun- derstood the generated text. 4.6 Reconstructed historical arithmetic The archived matrix contained 420 possible case-finding blocks. Exactly 369 were complete across all ten nominal conditions. The original procedure first deleted missing values independently within each condition and then truncated all remaining vectors to the shortest length, n = 383. Equal length did not restore matching. If one condition lacks key 2, for example, its second retained value belongs to key 3 while the second value of a complete condition still belongs to key 2. Every position after the first asymmetric gap can therefore compare different case-finding blocks. After joining by key, one common cohort of 369 blocks was used for all calculations. It yielded Cochran’s Q = 182.28998 with 9 degrees of freedom and a nominal p = 1.684× 10 −34 . For every implemented McNemar variant, 27 of 45 unadjusted comparisons had nominal p < 0.05, and 20 of 45 remained below 0.05 after Holm adjustment. The threshold counts were identical for the asymptotic, continuity-corrected, exact binomial, and exact mid-p variants; the public aggregate ledger does not identify whether the same pairs crossed the threshold in every variant. The reviewer-suggested 28/45 count was not reproduced. Allowing each pair to use its own avail- able blocks still yielded 27/45 unadjusted comparisons under the asymptotic, continuity-corrected, exact binomial, and exact mid-p variants; each route also yielded 20/45 after Holm adjustment. Pair-specific deletion is a legitimate descriptive route under an appropriate analysis plan, but here it changes the observations across contrasts. The audit selected the common 369-block set so that Cochran’s Q and the full 45-test family referred to one archived cohort. Neither route repairs prompt-condition identity, potentially informative missingness, or within-study clustering. Forensic reproducibility audit14 These values correct the reported Q = 154.73, 26/45 raw, and 18/45 adjusted counts. They do not validate the benchmark: the ten columns include two mislabeled Claude conditions, invalid image representations, an incompletely preserved partition, and unvalidated extracted labels. The nominal p-values are also uncalibrated for the nesting of findings within studies and repeated studies within two patients. They are retained as forensic outputs of a historical analysis, not evidence of population heterogeneity. 150170190 Q statistic (a) Cochran Q 154.73 Original 182.29 Keyed global +27.56 (17.8%) 010203040 Comparisons with p < 0.05 (of 45) Original Pairwise check Keyed common (b) McNemar counts Raw 26Holm 18 2720 2720 reviewer claim: 28 not reproduced Figure 3. Three versions of the historical arithmetic. The original report used positional truncation. The reviewer claim of 28 raw comparisons was not reproduced: all four pairwise available-case variants yielded 27 raw and 20 Holm-adjusted comparisons. The keyed reconstruction used one globally com- plete cohort for the omnibus test and all 45 McNemar comparisons. These values describe archived automated labels, not clinical performance. 4.7 Propagation into released artifacts The propagation audit found that source correction and public correction occurred separately. At the time of inspection, the Zenodo record with DOI 10.5281/zenodo.21427298 contained reproducibility_bundle_REVISED.zip, dated 18 July 2026. It predated later corrections. Stale model names remained in a README, generated results tables, performance figures, and the run- ner source. The archive did not contain the provider adapters needed to resolve the contradictory names. The stale labels also appeared in parallel journal and arXiv sources and in copy-ready metadata. Several manually maintained publication branches remained internally consistent while describing an obsolete state. Correcting the local manuscript did not change the bytes attached to the public record. The audit used a locally preserved copy of the inspected release. The corrective record provides a version-specific file inventory and checksums that distinguish the audited bundle from later public changes. 4.8 Impact on scientific claims Table 5 separates findings that remain supportable from claims that the retained evidence cannot identify. Forensic reproducibility audit15 Table 5. Disposition and public verifiability of the original headline claims. Original claim type StatusDecisive evidencePublic check System B reduces hallucinations Withdrawn The endpoint was report-label discordance, not image-adjudicated hallucination; routing also failed for Claude. The candidate includes only the withdrawal and aggregate failure count. Five-model performance ranking Withdrawn Provider-resolved identities, valid common inputs, and a clinical reference standard were not preserved. Absence is recorded; a ranking is not regenerated. Claude A/B behavior Withdrawn Both labels were executed with the same C prompt. Synthetic routing tests demonstrate the required guard. Scale or open/proprietary status explains performance Withdrawn The benchmark is invalid and proprietary parameter counts were unavailable. The aggregate claim-disposition ledger records the withdrawal; no quantitative explanation is regenerated. The archived matrix yields the reconstructed statistics Restricted The keyed calculation is defined for the archived automated labels, but its nominal reference distribution is not clinically calibrated. Aggregate values and dependency-free algorithm tests are included in the release candidate; the private historical matrix is not. The released bundle reproduces the paper Withdrawn Names, statistics, tables, and figures were not synchronized. The version-specific inventory, correction notice, and aggregate ledger document the discrepancy. The audit found multiple cross-layer divergences RetainedEach divergence is supported by code, headers, file state, or artifact text. The release candidate contains the aggregate ledger and guards; some source-level findings require controlled verification. The audit reports arithmetic tied to preserved inputs and withholds claims that require valid clinical measurement or prompt-condition identity. Numerical correction alone does not establish that the associated conclusion remains valid. 4.9 Executable safeguards A clean local run completed all 36 offline tests. They use synthetic record-level fixtures, the public aggregate ledger, and the public source tree; no restricted clinical record enters the suite. The tests require only Python’s standard library, while Matplotlib is needed to regenerate the figures. Table 6 links each family to the historical failure it is designed to expose. Forensic reproducibility audit16 Table 6. Offline regression-test inventory. Record-level fixtures are synthetic; aggregate and source- tree checks use release-candidate artifacts. Control familynRepresentative failureFail-closed guard Keyed statistics11Positional truncation or a changing matched denominator Unique scientific keys, one global cohort, known Q/McNemar/Holm results Artifact, privacy, release 11Withdrawn prose, private tokens, unsafe paths, or archive drift Claim scan, privacy patterns, closed manifest, and SHA-256 verification Cohort and pixels 4Duplicate patient, missing split, unexpected view, or uninverted MONOCHROME1 Manifest validation before inference Aggregate ledger4Contradictory totals or altered corrective values Schema and cross-field invariants Prompt routing3Condition label and request payload disagree Exact payload and prompt-hash assertions Output accounting 3A success flag masks an empty or missing file Terminal-state validation and planned-call reconciliation 024681012 Offline regression tests (n) Keyed statistics Cohort and pixels Prompt routing Artifact, privacy, and release Aggregate ledger Output accounting Scientific identity Release integrity 11 4 3 11 4 3 36 tests | 6 families Figure 4. Distribution of the 36 offline tests. Solid bars denote controls over scientific identity; hatched bars denote release-integrity controls. Counts are an inventory, not code coverage or clinical validation. 5 A machine-verifiable benchmark contract 5.1 Design principles and threat model The observed failures arose through different mechanisms, but each involved an unchecked mapping between a label and the object it was meant to identify. The contract checks four transitions: planning resolves sampling-unit, image, prompt, and model identity; execution creates complete atomic records; analysis requires annotation provenance and scientific keys; and release rebuilds and reconciles derived artifacts. “Fail closed” means that an unmet invariant changes the pipeline state to a named failure and prevents downstream publication. It does not mean that every unexpected observation is deleted. A lateral image, repeated patient, or length-limited response may be scientifically useful in a protocol Forensic reproducibility audit17 that anticipates it. The requirement is that the protocol declare how it is handled before inference and that the implementation enforce that declaration. The threat model covers accidental drift, stale configuration, incomplete materialization, am- biguous missingness, unsafe joins, and failed propagation. It excludes deliberate fabrication by an actor able to alter source data and manifests together, and it does not assess whether the clinical question is valid. The contract instead preserves the identity of the objects that a benchmark claims to compare and exposes divergence before result generation. 5.2 Eight contract clauses 1. Cohort contract. Each sampling unit receives an immutable case key, a pseudonymous patient key, source and split provenance, declared eligibility fields, and an inclusion decision. The contract must state whether repeated patients are prohibited or handled by a patient- level split and clustered analysis. In the audited project, this clause would have rejected an undocumented split, unexpected view values under the stated frontal-only rule, and an unacknowledged difference between studies and patients before any API call. 2. Pixel contract. Preprocessing parameters, relevant DICOM attributes, software version, conditional photometric transformation, and the final rendered-image hash are recorded. Ev- ery paired model or prompt condition must receive the same frozen bytes unless image trans- formation is itself the planned treatment. A syntactically valid hash establishes identity, not image quality; separate modality-specific tests and visual checks remain necessary. 3. Prompt contract. The nominal condition points to an immutable expected prompt text and hash. Every call records the text and hash at the request boundary, after template expansion and adapter logic. A test must inspect the actual request object or a faithful interceptor, because reading a module constant would not have detected the import-binding defect observed here. Conditions expected to differ fail if their executed prompt hashes collapse. 4. Model contract. The planned target, provider endpoint, requested identifier, provider- resolved identifier, SDK and adapter version, decoding settings, timestamp, and request ID remain linked to each call. Mutable aliases such as “latest” are rejected unless the provider response resolves and records a stable snapshot. When a provider does not expose a resolved identifier, the limitation is explicit and model-version claims are restricted. 5. Output contract. A planned-call ledger is reconciled with exactly one terminal record per call. Completion requires nonempty bytes, a request ID, and a terminal finish state. Empty, transport-failed, safety-blocked, invalid, and length-limited responses remain distinct statuses. Files are written atomically with sidecar metadata so that existence alone cannot masquerade as successful completion. 6. Annotation contract. The clinical reference, candidate extraction, reader decisions, adju- dication, extractor request and raw response, model and decoding configuration, text-length handling, and uncertainty policy remain separate fields. A derived TP, TN, FN, or FP cell is generated from these fields and never replaces them. This clause makes sensitivity to No Finding, uncertainty, and severity mapping inspectable instead of embedding those choices in one irreversible label. 7. Analysis contract. Joins operate on explicit scientific keys. The analysis plan names the sampling unit, missingness rule, denominator, clustering structure, multiplicity family, test variant, effect measure, and random seed where applicable. Invariants verify that an omnibus Forensic reproducibility audit18 test and its post-hoc family use the declared cohort. A changing denominator can still be allowed, but only as an explicitly named estimand rather than an unnoticed side effect of list deletion. 8. Release contract. One versioned registry supplies canonical names and corrective values to manuscript text, tables, figures, metadata, and archive builders. Every generated artifact records its input and script hash. A closed manifest rejects unlisted or changed files, while text guards reject withdrawn claims and stale headline values. Under this contract, each release receives a version and checksum; folder contents and modification times do not define release state. 5.3 Reference implementation and current coverage The release candidate is a reference implementation of selected guards, not a validated framework. Table 7 distinguishes executable controls from specifications so that 36 passing tests are not mis- taken for complete coverage of all eight clauses. Table 7. Implementation status of the proposed benchmark contract in the corrective release candi- date. ClauseStatusExecutable componentImportant limitation CohortPartial reference Case, patient, split, and view checks Rules are protocol parameters; split membership was not retained in the audited manifest, although controlled re-linkage may be possible. PixelPartial reference Photometry, inversion flag, and hash-format checks The validator does not re-render or assess image quality. PromptReference validator Expected text, payload hash, routing, and collapsed-condition checks Synthetic requests only; no historical payload ledger is public. ModelSpecification only Required fields are described in the manuscript No historical provider-resolved IDs can be reconstructed. OutputReference validator Plan reconciliation, byte count, request ID, status, and finish state Atomic file writing is specified but not simulated against a provider. AnnotationSpecification only Mapping risks are frozen in the aggregate ledger No public record-level labels, reader workflow, or raw extractor responses. AnalysisPartial reference Keyed complete matrix, Q, four McNemar variants, and Holm adjustment Cluster-aware inference and clinical estimands are not implemented. ReleaseReference build Claim, privacy, schema, test, figure-trace, clean manuscript, archive, and closed-manifest gates Canonical generation of every prose value is not yet automatic. The file artifact_contract.json encodes fail-closed text rules for submission artifacts. It is not a complete machine-readable representation of the eight-clause benchmark contract. A dependency- free validator enforces the JSON Schema during tests and release preflight, while Python tests enforce cross-field arithmetic that the schema does not express conveniently. 5.4 Practical adoption and limits A small project can implement the contract with a frozen CSV or JSON call plan, standard-library hashes, adapter tests of request objects, one sidecar per call, composite-key joins, and whitelist Forensic reproducibility audit19 release builds. The specific tools may vary, provided that each state transition has a machine- checkable precondition and leaves evidence for the next state. Hashes document whether an object changed and whether an archive matches a frozen list. Ex- pert review must still assess clinical appropriateness, consent, data-use compliance, and statistical adequacy. A reproducible pipeline may still implement an invalid experiment. The contract makes the evidence chain inspectable for review but does not establish validity by itself. Stage 1 — Inputs and build Stage 2 — Verify and release Release inputs aggregate ledger synthetic fixtures · source tree 01 Contract tests routing · cohort · statistics 02 Artifact rebuild figures · manuscript 03 Artifact checks claims · files · build log 04 Privacy + manifest pattern scan · SHA-256 Release artifacts Zenodo ZIP · arXiv TAR.GZ × Any failed gate blocks both archives. Figure 5. Local release gates and outcomes for the corrective artifacts. Versioned aggregate evidence, synthetic fixtures, and source files pass through four local gates before the Zenodo and arXiv archives are built. If any gate fails, neither archive is built. These gates verify the local package, not the external platforms. 6 Discussion 6.1 Interpretation of the reconstructed arithmetic Joining observations by explicit keys produced one historical Q value and the same 27/45 unadjusted and 20/45 Holm threshold counts across the four implemented McNemar variants. Those aggregate counts answer a narrow forensic question about one declared reduction of the archived matrix. They do not show that the columns represent the intended prompt-condition identities, that every row represents an eligible image, or that a binary agreement cell represents a clinical judgment. The estimand also depends on the treatment or condition, sampling unit, input representa- tion, measurement procedure, and population to which the result refers. Here, failures in prompt- condition identity, input eligibility, sampling-unit definition, partition provenance, and outcome measurement were cumulative. A corrected formula cannot reconstruct components that were never created or were not preserved. The nominal Q statistic therefore does not restore the original claim. Even before accounting for nested findings, it describes differences among columns with compromised identities. The correction should report the historical arithmetic accurately and withdraw interpretations unsupported by condition and measurement identity. Forensic reproducibility audit20 6.2 Why the failures interacted These failures interacted. Prompt routing and DICOM handling altered the conditions and inputs. Output state and extraction rules determined which measured cells existed, and positional trunca- tion changed how those cells were paired. Because the defects were not isolated, this audit does not estimate the contribution of any single one. Consider the prompt defect. The repository contained three real prompt strings and the run- ner printed two plausible condition labels. Output directories and analysis columns were named consistently with A and B. Most artifact-level checks would therefore see a balanced design. Only execution-path tracing showed that a value imported into another module remained C after the at- tempted patch. Because the prompt defined the experimental condition, the binding error changed the executed comparison. A one-call adapter test asserting the request text or hash would have exposed the failure before 60 outputs were produced. The DICOM finding has the same structure: MONOCHROME1 changes how sample values should be displayed [14], so omitting polarity inversion broke the equivalence between the source file and the model-visible input. The audit did not re-render or re-evaluate the images; the warranted conclusion is that the model inputs for four studies fell outside the intended pipeline, not that a particular performance penalty occurred. Missingness linked execution to analysis because file status, byte count, reference uncertainty, and candidate-label availability determined whether a block entered the matrix. Pairwise available-case analysis and the globally complete matrix used different denominators, but all four implemented pairwise variants and the selected common-cohort analysis yielded 27/45 raw and 20/45 Holm- adjusted comparisons. The reviewer’s 28/45 value was therefore not reproduced. Explicit keys and a named missingness rule make the analysis sets and this numerical discrepancy visible. 6.3 Measurement language and the term hallucination The original project used report-derived labels as both reference and candidate measurements. That design can be useful for scalable report comparison, but it does not directly observe image-level correctness. A source report may omit a visible finding, describe it with uncertain language, or reflect a different acquisition context. A generated report may express the same concept with wording that an extractor misses. The extractor may then add a third disagreement. Pooling No Finding with 13 pathology labels further shows that the meaning of a binary endpoint depends on its construction. A positive No Finding label has different semantics from a positive pathology label. Likewise, treating candidate uncertainty as non-positive is one possible decision rule, not a neutral fact. The observed directional sensitivities demonstrate that these conventions mattered in the archived matrix, but they do not establish which convention is clinically preferable. The manuscript therefore uses automated report-label discordance rather than hallucination. Re- cent radiology taxonomy work also calls for a precise definition of the evaluated contingency [22]. The narrower term describes the measurement without implying image-level adjudication, clinical severity, or a causal explanation for the generated text. 6.4 Release engineering is part of the method The public-release failure shows that source correction and release correction require separate veri- fication. Correcting one authoritative source did not update the repository ZIP, generated tables, figures, parallel submission variants, or copy-ready metadata. Without a machine-readable depen- dency graph, the author must identify and update each dependent object manually. The version- specific Zenodo bundle inspected in the audit confirms that the public correction was incomplete. Forensic reproducibility audit21 A whitelist build, input and script hashes, a closed manifest, and archive verification identify the source state that produced each released object. These controls make corrections traceable and allow reviewers to establish which artifact they assessed; they do not validate the underlying scientific claim. 6.5 Relationship to existing reproducibility frameworks The proposed contract complements the reporting, documentation, stewardship, provenance, and artifact-review frameworks discussed in the Introduction. The contract adds identity checks at execution boundaries: request hashes link prompt descriptions to request bytes; rendered-image hashes link DICOM selections to model-visible inputs; scientific keys preserve case-finding identity across conditions; and release manifests link public archives to source states. These controls add machine-verifiable provenance where narrative documentation does not establish object identity. The leakage problem was not limited to partition membership [23]. The condition label remained in filenames after the prompt binding collapsed. Accounting treated an ok status as completion even when a file was empty, and one manuscript was corrected while the public release retained the old value. These are provenance failures even when no classic train-test leakage occurs. 6.6 What can be published without rerunning the study Without a rerun, the preserved evidence supports a forensic case study, not a new benchmark claim. A new benchmark paper would require a prospectively frozen protocol, valid inputs, logged model and prompt identities, human-validated measurement, an appropriate sampling plan, and calibrated analysis. None is created by this audit. Presenting corrected performance estimates would therefore exceed the evidence. A forensic case study can document how preserved artifacts diverged, determine which historical calculations remain reconstructable, define the boundary between correction and non-identifiability, and test controls against synthetic reproductions of the failure classes. These contributions require explicit evidence grading and exclusion of outcomes that the retained evidence does not identify. When the evidence does not identify a replacement estimate, the correction should state that limit. Here, the post hoc 330-block boundary shows how far mechanical cleaning could proceed; because the subset was selected after the defects were known, it is not reported as a new benchmark estimate. 6.7 Implications for authors, reviewers, and repositories For authors, several controls can be applied before expensive inference: validate cohort and patient units, freeze rendered-image hashes, intercept one request from each adapter, and assert distinct condition hashes. During inference, record resolved model identity and terminal output states. Before analysis, require explicit keys and print a missingness table. Before release, rebuild from a whitelist and verify the archive after extraction. Reviewers cannot be expected to reverse-engineer every pipeline, but they can ask boundary questions that expose weak provenance: Where was the prompt captured, before or after adapter logic? Are the reported cases patients, studies, images, or finding blocks? Does every paired comparison use the same keys? What does a successful call mean when a file is empty? Can the authors show that the downloadable archive was built from the submitted source? Editors and repositories can support these checks by accepting compact machine-readable ledgers, preserving version history, displaying relations between corrected and superseded records, and encouraging executable artifact review. Persistent citation metadata does not validate a record’s contents or show that a later version reproduces an earlier state. Versioned correction notices and checksums help readers distinguish record persistence from content validation. Forensic reproducibility audit22 6.8 Limitations and researcher reflexivity This is a single case from one small radiology VLM project. It cannot estimate how often similar failures occur in medical AI, and the proposed controls should not be interpreted as a prevalence- derived taxonomy. The same author designed the original project, performed the audit, adjudicated claims, and prepared this report. That access enabled detailed reconstruction but creates risks of selective attention and confirmation bias. The audit was retrospective, not preregistered, and no independent laboratory repeated it. Several conclusions rely on evidence that is stronger than a manuscript label but weaker than a complete historical log. The C/C prompt result follows deterministically from the preserved import path, yet provider-side payloads were not retained. Model-resolved identifiers and historical account settings remain incomplete. The DICOM review established a missing MONOCHROME1 inversion in preserved code but did not validate the entire rendering chain. Cached extractor labels permitted semantic sensitivity checks, while raw responses and full decoding settings could not be reconstructed. The release candidate intentionally excludes record-level MIMIC material, so readers cannot independently regenerate the aggregate results from the candidate. Controlled verification by an appropriately credentialed reviewer may be necessary. The included privacy scanner is only a guardrail and does not certify data-use compliance. The historical matrix has an additional statistical limitation. Fourteen finding blocks are nested within each study, and two patients contributed repeated studies. The nominal Q and McNemar reference distributions do not account for this clustering. No population-level effect size, confidence interval, or power calculation is offered because the outcome estimand itself is not being reclaimed. The reported p-values are forensic reproductions of historical arithmetic, not a corrected inferential analysis. The benchmark contract was derived after observing these failures. Its offline tests show that se- lected validators react to synthetic record-level fixtures and the public aggregate or source artifacts; they do not establish prospective validity of the complete framework. Model and annotation clauses remain specifications, the JSON and text contracts do not encode every semantic requirement, and the workflow has not been tested by an independent project. Future work should prospectively im- plement the contract, measure false alarms and missed failures, and evaluate whether the controls remain usable at larger scale. 7 Conclusion The audited benchmark diverged from its reported identity at the prompt, pixel, cohort, output, annotation, analysis, and release layers. Recomputing the statistics corrected three historical values; the condition, input, cohort, and measurement defects remained. The paper therefore withdraws all clinical-performance, safety, ranking, and prompt-effect claims. The publishable contribution without rerunning the study is the documented failure chain and a contract designed to make the observed failure classes machine-detectable before release. Its reference validators are testable now; prospective validation remains future work. Data and code availability Release candidate 1.1.0 contains audit code, 36 offline tests, a dependency-free JSON Schema val- idator, an artifact contract, aggregate forensic findings, versioned figure-generation code, a privacy scanner, a gated deterministic archive builder, and SHA-256 manifests. Record-level test fixtures Forensic reproducibility audit23 are synthetic; other checks read only the aggregate ledger or source-tree artifacts included in this candidate. The tests run with Python’s standard library. The figure generator reads all numerical values from the aggregate ledger; Figure 5 is conceptual. The reproducibility boundary has three levels. Public verification covers aggregate accounting, schema and invariant checks, statistical algorithms on synthetic matrices, traced figure generation, claim and privacy guards, manuscript compilation, and archive checksums. Controlled verification may cover the record-level historical matrix, DICOM headers, and local label caches for an appro- priately credentialed reviewer. Provider-side payloads, raw extractor responses, and some resolved model identities were not retained and cannot be reconstructed at either level. Final split member- ship was absent from the audited manifest and is unavailable publicly; a controlled re-linkage may be possible if the necessary restricted keys survive. The release candidate intended for public deposit excludes MIMIC images, source reports, patient or study identifiers, generated reports, private development cases, and record-level derivatives. The inspected legacy release is available at 10.5281/zenodo.21427298 [13]. The corrective audit and archival reconstruction are available at 10.5281/zenodo.21629849 [24]. Declarations Ethics and governance. No new model inference, annotation, or data collection was performed for this audit, and no credentialed clinical material is redistributed. Historical PhysioNet/provider compliance has not been independently verified. No claim of ethics exemption is made. Funding. This work received no external funding. Competing interests. The author declares no competing interests. Author contributions. Mateusz Kozłowski: conceptualization, investigation, methodology, soft- ware, validation, visualization, data curation, and writing. Use of generative AI. OpenAI Codex assisted with code review, synthetic test scaffolding, figure generation code, and language editing. It did not supply clinical labels or new experimental observations. The author inspected the preserved evidence, verified numerical outputs and cited sources, reviewed every generated artifact, and retains responsibility for the manuscript. References [1] Ali S. Tejani, Michail E. Klontzas, Anthony A. Gatti, John T. Mongan, Linda Moy, Seong Ho Park, and Charles E. Kahn, Jr. Checklist for artificial intelligence in medical imaging (CLAIM): 2024 update. Radiology: Artificial Intelligence, 6(4):e240300, 2024. [2] Christian Bluethgen, Dave Van Veen, Cyril Zakka, Katherine E. Link, Aaron Hunter Fanous, Roxana Daneshjou, Thomas Frauenfelder, Curtis P. Langlotz, Sergios Gatidis, and Akshay Chaudhari. Best practices for large language models in radiology. Radiology, 315(1):e240528, 2025. [3] Geir Kjetil Sandve, Anton Nekrutenko, James Taylor, and Eivind Hovig. Ten simple rules for reproducible computational research. PLoS Computational Biology, 9(10):e1003285, 2013. [4] Morgan Taschuk and Greg Wilson. Ten simple rules for making research software more robust. PLoS Computational Biology, 13(4):e1005412, 2017. Forensic reproducibility audit24 [5] Benjamin J. Heil, Michael M. Hoffman, Florian Markowetz, Su-In Lee, Casey S. Greene, and Stephanie C. Hicks. Reproducibility standards for machine learning in the life sciences. Nature Methods, 18:1132–1135, 2021. [6] Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelz- imer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research: A report from the NeurIPS 2019 reproducibility program. Journal of Machine Learning Research, 22(164):1–20, 2021. [7] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 220–229, 2019. [8] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé I, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86–92, 2021. [9] Mark D. Wilkinson et al. The FAIR guiding principles for scientific data management and stewardship. Scientific Data, 3:160018, 2016. [10] Timothy Lebo, Satya Sahoo, Deborah McGuinness, Khalid Belhajjame, James Cheney, David Corsar, Daniel Garijo, Stian Soiland-Reyes, Stephan Zednik, and Jun Zhao. PROV-O: The PROV ontology. W3c recommendation, World Wide Web Consortium, April 2013. [11] Association for Computing Machinery. Artifact review and badging, version 1.1. https://w. acm.org/publications/policies/artifact-review-and-badging-current, August 2020. Accessed 27 July 2026. [12] Alistair E. W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Roger G. Mark, and Steven Horng. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 6:317, 2019. [13] Mateusz Kozłowski. Reproducibility bundle: Iterative prompt calibration of a vision-language model system for automated radiology report generation. https://zenodo.org/records/ 21427298, 2026. Historical release record, DOI 10.5281/zenodo.21427298; the audit used a locally preserved copy of the version-specific bundle inspected in July 2026. [14] National Electrical Manufacturers Association. DICOM PS3.3: Information object defi- nitions, image pixel module. https://dicom.nema.org/medical/dicom/current/output/ chtml/part03/sect_C.7.6.3.html, 2026. Current online edition, accessed 27 July 2026. [15] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):590–597, 2019. [16] William G. Cochran. The comparison of percentages in matched samples. Biometrika, 37(3– 4):256–266, 1950. [17] Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157, 1947. [18] Morten W. Fagerland, Stian Lydersen, and Petter Laake. Recommended tests and confidence intervals for paired binomial proportions. Statistics in Medicine, 33(16):2850–2875, 2014. Forensic reproducibility audit25 [19] Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979. [20] Hadley Wickham. Tidy data. Journal of Statistical Software, 59(10):1–23, 2014. [21] PhysioNet. Use of MIMIC data with large language models and online services. https: //physionet.org/news/post/llm-responsible-use/, September 2025. Accessed 27 July 2026. [22] Frank J. Brooks and Mark A. Anastasio. A taxonomy of machine hallucination in radiology. Radiology: Artificial Intelligence, 8(2):e250203, 2026. [23] Sayash Kapoor and Arvind Narayanan. Leakage and the reproducibility crisis in machine- learning-based science. Patterns, 4(9):100804, 2023. [24] Mateusz Kozłowski. VLM radiology prompt-calibration benchmark: Versioned reproducibility archive. https://doi.org/10.5281/zenodo.21629849, 2026.