Paper deep dive
6.5% of the Neuro-Symbolic Literature Can Be Reproduced from Its Published Artifacts, a Six-Stage Audit Framework and First Instantiation
Brandon Colelough, Vladimir Martirosyan, Ishan Tamrakar, William Regli, Aditya Kumar, Anh N. Nhu, Dhruv Dubey, Raj Ambavane, Haowei Deng
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/28/2026, 3:29:09 AM
Summary
This paper presents a six-stage audit framework for assessing the reproducibility of scientific claims in computer science, instantiated on the Neuro-Symbolic AI (NSAI) subdomain. The study audited 1,304 eligible NSAI records, identifying 455 with verifiable public code artifacts. Of these, only 85 (6.52% of the eligible corpus, 18.68% of attempted reruns) were fully or partially reproduced. The primary barriers to reproducibility were missing non-code artifacts (321 cases) and missing or unusable code repositories (42 cases). The authors argue for enforced, versioned, and permanently archived artifact bundles in future NSAI publications.
Entities (8)
Relation Signals (6)
Brandon Colelough → affiliatedwith → University of Maryland
confidence 99% · Brandon Colelough... Department of Computer Science, University of Maryland College Park
Neuro-Symbolic AI → hasreproducibilityrate → 6.52%
confidence 98% · We fully or partially reproduced 85 studies, 6.52% of the eligible corpus
Six-Stage Audit Framework → appliedto → Neuro-Symbolic AI
confidence 95% · instantiate our framework for the neuro-symbolic AI (NSAI) subdomain.
University of Maryland → owns → Zaratan
confidence 90% · executed on the University of Maryland Zaratan high-performance computing cluster
Zaratan → uses → Nvidia A100
confidence 90% · GPU jobs used the GPU partition, each limited to one full GPU node consisting of either 4 × NVIDIA A100
Zaratan → uses → NVIDIA H100
confidence 90% · or 4 × NVIDIA H100
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We present a six-stage framework for auditing the reproducibility of scientific claims across a research literature within the computer science domain, and instantiate our framework for the neuro-symbolic AI (NSAI) subdomain. Instantiating the framework on the NSAI subdomain produced a multi-year audit. Stage one retrieved 5,497 records and removed 3,018 duplicates. Stage two screened the 2,479 unique records at title and abstract, identifying 1,365 self-identified NSAI records, then removed a further 61 at full text for off-topic, non-research, no-quantitative-evaluation, or inaccessible-full-text reasons. Stage three sought a verifiable public code artifact for each of the 1,304 eligible records and found none for 849, leaving 455 to enter the artifact inventory and bounded rerun of stages four and five. We fully or partially reproduced 85 studies, 6.52% of the eligible corpus and 18.68% of attempted reruns. We found that 321 attempted reruns were blocked by missing non- code artifacts and 42 by missing or unusable code repositories. These figures quantify a persistent reproducibility deficit that survives even nominal "code available" declarations, and signal the need for enforced, versioned, and permanently archived artifact bundles in future NSAI publications. We argue that empirical NSAI papers should be required at submission time to provide complete, versioned, and permanently archived artifact bundles.
Tags
Links
- Source: https://arxiv.org/abs/2608.26236v1
- Canonical: https://arxiv.org/abs/2608.26236v1
Trouble viewing inline? Open PDF directly →
Full Text
92,261 characters extracted from source content.
Expand or collapse full text
6.5% of the Neuro-Symbolic Literature Can Be Reproduced from Its Published Artifacts, a Six-Stage Audit Framework and First Instantiation Brandon Colelough ∗ , Vladimir Martirosyan, Ishan Tamrakar, William Regli, Aditya Kumar, Anh N. Nhu, Dhruv Dubey, Raj Ambavane, Haowei Deng Department of Computer Science, University of Maryland College Park, Maryland, USA brandcol@umd.edu Abstract We present a six-stage framework for auditing the reproducibil- ity of scientific claims across a research literature within the computer science domain, and instantiate our framework for the neuro-symbolic AI (NSAI) subdomain. Instantiating the framework on the NSAI subdomain produced a multi-year audit. Stage one retrieved5,497records and removed3,018 duplicates. Stage two screened the2,479unique records at title and abstract, identifying1,365self-identified NSAI records, then removed a further61at full text for off-topic, non-research, no-quantitative-evaluation, or inaccessible-full-text reasons. Stage three sought a verifiable public code artifact for each of the1,304eligible records and found none for849, leaving455 to enter the artifact inventory and bounded rerun of stages four and five. We fully or partially reproduced85studies,6.52 % of the eligible corpus and18.68 %of attempted reruns. We found that321attempted reruns were blocked by missing non- code artifacts and42by missing or unusable code repositories. These figures quantify a persistent reproducibility deficit that survives even nominal “code available” declarations, and sig- nal the need for enforced, versioned, and permanently archived artifact bundles in future NSAI publications. We argue that empirical NSAI papers should be required at submission time to provide complete, versioned, and permanently archived artifact bundles. Introduction Can an independent team rerun the computational pipeline released with a published paper and recover its main reported result? We put that question to 455 neuro-symbolic stud- ies that advertised a public code artifact. Eighty-five reruns succeeded. In this paper, we undertake a study of the same- artifact rerun of published results in the emerging field of Neuro-Symbolic Artificial Intelligence. Specifically, we ask whether an independent team can rerun the computational pipeline released with a paper and recover the main reported result within a prespecified tolerance. NSAI is viewed by many as the emerging frontier that merges concepts from connectionist methods with those referred to informally as “Good Old Fashioned AI”, consisting of logic and formal tech- niques for reasoning. Integrations range from loose pipelines, where a neural component feeds a symbolic one, to tightly coupled systems trained jointly. The frontier science that aims ∗ Corresponding author. to integrate these two techniques is highly dynamic, and while the number of papers that claim to be creating contributions in this area is exploding in number, this paper reports that the results in the majority of these papers cannot be recovered by rerunning the released artifacts. This indicates that the research area comprising “Neuro-Symbolic AI” is experienc- ing a crisis of replication and lacks the community norms that would make released artifacts reliably executable as part of the publication of work. For the field to advance into the position of critical importance that many feel is inevitable, we should raise the bar on our scientific standards and require greater emphasis on how we can document reproducibility so that others may verify our claims. On the evidence below, we argue that empirical submissions in this field should carry a complete, versioned, and permanently archived artifact bundle at submission time. Background As has been documented often in the recent literature, many areas of science are currently experiencing a “reproducibility crisis” (Baker 2016) as studies have proven difficult or impos- sible to replicate (López-Nicolás et al. 2022), data sources are not available (Miłkowski, Hensel, and Hohol 2018), and ex- perimental assumptions are not made explicit (Hensel 2020). Social sciences, in which these phenomena have been doc- umented most extensively, have the additional issues posed by post hoc redesign of scientific hypotheses. Known as “Hy- pothesizing After the Results are Known”, or “p-hacking”, this occurs when a researcher forms or rewrites a hypothesis after seeing the data, and then presents that hypothesis as if it were specified before the data were collected (Rubin 2022). The field of computing, in theory, should be highly reproducible, as algorithms, code, data, and other artifacts can be easily shared and adopted. Conferences and journals have begun to require data sharing and other best practices to improve reproducibility (noa 2020). Without reproducible artifacts, computational research risks becoming unfalsifiable claims that cannot be independently verified and thus fall out- side the bounds of science. We need to be able to rigorously and independently test hypotheses laid out in papers in the computer science community in the same way we do in other domains, so that the work can have more credibility. arXiv:2608.26236v1 [cs.AI] 26 Aug 2026 Contributions and Research Questions We follow the ACM badging terminology (Association for Computing Machinery 2020) for reproducibility and refer to reproducibility as meaning that a team other than the original obtains a consistent result using the original team’s artifacts. We make two contributions. The first is a six-stage framework for auditing same-artifact rerun reproducibility across a liter- ature. The second is an instantiation of the audit framework itself, the largest of its kind, employing the framework on 1,304eligible NSAI records. The research questions we aim to address include: RQ1 What proportion of NSAI papers releasing code can be fully or partially reproduced? RQ2 How does artifact completeness (code / data / model weights) affect the probability of successful rerun? RQ3 Do reproduction outcomes vary systematically by publication year or venue family (conference, journal, preprint)? Related Work Reproducibility failures are well-documented across empiri- cal disciplines. Landmark studies in psychology and medicine have shown that a substantial fraction of published findings cannot be independently replicated (Open Science Collabo- ration 2015; Ioannidis 2005), and large-scale surveys across the social and biomedical sciences trace these failures to non-disclosed analytical choices, unavailable data, and se- lective reporting (Baker 2016; López-Nicolás et al. 2022; Hensel 2020). The interested reader is referred to that broader literature for a full treatment. Computer science presents a structurally different case, as algorithms, code, and data can in principle be shared exactly, meaning empirical claims ought to be among the most verifiable in science. In practice, however, the same failure modes recur, as the studies below illustrate. Vanderdonckt and Vatavu introduced Amplitum, a contextual framework that augments generic replication taxonomies with explicit descriptors of participants, devices, and physical settings (Vanderdonckt and Vatavu 2025). Their gesture-elicitation case study reproduced prior findings only after replicators matched the original laboratory environment and user cohort, underscoring that code and stimuli alone are insufficient when human behavior is the main dependent vari- able. Ferrari Dacrema et al. inspected 26 “state-of-the-art” (for the time) neural recommender papers (2015–2018) and were able to fully reproduce only 12 of the original 26, and in doing so discovered that eleven of those twelve fell behind well-tuned neighborhood, matrix-factorization, or sparse-linear baselines once a common experimental protocol was enforced (Fer- rari Dacrema et al. 2021). Ferrari et al. attribute the apparent performance gains to methodological weaknesses, including baselines left at default settings, inadvertent test-set leakage during epoch selection, and undocumented data-split choices. The ReproNLP shared task series asks independent teams to repeat published human evaluations using materials supplied by the original authors, and reports that agreement with the original figures is frequently not achieved (Belz and Thomson 2023). The organisers attribute divergence to underspecified evaluation protocols and to details of the original setup that were never documented. Henderson et al. evaluated several widely cited DRL algorithms and found that benchmark rank- ings could reverse when random seeds, hardware platforms, or training horizons were varied (Henderson et al. 2018). Their study therefore recommends reporting results over many seeds, applying formal significance tests, and disclosing every experimental detail. Pawlik et al. examined the longevity of public datasets and found that many links degrade, move, or silently change versions, rendering later experiments irrepro- ducible despite nominal accessibility (Pawlik et al. 2019). They argue for immutable storage, rigorous version identi- fiers, and provenance metadata. Reproduction breaks down whenever any piece of the experimental context is missing, and Neuro-Symbolic AI is no exception. An open-code link without hardware details, solver commits, pre-processing scripts, dataset splits, and hyperparameter schedules offers little more than performative compliance. Gains attributed to the symbolic–neural fusion may vanish once baselines re- ceive equal tuning or a knowledge base is revised; stochastic variation in the neural component could dominate the sym- bolic layer as well. Reliable reproduction, therefore, requires immutable, versioned artifact bundles, multi-seed evaluation, and full provenance for pretrained weights, logic programs, and curated knowledge graphs. Methodological Audit Framework We present our six-stage framework for auditing same-artifact rerun reproducibility across a literature and a realised example of our six-stage reproducibility framework below. Stage 1 constructs and screens the corpus. Stage 2 confirms full-text eligibility. Stage 3 identifies the code artifact and records a six-item artifact inventory before any execution is attempted. Stage 4 builds the released environment and checks executable integrity. Stage 5 reruns the primary experiment under a bounded repair allowance and a stated fidelity criterion. Stage 6 extracts data and audits outcome labels. Corpus Construction & Screening, Audit Protocol Stages 1-3 Stage 1 constructs the corpus. We began with a deliber- ately broad bibliographic sweep, designed to capture the full breadth of the neuro-symbolic domain. Guided by a PRESS-validated query (McGowan et al. 2016) centered on “neuro-symbolic OR NeSy OR NSAI”, we queried nine major digital libraries on 23 May 2025: Web of Science, Scopus, PubMed, Ei Compendex, IEEE Xplore, ACM DL, SpringerLink, Google Scholar, and arXiv. The query returned5,497records. This broad sweep increased coverage of emerging or lexically idiosyncratic work that narrower queries often overlook. A de-duplication pipeline removed 3,018duplicates (55% of raw hits), leaving2,479unique records. The substantial overlap across sources illustrates the cross-posting norm in NSAI research and underscores the need for multi-pass de-duplication. 1 Relevance screening was performed in a single-blind title-and-abstract pass. After a 20-record calibration, reviewers screened titles and abstracts and retained any paper whose authors explicitly described the work as “neuro-symbolic,” regardless of application domain. 1,365papers satisfied this broad criterion. Code availability 1 Pipeline tools: EndNote, Covidence, Zotero, SR-Accelerator, and Rayyan. Records identified from databases n = 5,497 Unique records after duplicates n = 2,479 Records screened (title/abstract) n = 2,479 Records on topic n = 1,365 Eligible full-text NSAI records n = 1,304 Attempted reruns n = 455 Attempted reruns n = 455 Reproduced and included n = 85 Not successfully reproduced n = 370 Duplicates removed n = 3,018 Title/abstract exclusions n = 1,114 Full-text eligibility exclusions n = 61 No verifiable public code artifact n = 849 Figure 1: Selection and eligibility flow for the NSAI audit. Late full-text eligibility exclusions are shown separately and are not counted as rerun outcomes. The dashed bottom box reports the number of successful reruns within the attempted set and is shown as an audit outcome. was not inferred from the abstract alone. Instead, repository verification was conducted during full-text eligibility assess- ment wherein annotators first inspected the full paper for repository or artifact links, and when none were present, they performed a structured external search using the paper title, author names, and method or domain keywords. This process eliminated849records for which no verifiable public code artifact could be identified and left455code-bearing studies for the rerun audit, an attrition of65.11 %that exposes the gap between open-science claims and delivered artifacts. The stages reduced the literature from2,479unique records to 1,365NSAI-relevant papers and further to455audit can- didates that at minimum linked or could be matched to a code artifact. Figure 1 visualizes each reduction step, linking corpus construction directly to our objective of quantifying verifiable progress in Neuro-Symbolic AI. Audit Protocol Stage 4-6 Inclusion criteria A paper entered the reproduction pipeline only when it satisfied all of the following;IC 1 - Neuro-symbolic integration – the study is self-described as Neuro-Symbolic.IC 2 - Empirical evaluation – the pa- per reports quantitative results on benchmarks, real-world datasets, or synthetic tasks and compares against baselines or ablations.IC 3 - CS relevance – the work contributes technical insight into the CS domainIC 4 - Auditable code claim – the paper provides a direct repository link or a uniquely iden- tifiable public code artifact for the reported system, sufficient to permit artifact audit.IC 5 - Full text Available – the paper has an accessible full text for audit. Exclusion criteria Papers were removed if they violated any of the following:E 1 Not written in English.E 2 Lit- erature review, review, survey, editorial, or otherwise not original empirical research.E 3 No verifiable public reposi- tory or archival code artifact for the reported system could be identified from the paper or via a structured external search.E 4 Missing indispensable artifacts required to rerun the primary experiment could not be identified, accessed, or reconstructed under the study protocol.E 4 applies where the paper or its linked repository shows an indispensable artifact to be unavailable before any rerun is attempted and absences discovered during an attempted rerun are recorded asO 4 .E 5 Lacks quantitative evaluation.E 6 Outside the scope of neuro-symbolic methods.E 7 Duplicate, superseded, or version-of-record already retained.E 8 No full-text access (pay-walled or retracted). ExclusionsE 1 –E 8 are applied only at title-and-abstract screening and at full-text eligibility assess- ment. A record failing any of them never enters the attempted rerun pool. Once a record does enter, every subsequent failure is recorded as an audit outcome underO 1 –O 5 and never as an exclusion, so that sample selection is never confounded with audit results. Figure 1 reports counts for both. Reproduction Procedure Post-entry rerun outcomes are reported in Figure 2. Every study with a publicly accessible repository was evaluated under the six-stage protocol that com- pressed the workflow into discrete, auditable checkpoints 2 . All annotators began with a one-hour onboarding workshop 3 that introduced the audit workflow and extraction template. This was followed by two 4-hour live sessions (covering environment builds and dependency management, metric verification, and licensing constraints) and a 3-hour super- vised drop-in lab in which each participant could reproduce exemplar studies end-to-end in a supervised environment. A worked example demonstrating application of the study’s data-extraction form is also available. 4 All rerun attempts and artifact assessments were conducted by a team of eight trained graduate student annotators over a nine-month period, who completed the study’s standardized onboarding work- shop and calibration protocol. Each annotator handled an average of 64 records, comprising the455attempted reruns and the 61 late eligibility exclusions distributed across eight annotators. Workload was allocated randomly across the team. A ten-paper calibration pilot produced Cohen’sκ = 0.82(a standard measure of inter-rater agreement, where 1.0 is perfect agreement and values above 0.80 are considered strong), and outstanding disagreements were reconciled in group discus- 2 NSAI survey and audit index 3 annotator training and calibration materials in the [https://doi.org/10.5281/zenodo.21779234] 4 NSAI-2025 data-extraction worked example sion. During the nine-month audit phase, the team met every second week to review edge cases and realign on the reproduc- tion protocol. Outcome labels were assigned after entry into the attempted rerun pool as follows: O1) Fully reproduced. The primary experiment was executed successfully, and the reproduced result satisfied the study’s fidelity criterion. O2) Partially reproduced. The core pipeline executed, and the paper’s main qualitative claim was preserved, but one or more quantitative results fell outside the acceptance band, or only a subset of the headline experiments could be rerun. O3) Executed but did not reproduce within tolerance. The system ran to completion, but the reproduced results materially exceeded the acceptance threshold or contradicted the paper’s main quantitative claim. O4) Not executable due to missing or inaccessible artifacts. The attempted rerun could not proceed because one or more indispensable artifacts were unavailable, inaccessible, or not reconstructable under the study protocol. O5) Not executable due to environment or code failure. Attempted rerun failed because the released code or environment could not be built or executed under the protocol despite the permitted minimal fixes. For this study, an artifact was classified as missing only when it was an indispensable input to the primary experiment and could not be reconstructed from the paper and released materials under the study protocol. Indispensable inputs included fixed datasets or splits, preprocessing scripts that alter data seman- tics, checkpoints or weights when evaluation depended on a fixed trained model state, rule sets or knowledge bases, configuration files, environment descriptors, and evaluation assets. All reproduction logs and extracted data are version- controlled, and a consolidated record is available in the public reproduction and data-extraction sheet. Evaluation Design We report two outcomes and keep them separate. Full repro- duction (O1) is the primary result and requires recovering the paper’s headline number. Partial recovery (O2) is reported alongside it. Only papers that entered the attempted rerun pool count toward either. To qualify as a full reproduction, a repository must execute under the author-supplied envi- ronment (or a minimally updated equivalent), with a max 7 day wall-clock time and on the audit team’s GPU pool, and reproduce the paper’s primary metric to within±5% absolute error (or inside the authors’ 95 % confidence interval), yield outputs consistent with the paper’s headline claims, and re- quire no correction of bugs intrinsic to the model architecture. Annotators were authorized to (i) update deprecated package versions or apply path fixes, provided it is minimal, (i) adjust file paths, (i) supply a lightweight evaluation harness when none was provided, and (iv) patch minor scripting errors. Malformed or undocumented environments, missing indis- pensable post-entry artifacts, or architecture-level defects were recorded as O4 or O5 audit outcomes, not as exclusions. Protocol Stages This study was designed around six central stages, which included: Stage 1 –Full-text eligibility confirmation Using the eli- gibility criteria above and the reproduction component of the data extraction Form, annotators re-checked language, topical relevance, quantitative evaluation, and full-text accessibility. Failure on any item triggered eligibility exclusion. Stage 2 – Repository identification and full-artifact ver- ification For papers passing Stage 1, annotators identified the code artifact from the paper itself or via structured external search, recorded the repository URL, license, and commit hash where available, and then verified whether all artifacts required for reproduction (e.g., code, data, weights, environ- ment files, and documentation) were publicly accessible. If an essential artifact was available but required gated access (e.g., proprietary data), up to two e-mail requests were sent over 14 days, and a lack of response led to a missing-artifact access outcome. Stage 3 – Environment Build & Executable Integrity. Repositories passing Stage 2 were rebuilt with the authors’ en- vironment file(s) (e.g., Dockerfile,requirements.txt, environment.yml, etc.). Builds failing due to irreconcil- able dependencies were recorded as O5 outcomes. Successful builds proceeded to unit or smoke tests, and fatal code-level errors were likewise recorded as O5 outcomes. Stage 4 – Result Re-execution. The primary experiment was rerun. A study was marked Accepted if reproduced metrics were faithful to the reported results from the authors and reported on a 5-point Likert scale of Fully Reproducible Partial, Not reproduced (large deviation / failed run), and Unable to attempt/pending whilst also reporting the exact percentage gap between reported and reproduced results. Stage 5 – Data Extraction & Audit. Papers assigned O1 or O2 triggered data extraction utilizing Sections 4–8 of the reproduction and data extraction Form, capturing neural/symbolic design details, datasets, compute budget, metrics, and evaluation protocol. Upon completion of the reproduction work, one annotator later reviewed 100 % of O1/O2 papers and a stratified sample of O3/O4/O5 papers. Technical Infrastructure, Limitations, and Data Management All reproductions were executed on the University of Mary- land Zaratan high-performance computing cluster under its default quality-of-service limits. Jobs without GPUs ran in the standardpartition on up to a single 128-core Zen 3 node with 512 GiB RAM (7-day wall-clock cap). GPU jobs used the GPUpartition, each limited to one full GPU node consisting of either 4 × NVIDIA A100 (40 GiB each, 128 Zen 3 CPU cores, 512 GiB RAM) or 4 × NVIDIA H100 (80 GiB each, 96 Sapphire Rapids cores, 512 GiB RAM), also with a 7-day limit. Software stacks were loaded per job via the cluster’s modulesystem. Author-supplied Dockerfiles were converted to Apptainer images usingdocker2singularity, and when conversion failed, the build was completed on a local machine. Limitations The primary construct of this paper is repro- ducibility of reruns of same-artifacts to determine whether an independent team can obtain the reported result using the artifacts released by the authors. Under this construct, trained Other missing 25 (5.5%) Incomplete codebase 67 (14.7%) Environment issues 79 (17.4%) Missing instructions 25 (5.5%) Missing data 36 (7.9%) Fully reproducible 48 (10.5%) Partial (above threshold) 37 (8.1%) Has all artifacts but failed 7 (1.5%) Missing code 42 (9.2%) Missing model checkpoints / weights 89 (19.6%) Attempted n = 455 Successful reruns and failure modes within the attempted set Figure 2: Rerun outcomes for the attempted same-artifact audit set (n = 455). checkpoints were treated as indispensable when the published evaluation depended on a fixed trained model state. We did not substitute retraining in the primary analysis because retraining changes the experimental object and introduces additional stochastic, software, and platform variance. However, some papers without released checkpoints may remain reproducible in principle via retraining; we therefore interpret missing- checkpoint cases as non-rerunnable under this protocol rather than as proof that the underlying method is scientifically irre- producible in every broader sense. This framing is consistent with ACM’s current artifact terminology, under which repro- ducibility means an independent group obtaining the same result using the authors’ own artifacts, with agreement judged within an acceptable tolerance rather than exact identity. 5 Additionally, although annotators could patch environment files and author evaluation harnesses when absent from the original release, severe lack of documentation for code and environments, together with architecture-level defects intrin- sic to the framework itself (for example, a fresh clone that crashes with tensor shape mismatches or NaN losses despite the original environment), remained failure conditions under the protocol. Data Management & Transparency All repositories, weights, patched scripts, environment manifests, and execu- tion logs required for reproduction of examined manuscripts 5 https://w.acm.org/publications/policies/artifact-review- and-badging-current remain on the University of Maryland Zaratan project scratch file system under institutional retention policy. Summary statistics are mirrored in the public extraction sheet 6 . The complete audit trail for this systematic reproduction review (including per-paper reproduction logs, annotator tagging logs, and the reproduction protocol and supporting documen- tation) is archived in a long-term online repository 7 , together with the scripts required to regenerate every reported aggre- gate and figure. We additionally publish the raw outcomes via an interactive webpage 8 to enable paper-level inspection and independent recomputation of the headline counts reported in the manuscript. Results System-Level Reproducibility Figure 2 reports outcomes for the attempted rerun set only (n = 455). Within this set, 48 studies (10.55 %) were fully reproduced and 37 (8.13 %) were partially reproduced, yield- ing 85 successful reruns overall (18.68 %). Failures within the attempted set comprised 321 papers (70.55 %) blocked by missing non-code artifacts, 42 (9.23 %) blocked by unavail- able or unusable code repositories, and 7 (1.54 %) that had the nominal artifact set but failed because of environment or 6 NSAI reproduction and data-extraction results 7 [https://doi.org/10.5281/zenodo.21779234] 8 interactive results browser 1998 ... 2014 ... 201720182019202020212022202320242025 Publication year 0 20 40 60 80 100 120 140 160 Number of papers Rep: 0 Not: 1 (100.0%) Rep: 0 Not: 1 (100.0%) Rep: 0 Not: 4 (100.0%) Rep: 0 Not: 2 (100.0%) Rep: 1 (11.1%) Not: 8 (88.9%) Rep: 3 (10.3%) Not: 26 (89.7%) Rep: 5 (16.7%) Not: 25 (83.3%) Rep: 12 (18.2%) Not: 54 (81.8%) Rep: 19 (22.1%) Not: 67 (77.9%) Rep: 25 (18.1%) Not: 113 (81.9%) Rep: 20 (22.5%) Not: 69 (77.5%) Timeline not to scale Attempted Rerun Outcomes by Publication Year (Reproduced & Included vs Not Successfully Reproduced) Reproduced (full or partial, included) Not successfully reproduced Figure 3: Reproduction outcomes over time for the attempted NSAI papers. Stacked bars show, for each publication year, the number of papers that were successfully reproduced (full or partial) versus not reproduced under the study protocol, illustrating the growth of the field alongside a reproduction rate that shows no sustained upward trend. code defects. The 61 records that failed late full-text eligi- bility checks are reported separately in Figure 1 and are not counted as rerun outcomes. The rerun success rate is therefore 85/455 = 18.68%(Wilson 95% CI 15.4–22.5). Relative to the eligible corpus (n = 1,304), the rate is85/1304 = 6.52% (Wilson 95% CI 5.3–8.0). The primary obstacle to success- ful same-artifact reruns was incomplete artifact availability. Within the attempted set, 321 papers were blocked by missing non-code artifacts, and 42 by missing or unusable code repos- itories. A residual 7 studies shared the nominal artifact set yet still failed because of environment or code defects. By con- trast, when code, data, and weights were all available, reruns succeeded in 85 of 92 cases (92.4 %, Wilson 95% CI 85.1– 96.3). Missing artifacts concentrated in three components. Absent model checkpoints or weights blocked 89 attempts (19.56 % of attempted reruns), environment or specification problems blocked 79 (17.36 %), and incomplete codebases blocked 67 (14.73 %). These three categories account for 235 of 321 missing-artifact failures (73.2 %), leaving 86 failures distributed across missing datasets (36,7.91 %), missing or incomplete documentation (25,5.49 %), and other missing items (25,5.49 %). The skew implies that most failures arose from predictable artifact-release gaps. Year-Wise Reproducibility Patterns Figure 3 tracks NSAI reproduction outcomes by publication year and shows a steep rise in paper volume, from single digits before 2019 to138publications in 2024 and89in the partial-year 2025 cohort. From 2019 onward the annual full- or-partial rate ranges from10.3 %(2020) to22.1 %(2023), with intermediate values of11.1 %(2019),16.7 %(2021), 18.2 %(2022), and18.1 %(2024). Because the 2025 cohort covers only January–May, its denominator is still evolving; we therefore treat the22.5 %figure as provisional and exclude 2025 when describing the trend. The years before 2019 contributed8attempted reruns in total, none of which were reproduced, and we exclude them as well. Within the 2019 to 2024 window, the proportion of studies reproduced (fully or partially) remains in a narrow band of roughly10 %to 22 %of the yearly output, and the stacked-bar profiles reveal 020406080100120 Number of papers KR Other preprints/tech reports Sage journals Frontiers journals Nature journals UAI Semantic Web Publisher platforms (Science Direct/PubMed/IOS) Robotics/Multi-agent NeSy Domain-specific journals (Engineering/Bio) KDD/WebConf/IR Elsevier journals CVPR/ICCV/ECCV/WACV (vision) ACM conferences/journals ICML Springer journals/proceedings IJCAI Other conferences (CS) Workshops (CEUR/PMLR) Other journals (CS) ICLR AAAI family NeurIPS IEEE journals/proceedings ACL/EMNLP family arXiv Rep: 2 (100.0%) Not: 2 (100.0%) Not: 2 (100.0%) Rep: 1 (33.3%) Not: 2 (66.7%) Rep: 2 (66.7%) Not: 1 (33.3%) Rep: 2 (66.7%) Not: 1 (33.3%) Not: 4 (100.0%) Rep: 2 (40.0%) Not: 3 (60.0%) Not: 5 (100.0%) Rep: 2 (28.6%) Not: 5 (71.4%) Rep: 1 (12.5%) Not: 7 (87.5%) Rep: 2 (22.2%) Not: 7 (77.8%) Rep: 1 (10.0%) Not: 9 (90.0%) Rep: 1 (7.1%) Not: 13 (92.9%) Rep: 4 (26.7%) Not: 11 (73.3%) Rep: 7 (46.7%) Not: 8 (53.3%) Rep: 3 (20.0%) Not: 12 (80.0%) Rep: 4 (25.0%) Not: 12 (75.0%) Rep: 1 (5.9%) Not: 16 (94.1%) Rep: 2 (11.8%) Not: 15 (88.2%) Rep: 2 (10.5%) Not: 17 (89.5%) Rep: 5 (23.8%) Not: 16 (76.2%) Rep: 5 (21.7%) Not: 18 (78.3%) Rep: 5 (17.9%) Not: 23 (82.1%) Rep: 4 (12.5%) Not: 28 (87.5%) Rep: 11 (20.0%) Not: 44 (80.0%) Rep: 16 (15.2%) Not: 89 (84.8%) Attempted Rerun Outcomes by Venue (Reproduced & Included vs Not Successfully Reproduced) Reproduced (full or partial, included) Not successfully reproduced Figure 4: Reproduction outcomes by venue group for the attempted NSAI papers. Stacked bars show, for each venue family, the number of papers that were successfully reproduced (full or partial) versus not reproduced. Venues are ordered by number of attempted reruns. no sustained upward trajectory. In practical terms, the field is publishing substantially more work each year, but the likelihood that any given paper can be reproduced under our protocol has not materially improved, suggesting that the surge in publications has not, by itself, improved real-world reproducibility under our protocol. Venue-Level Impacts on Reproducibility Figure 4 compares reproduction outcomes across twenty- seven venue families and shows no detectable advantage for journals, conferences, or preprint servers at the class level (χ 2 = 2.26,df = 2,p = .32). Large outlets illus- trate this point as arXiv achieves 16/105 successes (15.2 %), ACL/EMNLP conference proceedings 11/55 (20.0 %), IEEE journals 4/32 (12.5 %), and NeurIPS 5/28 (17.9 %). Compara- ble rates are observed for AAAI family journals/conferences (21.7 %) and for ICLR (23.8 %) despite differing review mod- els, while domain-specific journals, vision conferences, and publisher platforms all cluster in the low-to-mid-teens. The only apparent outlier is ICML at 7/15 (46.7 %) however, its small denominator limits generality. Aggregating by broad class, conferences reproduce at 21.1 % (53/251, Wilson 95% CI 16.5–26.6), journals at 16.5 % (16/97, CI 10.4–25.1), and preprints at 15.0 % (16/107, CI 9.4–22.9). The three intervals overlap substantially. Publication venue alone is not a reliable predictor of NSAI reproducibility. Artifact completeness, not outlet type, is what distinguishes the attempts that succeeded from those that did not. Failure modes behind missing artifacts Figure 5 assigns each of then = 363papers that were non-reproducible due to incomplete artifact provision to a Dead link or artifact removed Build or install failure (deps not pinned) Underspecified setup (splits/seeds/hparams) Restricted access (login/approval required) Missing or unclear instructions Non-portable code (hard-coded paths) Other cause Files missing with no reason provided Missing Artefact Cause (why it was missing) Code/repo Data Docs/instructions Env specs (reqs, Docker) Eval assets KB/rules Model weights Other Preprocessing Missing Artefact Type (what was missing) 276881642636 6651152122 1146189 6133132111 2115 11 661121279 9 1123 Missing artefact type × cause (V2, n=363 papers; multi-label) 0 10 20 30 40 50 60 70 Count (papers) Figure 5: Rows indicate what was missing (e.g., data, model weights, environment specifications, documentation), while columns indicate why it was missing (e.g., dead links, restricted access, an under-specified setup, or files absent with no explanation) missing artifact type and a primary cause. The largest cell is missing model weights with files missing with no reason provided (79 papers), and the second is missing environment specifications linked to build or install failure from unpinned dependencies (61 papers). For model weights, restricted access and dead links are rare (1 and 6 papers, respectively), so the bottleneck is usually not permissioning but non-release or decay, and the claim that a repository enables rerunning the reported pipeline is falsified by inspection in the majority of weight-missing cases. Environment failures concentrate on dependency drift, which is consistent with repositories that omit a locked descriptor or ship one that no longer resolves under current tooling. Code and repository issues are also common, with files missing with no reason provided (36) and dead links or removed artifacts (27), plus a substantial remainder attributed to other causes (26), so “code available” is an unreliable proxy for an executable codebase. Data failures split between restricted access (11) and unexplained absence (22), while documentation failures cluster in missing or unclear instructions (18) and build failures (14). Citation impact vs reproducibility Figure 6 demonstrates whether citation impact is a use- ful proxy for practical reproducibility by comparing DOI- resolved citation counts across the same reproduction outcome and exclusion buckets shown in figures 2 and 5 above. Ci- tation counts were obtained by DOI lookup for 427 of the 455attempted records. The remaining 28 lacked a resolvable DOI and are excluded here. The distribution of citations from the remaining 427 papers, as shown in Figure 6, indicates substantial overlap across all buckets, with a large mass of zero and low citation papers in every outcome and a long right tail that inflates the average number of citations per bucket. The fully reproducible bucket has the highest mean citation count at14.1but a median of2, and comparably high means occur in dominant non-reproducible buckets such as missing model checkpoints or weights with mean10.0, median1, max 234and environment or specification issues with mean10.5, median2, max233, showing that incomplete artifact release shows no clear relationship with number of citations that a paper may receive. 0131030100300 Number of citations (log scale) Fully reproducible (n=44) Partial (above threshold) (n=36) Has all artifacts but failed (n=7) Missing code (n=37) Missing model checkpoints / weights (n=85) Missing data (n=36) Missing instructions (n=22) Environment issues (n=73) Incomplete codebase (n=62) Other missing (n=25) Attempted-rerun outcome Citations by attempted-rerun outcome (n = 427 papers with citation data) 5 10 15 20 25 30 Papers at same citation count (in bucket) Figure 6: Citation counts by reproduction outcome/exclusion reason for NSAI papers with available citation data (n = 427) shown on a log scale. The red marker indicates the mean citation count for each bucket. Answers to Research Questions RQ1 – RQ3 For RQ1 [proportion of NSAI papers that can be reproduced], within the attempted rerun set (n = 455), 48 studies met the strict full-reproduction criterion (10.55 %, Wilson 95% CI 8.1–13.7) and a further 37 met the partial criterion (8.13 %). The full-or-partial composite is 85/455 (18.68 %). Relative to the full eligible corpus (n = 1304), these fall to 3.68% (full; Wilson 95% CI 2.8–4.8) and 6.52% (full or partial; Wilson 95% CI 5.3–8.0). For RQ2 [effect of artifact completeness on reproduction success], artifact completeness is the dominant factor. When code, data, and weights were all present, repro- duction succeeded in 85 of 92 cases (92.4%). When artifacts were incomplete, 321 of 455 reruns (70.55%) were blocked, with missing model weights (n = 89), environment issues (n = 79), and incomplete codebases (n = 67) accounting for 73.2% of those failures. For RQ3 [variation in outcomes by publication year and venue family], neither publication year nor venue predicts reproducibility. Success rates remained flat between 10.3% and 22.1% from 2019 to 2024 despite rapid growth in paper volume, and large outlets cluster in a similar narrow band, with arXiv at 15.2%, ACL/EMNLP at 20.0%, IEEE at 12.5%, and NeurIPS at 17.9%, and aggregate class rates of 21.1% for conferences, 16.5% for journals, and 15.0% for preprints (χ 2 = 2.26, df = 2, p = .32) Discussion Summary of Principal Findings in Context Every study in the attempted set (n= 455) carried a verifiable public code artifact at screening. Fewer than one in five could be rerun. Trends and Patterns Despite adopting an audit protocol that permits limited depen- dency updates, path corrections, and lightweight evaluation harnesses to be supplied by auditors, the proportion of NSAI papers that reproduced remained stubbornly flat, oscillating between10.3 %and22.1 %per year from 2019 through 2024. Figure 2 attributes most failures to missing non-code artifacts (70.55 %) and, to a lesser extent, to absent code (9.23 %), yet when the full range of artifacts required for reproduction is present, reproduction succeeds in92.4 %of cases. Perhaps most striking is the insignificance of venues as a determinant for whether a manuscript will provide the resources required to reproduce the work. Conferences, journals, and preprints cluster at21.1 %,16.5 %, and15.0 %respectively, and the three confidence intervals overlap (Figure 4). The evidence points to artifact availability, rather than venue type or method- ological novelty, as the main bottleneck to reproducibility in current NSAI research. The analysis, therefore, shifts the conversation from improving experimental technique to en- forcing comprehensive artifact release across all publication outlets. “Dead” Codebase Links Despite passing the initial NSAI relevance screen and subse- quent repository-identification step, 42 of the in-scope papers ultimately fell into the Missing Code category because the referenced repository was dead, private, empty, unrelated, or otherwise unusable at audit time. In most instances, the hyperlink supplied in the manuscript was (i) dead or resolv- ing to a private or removed repository, (i) redirected to a project homepage that described the system but held no source files, (i) pointed to a repository that never held any source code, or (iv) led to code that was unrelated or insufficient to implement the reported method. These false-positive disclo- sures illustrate that nominal compliance with “code available” guidelines is not enough, and that persistent, content-verified repositories and explicit version tags are essential if claims of openness are to translate into practical reproducibility. Reproducibility requirements for the domain of Neuro-Symbolic AI as a science Practical implications. Systematic examination of the AAAI Author Reproducibility Checklist 9 , the NeurIPS Paper- Checklist Guidelines 10 , the IJCAI Reproducibility Guide- lines 11 , the NeurIPS report (Pineau et al. 2021), and the AI Magazine survey of reproducibility barriers and drivers (Sem- melrock et al. 2025) shows that the five documents converge on a common baseline of artifacts that an empirical paper must provide: (i) runnable source code; (i) the exact datasets or immutable links to them; (i) pre-trained checkpoints; (iv) a single command or script that reproduces the reported metrics (an evaluation harness) (v) a machine-readable environment file (environment.yml,requirements.txt, Dock- erfile, etc.) and (vi) documentation in the form of instructions 9 https://aaai.org/conference/aaai/aaai-26/reproducibility- checklist/ 10 https://neurips.c/public/guides/PaperChecklist 11 https://w.ijcai.org/reproducibility that integrate items (i)–(v) into one executable workflow. All checklists emphasize persistent identifiers, explicit version locking, and permissive licensing. Policy levers Existing venue initiatives fall into two cate- gories. ACM artifact badges 12 include an external review of the repository, whereas the AAAI, NeurIPS, and IJCAI checklists are self-attested by the authors. Each submission to a peer-reviewed conference, journal, and even preprint venues should undergo an automated repository audit before peer review, confirming that all six artifacts are present and executable. Only 7 of 455 attempts failed with a complete artifact set, so such a check would address 363 of the 370 failures we observed. Future directions for NSAI Research Neuro-Symbolic models integrate multiple components in complicated systems to deliver functional systems, so a single missing artifact can invalidate the entire pipeline. An absent checkpoint severs logic bindings, and undocumented rule sets could compromise evaluation and as such, authors must archive the complete artifact bundle (codebase, datasets, model weights, evaluation scripts, environment files and documentation) in a DOI-minting repository (e.g., Zenodo, HuggingFace- Hub, Figshare, OSF, or Mendeley Data). 13 Computer science research is only science when its results withstand independent execution. A manuscript and a transient GitHub link do not satisfy this condition. Requiring the six-item artifact bundle, bound by immutable URIs for all publication venues (including pre-publication), is the most immediate, low-cost intervention available to the community. The good news is that the barrier to improvement is low. The included papers from this study demonstrate that full reproducibility is achievable at any publication venue when authors commit to a complete artifact bundle from the outset. The community can act now by depositing code, data, model weights, environment files, and a single-command evaluation script in a DOI-minting repository such as Zenodo or HuggingFace-Hub at submission time, not as an afterthought. Venues can reinforce this by requiring automated artifact checks before peer review begins, a low- cost intervention relative to the effort wasted reproducing or discarding non-reproducible work. Case Studies Case Study 1: Scallop-A Badge-Awarded Benchmark for Reproducibility The Scallop PLDI 2023 paper (Li, Huang, and Naik 2023) earned both the ACM Artifacts Available and Artifacts Evaluated—Reusable badges, and its release exemplifies full-stack reproducibility. A Zenodo snapshot (DOI 7804200) freezes the exact commit, dataset splits, pretrained weights, and SHA-256 hashes. Docker and Conda manifests plus a Rust nightly lockfile reconstitute the software stack, while a single script (run_all.sh) rebuilds, trains, and evaluates the eight-task benchmark. GitHub CI recompiles the core library, runs unit tests, and executes a 12 https://w.acm.org/publications/policies/artifact-review- and-badging-current 13 These services issue persistent identifiers, maintain long-term storage, and provide metadata suitable for citation. smoke benchmark on every push, preventing configuration drift. Our audit reproduced all eight tasks on a single A100 GPU with a median absolute deviation≤3 % from the reported metrics and required no manual intervention. Scallop, therefore, is a fantastic demonstration that badge-level artifact curation and automated validation can turn a complex NSAI system into a reliably re-runnable research object. Case Study 2: LogiCity-Conference-Level Reproducibility without Formal Badging Unlike Scallop, LogiCity (Li et al. 2024) earned no external badge, yet its NeurIPS 2024 Datasets and Benchmarks package replicated on the first attempt. The authors froze code, data, and checkpoints in a version-tagged release, shipped Docker and Conda manifests, and wired a one-command launcher that rebuilds the simula- tor, trains agents, and scores results. Continuous-integration runs the full smoke benchmark on every commit, and SHA- 256 checks ensure downloads match the snapshot. On our A100 test node, the Safe-Path-Following and Visual-Action- Prediction tasks reproduced within±2 p of the reported scores and retained the original baseline ordering. LogiCity therefore demonstrates that rigorous curation and automated checks can deliver badge-level reproducibility at a premier conference—even when no formal badging program exists. Case Study 3: MARS-Preprint-Level Reproducibility through Curated artifacts The Mechanism-of-Action Re- trieval System (MARS) (DeLong et al. 2024) appears only as an arXiv preprint posted in March 2024, yet its artifact package met every criterion for a gold standard reproducible package. A version-tagged GitHub repository provides Docker and Conda descriptors, immutable data archives, and pre- trained checkpoints, and a singlerun.shscript rebuilds the biomedical-graph pipeline, trains the models, and computes evaluation metrics. Re-execution on one NVIDIA A100 GPU reproduced the MoA-Net benchmark with MRR 0.315 versus the reported 0.318 and Hits@10 0.672 versus 0.685 (absolute deviation≤2 percentage points), preserving the published ranking of baselines. All code, data, and weights are protected by SHA-256 hashes, permitting table verification without retraining. Although the release has not yet undergone an external artifact audit, this example indicates that a fully versioned, one-command artifact stack can deliver robust reproducibility irrespective of publication venue. Cross-Case Comparison and Venue Implications The three case studies span the principal publication strata in computer science, as a badge-audited archival proceedings (Scallop), a top-tier conference track without external certifica- tion (LogiCity), and an open-access arXiv preprint (MARS), yet each enabled low-effort, reliable replication by adhering to the same practical principles. Each artifact package (i) freezes the software stack with Docker or Conda, (i) archives immutable data and pretrained checkpoints under permanent identifiers, (i) offers a one-command script that rebuilds and evaluates the full pipeline, (iv) discloses every configuration parameter, and (v) offers a good level of documentation with instructions on environment build requirements. With these ingredients in place, our audit reproduced all reported metrics with an absolute error no greater than three percentage points for all three frameworks. Scallop achieves this through a formal ACM badge review, LogiCity relies on conference- driven community norms and continuous-integration tests, and MARS depends on author-integrity. These results indicate that enforceable artifact standards, not venue prestige, are the decisive factor in achieving reliable reproducibility. Conclusion This study provides, to the best of our knowledge, the first longitudinal, large-scale assessment of same-artifact rerun reproducibility within neuro-symbolic AI. Across1,365self- identified NSAI records identified by our search,1,304met full-text eligibility and455entered the attempted same- artifact rerun audit and85were fully or partially reproduced. Failure analysis shows that reproducibility typically collapses when essential artifacts such as full codebases, datasets, pretrained model weights, evaluation harness scripts, environ- mental files or documentation are absent at release. Computer science research attains scientific legitimacy only when inde- pendent investigators can verify its empirical claims. To that end, we recommend that all empirical submissions, regard- less of venue, be required to pass an automated artifact audit prior to peer review. The audit should confirm the availability of six items including (i) full source code, (i) immutable datasets, (i) pretrained model checkpoints, (iv) an executable evaluation script, (v) a machine-readable environment descrip- tor, and (vi) concise documentation sufficient to invoke the pipeline. Persistent DOI-minting repositories such as Zenodo, HuggingFace-Hub, Figshare, OSF, or Mendeley Data already provide the necessary infrastructure to host these artifacts at scale and with minimal cost to authors. Adopting this minimal standard would align computer science publication practice with the norms of experimental science and render future advances in Neuro-Symbolic AI transparent, verifiable, and readily extensible. The community must therefore institution- alize immutable artifact disclosure, requiring and archiving scientific works prior to publication, and thereby ensure that progress is measurable and cumulative by lowering the barrier for full pipeline reproduction of Neuro-Symbolic systems. References 2020. Research, reuse, repeat. Nat. Mach. Intell., 2(12): 729–729. Association for Computing Machinery. 2020. Artifact Review and Badging, Version 1.1. https://w.acm.org/publications/ policies/artifact-review-and-badging-current. Baker, M. 2016. 1,500 scientists lift the lid on reproducibility. Nature, 533(7604): 452–454. Belz, A.; and Thomson, C. 2023. The 2023 ReproNLP Shared Task on Reproducibility of Evaluations in NLP: Overview and Results. In Belz, A.; Popović, M.; Reiter, E.; Thomson, C.; and Sedoc, J., eds., Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems, 35–48. Varna, Bulgaria: INCOMA Ltd., Shoumen, Bulgaria. DeLong, L. N.; Gadiya, Y.; Galdi, P.; and Fleuriot, J. D. 2024. Mars: A neurosymbolic approach for interpretable drug discovery. arXiv. Ferrari Dacrema, M.; Boglio, S.; Cremonesi, P.; and Jan- nach, D. 2021. A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research. ACM Trans. Inf. Syst., 39(2). Henderson, P.; Islam, R.; Bachman, P.; Pineau, J.; Precup, D.; and Meger, D. 2018. Deep reinforcement learning that matters. In Proceedings of the Thirty-Second AAAI Con- ference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial In- telligence, AAAI’18/IAAI’18/EAAI’18. AAAI Press. ISBN 978-1-57735-800-8. Hensel, W. M. 2020. Double trouble? The communication dimension of the reproducibility crisis in experimental psy- chology and neuroscience. Eur. J. Philos. Sci., 10(3). Ioannidis, J. P. A. 2005. Why most published research findings are false. PLoS Med., 2(8): e124. Li, B.; Li, Z.; Du, Q.; Luo, J.; Wang, W.; Xie, Y.; Stepputtis, S.; Wang, C.; Katia, P. S.; Ravikumar, P. K.; Gray, A. G.; Si, X.; and Scherer, S. 2024. LogiCity: Advancing Neuro-Symbolic AI with Abstract Urban Simulation. Neurips. Li, Z.; Huang, J.; and Naik, M. 2023. Scallop: A language for neurosymbolic programming. ACM. López-Nicolás, R.; López-López, J. A.; Rubio-Aparicio, M.; and Sánchez-Meca, J. 2022. A meta-review of transparency and reproducibility-related reporting practices in published meta-analyses on clinical psychological interventions (2000- 2020). Behav. Res. Methods, 54(1): 334–349. McGowan, J.; Sampson, M.; Salzwedel, D. M.; Cogo, E.; Foerster, V.; and Lefebvre, C. 2016. PRESS peer review of electronic search strategies: 2015 guideline statement. J. Clin. Epidemiol., 75: 40–46. Miłkowski, M.; Hensel, W. M.; and Hohol, M. 2018. Repli- cability or reproducibility? On the replication crisis in com- putational neuroscience and sharing only relevant detail. J. Comput. Neurosci., 45(3): 163–172. Open Science Collaboration. 2015. PSYCHOLOGY. Esti- mating the reproducibility of psychological science. Science, 349(6251): aac4716. Pawlik, M.; Hütter, T.; Kocher, D.; Mann, W.; and Augsten, N. 2019. A Link is not Enough – Reproducibility of Data. Datenbank-Spektrum, 19. Pineau, J.; Vincent-Lamarre, P.; Sinha, K.; Larivière, V.; Beygelzimer, A.; d’Alché Buc, F.; Fox, E.; and Larochelle, H. 2021. Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program). J. Mach. Learn. Res., 22(1). Rubin, M. 2022. The costs of HARKing. Br. J. Philos. Sci., 73(2): 535–560. Semmelrock, H.; Ross-Hellauer, T.; Kopeinik, S.; Theiler, D.; Haberl, A.; Thalmann, S.; and Kowald, D. 2025. Re- producibility in machine-learning-based research: Overview, barriers, and drivers. AI Mag., 46(2). Vanderdonckt, J.; and Vatavu, R.-D. 2025. Context is Key for Reproducibility of Empirical Studies in Human-Computer Interaction. In Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, ACM REP ’25, 41–50. New York, NY, USA: Association for Computing Machinery. ISBN 9798400719585. Appendix Data Availability. The supplementary materi- als supporting this study are deposited in the [https://doi.org/10.5281/zenodo.21779234]. The archive contains the paper-level reproduction and data-extraction records, missing-artifact annotations, audit protocols, anno- tator documentation, analysis scripts, generated aggregate results, and an offline copy of the results browser. A worked example demonstrating application of the data-extraction form is available through the NSAI-2025 data-extraction worked example. Paper-level outcomes can also be inspected through the interactive results browser. Broader impacts This work can positively affect the machine-learning com- munity by encouraging more transparent, verifiable, and cumulative Neuro-Symbolic AI research. At the same time, stronger artifact-release requirements may increase burdens on authors, especially under-resourced groups, and may create pressure to release sensitive, proprietary, or legally restricted artifacts. These risks can be mitigated through anonymized review artifacts, persistent metadata, gated access, and docu- mented exemptions where full public release is not ethically or legally possible. Final List of Reproduced Works A full list of the papers that were able to be reproduced for this study can be found here: NSAI-2025-reproducibility-study. CitationTitleDescription Acharya et al. (2025) Codebase Neurosymbolic AI for Travel De- mand Prediction: Integrating Deci- sion Tree Rules into Neural Net- works This paper introduces a neuro-symbolic AI framework that blends decision tree rules with neural networks to predict travel demand. By combining interpretability with deep learning power, it achieves more accurate and explainable results for transportation planning and resource optimization. Ishay et al. (2023) Codebase Leveraging Large Language Models to Generate Answer Set Programs They use LLMs to work through logic puzzle solving in a step-by-step manner. Murali et al. (2019) Codebase Composing Neural Learning and Symbolic Reasoning with an Appli- cation to Visual Discrimination propose a compositional neurosymbolic framework that com- bines a neural network to detect objects and relationships with a symbolic learner that finds interpretable discriminators. Ahmed et al. (2022) Codebase Semantic probabilistic layers for neuro-symbolic learning The paper presents Semantic Probabilistic Layers (SPLs), a neural network module that ensures structured-output predic- tions are always consistent with logical constraints, enabling accurate and tractable neuro-symbolic learning. SPLs modu- larly combine probabilistic inference and logical reasoning, outperforming previous methods in tasks requiring strict output validity. Ahmed et al. (2022) Codebase Neuro-symbolic entropy regulariza- tion This paper presents neuro-symbolic entropy regularization, a unified framework that combines entropy regularization and neuro-symbolic learning for structured prediction tasks. By constraining entropy minimization to outputs that form valid structures (as defined by logical circuits), the approach yields models that are both more accurate and more likely to produce valid predictions, demonstrated across semi-supervised and fully-supervised experiments. Akl (2024) Codebase NeSy is alive and well: A LLM- driven symbolic approach for better code comment data generation and classification This paper presents a neuro-symbolic workflow combining semantic rule–based decomposition with an LLM to generate controlled synthetic data for C-code comment classification. Empirical results show that this augmentation improves ML models (Voting Classifier, Random Forest, MLP) performance measured in F1 score. Alam et al. (2024) Codebase A Walsh Hadamard Derived Linear Vector Symbolic Architecture The paper introduces the Hadamard-derived Linear Binding (HLB), a novel vector symbolic architecture that leverages properties of the Walsh-Hadamard transform for efficient, numerically stable vector binding in neuro-symbolic AI. HLB achieves state-of-the-art performance on both classical VSA benchmarks and selected deep learning tasks, outperforming previous VSA methods in terms of computational complexity, accuracy, and differentiability for modern neural architectures. Daniele and Luciano (2022) Codebase Knowledge Enhanced Neural Net- works for relational domains Paper extends knowledge enhanced NNs to handle relational data and shows that stacking multiple KE layers deals with rule dependencies, achieving better accuracy than baseline NNS on Citeseer citation network classification while also being faster than baseline Semantic Based Regularization (SBR) and Relational Neural Machines (RNM) Ibrahimzada et al. (2024) Codebase AlphaTrans: A Neuro-Symbolic Compositional Approach for Repository-Level Code Translation and Validation Solves the task of code translation from one programming language to another. It does so using neural symbolic frame- work that breaks down the source code into fragments and also utilized the test code to ensure code was properly translated. Alon et al. (2022) Codebase Neuro-symbolic language modeling with automaton-augmented retrieval The paper presents RETOMATON, a neuro-symbolic system that approximates costly nearest-neighbor datastore searches in retrieval-based language models using automaton states and pointer links, enabling substantial speed-ups while maintaining or improving perplexity. CitationTitleDescription Asai and Muise (2020) Codebase Learning neural-symbolic descrip- tive planning models via cube- space priors: The voyage home (to STRIPS) neuro-symbolic architecture is trained end-to-end to produce a succinct and effective discrete state transition model from images alone. Aspis et al. (2022) Codebase Embed2sym-scalableneuro- symbolic reasoning via clustered embeddings The paper presents Embed2Sym, a scalable framework that combines neural perception and symbolic reasoning through clustered embeddings, enabling fast training, interpretability, and generalization in tasks that exceed the scalability of prior neuro-symbolic systems. Embed2Sym achieves state-of-the- art results and significantly reduces training time on complex reasoning tasks involving visual and symbolic inputs. Badreddine et al. (2022) Codebase Logic Tensor NetworksThis paper presents Logic Tensor Networks (LTN), a neu- rosymbolic framework that supports querying, learning and reasoning with both rich data and abstract knowledge about the world. LTN introduces a fully differentiable logical language, called Real Logic, whereby the elements of a first-order logic signature are grounded onto data using neural computational graphs and first-order fuzzy logic semantics Bairi et al. (2024) Codebase CodePlan: Repository-Level Coding using LLMs and Planning The paper introduces CodePlan, a framework that treats large- scale repository-level code edits (e.g., API migrations or temporal edits across many files) as a planning problem: it uses an LLM guided by static dependency and impact analysis to generate a chain of edits until the repository satisfies a correctness oracle. They evaluate on C# and Python codebases and show that it outperforms no-planning baselines in build success rate and matching ground truth edits. Barbiero et al. (2023) Codebase Interpretable neural-symbolic con- cept reasoning The paper proposes the Deep Concept Reasoner (DCR), an interpretable concept-based model that uses neural networks to generate fuzzy logic rules from concept embeddings, and then executes those rules on concept truth degrees to make semantically meaningful and differentiable predictions. Barnaby et al. (2023) Codebase ImageEye: Batch Image Processing using Program Synthesis Batch editing of images such as cropping out a desired ob- ject/person in a batch of 100+ images has not been an easy task. The paper describes a neuro symbolic approach to generate programs from user demonstrations that perform such tasks. They show the program can automate 96% of these tasks. Biggio et al. (2021) Codebase Neural symbolic regression that scales The paper presents NeSymReS, a neural symbolic regression model that leverages large-scale pre-training of Transform- ers on procedurally generated equations and data, enabling scalable, efficient, and robust discovery of symbolic equations from input-output data pairs. The approach outperforms tradi- tional and neural symbolic regression methods across diverse evaluation benchmarks. Li et al. (2024) Codebase LogiCity: Advancing Neuro- Symbolic AI with Abstract Urban Simulation LogiCity models diverse urban elements using semantic and spatial concepts, such as IsAmbulance(X) and IsClose(X, Y). These concepts are used to define FOL rules that govern the behavior of various agents. Since the concepts and rules are abstractions, they can be universally applied to cities with any agent compositions, facilitating the instantiation of diverse scenarios Carraro et al. (2024) Codebase Mitigating data sparsity via neuro- symbolic knowledge transfer Paper uses Logic Tensor Networks (LTN) to mitigate data sparsity issues in recommender systems. It combines Matrix Factorization (neural) models with First-Order Logic axioms (symbolic). CitationTitleDescription Chanin and Hunter (2023) Codebase Neuro-symbolic commonsense so- cial reasoning present a novel system for taking social rules of thumb (ROTs) in natural language from the Social Chemistry 101 dataset and converting them to first-order logic where reasoning is performed using a neuro-symbolic theorem prover. Dickens et al. (2024) Codebase Convex and Bilevel Optimization for Neuro-Symbolic Inference and Learning The paper presents a bilevel, gradient-based optimization framework for neural-symbolic (NeSy) learning, introducing a dual block coordinate descent algorithm and smooth formula- tion for efficient and scalable parameter learning, empirically validated on multiple datasets. Chen and Jabbarvand (2024) Codebase Neurosymbolic Repair of Test Flaki- ness Introduces FlakyDoctor, a neuro-symbolic repair system that couples LLM-based patch synthesis with static analysis, com- pilation “stitching,” and targeted validation to repair order- dependent (OD) and implementation-dependent (ID) flaky tests. Evaluated on 873 flaky tests from 243 projects, achiev- ing ~57% OD and 59% ID repair success and producing 79 previously un-fixed patches (19 merged PRs) Chen et al. (2022) Codebase ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversa- tional Finance Question Answering Introduces a new dataset to study chain of numerical reasoning in question-answering. Glanois et al. (2021) Codebase Neuro-Symbolic Hierarchical Rule Induction This paper proposes an efficient and interpretable neuro- symbolic model called Hierarchical Rule Induction (HRI) to solve Inductive Logic Programming (ILP) problems. This model is built from a set of meta-rules organized in a hier- archical structure, and it invents first-order rules by learning embeddings to match facts and predicates. Yin et al. (2024) Codebase MDD-5k: A New Diagnostic Conver- sation Dataset for Mental Disorders Synthesized via Neuro-Symbolic LLM Agents Introduces a new framework to generate a dataset of diagnostic conversations between a patient and a doctor. It also provides a dataset with 5k high-quality long conversations with diagnosis results and treatment options. Pryor et al. (2022) Codebase NeuPSL: Neural Probabilistic Soft Logic Introduce Neural Probabilistic Soft Logic (NeuPSL), a novel neurosymbolic (NeSy) framework that unites state-of-the-art symbolic reasoning with the low-level perception of deep neural networks. To model the boundary between neural and symbolic representations, they propose a family of energy- based models, NeSy Energy-Based Models, and show that they are general enough to include NeuPSL and many other NeSy approaches. Dagan et al. (2023) Codebase Dynamic Planning with a LLMpresents LLM Dynamic Planner (LLM-DP): a neuro-symbolic framework where an LLM works hand-in-hand with a tra- ditional planner to solve an embodied task. Given action- descriptions, LLM-DP solves Alfworld faster and more effi- ciently than a naive LLM ReAct baseline. Daggitt et al. (2024) Codebase Vehicle: Bridging the embedding gap in the verification of neuro- symbolic programs This paper presents Vehicle, a tool designed to bridge the "embedding gap" in the verification of neuro-symbolic pro- grams—programs combining neural (machine learning) and symbolic (traditional code) components. Vehicle provides a unified, dependently-typed language and compiler that en- ables formal specifications for neural components to be in- tegrated across different verification and training backends, demonstrated by verifying a neural network controller for an autonomous car. Li et al. (2022) Codebase Learning Signal Temporal Logic through Neural Network for Inter- pretable Classification design a novel time function and sparse softmax function to improve the soundness and precision of the neural-STL framework. As a result, we can efficiently learn a compact STL formula for the classification of time-series data through off-theshelf gradient-based tools. CitationTitleDescription DeLong et al. (2024) Codebase Mars: A neurosymbolic approach for interpretable drug discovery Mechanism of Action Retrieval System (MARS) is a neu- rosymbolic approach for drug discovery that uses logical rules with learned rule weights. The model has better interpretability through dynamic weight learning from logical rules. Dhanraj and Eliasmith (2025) Codebase Improving Rule-based Reasoning in LLMs via Neurosymbolic Represen- tations introduces a novel neurosymbolic method that improves LLM reasoning by encoding hidden states into neurosymbolic vec- tors, enabling problem-solving within a neurosymbolic vector space Done et al. (2023) Codebase CORRPUS: Code-based Structured Prompting for Neurosymbolic Story Understanding CoRRPUS shows the usefuleness of code-based symbolic representations for enabling LLMs to perofrm better on story reasoning tasks. Dong et al. (2019) Codebase Neural Logic Machinespropose the Neural Logic Machine (NLM), a neural-symbolic architecture for both inductive learning and logic reasoning. NLMs exploit the power of both neural networks—as function approximators, and logic programming—as a symbolic pro- cessor for objects with properties, relations, logic connectives, and quantifiers. Eiter et al. (2023) Codebase A Modular Neurosymbolic Ap- proach for Visual Graph Question Answering This paper presents a modular neuro-symbolic architecture that processes images of graph-structured data (rather than symbolic graphs) by first using optical graph recognition and OCR to extract nodes/edges and labels, then parsing the question and using answer-set programming (ASP) for logical reasoning. This paper also introduces a new VGQA task and dataset (in 3 sets: tiny, small, medium) and establishes a baseline of 73% accuracy. Misino et al. (2022) Codebase VAEL: Bridging Variational Autoen- coders and Probabilistic Logic Pro- gramming This paper presents VAEL, a neuro-symbolic generative model integrating variational autoencoders (VAE) with the reasoning capabilities of probabilistic logic (L) programming. Besides standard latent subsymbolic variables, their model exploits a probabilistic logic program to define a further structured representation, which is used for logical reasoning. Marconato et al. (2024) Codebase BEARS Make Neuro-Symbolic Models Aware of their Reasoning Shortcuts They propose to ensure NeSy models are aware of the semantic ambiguity of the concepts they learn, thus enabling their users to identify and distrust low-quality concepts. Starting from three simple desiderata, they derive bears (BE Aware of Reasoning Shortcuts), an ensembling technique that calibrates the model’s concept-level confidence without compromising prediction accuracy, thus encouraging NeSy architectures to be uncertain about concepts affected by RSs. van Krieken et al. (2025) Codebase Neurosymbolic Diffusion ModelsTo overcome the limitations of the independence assump- tion, this paper introduces neurosymbolic diffusion models (NESYDMS), a new class of NeSy predictors that use dis- crete diffusion to model dependencies between symbols. their approach reuses the independence assumption from NeSy pre- dictors at each step of the diffusion process, enabling scalable learning while capturing symbol dependencies and uncertainty quantification. van Krieken et al. (2022) Codebase A-NeSI: A Scalable Approximate Method for Probabilistic Neurosym- bolic Inference Introduce Approximate Neurosymbolic Inference (A-NESI): a new framework for PNL that uses neural networks for scal- able approximate inference. A-NESI 1) performs approximate inference in polynomial time without changing the semantics of probabilistic logics; 2) is trained using data generated by the background knowledge; 3) can generate symbolic explana- tions of predictions; and 4) can guarantee the satisfaction of logical constraints at test time, which is vital in safety-critical applications. CitationTitleDescription Endo et al. (2023) Codebase Motion Question Answering via Modular Motion Programs Proposes a neuro symbolic framework to reason about motion sequences in human actions via symbolic reasoning and mod- ular design. They say that it grounds motion through learning motion concepts, attribute neural operators and temporal rela- tions. The task they chose is human motion QA for evaluate their NSPose method. Karabulut et al. (2025) Codebase Neurosymbolic Association Rule Mining from Tabular Data Aerial+ trains an autoencoder on tabular data, then extracts association rules by feeding in test vectors with features set to 1. If the autoencoder reconstructs other features with high probability, it creates a rule. Generates way fewer rules than FP-Growth (2-10x less) with better coverage and runs faster on big datasets, plus makes rule-based classifiers like CORELS train faster without losing accuracy. Evans et al. (2021) Codebase https://w.sciencedirect. com/science/article/pii/ S0004370221000722?via% 3DihubMaking sense of raw input The central contribution of this paper is a neuro-symbolic framework for distilling interpretable theories out of streams of raw, unprocessed sensory experience. Feinman and Lake (2020) Codebase Learning task-general represen- tations with generative neuro- symbolic modeling develop a generative neuro-symbolic (GNS) model of hand- written character concepts that uses the control flow of a probabilistic program, coupled with symbolic stroke primi- tives and a symbolic image renderer, to represent the causal and compositional processes by which characters are formed. Feng et al. (2022) Codebase Neuro-symbolic natural logic with introspective revision for natural lan- guage inference Presents a NeSy framework to integrate natural logic with NNs to do natural language inference. They use RL to help guide this natural logic. Overall this improves the intuitive inference understandability of humans. Gong et al. (2024) Codebase Neuro-Symbolic Embedding for Short and Effective Feature Selec- tion via Autoregressive Generation Proposes a neuro-symbolic framework that models feature selection as an autoregressive token generation problem, learn- ing embeddings for short and effective feature subsets with superior predictive performance compared to classical base- lines. Hakim et al. (2025) Codebase ANSR-DT: An Adaptive Neuro- Symbolic Learning and Reasoning Framework for Digital Twins ANSR-DT presents an adaptive neuro-symbolic framework in- tegrating deep learning (CNN-LSTM) and symbolic reasoning (Prolog) for digital twins. The system generates interpretable insights, learns rules from data, and provides knowledge graph visualizations to support decision-making in industrial scenar- ios. Princis et al. (2024) Codebase Enhancing SQL Query Generation with Neurosymbolic Reasoning Find a way to use language models to generate SQL queries. They introduce a new tool called Xander which helps Ho Fung et al. (2024) Codebase SymbolNet: Neural Symbolic Re- gression with Adaptive Dynamic Pruning SymbolNet combines neural networks with symbolic regres- sion through adaptive pruning to extract interpretable mathe- matical expressions while achieving high model compression. The method uses a custom training loop with threshold-based pruning to identify and eliminate less important network connections, enabling recovery of human-readable symbolic formulas from trained models without sacrificing predictive accuracy. Howard et al. (2023) Codebase Neurocomparatives:Neuro- symbolic distillation of comparative knowledge This paper presents NeuroComparatives, a dataset of 8.8M comparative commonsense statements (e.g., "cats are typically smaller than dogs") generated using constrained beam search with GPT-2 and filtered using a discriminator trained on human annotations. The dataset aims to provide high-validity comparative knowledge for commonsense reasoning tasks. Hu et al. (2023) Codebase A multi-grained self-interpretable symbolic-neural model for single/multi-labeled text classifica- tion propose a Symbolic-Neural model that can learn to explicitly predict class labels of text spans from a constituency tree without requiring any access to spanlevel gold labels CitationTitleDescription Ishay et al. (2024) Codebase Think before You Simulate: Sym- bolic Reasoning to Orchestrate Neu- ral Computation for Counterfactual Question Answering The paper discusses a novel approach to do causal and temporal reasoning in video particularly for counterfactual reasoning. They make use of a causal graph and ASP to coordinate between perception and simulation modules. Maene et al. (2024) Codebase KLay: Accelerating Arithmetic Cir- cuits for Neurosymbolic AI The paper presents KLay, a scalable data structure and layeriza- tion algorithm that accelerates arithmetic circuit computation in neurosymbolic AI systems, enabling efficient forward and backward passes. It demonstrates near-linear runtime scaling across five orders of magnitude, significantly outperforming baseline methods on logic-based inference tasks. Ying et al. (2025) Codebase A Neural Symbolic Model for Space Physics PhyE2E presents an end-to-end transformer-based framework for physics-informed symbolic regression that incorporates dimensional analysis as an inductive bias through oracle- guided divide-and-conquer strategies combined with MCTS refinement. The method achieves state-of-the-art performance on the Feynman equations benchmark, improving symbolic accuracy by 10-29% over existing methods like PySR, uDSR, TPSR, and LaSR. Kricheli et al. (2024) Codebase Error Detection and Constraint Re- covery in Hierarchical Multi-Label Classification without Prior Knowl- edge his paper presents an approach that uses Error Detection Rules (EDR) to learn the failure modes of machine learning models, relaxing the common assumption that hierarchical constraints must exist beforehand. These learned rules are effective at detecting classifier errors and can be leveraged as constraints for Hierarchical Multi-label Classification (HMC), allowing for the recovery of explainable constraints even when they are not provided initially. Ma et al. (2020) Codebase Knowledge-driven Data Construc- tion for Zero-shot Evaluation in Commonsense Question Answering vary the set of language models, training regimes, knowledge sources, and data generation strategies, and measure their impact across tasks. Extending on prior work, we devise and compare four constrained distractor-sampling strategies Katz et al. (2021) Codebase Tunable Neural Encoding of a Sym- bolic Robotic Manipulation Algo- rithm present a neurocomputational controller for robotic manipula- tion based on the recently developed “neural virtual machine” (NVM). Authors program the NVM with a symbolic algorithm that solves blocks-world restacking problems, and execute it in a robotic simulation environment Kelly et al. (2023) Codebase There and back again: extracting for- mal domains for controllable neu- rosymbolic story authoring Demonstrate that languagemodels can be used to author nar- rative planning domainsfrom natural language stories with minimal human intervention. Second, authors explore the re- verse, demonstrating that we can use logical story domains and plans to produce storiesthat respect the narrative commitments of the planne Kohaut et al. (2024) Codebase Probabilistic Mission Design in Neuro-Symbolic Systems Authors propose Probabilistic Mission Design (ProMis), a novel neuro-symbolic approach to navigating UAS within legal frameworks. ProMis is an interpretable and adaptable system architecture that links uncertain geospatial data and noisy perception with declarative, Hybrid Probabilistic Logic Programs (HPLP) to reason over the agent’s state space and its legality Kohaut et al. (2024) Codebase The Constitutional Filterintroduces an approach for Bayesian estimation of agents ex- pected to comply with a human-interpretable neuro-symbolic model we call its Constitution. Hence, autors present the Con- stitutional Filter (CoFi), leading to improved tracking of agents by leveraging expert knowledge, incorporating deep learning architectures, and accounting for environmental uncertainties CitationTitleDescription Kouvaros and Botoeva (2024) Codebase Formal verification of parameterised neural-symbolic multi-agent sys- tems This paper presents VENMAS, a formal verification toolkit for multi-agent systems with neural components, enabling bounded temporal logic verification of parameterized neural- symbolic agents. The work extends existing verification meth- ods to handle systems with an arbitrary number of homoge- neous agents using symbolic abstraction techniques. Kulmanov et al. (2021) Codebase Semantic similarity and machine learning with ontologies provide an overview over the methods that use ontologies to compute similarity and incorporate them in machine learn- ing methods; in particular, the authors outline how semantic similarity measures and ontology embeddings can exploit the background knowledge in ontologies and how ontologies can provide constraints that improve machine learning models. Kulmanov et al. (2024) Codebase Protein function prediction as ap- proximate semantic entailment The Gene Ontology (GO) is a formal, axiomatic theory with over 100,000 axioms that describe the molecular functions, biological processes and cellular locations of proteins in three subontologies. Developed DeepGO-SE, a method that predicts GO functions from protein sequences using a pretrained large language model. DeepGO-SE generates multiple approximate models of GO, and a neural network predicts the truth values of statements about protein functions in these approximate models. Lee et al. (2024) Codebase Diakop: Dialogue-based knowledge- oriented programming for neural- symbolic knowledge base question answering present Dialogue-based Knowledge-oriented Programming system (DiaKoP), a system with a chat interface designed for multi-turn knowledge base question answering (KBQA). DiaKoP enables users to decompose complex questions into multiple simpler follow-up questions and interact with the system to obtain answers. Li et al. (2020) Codebase Closed loop neural-symbolic learn- ing via integrating neural perception, grammar parsing, and symbolic rea- soning This paper proposes a Neural-Grammar-Symbolic (NGS) model that improves upon inefficient reinforcement learn- ing approaches by using a grammar model to bridge neural perception and symbolic reasoning, along with a novel back- search algorithm to learn from incorrect predictions. The experiments, conducted on handwritten formula recognition and visual question answering tasks, demonstrate that this method significantly outperforms reinforcement learning mod- els in terms of performance, convergence speed, and data efficiency. Li et al. (2022) Codebase AdaLoGN: Adaptive Logic Graph Network for Reasoning-Based Ma- chine Reading Comprehension present a neural-symbolic approach which, to predict an answer, passes messages over a graph representing logical relations between text units. Li et al. (2023) Codebase Scallop: A language for neurosym- bolic programming present Scallop, a language which combines the benefits of deep learning and logical reasoning. Scallop enables users to write a wide range of neurosymbolic applications and train them in a data- and compute-efficient manner. It achieves these goals through three key features: 1) a flexible symbolic representation that is based on the relational data model; 2) a declarative logic programming language that is based on Datalog and supports recursion, aggregation, and negation; and 3) a framework for automatic and efficient differentiable reasoning that is based on the theory of provenance semirings. CitationTitleDescription Li et al. (2024) Codebase Relational Programming with Foun- dation Models implement VIEIRA by extending the SCALLOP compiler with a foreign interface that supports foundation models as plugins. We implement plugins for 12 foundation models including GPT, CLIP, and SAM. We evaluate VIEIRA on 9 challenging tasks that span language, vision, and structured and vector databases. Our evaluation shows that programs in VIEIRA are concise, can incorporate modern foundation models, and have comparable or better accuracy than competitive baselines. Murphy et al. (2024) Codebase Autoformalizing Euclidean Geome- try use theorem provers to fill in such diagrammatic information automatically, so that the LLM only needs to autoformalize the explicit textual steps, making it easier for the model. Manginas et al. (2024) Codebase NeSyA: Neurosymbolic Automatahow that symbolic automata can be integrated with neural- based perception, under probabilistic semantics towards an end-to-end differentiable model. Their proposed hybrid model, termed NESYA (Neuro Symbolic Automata) is shown to either scale or perform more accurately than previous NeSy systems in a synthetic benchmark and to provide benefits in terms of generalization compared to purely neural systems in a realworld event recognition task Mejri et al. (2024) Codebase LARS-VSA: A Vector Symbolic Ar- chitecture For Learning with Ab- stract Rules The paper presents LARS-VSA, a neuro-symbolic frame- work leveraging hyperdimensional computing for abstract rule learning with compositional generalization. It introduces a high-dimensional attention mechanism and demonstrates supe- rior generalization, efficiency, and accuracy over contemporary neural and neuro-symbolic baselines on multiple relational reasoning and math tasks. Olausson et al. (2023) Codebase (sic) LINC: A Neurosymbolic Ap- proach for Logical Reasoning by Combining Language Models with First-Order Logic Provers investigate the validity of instead reformulating such tasks as modular neurosymbolic programming, which we call LINC: Logical Inference via Neurosymbolic Computation. In LINC, the LLM acts as a semantic parser, translating premises and conclusions from natural language to expressions in first-order logic. These expressions are then offloaded to an external theo- rem prover, which symbolically performs deductive inference. Post et al. (2024) Codebase Accelerating UMR adoption: Neuro- symbolic conversion from AMR-to- UMR with low supervision The paper proposes a neuro-symbolic method for converting AMR (Abstract Meaning Representation) roles to UMR (Uni- form Meaning Representation) roles by integrating animacy parsing and logic rules with a neural network. The approach addresses non-deterministic role mappings with minimal hu- man supervision, achieving 75.81% accuracy compared to a 62.35% baseline neural network. Premsri and Kordjamshidi (2024) Codebase Neuro-symbolic Training for Rea- soning over Spatial Language This paper presents SpaRTUNQChain, a neuro-symbolic framework that combines BERT embeddings with spatial logic constraints for question-answering over spatial language. The system uses the DomiKnowS framework to enforce com- positional reasoning rules during training, improving accuracy on spatial reasoning benchmarks. Quan et al. (2025) Codebase Peirce: Unifying material and for- mal reasoning via llm-driven neuro- symbolic refinement introduce PEIRCE, a neuro-symbolic framework designed to unify material and formal inference through an iterative conjecture–criticism process. Roig Vilamala et al. (2023) Codebase DeepProbCEP: A neuro-symbolic approach for complex event process- ing in adversarial settings Introduces DeepProbCEP, a differentiable ProbLog + CNN hybrid that trains end-to-end from complex-event labels. CitationTitleDescription Skryagin et al. (2024) Codebase Scalable Neural-Probabilistic An- swer Set Programming This paper introduces Answer Set Networks (ASNs), a novel neural-symbolic solver based on Graph Neural Networks (GNNs) designed to overcome the high computational costs and CPU-bound nature of traditional Answer Set Programming (ASP) solvers. By translating ASP problems into a graph format that leverages GPU parallelization, ASNs outperform existing systems and enable new applications like fine-tuning Large Language Models (LLMs) with logic and solving large-scale drone navigation tasks. Tammet et al. (2023) Codebase An Experimental Pipeline for Au- tomated Reasoning in Natural Lan- guage (Short Paper) The paper presents NLPSolver, an experimental end-to-end pipeline that parses English text with a neural UD parser, converts it into extended first-order logic with defaults and confidences, performs reasoning with a high-performance default logic engine, and converts proofs back into natural language answers and explanations. The authors evaluate the system on toy NLI/QA examples, a subset of HANS, and AllenAI ProofWriter demos, showing strong performance there and highlighting the pipeline as a basis for future hybrid neuro-symbolic systems. Winters et al. (2021) Codebase DeepStochLog: Neural Stochastic Logic Programming This paper introduces DeepStochLog, a novel neural-symbolic framework that integrates neural networks into stochastic defi- nite clause grammars (SDCGs) to define a probability distri- bution over possible derivations. The experimental evaluation shows this approach scales significantly better than methods based on neural probabilistic logic programs and achieves state-of-the-art results on several challenging neural-symbolic tasks. Trinh et al. (2024) Codebase Solving olympiad geometry without human demonstrations AlphaGeometry is a neuro-symbolic system that uses a neu- ral language model, trained from scratch on our large-scale synthetic data, to guide a symbolic deduction engine through infinite branching points in challenging problems. Wu et al. (2024) Codebase LLM Meets Bounded Model Check- ing: Neuro-symbolic Loop Invariant Inference proposes LaM4Inv, a neuro-symbolic framework that combines large language models (Llama-3-8B, GPT-3.5, GPT-4, GPT- 4-Turbo) with bounded model checking (ESBMC + SMT solvers) to automatically infer loop invariants for C programs. It evaluates LaM4Inv on an expanded benchmark of 316 loop- invariant problems, showing large gains in the number of verified loops compared to classical invariant generators and recent LLM-based baselines. Wu and Nakayama (2024) Codebase MILE: Memory-Interactive Learn- ing Engine for Neuro-Symbolic So- lutions to Mathematical Problems MILE is a neuro-symbolic math word problem solver that predicts formulas using a memory-based decoder instead of tree-structured decoding, letting it handle more complex computation graphs. It beats existing methods on Math23K by learning formulas as rules. Kang et al. (2025) Codebase Neuro-Conceptual Artificial Intelli- gence: Integrating OPM with Deep Learning to Enhance Question An- swering Quality introduceNeuro-ConceptualArtificialIntelli- gence(NCAI),aspecializationof the neuro-symbolicAI approach that integrates conceptual modeling usingObject- Process Methodology (OPM) ISO19450:2024with deeplearningtoenhancequestion-answering (QA)quality Yang et al. (2023) Codebase Learning to Solve Constraint Sat- isfaction Problems with Recurrent Transformer Constraint satisfaction problems (CSPs) are about finding values of variables that satisfy the given constraints. We show that Transformer extended with recurrence is a viable approach to learning to solve CSPs in an end-to-end manner, having clear advantages over state-of-the-art methods such as Graph Neural Networks, SATNet, and some neuro-symbolic models. CitationTitleDescription Yang et al. (2023) Codebase Injecting Logical Constraints into Neural Networks via Straight- Through Estimators This paper introduces CL-STE, a method to inject discrete logical constraints into neural networks by representing the constraints as a loss function and using a Straight-Through Estimator (STE) to enable gradient-based optimization. By leveraging GPUs and avoiding heavy symbolic computation, this technique scales significantly better than existing neuro- symbolic methods and allows various architectures, like CNNs and GNNs, to learn from constraints with fewer or no labeled data. Li et al. (2025) Codebase Proving Olympiad Inequalities by Synergizing LLMs and Symbolic Reasoning we introduce a neuro-symbolic tactic generator that synergizes the mathematical intuition learned by LLMs with domain- specific insights encoded by symbolic methods.