Paper deep dive
GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation
Murad Hossen, Tasneem Selim, Gurur Gamgam, Tuga Yousif, Abderrahmane Kasmi, Ikram Aissiou, Mubaraq Onipede, Faran Taimoor Butt, Sanae Zrigui, Rosa Y. G. Paccotacya-Yanque, Ignatius Balayo, Ikram Elhouiti, Hadil Affes, Bijay Adhikari, Sargam Goyal, Muhammad Ibrahim Isah, Mohammad Idrees Bhat, Samuel Kangoni Matia, Peguy Kem-Meka Tiotsop Kadzue, Maha Trabelsi, Emmanuel Owusu, Vinit, Nour Majdoub, Tamiru Alemnew, Islem Rekik
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) have demonstrated strong performance on coding and reasoning benchmarks; however, their ability to solve graph-structured machine learning problems remains largely unexplored. In particular, no benchmark currently evaluates whether LLMs can autonomously solve end-to-end Graph Neural Network (GNN) coding tasks under realistic competition settings. To address this gap, this paper introduces GNN-CB, the first competition-based benchmark for evaluating both humans and LLMs on GNN coding tasks. GNN-CB consists of 18 curated competitions spanning node-, edge-, and graph-level prediction across diverse graph categories, domains, and difficulty tiers. All submissions are evaluated through a unified automated pipeline with hidden test sets and standardized scoring. Human participants solve tasks under controlled competition constraints, while LLMs are evaluated using a frozen zero-shot prompting protocol based on a plan-then-code paradigm with bounded execute-and-repair loops. The benchmark additionally supports both non-agent and autonomous agent-based evaluation within the same protocol. Under our evaluated protocol, LLMs rarely match Human Top performance and show less stable performance across competitions. No single model dominates: a few competitions are won by LLMs, yet humans still hold the top score on most tasks. We release GNN-CB as a living benchmark with automated evaluation infrastructure, dynamic leaderboards, and reproducible execution pipelines. Beyond benchmarking, GNN-CB provides a practice-oriented resource for studying GNN implementation across progressively diverse graph-learning tasks. The benchmark and evaluation framework are publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.05387v1
- Canonical: https://arxiv.org/abs/2610.05387v1
Trouble viewing inline? Open PDF directly →
Full Text
112,492 characters extracted from source content.
Expand or collapse full text
GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation Murad Hossen1, Tasneem Selim222footnotemark: 2 , Gurur Gamgam322footnotemark: 2 , Tuga Yousif422footnotemark: 2 , Abderrahmane Kasmi522footnotemark: 2 , Ikram Aissiou6, Mubaraq Onipede7, Faran Taimoor Butt8, Sanae Zrigui9, Rosa Y. G. Paccotacya-Yanque10, Ignatius Balayo11, Ikram Elhouiti12, Hadil Affes13, Bijay Adhikari14, Sargam Goyal15, Muhammad Ibrahim Isah16, Mohammad Idrees Bhat17, Samuel Kangoni Matia18, Peguy Kem-Meka Tiotsop Kadzue19,20,21, Maha Trabelsi13, Emmanuel Owusu22, Vinit23, Nour Majdoub24, Tamiru Alemnew25, Islem Rekik2611footnotemark: 1 1University of Houston, Texas,USA ⋅· 2Department of Mathematics and Computer Science, Faculty of Science, Alexandria University, Alexandria, Egypt ⋅· 3Bogazici University, Turkey ⋅· 4Ankara Yıldırım Beyazıt University, Turkey ⋅· 5ESI, Algiers, Algeria ⋅· 6University of Algiers 1, Algeria ⋅· 7York St John University London, UK ⋅· 8Air University, Islamabad, Pakistan ⋅· 9LSISI, ENSA, Mohammed Premier University, Oujda, Morocco ⋅· 10Universidad Católica San Pablo, Arequipa, Peru ⋅· 11Busitema University, Uganda ⋅· 12University of Laghouat, Algeria ⋅· 13ISI, University of Tunis El Manar, Tunisia ⋅· 14Tribhuvan University, Nepal ⋅· 15IIT Roorkee, India ⋅· 16Shobhit Institute of Engineering and Technology, India ⋅· 17MIT World Peace University, Pune, India ⋅· 18University of Kinshasa, DR Congo ⋅· 19University of Bertoua, Cameroon ⋅· 20University of the Witwatersrand, Johannesburg, South Africa ⋅· 21African Institute for Mathematical Sciences, Research and Innovation Centre (AIMS RIC), Kigali, Rwanda ⋅· 22KNUST, Ghana ⋅· 23NSUT Delhi, India ⋅· 24ISIMM Monastir, Tunisia ⋅· 25Addis Ababa University, Ethiopia ⋅· 26BASIRA Lab, Department of Computing, Imperial College London, United Kingdom All authors are co-affiliated with the BASIRA Lab, Department of Computing, Imperial College London, United Kingdom. †thanks: Corresponding authors: mhossen2@cougarnet.uh.edu, i.rekik@imperial.ac.uk; https://basira-lab.com/†thanks: Co-first authors: These authors contributed equally. Abstract Large language models (LLMs) have demonstrated strong performance on coding and reasoning benchmarks; however, their ability to solve graph-structured machine learning problems remains largely unexplored. In particular, no benchmark currently evaluates whether LLMs can autonomously solve end-to-end Graph Neural Network (GNN) coding tasks under realistic competition settings. To address this gap, this paper introduces GNN-CB, the first competition-based benchmark for evaluating both humans and LLMs on GNN coding tasks. GNN-CB consists of 18 curated competitions spanning node-, edge-, and graph-level prediction across diverse graph categories, domains, and difficulty tiers. All submissions are evaluated through a unified automated pipeline with hidden test sets and standardized scoring. Human participants solve tasks under controlled competition constraints, while LLMs are evaluated using a frozen zero-shot prompting protocol based on a plan-then-code paradigm with bounded execute-and-repair loops. The benchmark additionally supports both non-agent and autonomous agent-based evaluation within the same protocol. Under our evaluated protocol, LLMs rarely match Human Top performance and show less stable performance across competitions. No single model dominates: a few competitions are won by LLMs, yet humans still hold the top score on most tasks. We release GNN-CB as a living benchmark with automated evaluation infrastructure, dynamic leaderboards, and reproducible execution pipelines. Beyond benchmarking, GNN-CB provides a practice-oriented resource for studying GNN implementation across progressively diverse graph-learning tasks. The benchmark and evaluation framework are publicly available at https://basiralab.github.io/GNN-CB/. 1 Introduction GNN-CB: benchmark overview18 competitions ⋅· 3 protocols ⋅· unified hidden-test evaluationCompetition taxonomyCategoryHomogeneousHeterogeneousTemporalSpatialHierarchicalMolecularGeneralizationDifficultyEasyMediumHardTask typeNode classificationEdge classificationGraph classificationGraph regression18 competitionsC01C02C03C04C05C06C07C08C09C10C11C12C13C14C15C16C17C18DomainsBiomedicalFinanceSocialCybersecurityTransportAcademicNeuroimagingBioprinting18 competitions7 graph categories3 difficulty tiersEvaluation protocolsProtocol 1 · Competition Design Repository README ⋅· datasets submission spec ⋅· tree Hidden test sets GitHub Actions pipeline Live leaderboard Protocol 2 · Human Evaluation Constraints GNN only 3 CPU hrs seed=25 Without LLMs ≤5 runs Participants 13–17 per task 1 final submission Output predictions.csv (encrypted) Protocol 3 · LLM Evaluation Slot fill README tree · data spec · reqs Frozen prompt zero-shot no tuning Plan-then- code ⟨ ⟩ ⟨ ⟩ Repair loop k≤5k≤5 iters traceback Agent mode (autonomous repo exploration) | Non-agent mode (manual prompt filling) Evaluated modelsClaude OpusGemini 2.5GPT 5.4Gemini 3 flashDeepSeek v4Llama 3.3Qwen 2.5Kimi k 2.6Unified hidden-test evaluation pipelineGitHub Actions ⋅· automated scoring ⋅· identical conditions for all submissionsLeaderboard + per-task metricshuman vs. LLM ⋅· live updates Figure 1: Overview of GNN-CB. Large language models (LLMs) have recently demonstrated strong capabilities in coding and reasoning tasks, including program synthesis, competitive programming, software engineering, and autonomous code repair (Chen et al., 2021; Li et al., 2022; Jimenez et al., 2023; Zhuo et al., 2024). However, existing evaluations mainly focus on general-purpose programming while largely overlooking graph-structured machine learning workflows. Graph Neural Networks (GNNs) are widely used for learning over graph-structured data in domains such as molecular modeling, recommender systems, social networks, neuroimaging, and knowledge graphs (Gilmer et al., 2017; Hamilton et al., 2017; Li et al., 2018; Ying et al., 2018; Li et al., 2021; Dauparas et al., 2022; Schlichtkrull et al., 2018). Unlike conventional coding tasks, GNN problems require understanding graph topology, building relational pipelines, selecting message-passing architectures, and optimizing under task-specific constraints, making them more challenging than isolated algorithmic tasks. Despite the growing importance of graph machine learning, no existing benchmark evaluates whether LLMs can autonomously solve realistic end-to-end GNN coding tasks. Benchmarks such as HumanEval (Chen et al., 2021), CodeContests (Li et al., 2022), SWE-bench (Jimenez et al., 2023), and BigCodeBench (Zhuo et al., 2024) focus on algorithmic reasoning or software engineering workflows without graph-learning components, while graph benchmarks such as OGB (Hu et al., 2020) and LRGB (Dwivedi et al., 2022) are designed to evaluate the predictive performance of GNN models rather than their implementation. To address this gap, we introduce GNN-CB, the first competition-based benchmark designed to evaluate both humans and LLMs on GNN coding tasks. As illustrated in Figure 1, GNN-CB contains 18 human-curated competitions spanning node-, edge-, and graph-level prediction tasks across multiple graph categories, domains, and difficulty tiers. Each competition provides standardized repositories, datasets, hidden test sets, and automated GitHub-based evaluation pipelines. To ensure fair and reproducible assessment, we define three complementary protocols: (i) competition design, (i) human evaluation, and (i) LLM evaluation. Human participants solve tasks under controlled competition constraints with fixed computational budgets and limited submissions. LLMs are evaluated using a frozen zero-shot prompting protocol based on a structured plan-then-code paradigm with bounded execute-and-repair loops. The benchmark additionally supports both non-agent and autonomous agent-based evaluation settings within the same protocol. Our contributions are summarized as follows. (i) GNN-CB is introduced as the first competition-based benchmark for evaluating both humans and LLMs on GNN coding tasks. (i) The benchmark contains 18 human-curated competitions spanning diverse graph categories, application domains, and difficulty tiers under unified hidden-test evaluation protocols. (i) A controlled LLM evaluation framework is proposed based on frozen prompting, plan-then-code generation, and bounded execute-and-repair loops, supporting both agent-based and non-agent settings. (iv) Evaluation of human participants and LLMs enables systematic analysis of performance variability, task difficulty, and implementation challenges in realistic GNN coding environments. (v) GNN-CB is released as a living benchmark with automated evaluation infrastructure, reproducible pipelines, and dynamic leaderboards to support future research on the capabilities and limitations of LLMs as autonomous GNN engineers, as well as serving as a structured resource for hands-on GNN practice across progressively diverse graph-learning tasks. 2 Related Work Recent benchmarks have significantly advanced evaluation of LLM-based code generation and reasoning. HumanEval (Chen et al., 2021) and CodeContests (Li et al., 2022) introduced standardized evaluation for functional correctness and competitive programming. Later works extended evaluation across programming languages and stricter testing settings, including HumanEval-X (Peng et al., 2024), MultiPL-E (Cassano et al., 2023), and EvalPlus (Liu et al., 2023). More realistic software engineering benchmarks such as DS-1000 (Lai et al., 2023), SWE-bench (Jimenez et al., 2023), BigCodeBench (Zhuo et al., 2024), and LiveCodeBench (Jain et al., 2024) evaluate repository-level reasoning, data science workflows, and instruction-following capabilities. Graph machine learning benchmarks have standardized evaluation for graph neural networks (GNNs). The Open Graph Benchmark (OGB) (Hu et al., 2020) provides datasets and evaluation protocols for node-, edge-, and graph-level prediction tasks, while Benchmarking GNNs (Dwivedi et al., 2023) and LRGB (Dwivedi et al., 2022) focus on structural generalization and long-range graph dependencies. However, these benchmarks do not assess whether LLMs can perform at a human level on GNN competitions. GNN-CB bridges these two research directions by introducing a competition-based benchmark for GNN coding tasks, supporting unified evaluation of both human participants and LLMs under controlled and reproducible conditions. 3 Benchmark Construction GNN-CB is a competition-based benchmark for evaluating GNN coding tasks under standardized settings. Unlike traditional code-generation benchmarks centered on isolated functions or short algorithmic problems, GNN-CB targets repository-level graph learning workflows where participants or LLMs must build complete pipelines spanning data processing, graph construction, architecture design, optimization, and hidden-test prediction generation. The benchmark construction consists of three protocols : (i) competition design protocol, (i) human evaluation protocol, and (i) LLM evaluation protocol. 3.1 Competition Design Protocol Table 1: Overview of all 18 GNN Coding Benchmark competitions. Cat. = Graph Category; Diff. = Difficulty; N = typical number of nodes; E = typical number of edges; Feat. = node feature dimensionality. For classification tasks, we report Macro-F1, Accuracy, and Area Under the Curve (AUC), depending on the competition specification. For regression tasks, we use Mean Absolute Error (MAE) for single-target regression and Normalized Mean Absolute Error (NMAE) for multi-target regression. # Competition Task Description Cat. Diff. N E Feat. Domain Metric Dataset C01 PROVEN-GNN Graph classification on source code representations Hetero Med 260K 1.7M 527 Cybersecurity Macro-F1 DiverseVul C02 CGCC Graph classification of city street-network layouts Homo Hard 9.396K 13.533K 3 Urban Macro-F1 OpenStreetMap C03 Real Or Fake! Binary graph classification for fake news detection Hier Easy 314K 308K 1078 Social Binary F1 GossipCop C04 GLIMPS-GNN Inductive cfRNA → preeclampsia classification Gen Hard 320 3200 6650 Biomedical F1 GEO C05 GNN_BACE Molecular graph classification for BACE-1 inhibition Mol Med 32 34 8 Medical Macro-F1 BACE (MoleculeNet) C06 DiaGraph Binary node classification for diabetes prediction Homo Hard 96K 1.2M 12 Medical Macro-F1 Kaggle Diabetes C07 GRAPE Binary graph classification for diabetic retinopathy Spatial Med 252 846 4 Medical Macro-F1 DRIVE/STARE/HRF C08 Liar Nodes Node Classification under Adversarial Feature Corruption for Cancer Single-Cell Analysis Homo Easy 3.7K 27K 1.7K Biomedical Accuracy CancerSCEM C09 GraphFMD Temporal inductive node classification (Bitcoin) Temp Med 25.5K — 165 Finance Macro-F1 Elliptic C10 NetLinkArena Link prediction in sparse citation graphs Homo Med 3.3K 9.1K 2.7K Academic AUC Citation C11 PARK-GNN Node classification for Parkinson’s detection Homo Hard 195 ∼ 1K 22 Medical Macro-F1 UCI Parkinson’s C12 GNN-CoRA Node classification on citation network Homo Med 2.7K 5.4K 1433 Academic Accuracy Cora C13 GTA Graph classification on ideal and perturbed conditions Gen Hard ∼ 18 ∼ 40 7 Bioinformatics Macro-F1 MUTAG C14 BrainAge Graph regression for age prediction from brain connectome Homo Easy 29.3K 478.8K 2 Neuroimaging MAE OASIS-3 C15 OctoNode Cup Node classification on GitHub social network Homo Med 37.7K 289K 42 Social Macro-F1 GitHub Social C16 Graph4ASD Binary classification of brain connectivity graphs Homo Med 127.4K 12676.3K 1 Neuroimaging Macro-F1 ABIDE C17 Mix2Print Learning material interactions for 3D bioprinting Homo Hard 423 740 442 Bioprinting Normalized-MAE CECT C18 THInC Inductive GNN for cell type classification Spatial Hard 51K 258K 4 Biomedical Macro-F1 NuCLS The competition design protocol organizes benchmark tasks across diverse graph learning settings. The benchmark contains 18 independent competitions spanning node classification, edge prediction, graph classification, and graph regression tasks (Figure 1). Competitions are organized along two axes: graph category and difficulty tier. The benchmark covers seven graph categories, including homogeneous, heterogeneous, temporal, spatial, hierarchical, molecular, and generalization graphs. Competitions are additionally grouped into Easy, Medium and Hard difficulty tiers based on implementation complexity, architectural requirements, and observed human performance. Table 1 summarizes all benchmark competitions, including graph categories, dataset scales, feature dimensionalities, application domains, and evaluation metrics. Detailed dataset specifications and repository descriptions are provided in Appendix A. Each competition is associated with a dedicated repository that specifies the task, submission format, evaluation metrics, and leaderboard configuration. An automated evaluation infrastructure is provided for submission validation, scoring, leaderboard updates, and execution logging, ensuring that all submissions are processed through a unified evaluation workflow across competitions. All submissions are evaluated exclusively on hidden test sets that remain inaccessible throughout the competition lifecycle. Evaluation is performed through automated GitHub-based pipelines that validate submission formats, execute scoring scripts, and update leaderboards without manual intervention. The infrastructure additionally supports secure submission mechanisms to prevent information leakage. Depending on repository configuration, participants either submit predictions through controlled external platforms or upload encrypted prediction files to designated repository directories, where they are automatically decrypted only during evaluation prior to scoring. 3.2 Human Evaluation Protocol The human evaluation protocol measures participant performance under standardized computational and submission constraints. Participants solve each competition independently using only GNN models, without external solutions, and submit a single final prediction file evaluated through the same automated hidden-test pipeline used for LLMs. LLM usage during the human track is restricted to low-level utility support without exposure to task-specific benchmark information. More detailed protocol and participant information are provided in Appendix B. Participants represent 18 universities across 12 countries (Figure 6, Appendix B), ensuring geographic diversity in benchmark design and human performance evaluation. 3.3 LLM Evaluation Protocol GNN-CB evaluates LLMs under a controlled and reproducible protocol designed to measure repository-grounded graph learning ability rather than prompt-engineering performance. All models are evaluated using a frozen prompt template shared across all competitions, without task-specific tuning or adaptive prompting (Zhuo et al., 2024). As illustrated in Figure 2, repository information is converted into a standardized prompt through slot filling (Jain et al., 2024), after which the model generates a structured reasoning plan and an executable code following a plan-then-code paradigm (Jiang et al., 2024). The generated solution is executed locally in a controlled environment and optionally refined through bounded execute-and-repair iterations (Yang et al., 2024). Final predictions are evaluated on hidden test sets using the same automated infrastructure used for human participants, enabling direct and fair comparison under identical conditions. The protocol consists of four components: (i) frozen prompt instantiation, (i) plan-then-code generation, (i) execution and repair, and (iv) evaluation settings. InputGenerationExecutionScoringRepositoryREADMEDataTreeSubmission Spec.Slot filling5 slotsFrozenpromptzero-shotLLM⟨plan⟩ plan ⟨code⟩ code -onlyerror?Submitpredictions.csvHidden-test scoringtask-specific metricLeaderboardlive updateRepair loop (k≤5k≤ 5)traceback → LLM → re-generatenoyes Figure 2: LLM evaluation pipeline. 3.3.1 Frozen Prompt Instantiation All models were evaluated using a standardized prompt template with limited, controlled task-specific context added to ensure executable outputs. The template contains predefined slots populated with repository-derived information such as the README, repository tree, dataset summary, submission specification, and dependency requirements. Repository components are inserted into structured slots without manual modification (Jain et al., 2024). The instantiated prompt requires the model to produce a structured <plan> followed by executable <code>, providing a standardized interface across all competitions. Figure 8 in Appendix C illustrates the slot-filling construction, and the full template is provided in Listing 1. 3.3.2 Plan-Then-Code Generation Given the instantiated prompt, the model generates: (i) a high-level reasoning plan and (i) an executable code. This plan-then-code strategy (Jiang et al., 2024) encourages explicit reasoning before implementation and reduces common failures such as invalid pipelines, hallucinated APIs, or malformed submissions. The generated plan typically specifies feature engineering, GNN architecture design, optimization procedures, validation strategy, and submission generation. The accompanying code implements the complete pipeline as a standalone executable solution. A representative plan-then-code output is shown in Figure 9 in Appendix C. 3.3.3 Execution and Repair Loop Generated code is executed locally in a controlled environment under CPU-only constraints. If execution fails due to syntax errors, runtime exceptions, dependency issues, or invalid outputs, an execute-and-repair loop (Yang et al., 2024) is triggered in which the execution traceback is appended to the conversation context and returned to the model for refinement. Repair attempts are bounded (k≤5k≤ 5) to maintain a controlled evaluation budget. 3.3.4 Agent-Based and Non-Agent Evaluation GNN-CB supports two complementary execution modes: non-agent and agent-based. In the non-agent mode, a lightweight orchestration script instantiates the frozen prompt with repository-derived slots, dispatches the API request, parses the response into <plan> and <code> blocks, executes the resulting solution.py under the CPU-only environment, and feeds runtime tracebacks back into the prompt for a bounded number of repair iterations; a simplified version of this control loop is reproduced in Listing 2 of Appendix C. In the agent-based mode, an autonomous coding system inspects repository contents, generates an executable solution, runs it, and iteratively repairs failures without human intervention, using the same execute-and-repair budget. Both modes consume identical repository information, enforce the same execution constraints, and submit through the same automated hidden-test pipeline, so that observed performance differences reflect the model’s reasoning and implementation ability rather than prompt-engineering effort or orchestration design. C01C02C03C04C05C06C07C08C09C10C11C12C13C15C16C18000.50.511ScoreLowAvgHighStd (a) Classification competitions (F1 / Accuracy / AUC). C14003366991212MAEC17000.050.050.10.10.150.15NMAE (b) Regression. Figure 3: Per-task human-performance statistics across all 18 GNN-CB competitions: minimum, average, maximum, and standard deviation of participant scores. Panel (a) covers the 16 classification competitions on a common 0–1 axis (F1 / Accuracy / AUC); panel (b) covers the two regression competitions (C14 BrainAge, MAE; C17 Mix2Print, NMAE) on independent axes due to different metric scales. Higher is better in (a); lower is better in (b). 3.3.5 LLM Implementations To ensure deterministic, reproducible outputs, all models are decoded with temperature = 0, applied where the provider’s API exposes this setting. API-only (non-agent) runs use a per-call output budget of max_output_tokens = 10K, while agent-mode runs follow each agent’s native session budget, which may consume substantially more tokens across multiple internal tool calls and extended-thinking traces. A full inventory of evaluated models, their API identifiers, and source links is provided in Table 4 (Appendix C). In the API-only mode, models are accessed through provider SDKs or OpenAI-compatible endpoints, including Together AI serverless inference, and receive the frozen prompt through a lightweight orchestration script that handles slot filling, code extraction, execution, and bounded repair iterations. Models evaluated in this mode include Gemini-3 Flash, Gemini 2.5 Pro, Llama-3.3 70B, Claude Opus 4.6, Qwen2.5-Coder 32B, Kimi k2.6, and DeepSeek v4 Pro. In the agent-based mode, models operate as autonomous coding agents that directly inspect repository contents, generate solutions, execute code, and iteratively repair failures without human intervention. Models evaluated in this mode include Claude Opus 4.7 and GPT-5.4 Codex. All generated solutions are executed locally under CPU-only constraints inside isolated environments. The framework automatically parses model outputs into <plan> and <code> sections, executes the generated solution.py, performs bounded repair iterations (k≤5k\!≤\!5), validates prediction files, and logs execution artifacts including runtime traces and generated submissions. This design enables reproducible evaluation across multiple model families without requiring local GPU deployment. 4 Results and Analysis We evaluate GNN-CB along two complementary tracks: a human-participant track (Section 3.2) with expert volunteers under standardized reproducibility constraints, and an LLM track (Section 3.3) with nine proprietary and open-source models under a frozen plan-then-code protocol with bounded execute-and-repair iterations. Both tracks use the same automated hidden-test pipeline and task-specific metrics: Macro-F1, accuracy, and AUC for classification, and MAE/NMAE for regression. We report human-cohort statistics, each LLM’s per-competition score and gap to the human top, and cross-task variability as a measure of consistency. Table 2: Descriptive statistics of human performance across competitions. Difficulty level (Diff), number of participants (P), average score (AVG), and standard deviation (STD) summarize overall human performance.High and Low columns report the maximum and minimum human scores observed, respectively, Metric indicates the evaluation criterion used for tasks. # Competition Diff. P AVG STD High Low Metric C01 PROVEN-GNN Med 17 0.792 0.101 0.900 0.435 F1 C02 CGCC Hard 17 0.383 0.086 0.537 0.205 F1 C03 Real Or Fake! Easy 15 0.945 0.053 0.980 0.767 F1 C04 GLIMPS-GNN Hard 16 0.389 0.094 0.600 0.28 F1 C05 GNN_BACE Med 14 0.496 0.088 0.674 0.391 F1 C06 DiaGraph Hard 15 0.570 0.125 0.822 0.477 F1 C07 GRAPE Med 16 0.707 0.131 0.830 0.4 F1 C08 Liar Nodes Easy 17 0.756 0.212 0.996 0.407 Acc. C09 GraphFMD Med 17 0.706 0.271 0.891 0.403 F1 C10 NetLinkArena Med 17 0.706 0.271 0.891 0.508 AUC C11 PARK-GNN Hard 14 0.773 0.116 0.933 0.548 F1 C12 GNN-CoRA Med 14 0.551 0.196 0.742 0.153 Acc. C13 GTA Hard 17 0.5816 0.1629 0.79 0.25 F1 C14 BrainAge Easy 14 6.531 1.474 10.638 4.984 MAE C15 OctoNode Cup Med 15 0.672 0.151 0.793 0.211 F1 C16 Graph4ASD Med 14 0.513 0.082 0.627 0.389 F1 C17 Mix2Print Hard 13 0.08 0.02 0.147 0.069 NMAE C18 THInC Hard 14 0.490 0.056 0.565 0.354 F1 Table 3: LLM Benchmark Leaderboard on GNN Competitions. Scores and gaps to Human Top are reported for 9 LLMs across 18 GNN competitions. Gaps are computed as score−human topscore-human top and sign-corrected for regression. Blue subscripts ↓ denote performance below Human Top; Green subscripts ↑ denote ties or improvements. For classification tasks, cells are tinted red (−5%-5\% gap <0<0%), light red (gap <5<5%), yellow (tied) and green (gap >0>0). For regression tasks, cells are colored as red and green for below and above Human Top, respectively. Underlining marks the best result within each model category, and boldface the best overall result per competition. The final row reports the average human–LLM gap across the nine models. Models Easy Medium Hard C03 C08 C14 C01 C05 C07 C09 C10 C11 C12 C15 C16 C02 C04 C06 C13 C17 C18 Real_Or_Fake Liar Nodes Brain-Age PROVEN GNN_BACE GRAPE Graph FMD NetLinkArena PARK-GNN GNN_CoRA OctoNode Graph4ASD CGCC GLIMPS-GNN DiaGraph GTA Mix2Print THInC val F1 acc. MAE mF1 F1 mF1 val F1 AUC val F1 acc. Mac-F1 Mac-F1 OOF F1 F1 Mac-F1 F1 (P.) NMAE Mac-F1 Proprietary Reasoning LLMs (Frontier) Gemini 2.5 Pro Google DeepMind (2025) 93.6↓4.5 92.7↓6.9 5.524↓0.540 75.5↓14.5 61.9↓5.5 83.00 83.6↑3.0 87.5↓1.6 71.2↓22.1 64.8↓9.4 54.1↓25.2 48.8↓13.9 40.7↓13.0 29.0↓31.0 48.9↓33.4 65.7↓14.2 0.093↓0.025 36.0↓20.6 Claude Opus 4.7 Anthropic (2026b) 97.6↓0.5 99.8↑0.2 5.504↓0.520 78.1↓11.9 55.4↓12.0 83.00 76.2↓4.4 69.6↓19.5 90.2↓3.0 55.1↓19.1 74.6↓4.8 60.8↓1.9 32.6↓21.0 44.0↓16.0 76.9↓5.3 79.1↓0.8 0.069↓0.000 57.2↑0.6 GPT-5.4 Codex OpenAI (2026) 70.7↓27.3 88.9↓10.6 5.847↓0.863 50.2↓39.8 53.4↓14.0 44.4↓38.5 77.6↓3.1 83.9↓5.2 84.6↓8.7 74.0↓0.2 71.8↓7.5 45.7↓17.0 36.2↓17.5 52.0↓8.0 72.5↓9.7 83.5↑3.6 0.074↓0.005 48.6↓7.9 Open-source Reasoning LLMs DeepSeek-v4 Pro DeepSeek-AI (2026) 95.2↓2.8 73.0↓26.6 5.512↓0.528 79.4↓10.6 53.5↓13.9 83.00 78.1↓2.5 66.5↓22.7 42.6↓50.6 70.0↓4.2 54.2↓25.1 32.9↓29.8 15.6↓38.1 30.0↓30.0 74.4↓7.9 39.7↓40.2 0.067↑0.002 43.1↓13.4 Proprietary Chat LLMs Gemini-3 Flash Google DeepMind (2026) 97.6↓0.5 98.8↓0.7 5.710↓0.726 81.1↓8.9 53.2↓14.2 76.2↓6.8 78.0↓2.7 68.3↓20.8 93.30 69.7↓4.5 74.9↓4.4 51.5↓11.2 33.5↓20.2 38.0↓22.0 49.5↓32.8 59.5↓20.4 0.069↑0.000 47.9↓8.6 Claude Opus 4.6 Anthropic (2026a) 95.4↓2.6 98.3↓1.3 5.668↓0.684 85.4↓4.6 45.2↓22.2 83.00 81.8↑1.2 78.1↓11.0 52.9↓40.3 81.6↑7.4 65.5↓13.9 55.4↓7.3 40.7↓12.9 28.0↓32.0 78.3↓3.9 69.3↓10.6 0.072↓0.003 54.2↓2.3 Open-source Chat LLMs Llama-3.3 70B Meta AI (2024) 79.4↓18.6 66.2↓33.4 7.449↓2.465 72.7↓17.3 39.1↓28.3 40.0↓43.0 72.8↓7.8 50.0↓39.1 51.1↓42.2 65.9↓8.3 42.9↓36.5 39.4↓23.3 46.7↓7.0 38.0↓22.0 32.9↓49.4 63.8↓16.1 0.070↓0.001 37.6↓18.9 Qwen2.5-Coder 32B Alibaba Cloud (2024) 96.6↓1.5 66.6↓33.0 9.502↓4.518 77.1↓12.9 39.1↓28.3 40.0↓43.0 69.8↓10.8 54.8↓34.3 42.6↓50.6 80.0↑5.8 52.5↓26.8 44.0↓18.6 19.6↓34.1 31.0↓29.0 50.2↓32.0 25.5↓54.4 0.117↓0.048 38.4↓18.1 Kimi-k2.6 Moonshot AI (2026) 96.4↓1.6 42.9↓56.7 6.139↓1.155 48.4↓41.6 52.1↓15.3 83.00 95.9↑15.3 71.4↓17.8 88.0↓5.3 70.5↓3.7 70.4↓9.0 64.0↑1.3 43.0↓10.7 48.0↓12.0 49.3↓33.0 79.1↓0.8 0.064↑0.005 57.2↑0.7 Human reference Human Top 98.0 99.6 4.98 90.0 67.4 83.0 80.6 89.1 93.3 74.2 79.3 62.7 53.7 60.0 82.2 79.9 0.069 56.5 Avg. human–LLM gap ↓6.6 6.6 ↓18.8 18.8 ↓1.333 1.333 ↓18.0 18.0 ↓17.1 17.1 ↓14.6 14.6 ↓1.3 1.3 ↓19.1 19.1 ↓24.8 24.8 ↓4.0 4.0 ↓17.0 17.0 ↓13.5 13.5 ↓19.4 19.4 ↓22.4 22.4 ↓23.0 23.0 ↓17.1 17.1 ↓0.008 0.008 ↓9.8 9.8 Figure 4: Best, Mean, and Worst Gap performance for leading LLMs on the classification tasks of the GNN-CB benchmark, grouped by model family. Best Gap denotes the best performance margin across the 16 classification competitions compared to the human top performance. Mean Gap is the average performance difference relative to the human top performance across all classification competitions. Worst Gap represents the most negative performance difference relative to the human top performance. 4.1 Human Performance Results Table 2 and Figure 3 together summarize the performance of human participants across all 18 competitions. Overall, human performance exhibits variability across tasks, reflecting differences in task difficulty, graph structure, and domain complexity. Easy classification tasks (e.g., C03) achieve high scores with low variance, indicating that standard GNN pipelines are implemented reliably when task specifications are clear. In contrast, harder competitions (e.g., C02, C04, C18) show lower performance and larger gaps between best and worst submissions, reflecting higher sensitivity to architectural and optimization choices. Several medium-difficulty tasks (e.g., C09, C10) also exhibit notable variability, suggesting that engineering and training decisions remain critical even for relatively standard formulations. The two regression tasks (C14 and C17) display similar behaviour, with larger relative performance gaps and less stable convergence compared to most classification settings. These observations suggest that human participants perform consistently on conventional GNN problems but encounter increasing difficulty as tasks require more advanced graph reasoning, domain-specific adaptation, and complex optimization strategies. 4.2 LLM Evaluation Results Table 3 reports the per-competition performance of the nine evaluated LLMs on all 18 GNN-CB tasks. The most important finding is that LLMs consistently fall short of Human Top performance across the benchmark under a zero-shot plan-then-code prompting protocol with a bounded execute-and-repair loop. Concretely, across all 9 evaluated models and 18 competition tasks (162 model–task pairs in total), LLMs surpass or tie the Human Top score in only 19 cases. In other words, in more than 88% of model–task pairs, the evaluated LLM does not reach Human Top performance. Among models achieving Human Top-level performance, we observe no clear model-family advantage, and the Human-Top level performance counts are roughly proportional to the number of models in each family. In addition, the average LLM performance exceeds the average human performance in only 6 of 16 competitions (37.5%), as shown in Figure 13, meaning that average human performance remains higher on most competitions under this protocol. To further assess LLM performance fairly, we analyze the gap to Human Top performance across 16 classification competitions. The evaluated open-source models exhibit larger average gaps and greater cross-task variability under this protocol. This trend is observed for both reasoning and chat models. DeepSeek-v4 Pro underperforms proprietary reasoning counterparts in mean gap, while open-source chat models (Llama-3.3 70B, Qwen2.5-Coder 32B) show larger gaps than proprietary chat models (Claude Opus 4.6, Gemini-3 Flash). An exception is Kimi-k2.6, which approaches the consistency of proprietary chat models. Within the proprietary models evaluated, neither reasoning nor chat models show a consistent benchmark-wide advantage, as also shown in Figure 4. In addition, LLMs exhibit larger performance gaps on Hard competitions than on Easy and Medium competitions, suggesting that the observed human–LLM gap increases with task difficulty under this protocol. We further examined severe underperformance in classification competitions, defined as scoring at least 30 percentage points below the best-performing LLM on the same task. Manual inspection identified 20 such cases, with Llama-3.3 70B and Qwen2.5-Coder 32B accounting for six and five cases, respectively. The error analysis showed that these failures were driven primarily by incorrect graph construction (8/20) and feature processing or utilization (7/20), rather than by minor coding issues or GNN architecture choice. The remaining cases involved indexing or data-reading errors (3/20) and unsuitable model selection (2/20). This suggests that severe degradation mainly arises when models fail to translate the input data into an appropriate graph representation.Further results are presented in Appendix C.5. 5 Discussion The findings in Table 3 and Figure 4 reveal the core limitations of LLMs as autonomous GNN engineers under the evaluated setting. All evaluated models show substantial variance across the 18 competitions, indicating limited transfer of GNN construction and reasoning strategies across heterogeneous settings.Proprietary models demonstrate better average gaps than their open-source counterparts, yet no meaningful difference emerges between reasoning and chat LLMs within either tier—suggesting that chain-of-thought reasoning capabilities do not confer a systematic advantage on GNN-specific tasks under our zero-shot, bounded-repair protocol. In addition, performance gaps widen on Hard tier competitions, which shows the inability of LLMs dealing with complex GNN-based tasks. These results suggest that task-aware model routing or ensembling may be more effective than scaling a single model. Competition-style evaluation also provides a discriminative measure of practical GNN coding ability and highlights the risks of deploying LLM-generated solutions without expert oversight. 6 Conclusion We introduced GNN-CB, the first competition-based benchmark for evaluating humans and LLMs on GNN coding tasks. It comprises 18 human-designed competitions spanning diverse graph categories, domains, and difficulty tiers, evaluated through a unified hidden-test pipeline. Three complementary protocols govern competition design, controlled human evaluation, and frozen-prompt LLM evaluation with bounded execute-and-repair iterations across API-only and agent-based settings. Under this protocol, current LLMs fall short of expert performance on most tasks and exhibit inconsistent performance across heterogeneous graph-learning problems. As models and evaluation methods continue to improve, the competition-based design of GNN-CB offers a challenging, extensible, and practically grounded testbed that can continuously distinguish advances in autonomous GNN engineering. Its public benchmark and dynamic leaderboards support future evaluation, comparison, and hands-on GNN coding practice. Limitations Due to API and compute-budget constraints, our LLM evaluation is limited to nine leading proprietary and open-source models across two execution modes (API-only and agent-based), and our human evaluation involves 13–17 expert volunteers per competition. The model and task selection was guided by a balance between inclusiveness and feasibility, aiming to cover diverse access types (proprietary vs. open-source), and model families (Claude, Gemini, GPT, Llama, Qwen, Kimi, DeepSeek) while maintaining reproducibility and accessibility for research groups with limited compute resources. We believe this setup provides a meaningful and representative evaluation of contemporary LLM capabilities on end-to-end GNN coding tasks, without requiring the extensive infrastructure typically available only to large industrial laboratories. Accordingly, our results should be interpreted as a broad and reproducible evaluation of contemporary LLM performance under the evaluated bounded-repair and agent-based protocols, rather than as an estimate of the maximum attainable performance of each model. Larger inference budgets, repeated sampling, and more advanced model-specific prompting or agentic pipelines may further improve performance and should be investigated in future work. A further limitation concerns the enforceability of the human-track constraint. Although the protocol restricts LLM use to low-level utility assistance and requires detailed reproducibility logs, local use of external assistance cannot be perfectly instrumented. Therefore, this constraint should be interpreted as practically auditable rather than fully enforceable. While GNN-CB covers a broad range of node-, edge-, and graph-level problems, certain emerging settings such as graph generation, quantum graph modelling, and self-supervised or reinforcement-learning paradigms are not included in the current release. The use of task-specific evaluation metrics (macro-F1F_1, accuracy, AUC, MAE, and NMAE) reflects the heterogeneity of real-world graph problems and limits direct cross-task comparison. To mitigate this, we report per-task gap-to-human-top scores and separate classification and regression results in our visualizations, particularly since the benchmark contains only two regression competitions. Although the two regression competitions broaden the task and metric diversity of GNN-CB, their limited number does not support strong general conclusions about LLM performance on graph-regression problems. Expanding both the number and diversity of regression competitions is therefore an important direction for future releases. Finally, because LLM capabilities evolve rapidly, the reported results represent model snapshots available as of May 2026, although the qualitative patterns we observe—difficulty stratification, the capability-versus-consistency gap, and family-level differences—are expected to remain informative as models continue to advance. These limitations can be viewed as opportunities for future extensions of GNN-CB toward broader graph-learning paradigms, larger model panels, and continuously updated evaluations. Future Directions for Extended LLM Evaluation The present study establishes GNN-CB as a zero-shot, frozen-prompt benchmark, and its infrastructure enables several deeper analyses that we did not pursue here due to computational and budgetary constraints. On the modelling side, future work can measure how performance scales beyond the current zero-shot protocol through few-shot prompting, retrieval-augmented generation, and task-specific fine-tuning, and can characterise stochastic variability via repeated sampling, confidence intervals, and pass@k-style metrics rather than the single greedy run reported here. Because every submission is executed and validated, the benchmark also supports a fine-grained error-mode taxonomy—incorrect graph construction, invalid message-passing architectures, preprocessing data leakage, mishandled class imbalance, and malformed submissions—that would reveal where and why LLMs break down beyond aggregate scores. On the evaluation side, the logged execution and repair traces enable a detailed token-economy and cost–latency analysis, particularly contrasting the per-call budget of API-only runs with the multi-step session budget of agent-based runs, alongside an analysis of agentic depth (tool-call counts, repair-loop depth, and the contribution of extended chain-of-thought). Rigour can be further strengthened through paired significance testing, bootstrap confidence intervals, and rank-correlation analysis of the human–LLM ordering. Finally, as a living benchmark with automated infrastructure and dynamic leaderboards, GNN-CB can be extended to broader paradigms (graph generation, self-supervised, temporal, and reinforcement-learning formulations) and used longitudinally to track whether the capability–consistency gap narrows as new models are released. Ethics Statement Participant Consent: All human participants provided informed consent for the publication of their performance results. Participants were expert volunteers who co-designed the benchmark; no monetary compensation was provided. Only aggregate counts and per-task scores are reported (Table 2); no personally identifying information about participants is released. Participant Role Disclosure: Each of the 18 benchmark competitions was designed and curated by a single coauthor. To avoid any designer advantage, no coauthor participated in the competition they personally designed. Instead, each coauthor solved a subset (approximately 14) of the remaining 17 competitions. Consequently, every human submission contributing to the Human Top, mean, and standard deviation reported for a given competition was produced by a participant with no design or curation role in that specific competition. Personally Identifying Information and Sensitive Content: All datasets are used in their publicly released forms. Sensitive biomedical datasets such as OASIS-3 LaMontagne et al. (2019) and ABIDE Craddock et al. (2013) follow their original de-identification protocols and data-use agreements; only aggregate per-task scores are reported. The GitHub Social Network dataset (Rozemberczki et al., 2019) contains public GitHub usernames that are already openly available; the usernames are not the target of prediction and no additional de-anonymization is performed. The Elliptic dataset (Weber et al., 2019) uses pseudo-anonymous transaction identifiers with no link to real identities. None of the datasets contain free-form user-generated text in their released versions, so no offensive-content screening was required. Licenses: All datasets are used under their original public licenses (MoleculeNet, Cora, OGB, OASIS-3 under data-use agreement, Elliptic, GossipCop, GitHub Social Network, etc.). LLMs are accessed under the providers’ API terms of service. GNN-CB itself, including the benchmark repositories, evaluation pipeline, and leaderboard infrastructure, will be released under the MIT license. Potential Risks: GNN-CB evaluates LLM-generated GNN code that, if deployed in safety-critical domains such as medical diagnosis, financial-fraud detection, or neuroimaging without expert verification, could amplify model failures or propagate dataset biases. We release the benchmark for controlled research and hands-on GNN coding practice, and discourage direct deployment of LLM-generated solutions without expert review. Use of AI Assistants. AI assistants were used solely for editorial and coding support during paper preparation, including grammar correction, LaTeX formatting, table and figure layout, and minor stylistic polishing. AI assistants were not used to generate experimental results, write benchmark code, or produce scientific claims. All scientific content—research questions, benchmark design, dataset curation, experimental design, analyses, and conclusions—was conceived and verified by the human authors. The AI assistants used for paper preparation are distinct from the LLMs evaluated in the benchmark. References Alibaba Cloud (2024) Alibaba Cloud Qwen2.5-Coder-32B-Instruct. Note: https://huggingface.co/Qwen/Qwen2.5-Coder-32B-InstructAccessed: 2026-05-10 Cited by: Table 4, Table 3. Amgad et al. (2022) M. Amgad, L. A. Atteya, H. Hussein, K. H. Mohammed, E. Hafiz, M. A. Elsebaie, A. M. Alhusseiny, M. A. AlMoslemany, A. M. Elmatboly, P. A. Pappalardo, et al. NuCLS: a scalable crowdsourcing approach and dataset for nucleus classification and segmentation in breast cancer. GigaScience 11, p. giac037. Cited by: Appendix A. Anthropic (2026a) Anthropic Claude opus 4.6. Note: https://w.anthropic.com/claude/opusAccessed: 2026-05-10 Cited by: Table 4, Table 3. Anthropic (2026b) Anthropic Claude opus 4.7. Note: https://w.anthropic.com/claude/opusAccessed: 2026-05-10 Cited by: Table 4, Table 3. Cassano et al. (2023) F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman, et al. Multipl-e: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering 49 (7), p. 3675–3691. Cited by: §2. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §1, §1, §2. Chen et al. (2023) Y. Chen, Z. Ding, L. Alowain, X. Chen, and D. Wagner Diversevul: a new vulnerable source code dataset for deep learning based vulnerability detection. In Proceedings of the 26th international symposium on research in attacks, intrusions and defenses, p. 654–668. Cited by: Appendix A. Craddock et al. (2013) C. Craddock, Y. Benhajali, C. Chu, F. Chouinard, A. Evans, A. Jakab, B. S. Khundrakpam, J. D. Lewis, Q. Li, M. Milham, et al. The neuro bureau preprocessing initiative: open sharing of preprocessed neuroimaging data and derivatives. Frontiers in Neuroinformatics 7 (27), p. 5. Cited by: Appendix A, Personally Identifying Information and Sensitive Content:. Dauparas et al. (2022) J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. I. M. Wicky, A. Courbet, R. J. de Haas, N. Bethel, P. J. Y. Leung, T. F. Huddy, S. Pellock, D. Tischer, F. Chan, B. Koepnick, H. Nguyen, A. Kang, B. Sankaran, A. K. Bera, N. P. King, and D. Baker Robust deep learning–based protein sequence design using proteinmpnn. Science 378 (6615), p. 49–56. Cited by: §1. DeepSeek-AI (2026) DeepSeek-AI DeepSeek-V4 Pro: a reasoning-optimised open-weights llm. Note: https://huggingface.co/deepseek-aiAccessed: 2026-05-10 Cited by: Table 4, Table 3. Desikan et al. (2006) R. S. Desikan, F. Ségonne, B. Fischl, B. T. Quinn, B. C. Dickerson, D. Blacker, R. L. Buckner, A. M. Dale, R. P. Maguire, B. T. Hyman, et al. An automated labeling system for subdividing the human cerebral cortex on mri scans into gyral based regions of interest. Neuroimage 31 (3), p. 968–980. Cited by: Appendix A. Dou et al. (2021) Y. Dou, K. Shu, C. Xia, P. S. Yu, and L. Sun User preference-aware fake news detection. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, p. 2051–2055. Cited by: Appendix A. Dwivedi et al. (2023) V. P. Dwivedi, C. K. Joshi, A. T. Luu, T. Laurent, Y. Bengio, and X. Bresson Benchmarking graph neural networks. Journal of Machine Learning Research 24 (43), p. 1–48. Cited by: §2. Dwivedi et al. (2022) V. P. Dwivedi, L. Rampášek, M. Galkin, A. Parviz, G. Wolf, A. T. Luu, and D. Beaini Long range graph benchmark. Advances in Neural Information Processing Systems 35, p. 22326–22340. Cited by: §1, §2. Fischl (2012) B. Fischl FreeSurfer. Neuroimage 62 (2), p. 774–781. Cited by: Appendix A. Gilmer et al. (2017) J. Gilmer, S. S. Schüt, P. F. Riley, O. Vinyals, and G. E. Dahl Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning (ICML), p. 1263–1272. Cited by: §1. Google DeepMind (2025) Google DeepMind Gemini 2.5 pro. Note: https://deepmind.google/technologies/gemini/pro/Accessed: 2026-05-10 Cited by: Table 4, Table 3. Google DeepMind (2026) Google DeepMind Gemini 3 flash. Note: https://deepmind.google/technologies/gemini/flash/Accessed: 2026-05-10 Cited by: Table 4, Table 3. Hamilton et al. (2017) W. L. Hamilton, Z. Ying, and J. Leskovec Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 30. Cited by: §1. Hu et al. (2020) W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec Open graph benchmark: datasets for machine learning on graphs. Advances in neural information processing systems 33, p. 22118–22133. Cited by: §1, §2. Jain et al. (2024) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica Livecodebench: holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974. Cited by: §2, §3.3.1, §3.3. Jiang et al. (2024) X. Jiang, Y. Dong, L. Wang, Z. Fang, Q. Shang, G. Li, Z. Jin, and W. Jiao Self-planning code generation with large language models. ACM Transactions on Software Engineering and Methodology 33 (7), p. 1–30. Cited by: §3.3.2, §3.3. Jimenez et al. (2023) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan Swe-bench: can language models resolve real-world github issues?. In The twelfth international conference on learning representations, Cited by: §1, §1, §2. Kerepesi et al. (2017) C. Kerepesi, B. Szalkai, B. Varga, and V. Grolmusz The braingraph.org database of high resolution structural connectomes and the brain graph tools. Cognitive Neurodynamics 11, p. 483–486. External Links: Document Cited by: Appendix A. Lai et al. (2023) Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W. Yih, D. Fried, S. Wang, and T. Yu DS-1000: a natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, p. 18319–18345. Cited by: §2. LaMontagne et al. (2019) P. J. LaMontagne, T. L. Benzinger, J. C. Morris, S. Keefe, R. Hornbeck, C. Xiong, E. Grant, J. Hassenstab, K. Moulder, A. G. Vlassenko, et al. OASIS-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and alzheimer disease. medrxiv, p. 2019–12. Cited by: Appendix A, Personally Identifying Information and Sensitive Content:. Li et al. (2021) X. Li, Y. Zhou, N. Dvornek, M. Zhang, S. Gao, J. Zhuang, D. Scheinost, L. H. Staib, P. Ventola, and J. S. Duncan BrainGNN: interpretable brain graph neural network for fmri analysis. Medical Image Analysis 74, p. 102233. External Links: ISSN 1361-8415 Cited by: §1. Li et al. (2018) Y. Li, R. Yu, C. Shahabi, and Y. Liu Diffusion convolutional recurrent neural network: data-driven traffic forecasting. In International Conference on Learning Representations (ICLR), Cited by: §1. Li et al. (2022) Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode. Science 378 (6624), p. 1092–1097. Cited by: §1, §1, §2. Little (2007) M. Little Parkinsons. Note: UCI Machine Learning RepositoryDOI: https://doi.org/10.24432/C59C74 Cited by: Appendix A. Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in neural information processing systems 36, p. 21558–21572. Cited by: §2. Meta AI (2024) Meta AI Llama 3.3 70b instruct. Note: https://huggingface.co/meta-llama/Llama-3.3-70B-InstructAccessed: 2026-05-10 Cited by: Table 4, Table 3. Moonshot AI (2026) Moonshot AI Kimi K2.6: a long-context open-weights chat llm. Note: https://kimi.moonshot.cn/Accessed: 2026-05-10 Cited by: Table 4, Table 3. OpenAI (2026) OpenAI GPT-5.4 Codex. Note: https://openai.com/index/gpt-5/Accessed: 2026-05-10 Cited by: Table 4, Table 3. Peng et al. (2024) Q. Peng, Y. Chai, and X. Li Humaneval-xl: a multilingual code generation benchmark for cross-lingual natural language generalization. In Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (LREC-COLING 2024), p. 8383–8394. Cited by: §2. Rozemberczki et al. (2019) B. Rozemberczki, C. Allen, and R. Sarkar Multi-scale attributed node embedding. External Links: 1909.13021 Cited by: Appendix A, Personally Identifying Information and Sensitive Content:. Schlichtkrull et al. (2018) M. Schlichtkrull, T. N. Kipf, P. Bloem, R. van den Berg, I. Titov, and M. Welling Modeling relational data with graph convolutional networks. In The Semantic Web, p. 593–607. Cited by: §1. Sen et al. (2008) P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Galligher, and T. Eliassi-Rad Collective classification in network data. AI magazine 29 (3), p. 93–93. Cited by: Appendix A, Appendix A. Shu et al. (2020) K. Shu, D. Mahudeswaran, S. Wang, D. Lee, and H. Liu Fakenewsnet: a data repository with news content, social context, and spatiotemporal information for studying fake news on social media. Big data 8 (3), p. 171–188. Cited by: Appendix A. Weber et al. (2019) M. Weber, G. Domeniconi, J. Chen, D. K. I. Weidele, C. Bellei, T. Robinson, and C. E. Leiserson Anti-money laundering in bitcoin: experimenting with graph convolutional networks for financial forensics. arXiv preprint arXiv:1908.02591. Cited by: Appendix A, Personally Identifying Information and Sensitive Content:. Wu et al. (2018) Z. Wu, B. Ramsundar, E. N. Feinberg, J. Gomes, C. Geniesse, A. S. Pappu, K. Leswing, and V. Pande MoleculeNet: a benchmark for molecular machine learning. Chemical science 9 (2), p. 513–530. Cited by: Appendix A. Yang et al. (2024) J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press Swe-agent: agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37, p. 50528–50652. Cited by: §3.3.3, §3.3. Ying et al. (2018) R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), p. 974–983. Cited by: §1. Zeng et al. (2022) J. Zeng, Y. Zhang, Y. Shang, J. Mai, S. Shi, M. Lu, C. Bu, Z. Zhang, Z. Zhang, Y. Li, et al. CancerSCEM: a database of single-cell expression map across various human cancers. Nucleic acids research 50 (D1), p. D1147–D1155. Cited by: Appendix A. Zhuo et al. (2024) T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, et al. Bigcodebench: benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877. Cited by: §1, §1, §2, §3.3. Appendix A Competition Specifications This appendix provides detailed descriptions of the competitions included in GNN-CB, each following a standardized structure (task definition, dataset statistics, evaluation metric, and submission protocol) and a unified evaluation pipeline based on hidden test sets, secure submission handling, and automated GitHub Actions scoring to ensure reproducibility and fairness. To mitigate the potential effects of data contamination from widely used public datasets, benchmark competitions are not restricted to canonical dataset configurations. Instead, most competitions in GNN-CB apply controlled modifications before constructing the benchmark task. These modifications fall into three main categories: • Feature and data modifications. Some competitions modify the original data to increase task difficulty and reduce the transferability of memorized solutions. For example, C12 (Cora-GNN) injects Gaussian noise into node features, C10 (NetLinkArena) applies feature permutation and noise injection, and C13 (GraphFMD) combines feature perturbation with a temporal split and adjusted class balance. • Distribution changes and new data splits. Several competitions construct new training and testing settings that differ from the original benchmark. For example, C05 (GNN-BACE) introduces severe class imbalance, applies scaffold splitting, and removes pre-computed descriptors. C16 (Graph4ASD) introduces a cross-site distribution shift, while C14 (BrainAge) uses new age-stratified subject splits and a restricted set of imaging features. • Converting non-graph datasets into graph-learning tasks. Some competitions reformulate the original dataset into a graph-learning problem. For example, C11 (PARK-GNN) transforms the original tabular Parkinson’s dataset into a graph using similarity-based edges. Likewise, C06 (DiaGraph) and C07 (GRAPE) convert non-graph data into graph representations. Although the underlying data remain unchanged, the resulting graph-learning tasks have no direct public reference solutions. Across all competitions, the goal is the same: to reduce the direct transferability of memorized solutions from LLM pretraining while creating more realistic and challenging graph-learning tasks. The exact modification depends on the characteristics of each dataset and application. The benchmark spans a diverse set of graph problem categories, including homogeneous, heterogeneous, temporal, spatial, hierarchical, molecular, and generalization graphs. To fomally define, Homogeneous : Graph with uniform node and edge types, Heterogeneous : Graph with multiple node and edge type, Temporal : Graph changes over time, Spatial: Graph is embedded in a spatial domain where nodes and edges reflect physical or geometric proximity, Hierarchical: Graph with multi-level structure among nodes, Molecular: Graph represents molecular structures, where nodes are atoms and edges are chemical bonds, Generalization: Test graph is drawn from distributions different from the training graph. We acknowledge that some categories may overlap; however, we assign each competition to its most distinct category. For example, a molecular graph is also homogeneous, but it is labeled as Molecular to distinguish it from other types. Across all competitions, a unified submission and evaluation protocol is enforced to ensure fairness, reproducibility, and consistent leaderboard construction. Each participant is allowed a single final submission per competition version, and only the latest valid submission is considered for evaluation. Submissions that are not properly encrypted, fail decryption, or do not conform to the required format are automatically rejected and excluded from scoring. Leaderboard ranking follows a Kaggle-style tie-breaking rule, where identical scores are assigned the same rank, and no additional ordering is enforced between tied entries. C01: PROVEN-GNN PROVEN-GNN (PROgram Vulnerability Examination Network using Graph Neural Networks) is a graph classification task on heterogeneous graph representations of code functions. Each source code function is represented as a graph Gk=(Vk,Ek)G_k=(V_k,E_k), where nodes denote code fragments (methods, variable declarations, etc.) and edges represent the relationships between them (AST, CFG, etc.). The objective is to detect vulnerable code functions based on these representations. The dataset consists of 31093109 functions from the DiverseVul dataset Chen et al. [2023], with vulnerable functions representing 29%29\% of the data. The graph representations were generated using the tool Joern. It allows generating code property graphs that capture syntactic properties, control flows, and data dependencies. Each node is represented with 512512 features, while edge features are represented using a one-hot encoding of the edge type (1212 features). Participants receive training and test graph files containing node embeddings, edge indices, and edge features. The labels (11 for vulnerable and 00 for non-vulnerable) are provided only for the training graphs. Participants are then required to generate predictions for the test graphs and submit them in an encrypted form using the provided public key. An automated pipeline then validates, decrypts, scores, and updates the leaderboard, evaluating each submission using the macro F1-score. C02:CGCC The CGCC (City Graph Classification Challenge) is a graph-level classification task defined over real-world urban street networks. Each sample is a city represented as a homogeneous graph Gk=(Vk,Ek)G_k=(V_k,E_k), wherenodes denote road intersections and edges denote the road segments connecting them. Given a collection of such graphs =GkG=\G_k\, the objective is to learn a mapping f(Gk)→ykf(G_k)→ y_k that assigns each city to one of three layout classes: organic (0), grid (1), or hybrid (2). The dataset comprises street networks for 120 cities distributed across North America, South America, Europe, Asia, Africa, Oceania, the Middle East, and Central Asia, extracted from OpenStreetMap via OSMnx within a 500 m buffer around each city centre and restricted to driveable roads. The class distribution reflects naturally occurring urban patterns and is moderately imbalanced, with 37 grid, 31 organic, and 52 hybrid cities. Each graph retains node coordinates, edge geometries and lengths, and graph-level attributes such as the city name and layout type; for the evaluation split, the layout label is withheld. The corpus is partitioned 70/3070/30 into training and test sets with stratification by class. Participants are provided with serialized city graphs and a label table that associates each training graph with its target class. No engineered urban-form descriptors (e.g., circuity, intersection density, orientation entropy) are provided. Participants are required to generate a prediction file mapping each test city to its predicted class, encrypt it locally using the provided public key, and submit encrpted file through a pull-request-based workflow. An automated pipeline then validates, decrypts, and scores the submissions, updating the leaderboard. Each submission is evaluated using the macro F1-score. C03: Real Or Fake! The Real or Fake challenge is a graph classification task based on Twitter news propagation data from the GossipCop dataset [Shu et al., 2020, Dou et al., 2021]. The objective is to train a Graph Neural Network (GNN) that predicts whether a news article is real or fake by leveraging both node features and the underlying propagation structure. Given a collection of hierarchical graphs =GkG=\G_k\, where each graph Gk=(Vk,Ek)G_k=(V_k,E_k) represents the spread of a single news article, the task is to predict the label yky_k for each graph :f(Gk)→ykf(G_k)→ y_k where yk∈0,1y_k∈\0,1\, with 11 indicating verified news and 00 indicating fake news. In this setting, nodes correspond to either the root news article or users who interacted with it, while edges capture interactions such as retweets. Each node is associated with rich features, including textual embeddings (e.g., BERT and spaCy) and user profile attributes. The dataset is split into 3,826 training graphs, 546 validation graphs, and 1,092 test graphs. Participants are provided with the graph structure, node features, and data splits distributed across multiple files. They are then expected to generate predictions for the test graphs, encrypt them locally using the provided public key and and submit through a pull-request. An automated pipeline subsequently validates, decrypts, and evaluates the submissions, updating the leaderboard accordingly. Each submission is assessed using the accuracy score. C04: GLIMPS-GNN The GLIMPS-GNN (Graph-based Liquid-biopsy Inductive Modeling for PreeclampSia) challenge is an inductive node-classification task on a biological graph. Given G=(V,E)G=(V,E) with feature matrix ∈ℝ|V|×dX ^|V|× d, the goal is to learn a mapping f(vi)→yif(v_i)→ y_i where each node vi∈v_i is assigned a label yi∈0,1y_i∈\0,1\ corresponding to Control (0) and Preeclampsia (1). The training subgraph comprises cell-free RNA (cfRNA) from maternal plasma, while the inductive test set contains unseen placental RNA-seq samples. Models must therefore generalize across a substantial domain shift between liquid-biopsy and tissue transcriptomics, making this competition fall under the generalization category. Data are sourced from GEO: maternal plasma cfRNA (GSE192902) and placental RNA-seq (GSE234729). Nodes are individual samples; edges encode similarity and ancestry. The graph contains 320 nodes and 3200 edges. Node features are harmonized gene-expression profiles. The training set has 209 cfRNA samples; the test set has 111 placenta samples with moderately imbalanced classes. Participants are provided with the graph structure, node features, and data splits distributed across multiple files. They are then expected to generate predictions for the test graphs and encrypt them locally using the provided public key. After submission, an automated GitHub Actions pipeline subsequently validates, decrypts, and evaluates the submissions, updating the leaderboard accordingly. Each submission is assessed using the macro-averaged F1-score. C05: GNN_BACE GNN_BACE (BACE Inhibition Prediction) is a graph-level binary classification task whose primary objective is to predict whether a given molecule acts as an inhibitor of the β-secretase 1 (BACE-1) enzyme, a critical target in Alzheimer’s disease research. Formally, the objective is to learn a mapping f(Gk)→ykf(G_k)→ y_k for each molecule represented as molecular graphs Gk=(Vk,Ek)G_k=(V_k,E_k). Molecules are represented as undirected graphs Gk=(Vk,Ek)G_k=(V_k,E_k), where nodes correspond to atoms and edges correspond to chemical bonds. Each node is described by an 8-dimensional one-hot atom-type feature vector, while edges include simple bond-level attributes. Label is 11 for active inhibitors and inactive compounds as 00. Participants are provided only with atom features and connectivity information, without any pre-computed molecular descriptors (e.g., MolLogP, Polar Surface Area). The dataset for this challenge is derived from the BACE dataset in the widely used MoleculeNet benchmark suite [Wu et al., 2018]. Participants are then required to generate predictions for the test graphs and encrypt them locally using the provided public key. After submitting encrypted file using pull-request, an automated GitHub Actions pipeline subsequently validates, decrypts, and evaluates the submissions, updating the leaderboard accordingly. Each submission is assessed using the Macro F1F_1-Score. C06: DiaGraph The DiaGraph (Robust Diabetes Prediction on Noisy Graphs) is a transductive node-level classification task where each sample corresponds to a patient represented as a node in a homogeneous graph G=(V,E)G=(V,E). Nodes denote patients, while edges encode similarity relationships based on latent clinical affinities. The objective is to learn a mapping f(vi)→yif(v_i)→ y_i that predicts whether a patient is diabetic (11) or non-diabetic (00). Participants are provided only with node features and the adjacency matrix for all nodes, while labels are available only for the training nodes. For evaluation, participants generate a prediction file containing binary labels for all nodes in the hidden test set. The predictions are encrypted locally using the provided public key prior to submission, and only encrypted files are submitted to the repository. An automated evaluation pipeline decrypts the submissions, validates their format, computes the official score against hidden test labels, and updates the public leaderboard. Each submission is evaluated using the macro F1-score. C07: GRAPE The GRAPE (Graph Retinal Analysis for Prediction and Evaluation) challenge is a graph-level classification benchmark designed for diabetic retinopathy (DR) detection from retinal vessel graphs. Each sample corresponds to a retinal vascular network represented as a spatial graph Gk=(Vk,Ek)G_k=(V_k,E_k), where nodes denote vessel bifurcation or intersection points and edges represent vessel segments connecting them. Given a collection of retinal graphs =GkG=\G_k\, the objective is to learn a mapping f(Gk)→ykf(G_k)→ y_k that predicts whether the retinal image corresponds to a healthy subject (00) or a diabetic retinopathy patient (11). The benchmark is constructed from retinal vessel graphs extracted from three publicly available retinal imaging datasets: DRIVE, STARE, and HRF. In total, the dataset contains 70 graphs, comprising 48 healthy and 22 diabetic retinopathy cases. Specifically, the DRIVE subset contributes 20 graphs (17 healthy, 3 DR), STARE contributes 20 graphs (16 healthy, 4 DR), and HRF contributes 30 graphs (15 healthy, 15 DR). The dataset is partitioned into 55 training graphs and 15 hidden test graphs using stratified sampling to preserve class proportions across splits. Each retinal graph preserves the vascular connectivity structure derived from vessel segmentation outputs. Participants are provided with node features and the adjacency matrix for all graphs, while labels are available only for the training graphs. Participants are expected to generate a prediction file containing binary labels for all graphs in the hidden test set. The predictions are encrypted locally using the provided public key prior to submission, and only encrypted files are submitted to the repository. An automated GitHub Actions workflow securely injects the private decryption keys, decrypts the submissions, validates their format, computes the official score against the hidden test labels, and updates the public leaderboard. Each submission is evaluated using the macro F1-score. C08: Liar Nodes The Liar Nodes challenge is formulated as a transductive node classification problem over a homogenous graph derived from single-cell cancer transcriptomics data. Each node represents an individual cell, while graph edges encode contextual or biological similarity relationships between cells within the tumor microenvironment. Given a graph G=(V,E)G=(V,E), the objective is to learn a mapping f(vi)→yif(v_i)→ y_i that predicts whether a cell is cancerous or non-cancerous. The underlying data originate from the Cancer Single-cell Expression Map (CancerSCEM) resource [Zeng et al., 2022], which contains large-scale single-cell RNA sequencing data across multiple cancer types. The original dataset provides UMI count matrices, cell-type annotations, functional molecular descriptors, and cell-cell interaction information. For the purposes of this benchmark, the data are transformed into graph learning instances suitable for GNN training and evaluation. Node embeddings are intentionally altered and partially corrupted in order to create contradictions between intrinsic node representations and graph context. Participants are provided with graph-structured data derived from the CancerSCEM resource, together with training labels for supervised learning. Final predictions are submitted via a designated Google Form for evaluation and leaderboard ranking. An automated evaluation pipeline computes accuracy on hidden test labels and updates the public leaderboard accordingly. C09: GraphFMD The GraphFMD (Graph-based Financial Misconduct Detection) challenge is a temporal inductive node classification task defined over the Bitcoin transaction network. Each sample is a transaction represented as a node viv_i in a directed temporal graph G=(V,E)G=(V,E), where nodes denote individual Bitcoin transactions and edges denote the flow of Bitcoin between them. The objective is to learn a mapping f(vi)→yif(v_i)→ y_i that assigns each test-time transaction to one of two classes: illicit (1) or licit (2). The dataset is derived from the Elliptic Bitcoin transaction graph Weber et al. [2019] and is partitioned temporally: training nodes span time steps 1–34, comprising 16,658 labelled transactions, while test nodes are drawn from time steps 35 and beyond, comprising 8,896 transactions. The graph contains approximately 5 million directed edges. Each node is associated with a 165-dimensional feature vector consisting of 94 local transaction features (e.g., transaction amounts, fees, and input/output counts) and 71 aggregated neighbourhood features derived from one-hop neighbours; Gaussian noise has been injected into these features to simulate real-world data degradation. The dataset exhibits strong class imbalance, with illicit transactions constituting only 1–3% of all nodes. Participants are provided with node feature files and an edge list for the full graph, together with ground-truth labels for the training split. No hand-crafted forensic descriptors (e.g., mixing patterns, chain-length heuristics, or address-clustering signals) are supplied. Participants are expected to generate a prediction table mapping each test transaction identifier to its predicted class 1,21,2. Submissions are encrypted and uploaded via a secure Google Form, which triggers a GitHub Actions workflow that validates the submission, computes the performance against the hidden label file, and updates the public leaderboard automatically. Performance is evaluated on the hidden test split using the macro F1-score. C10: NetLinkArena The NetLinkArena (Link Prediction Challenge) is a link prediction task defined on a real-world scientific citation network. Each sample corresponds to a pair of scientific papers represented within a homogenous citation graph G=(V,E)G=(V,E), where nodes denote research papers and edges represent citation relationships between them. Given a partially observed graph and a set of candidate node pairs (u,v)(u,v), the objective is to learn a mapping f(u,v)→yf(u,v)→ y that predicts whether a citation link exists between the two papers, where y∈0,1y∈\0,1\, with 11 indicating the presence of a link and 00 indicating no link. The dataset is constructed from the original CiteSeer dataset Sen et al. [2008], in which additional perturbations and synthetic noise were introduced to generate a more challenging and realistic link prediction benchmark. The graph contains 3,327 scientific papers, each represented by a 2,742-dimensional feature vector that captures the semantic and structural properties of each document. Citation relationships are provided as source-target node pairs, where positive edges correspond to observed citations and negative edges represent non-citation pairs sampled from the graph. The dataset is partitioned into training, validation, and hidden test splits for supervised evaluation. Participants are provided with the node feature matrix, labeled source–target node pairs for training and validation, and unlabeled node pairs for inference. Participants are expected to generate a file mapping each test node pair to a predicted probability score indicating the likelihood of a citation link. Submissions are entered through a Google Sheets-based workflow, where each team records predictions in a designated submission sheet. A GitHub Actions workflow then validates the submission format and updates the leaderboard automatically. Performance is evaluated on the hidden test split using the AUC score. C11: PARK-GNN The PARK-GNN Challenge (Parkinson’s Acoustic Representation & Knowledge with Graph Neural Networks) is a transductive node classification task defined over a homogenous graph constructed from acoustic voice recordings. Each node viv_i in the graph G=(V,E)G=(V,E) represents a single voice recording, characterized by a 22-dimensional feature vector of acoustic measurements including vocal fundamental frequency statistics, jitter and shimmer variations, harmonics-to-noise ratios, and nonlinear dynamical complexity features. Edges are formed by a k-nearest-neighbor criterion (k=5k=5) over acoustic similarity in feature space, supplemented by subject-level connections that link recordings from the same individual. The objective is to learn a mapping f(vi)→yi∈0,1f(v_i)→ y_i∈\0,1\ that assigns each voice recording to either a healthy control (0) or a Parkinson’s Disease patient (1). The dataset is derived from the UCI Parkinson’s Dataset [Little, 2007], comprising 195 voice recordings collected from 31 subjects, of whom 23 were diagnosed with Parkinson’s Disease and 8 were healthy controls. The corpus is partitioned into a training set of 156 nodes (approximately 80%) and a hidden test set of 39 nodes (approximately 20%). Training labels are provided alongside the full graph structure; test labels are withheld and used exclusively within the automated evaluation pipeline. Participants are provided with the serialized graph structure, node feature matrices, and a label table mapping each training node to its ground-truth class. No engineered biomarker composites or pre-computed diagnostic descriptors are supplied.Participants are expected to generate a prediction table mapping each test node identifier to its predicted class. Submissions are encrypted locally using the provided public-key scheme and uploaded through a pull-request-based workflow. An automated GitHub Actions pipeline then validates, decrypts, scores, and updates the public leaderboard. Performance is evaluated on the hidden test split using the macro-averaged F1-score. C12: GNN-CoRA The GNN-CoRA (Cora Node Classification Challenge) competition is a node classification task defined over a homogenous citation-network graph derived from the Cora benchmark dataset [Sen et al., 2008]. The objective is to train a GNN that predicts the research-topic label of each publication node from both its textual feature representation and the surrounding citation structure. Formally, given a graph G=(V,E)G=(V,E) in which nodes correspond to publications and edges denote citation relationships, the task is to learn a mapping f(vi)→yif(v_i)→ y_i that assigns each node viv_i to one of seven research-topic classes. The dataset consists of 2,708 publication nodes connected by 5,429 citation edges. Each node is associated with a high-dimensional sparse bag-of-words representation extracted from the publication text, and the graph structure captures citation links between papers. The benchmark follows the standard citation-network split, containing 140 labelled training nodes, 500 validation nodes, and 1,000 hidden test nodes. To increase task difficulty and evaluate robustness, Gaussian noise is injected into the node-feature representations during benchmark construction. Participants are provided with graph connectivity information, node feature matrices, training and validation label tables, and identifiers for the hidden test nodes. The ground-truth labels for the test split remain concealed and are used exclusively within the automated evaluation pipeline. Participants are expected to generate a prediction table mapping each hidden test node to one of the seven target classes. Submissions are encrypted locally using the benchmark’s public-key encryption scheme and submitted through a pull-request-based workflow. An automated GitHub Actions evaluation pipeline then validates, decrypts, scores, and updates the leaderboard accordingly. Performance is evaluated on the hidden test set using classification accuracy. C13 : GTA The GTA (Graph Topology Ablation) challenge is formulated as a supervised binary graph classification problem over molecular graphs derived from the MUTAG dataset, a widely used benchmark in chemical informatics. Each graph Gk=(Vk,Ek)G_k=(V_k,E_k) represents a chemical compound in which nodes correspond to atoms and edges correspond to chemical bonds. The objective is to learn a mapping f(Gk)→ykf(G_k)→ y_k that predicts whether a molecule is mutagenic or non-mutagenic with respect to Salmonella typhimurium. The dataset contains 188 molecular graphs with an average of approximately 18 nodes and 40 edges per graph. Node attributes consist of categorical atom labels interpreted as node features. The benchmark is divided into two evaluation settings. In the ideal graph topology setting, participants evaluate their models on clean and unmodified node features. In the perturbed graph topology setting, node features are corrupted through a combination of feature distribution shift and Gaussian noise injection. Specifically, a constant offset δ is added to node features to simulate systematic domain shift: x←x+δ,x← x+δ, where δ corresponds to a predefined feature shift parameter. In addition, Gaussian noise is injected into node attributes according to: x←x+ϵ,ϵ∼(0,σ2),x← x+ε, ε (0,σ^2), where σ controls the noise magnitude. This competition falls under the generalization category due to pertubred test graphs. The dataset is partitioned into training and hidden evaluation splits using stratified sampling. The training split contains labeled molecular graphs together with graph identifiers and binary labels, while the evaluation split provides only graph identifiers and hides the ground-truth labels used for leaderboard scoring. Models are therefore trained exclusively on clean graph features and subsequently evaluated on both clean and perturbed graph representations. Performance is evaluated using the macro F1 score, which balances precision and recall across both classes and provides a more reliable evaluation criterion than classification accuracy alone. Each submission is scored separately under the clean and perturbed settings, yielding two metrics: F1idealF_1^ideal and F1perturbedF_1^perturbed. To quantify robustness, the benchmark additionally computes the Robustness Gap: RobustnessGap=F1ideal−F1perturbed.Robustness\ Gap=F_1^ideal-F_1^perturbed. Smaller robustness gaps indicate models that maintain stable performance under feature corruption. Participants submit encrypted prediction files for both clean and perturbed evaluation settings. An automated evaluation pipeline decrypts submissions securely, computes official metrics against hidden labels, and updates the public leaderboard automatically. Leaderboard ranking prioritizes the highest perturbed macro-F1F_1 score, followed by the smallest robustness gap as a secondary criterion. C14: BrainAge The BrainAge (Brain-Age Prediction Challenge) challenge is a graph-level regression task defined over structural neuroimaging data. The dataset consists of homogenous brain graphs =GkG=\G_k\, where each graph Gk=(Vk,Ek)G_k=(V_k,E_k) is constructed from structural and diffusion MRI data from a specific subject’s imaging session. Nodes VkV_k correspond to cortical brain regions defined by the Desikan-Killiany atlas Desikan et al. [2006] (N=68N=68), and edges EkE_k encode structural connectivity derived from diffusion MRI. Given a graph GkG_k, the objective is to learn a mapping f(Gk)→ykf(G_k)→ y_k, where yk∈ℝy_k denotes the age of an individual at the time of image acquisition. The dataset is constructed from the OASIS-3 longitudinal neuroimaging cohort LaMontagne et al. [2019], restricted to subjects who remained cognitively normal across all visits. Cortical measurements were obtained from preprocessed FreeSurfer Fischl [2012] outputs available through the Neuroimaging Informatics Tools and Resources Clearinghouse (NITRC). Structural connectivity information is sourced from the PIT Bioinformatics Group Kerepesi et al. [2017]. The final dataset comprises 431 MRI sessions from 288 unique subjects across multiple longitudinal time points. For every cortical region, two node features are provided: cortical thickness and gray matter volume. Edge weights are defined by diffusion MRI streamline counts between regions. To prevent information leakage arising from repeated longitudinal observations of the same participant, training, validation, and test splits were performed at the subject level using age-stratified sampling. For evaluation, participants generate age predictions for test MRI sessions and submit them as an encrypted prediction table through a pull-request-based GitHub workflow. The automated GitHub pipeline validates the submission, decrypts the predictions, computes the evaluation score, and updates the leaderboard upon merge. Since this competition involves a single regression target, performance is evaluated on the hidden test split using MAE. C15: OctoNode Cup OctoNode Cup (GitHub Social Network Classification) is a transductive node classification task focused on classifying users in a homogenous GitHub social network. In this network, represented as a graph G=(V,E)G=(V,E), each node corresponds to a GitHub user and edges represent connections between users. The task aims to predict a binary label for each node based on its associated features. The objective is to learn a mapping f(vi)→yif(v_i)→ y_i where viv_i denotes a GitHub user and yi∈0,1y_i∈\0,1\ represents the predicted class, with 11 corresponding to a Web Developer and 00 corresponding to a Machine Learning Developer. The dataset is derived from a real GitHub developer network introduced by [Rozemberczki et al., 2019] and represents the platform as a graph. Each node is identified by a unique ID and is associated with a set of features used for prediction. The dataset contains approximately 37,700 nodes and 289,000 edges. It is divided into two partitions: a training set comprising 70% of the data, which includes both features and labels, and a test set comprising the remaining 30%, where only the features are provided while the labels remain hidden for evaluation purposes. Participants are provided with input files required for training and evaluation. Predictions must be generated locally and encrypted prior to submission using the provided encryption method. Only encrypted submission files are uploaded through the repository-based submission system. An automated GitHub pipeline validates the submission, decrypts the predictions, and computes the evaluation score. Submissions are evaluated on the hidden test set using the Macro F1-score. C16: Graph4ASD Challenge The Graph4ASD (Graph Neural Networks for the classification of Autism Spectrum Disorder) challenge is a binary graph classification task over brain networks derived from resting-state functional magnetic resonance imaging (rs-fMRI) recordings of the human brain. Each sample corresponds to a single subject and is represented as a homogenous graph Gk=(Ak,Xk)G_k=(A_k,X_k), where Ak∈ℝn×nA_k ^n× n is the weighted adjacency matrix encoding functional connectivity between brain regions and XkX_k is the node feature matrix. Given a collection =GkG=\G_k\, the task is to learn a mapping f(Gk)→yk∈0,1f(G_k)→ y_k∈\0,1\ that assigns each subject to one of two diagnostic classes: Typical Control (0) or Autism Spectrum Disorder (1). The dataset is drawn from the Autism Brain Imaging Data Exchange (ABIDE) Craddock et al. [2013], a multi-site repository of anonymised rs-fMRI scans collected at seventeen international acquisition sites. The full corpus contains 1,009 subjects (516 ASD, 51.14%; 493 controls, 48.86%), yielding a near-balanced two-class distribution. Each subject’s brain volume is parcellated with the Craddock 200 atlas, so every graph has exactly n=200n=200 nodes corresponding to predefined regions of interest, with identical node ordering across samples. A challenge-specific subset, preprocessed exclusively for this benchmark, is partitioned into 484 training and 153 evaluation graphs; participants are required to use the data as released, without alternative preprocessing pipelines or auxiliary neuroimaging corpora. For every subject in the training and evaluation partitions, participants receive a connectivity matrix and a node feature matrix; the training partition is additionally accompanied by subject-level diagnostic labels. Node features are restricted to identity descriptors, so any discriminative signal must be recovered from the connectivity structure over the 200 Craddock regions alone. The training and evaluation partitions are drawn from non-identical site distributions, requiring models to learn site-invariant connectivity representations. For evaluation, participants produce a prediction file mapping each test subject to a predicted class. The file is encrypted locally using the provided public key and submitted through a pull-request-based workflow. An automated pipeline validates, decrypts, scores, and updates the leaderboard. Performance on the hidden test split is reported using the macro F1-score. C17: Mix2Print Mix2Print (Learning Material Interaction Physics for identifying parameters of 3D Bioprinting) is a graph regression challenge focused on predicting optimal 3D bioprinting parameters from bioink formulation graphs. The competition models each bioink formulation as a homogenous graph, where nodes correspond to biomaterials and edges represent potential interactions between materials in the mixture. The task aims to learn the underlying material interaction physics governing extrusion-based bioprinting. Given a graph formulation Gi=(Vi,Ei,Xi)G_i=(V_i,E_i,X_i), participants are required to predict three continuous targets: f(Gi)→yif(G_i)→ y_i where yi=(pressure,temperature,speed)y_i=(pressure,temperature,speed) corresponding to Pressure (kPa), which represents extrusion force; Temperature (∘C), which represents printing temperature; and Speed (m/s), which represents print-head velocity. The dataset represents bioink formulations as fully connected graphs, reflecting the assumption that all biomaterials in a formulation can potentially interact with each other. Node features consist of one-hot encoded material identity features and normalized concentration value. Thus, the total node feature dimensionality is approximately D≈31D≈ 31. The dataset contains 423 bioink formulations collected from peer-reviewed publications, 30 biomaterials appearing at least five times ,303 training samples and 120 test samples. The train-test split follows a stratified grouping strategy based on printing temperature regimes (hydrogel versus thermoplastic formulations). The dataset captures diverse biomaterial categories including alginates, gelatins, polymers, ceramics, and natural biomaterials. Participants are provided with graph-based input files for model development and evaluation. Participants are required to generate predictions for the hidden test set in CSV format. To preserve leaderboard integrity, all submissions must be encrypted prior to upload using the provided encryption utility. The encrypted submission file is then uploaded through a GitHub pull request. Since this competition involves three regression targets that may have different value ranges, submissions are evaluated using NMAE, which normalizes the errors across targets before aggregation. C18: THInC THInC (Tumor Histopathology Inductive Classification) is formulated as an inductive node classification problem over spatial cell graphs derived from H&E-stained histology images. Each graph Gk=(Vk,Ek)G_k=(V_k,E_k) corresponds to a single patient or tissue slide, where nodes represent individual cell nuclei and edges encode spatial relationships between nearby cells. Given a set of training graphs trainG_train, the goal is to learn a mapping f(vi)→yif(v_i)→ y_i that assigns each cell to one of four classes: Tumor (0), Stromal (1), Lymphocyte (2), or Macrophage (3). Importantly, predictions must generalize to entirely unseen graphs corresponding to new patients, making this an inductive rather than transductive learning setting. The dataset is derived from the NuCLS breast cancer histopathology dataset [Amgad et al., 2022] and is transformed into spatial cell graphs using a biologically grounded pipeline. Starting from raw histology images, cell nuclei are detected and represented as bounding boxes, which are then converted into centroid coordinates (x,y)(x,y) forming graph nodes. Graph edges are constructed using a k-nearest neighbor (k-N) approach with k=5k=5, where each cell is connected to its closest spatial neighbors within the same tissue image. The inductive split is performed at the image level rather than the node level. Specifically, 80% of tissue images are used for training, while the remaining 20% are held out as completely unseen test patients. Node features consist of normalized spatial coordinates (x,y)(x,y) along with morphological descriptors such as nucleus width and height, standardized to zero mean and unit variance to ensure stable training. To ensure data privacy, participants are required to encrypt their submission CSV files prior to upload. The evaluation system decrypts and scores predictions in a secure environment, preventing exposure of ground-truth labels from unseen patients during the leaderboard process. Performance is evaluated using accuracy. Appendix B Human Evaluation Details This appendix provides additional implementation details for the human evaluation track of GNN-CB. B.1 Human Implementations All human participants follow a uniform implementation protocol designed to ensure reproducibility, prevent test-set leakage, and isolate model design choices as the primary source of performance variation across submissions. Each competition enforces a fixed wall-clock budget of three hours of CPU time per task, with hyperparameter tuning capped at five training runs per task and a single final submission per benchmark version to discourage test-set overfitting. All random seeds are fixed to seed = 25, training loss curves must be plotted to verify proper convergence and detect overfitting (with early stopping permitted), and the exact source code that produced each submitted prediction file is retained alongside the recorded training time and hardware specifications. Human participants never access the hidden test instances, their labels, or any hidden-test performance signal. They receive only the designated training and validation splits, and all model selection and hyperparameter tuning are conducted locally using validation-set performance within the five-run budget. Each submitted solution is evaluated only once through the unified hidden-test evaluation pipeline. Submitted solutions must be implemented using Graph Neural Network architectures only; convolutional, recurrent, and other non-graph spatial deep-learning models are prohibited, and copy-pasting external solutions is not permitted. LLM assistance is restricted to low-level utility functions and must not expose the model to any task-specific information (e.g., competition name, dataset identity, or feature schema) that could induce data leakage. For each competition, every participant maintains a single documentation file containing the hyperparameter-search strategy used (manual, grid, or random), all configurations attempted within the five-run budget, and the exact prompts and LLM-generated snippets reused in the solution. Together, these records provide a post-hoc audit trail for verifying that the submitted prediction file is consistent with the declared GNN-based solution, documented hyperparameter search, and permitted low-level LLM assistance. B.2 Benchmark Composition Attempted by Human Participants Figure 5 summarizes the composition of the 18 GNN-CB competitions attempted by human participants across four characterizing dimensions: graph category, difficulty tier, task type, and application domain. The benchmark is dominated by homogeneous graphs (55.6%55.6\%), with the remainder distributed across spatial, generalization, temporal, heterogeneous, hierarchical, and molecular settings. Difficulty tiers are skewed toward Medium (44.4%44.4\%) and Hard (38.9%38.9\%) competitions, with only three Easy tasks. Node classification and graph classification together account for 83.3%83.3\% of the tasks, complemented by one edge-prediction and two graph-regression competitions. Application domains span medical, biomedical, academic, neuroimaging, social, bioinformatics, bioprinting, cybersecurity, finance, and urban data, reflecting the benchmark’s emphasis on real-world heterogeneity rather than a single application focus. Figure 5: Composition of the 18 GNN-CB competitions attempted by human participants, summarized along four axes: graph category (top-left), difficulty tier (top-right), task type (bottom-left), and application domain (bottom-right). Per-slice counts and competition codes are listed in the legend below each donut. B.3 Geographic Distribution of Participants GNN-CB draws on a globally distributed cohort of human contributors. Figure 6 shows the locations of the 18 participating universities across 12 countries spanning four continents, with each country annotated by the competition(s) co-designed by its contributing team(s). The cohort represents Africa (Algeria, Egypt, Ghana, Tunisia, Uganda), the Americas (Peru, USA), Asia (India, Nepal, Pakistan), and Europe (Turkey, UK), reflecting the benchmark’s emphasis on broad geographic and institutional representation rather than concentration in a single research hub. This geographic diversity is intentional: it ensures that the curated competitions, datasets, and domain conventions in GNN-CB are not dominated by the priorities of any single regional research community, and that the human performance baseline reported in Table 2 reflects a heterogeneous expert pool rather than a homogeneous local cohort. Figure 6: Global distribution of the 18 participating universities that co-designed and evaluated the GNN-CB benchmark. Each highlighted country is annotated with the competition(s) contributed by its team(s); the colour key appears in the legend. The cohort spans 12 countries across Africa, the Americas, Asia, and Europe. B.4 Human Evaluation Pipeline Figure 7 summarizes the end-to-end human evaluation workflow used in GNN-CB, from repository fork through encrypted pull-request submission to automated GitHub Actions scoring. InputDevelopmentSubmissionScoringData filesnodes ⋅· edgesadjacency ⋅· featuresPlan & designGNN-onlyarchitectureloss, metricTrainseed = 253 CPU hrsHP tune≤5≤ 5 runsearly stopPredictpredictions.csvEncryptpredictions.csv.encmetadata.jsonUpload to GitHubpush to fork branchPull requestagainst main repoGitHub Actions scoringdecrypt → score on hidden testLive leaderboardPR auto-closedDocumentation log (maintained throughout Development)hyper. tuinng ⋅· runtime ⋅· hardware ⋅·⋅· epoch Figure 7: Human evaluation pipeline of GNN-CB. Participants prepare graph data (nodes, edges, adjacency, and feature matrices) from the released data files, then design and train a GNN-only solution under fixed reproducibility constraints (seed = 25, three CPU hours, at most five hyperparameter-tuning runs, early stopping permitted). They generate a predictions.csv file, encrypt it locally into predictions.csv.enc together with a metadata.json, push the encrypted submission to their GitHub fork, and open a pull request against the main repository. An automated GitHub Actions workflow decrypts the submission, scores it against the hidden test labels, updates the live leaderboard, and closes the pull request. A documentation log—maintained throughout development—records the hyperparameter-search strategy, runtime, hardware specifications, loss-curve verification, and any LLM-assisted utility usage. Appendix C LLM Evaluation Details This appendix provides additional implementation details for the LLM evaluation framework used in GNN-CB, including the frozen prompt construction, a representative plan-then-code example, the prompt template, and the orchestration pipeline used during API-based evaluation. As in the human evaluation track, LLMs never access the hidden test set or its labels. Their only pre-submission feedback consists of local execution tracebacks generated during the bounded execute-and-repair loop. Table 4 summarizes the set of LLMs evaluated in GNN-CB, grouped by access type (proprietary vs. open-source) and functional category (chat vs. reasoning). For each model, we report the exact API engine identifier used in our orchestration pipeline along with a reference to the official provider documentation. Table 4: List of LLMs evaluated in GNN-CB, grouped by access type (proprietary vs. open-source). Default decoding parameters are temperature = 0 and max_output_tokens = 10K, applied where the provider’s API exposes these settings. API-only (non-agent) runs follow this per-call output budget, while agent-mode runs may consume substantially more tokens across multiple internal tool calls and extended-thinking traces, governed by each agent’s native session budget rather than the per-call cap. The Model Engine Name column lists the exact API identifier used in our orchestration scripts; the Source column links to the provider’s official model documentation. # Model Name Model Engine Name Source Proprietary Chat LLMs 1 Claude Opus 4.6 Anthropic [2026a] claude-opus-4-6 Link 2 Gemini-3 Flash Google DeepMind [2026] gemini-3.0-flash Link Open-source Chat LLMs 3 Llama-3.3 70B Meta AI [2024] Llama-3.3-70B-Instruct Link 4 Qwen2.5-Coder 32B Alibaba Cloud [2024] Qwen2.5-Coder-32B-Instruct Link 5 Kimi k2.6 Moonshot AI [2026] kimi-k2.6 Link Open-source Reasoning LLMs 6 DeepSeek v4 Pro DeepSeek-AI [2026] deepseek-v4-pro Link Proprietary Reasoning LLMs 7 Gemini 2.5 Pro Google DeepMind [2025] gemini-2.5-pro Link 8 Claude Opus 4.7 Anthropic [2026b] claude-opus-4-7 Link 9 GPT-5.4 Codex OpenAI [2026] gpt-5.4-codex Link C.1 Frozen Prompt Construction RepositoryREADME.md/data/tree.txtsubmit_spec.mdrequirements.txt.github/workflow/prompt.mdslot filling README DATA TREE SUBMIT REQS Frozen prompttask-agnosticnever modifiedshared acrossall 18 competitionszero-shotLLMGPT-5.4Claude opusGemini 2.5Llama-3-70BOutput ⟨ ⟩ ⟨ ⟩ standaloneexecutableextractinstantiatepromptgenerate Figure 8: A frozen prompt construction is instantiated from the repo, and then it is passed to the LLM, which outputs a plan and code. Figure 8 illustrates how repository components are inserted into structured slots without manual modification, producing an instantiated prompt shared across all competitions. C.2 Plan-Then-Code Example 1⃝ Plan (zero-shot, plan-then-code) Step 1 Feature engineering Concatenate BERT (768) + spaCy (300) + profile (10) → 1078-dim; StandardScaler fit on training nodes only. Step 2 Graph construction Custom Dataset builds per-graph Data(x, edge_index, y, root_idx); node indices remapped to local 0..N−1N-1. Step 3 Model architecture Stacked GATConv + global_mean_pool, concatenated with a dedicated root-node MLP; sigmoid head. Step 4 Training Adam (η=10−3η\!=\!10^-3), BCELoss, batch = 64, early stopping on val accuracy (patience = 5). Step 5 Submission Inference on hidden test split → write predictions.csv with columns (id, y_pred). 2⃝ Code (LLM-generated, abridged) ⬇ # Step 1: features x = np.hstack([bert, spacy, StandardScaler().fit_transform(profile_train)]) # 1078-d # Step 2-3: model class GossipGNN(n.Module): def __init__(self, in_ch, hid=128): super().__init__() self.gat1 = GATConv(in_ch, hid, heads=4) self.gat2 = GATConv(hid * 4, hid) self.root = n.Linear(in_ch, hid) self.head = n.Linear(hid * 2, 1) def forward(self, x, edge_index, batch, root_idx): h = F.elu(self.gat1(x, edge_index)) h = self.gat2(h, edge_index) g = torch.cat([global_mean_pool(h, batch), self.root(x[root_idx])], dim=1) return torch.sigmoid(self.head(g)) # Step 4: training loop with early stopping opt = Adam(model.parameters(), lr=1e-3) for epoch in range(200): train_one_epoch(model, train_loader, opt, BCELoss()) if early_stop(evaluate(model, val_loader)): break # Step 5: submission preds = (model(test_batch).sigmoid() > 0.5).long() pd.DataFrame("id": test_ids, "y_pred": preds)\ .to_csv("predictions.csv", index=False) Figure 9: Plan-then-code paradigm illustrated for C03 (Real or Fake!): a 5-step structured plan (left) and the LLM-generated, executable implementation skeleton (right). Figure 9 presents a representative plan-then-code output for C03 (Real or Fake!), with the high-level reasoning plan on the left and the LLM-generated executable code on the right. C.3 Frozen Prompt Template To minimize prompt-engineering bias and ensure cross-model comparability, every LLM evaluation in GNN-CB uses the same task-frozen prompt template, instantiated through automatic slot filling from the competition repository. The template encodes the inputs the model receives, the structured output it must produce, and the execution constraints under which the generated solution will be evaluated. Listing 1 reproduces a simplified version of this template; the same string is shared verbatim across all 18 competitions, with only the slot contents (README, repository tree, dataset summary, submission specification, and allowed libraries) varying per task. C.4 LLM Evaluation Pipeline Listings The conceptual distinction between the non-agent (API-only) and agent-based evaluation modes is described in Section 3.3.4. Listing 2 reproduces the simplified Python orchestration loop used in the non-agent mode; the agent-based mode has no equivalent participant-side script, as it relies on the autonomous coding system’s native repository inspection, code execution, and repair capabilities, governed by each agent’s session budget. C.5 Performance Analysis of LLMs Across Competitions To assess the comparative performance of leading LLMs against human baselines, we evaluate their capabilities across Easy, Medium, and Hard classification tasks in the GNN-CB benchmark from three perspectives: a gap analysis quantifying performance differences relative to the human-top score, an average human–LLM comparison, and a category-based comparison of how different model types perform across difficulty levels. Gap analysis: For Easy classification tasks (Figure 10), proprietary models except from GPT 5.4 Codex show minimal deviation, with best-case gaps approaching zero and worst-case gaps remaining moderate. In contrast, open-source models exhibit larger negative gaps across all models, indicating instability and weaker alignment with human-top performance. On Medium tasks (Figure 11), the performance gap widens across all model categories. Proprietary models continue to outperform open-source models in terms of mean gap and exhibit more consistent behavior overall, whereas open-source models show substantially larger negative gaps and higher variability across competitions. ⬇ You are solving a GNN coding competition. You will receive: (1) competition README (2) repository tree (3) dataset summary (4) submission format (5) allowed libraries Produce: <plan> ...structured reasoning plan... </plan> <code> ...standalone executable implementation... </code> Constraints: - runnable as ‘python solution.py‘ - CPU-only execution - no external downloads - must generate predictions.csv Listing 1: Simplified frozen prompt template used across all 18 GNN-CB competitions. On Hard tasks (Figure 12), the same overall trend persists, although the gap further increases for nearly all models. Notably, the performance of Gemini 2.5 Pro drops considerably, while Kimi k2.6 reaches a comparable level on several tasks. Despite these improvements, the overall gap between LLMs and human participants remains large on hard competitions. Across all difficulty levels, no substantial performance difference is observed between chat and reasoning LLMs. ⬇ prompt = instantiate_frozen_prompt(repo) for attempt in range(REPAIR_LIMIT): # k <= 5 response = call_llm(prompt) # API request plan = extract_plan(response) code = extract_code(response) write_solution(code) # -> solution.py result = execute_solution() # CPU-only sandbox if valid_submission(result): # schema + content break prompt += format_traceback(result) # appended for repair Listing 2: Simplified non-agent orchestration loop with bounded execute-and-repair iterations (k≤5k\!≤\!5). Average human–LLM comparison: Figure 13 analyzes LLM and human average scores across Easy, Medium, and Hard competitions on a per-competition basis. Although LLM averages occasionally surpass human averages on individual competitions, humans still achieve higher average scores in 10 out of the 16 classification competitions, indicating that human participants remain more consistently reliable overall. Moreover, as task difficulty increases from Easy to Hard, LLM performance declines similarly to human performance, with larger drops observed on harder tasks. Category-based comparison: Finally, we analyse model performance by category across the Easy, Medium, and Hard classification tasks using each model’s best, mean, and worst accuracy. On Easy tasks (Figure 14), proprietary models achieve high performance with a small variablity between best and worst scores, indicating strong consistency, while open-source models remain competitive but display wider variability. On Medium and Hard tasks (Figures 15 and 16), mean scores decline across nearly all models and the best–worst variance widens, indicating reduced stability as complexity increases. The steep decline in mean and worst-case accuracy highlights the limitations of current foundational architectures under higher complexity. Figure 10: Best, Mean, and Worst Gap performance for leading LLMs on the Easy classification tasks of the GNN-CB benchmark, grouped by model family, where the Gap is defined as Gap = Score - Human Top Figure 11: Best, Mean, and Worst Gap performance for leading LLMs on the Medium classification tasks of the GNN-CB benchmark, grouped by model family, where the Gap is defined as Gap = Score - Human Top Figure 12: Best, Mean, and Worst Gap performance for leading LLMs on the Hard classification tasks of the GNN-CB benchmark, grouped by model family, where the Gap is defined as Gap = Score - Human Top Figure 13: LLM vs Human Average Performance across difficulty levels Figure 14: LLM performance grouped by model family on Easy classification tasks. Performance metrics capture the best, mean, and worst scores achieved by each model. Figure 15: LLM performance grouped by model family on Medium classification tasks. Figure 16: LLM performance grouped by model family on Hard classification tasks. Error Analysis: Severe underperformance was concentrated primarily among open-source models: 17 of the 20 cases (85%) were produced by open-source LLMs, compared with only three (15%) by proprietary models. When normalized by the number of evaluated model–task pairs, severe failures occurred in 17/64 (26.6%) open-source pairs and 3/80 (3.8%) proprietary pairs. Chat models accounted for 14 cases and reasoning models for six, corresponding to failure rates of 14/80 (17.5%) and 6/64 (9.4%), respectively. However, given the small and unbalanced number of models across categories, particularly the presence of only one open-source reasoning model, these differences should be interpreted as descriptive patterns rather than inherent effects of model family or accessibility. Acknowledgments This research was carried out as part of the BASIRA Lab Global Consortia Projects within the GNNs for Rising Stars Program, which Dr Rekik founded, organized, and taught, and which gave the authors their shared foundation in graph representation learning and directly shaped the design of the competitions presented here; further details on the program and its competitions are available at https://basiralab.github.io/GNN-CB/. We are grateful to have carried out this work as a global consortium of early-career researchers spanning 24 institutions across 16 countries.