Paper deep dive
WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting
Zhaokai Wang, Tianlin Gui, Jiayuan Rao, Shangzhe Di, Yihong Tang, Dingli Liang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 7/21/2026, 6:17:41 AM
Summary
The paper introduces WorldCupArena, a dynamic benchmark for evaluating language models and deep-research agents on football forecasting using the 2026 FIFA World Cup. It assesses models across five layers: match result/score, players/lineups, events, tactics/statistics, and competition outcome. The study compares 13 systems under two evidence settings (standardized context vs. self-directed search) against betting-market and human baselines, finding that while result accuracy gains are small, detailed predictions reveal significant model differences.
Entities (10)
Relation Signals (7)
WorldCupArena → evaluates → 2026 FIFA World Cup
confidence 95% · The 2026 FIFA World Cup is its first evaluation...
WorldCupArena → usesmetric → Result Accuracy
confidence 90% · We report result accuracy, exact-score accuracy...
WorldCupArena → usesmetric → Scoreline Score
confidence 90% · We report result accuracy... and a scoreline score...
Polymarket → comparedwith → WorldCupArena
confidence 85% · Compared with betting-market and human-fan baselines...
S1 → providesevidence → Claude Opus 4.7
confidence 85% · Claude Opus 4.7 (Thinking) [1] S1
S2 → providesevidence → GPT-5.4
confidence 85% · GPT-5.4 (Search) [30] S2
Claude Opus 4.7 → achieveshigherscoreline → GPT-5.4
confidence 80% · Table 2 shows Claude Opus 4.7 Scoreline 63.16 vs GPT-5.4 Scoreline 56.05
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Predicting a football match before kickoff requires more than knowing past results: a model must use changing information and make a clear prediction before the answer is available. We present WorldCupArena, a dynamic benchmark for language models and deep-research agents. The 2026 FIFA World Cup is its first evaluation, and the same process can be reused for future leagues and cups. Before each match, a model either receives a common evidence package or searches for information itself. It predicts the result and score, likely players and events, match statistics, and the outcome of the competition. After the match, these predictions are compared with the recorded result. We report result accuracy, exact-score accuracy, and a scoreline score that gives some credit when a predicted score is close but not exact, together with scores for the other prediction tasks. Across 104 matches and 13 systems, models with similar result accuracy differ more clearly on detailed predictions. Compared with betting-market and human-fan baselines, the best system shows only small gains in result and exact-score accuracy, but a clearer gain in Scoreline. New schedules can be added as they begin, allowing the benchmark to evaluate future models without using outcomes that are already known. Code, prompts, predictions, and evaluation scripts are open sourced at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2607.18084v1
- Canonical: https://arxiv.org/abs/2607.18084v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
56,120 characters extracted from source content.
Expand or collapse full text
WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting Zhaokai Wang Shanghai Jiao Tong University Shanghai, China wangzhaokai@sjtu.edu.cn Tianlin Gui Shanghai Jiao Tong University Nanjing University Shanghai, China Jiayuan Rao Shanghai Jiao Tong University Shanghai, China Shangzhe Di Shanghai Jiao Tong University Shanghai, China Yihong Tang McGill University Montreal, Canada Dingli Liang University College London London, UK Abstract Predicting a football match before kickoff requires more than know- ing past results: a model must use changing information and make a clear prediction before the answer is available. We present World- CupArena, a dynamic benchmark for language models and deep- research agents. The 2026 FIFA World Cup is its first evaluation, and the same process can be reused for future leagues and cups. Before each match, a model either receives a common evidence package or searches for information itself. It predicts the result and score, likely players and events, match statistics, and the outcome of the competition. After the match, these predictions are compared with the recorded result. We report result accuracy, exact-score accuracy, and a scoreline score that gives some credit when a predicted score is close but not exact, together with scores for the other prediction tasks. Across all 104 matches and 13 systems, models with similar result accuracy differ more clearly on detailed predictions; four systems predicted champion Spain, and two of them also recov- ered the exact final pairing. Compared with betting-market and human-fan baselines, the best system shows only small gains in result and exact-score accuracy, but a clearer gain in Scoreline. New schedules can be added as they begin, allowing the benchmark to evaluate future models without using outcomes that are already known. Code, prompts, predictions, and evaluation scripts are open sourced at https://github.com/wzk1015/WorldCupArena. Keywords sports forecasting, large language models, deep research agents, football, evaluation, benchmark 1 Introduction Most language-model benchmarks ask questions whose answers are already known. They are useful for testing knowledge and rea- soning, but they do not test whether a model can make a useful prediction before an event happens. Recent dynamic benchmarks update their questions, and forecasting benchmarks wait for future outcomes before assigning scores [20–22,36]. Most of them, how- ever, still focus on a single answer or a coarse event outcome. They say little about whether a model can find current evidence and use it consistently across several related predictions. Football provides a practical setting for this question. Matches follow a public schedule, useful information changes before kickoff, and the final record contains much more than the winner. It also contains the score, lineups, goals and cards, match statistics, and the path through a league or cup. Existing sports benchmarks mainly study completed events [8,26,34,37], while traditional football fore- casting usually focuses on the result or score distribution [3,9,16]. Evaluating all of these outputs together shows whether a plausible match prediction is supported by plausible details (Figure 1). WorldCupArena evaluates models in this setting. Twenty-four hours before a match, each model either receives the same evi- dence package or searches for its own sources. It then submits probabilities, a predicted score, detailed match predictions, and a written explanation. Once the match is over, the prediction is scored against the official record. The same procedure can be applied to a new league or cup, so the benchmark can continue to test models released after the World Cup. This paper reports 13 systems over all 104 World Cup matches. The main finding is that result accuracy alone hides important dif- ferences: models that predict a similar number of winners correctly can differ on scorelines, players, events, and statistics. We also find no reliable gain from adding search to match prediction, and several models fail on the same upsets. Against betting-market and human- fan baselines, the best model improves strict result prediction only slightly, while its larger Scoreline gain comes from making closer score predictions when it misses. Our contributions are threefold: • We build a benchmark that can be reused for future football competitions and report its first evaluation on all 104 matches of the 2026 World Cup. •We compare models given the same evidence with agents that search for evidence themselves, while asking both groups to make the same match and competition predictions. •We score exact answers and close misses separately, exclude unavailable match records rather than counting them as errors, and publish the predictions and scoring code needed to check the results. 2 Related Work LLMs and deep-research agents. Language agents can browse the web, call tools, and combine information from several sources. Existing benchmarks test these abilities through browsing, plan- ning, and long research tasks [28,38,39]. Deep Research Bench, for example, evaluates long web investigations on a fixed collection of arXiv:2607.18084v1 [cs.AI] 20 Jul 2026 WorldCupArena, 2026Zhaokai Wang, Tianlin Gui, Jiayuan Rao, Shangzhe Di, Yihong Tang, and Dingli Liang One match (result & score) Increasing prediction scope and structural richness Whole competition T1 Result & score Headline score Home win probability T2 Players & lineups Starting eleven Formation 4-3-3 Scorer T3 Events & timeline Goal at 63 min Substitution Card T4 Tactics & statistics Possession 58% Shots 14 - 9 Corners T5 Competition outcome Group order Knockout pairings Champion Match result Figure 1: Visual taxonomy of the five evaluation layers, moving from a match result and score through players and lineups, events, tactics and statistics, to the outcome of a whole competition. pages [4]. Our question is different: does searching for more infor- mation improve a prediction about a match that has not happened yet? We compare models that receive the same prepared evidence (S1) with agents that search for evidence themselves (S2), and score both after the match. Sports understanding and forecasting. Sports benchmarks in- creasingly test whether models can understand video, commentary, and questions about a game. SoccerNet-v2, MatchTime, and UniSoc- cer cover tasks such as finding actions in video and connecting a broadcast to commentary [8,33,34]. Sports-QA and SPORTU test several forms of reasoning about sports [26,37]. In each case, the game has already been played when the model is evaluated. Football forecasting usually asks a narrower question. Beal et al. predict Premier League results from statistics and news, while Groll et al. simulate international tournaments with a hybrid random forest [3,16]. WorldCupArena keeps the before-match setting but asks general-purpose models for a fuller prediction, including the score, players, events, statistics, and the competition outcome. Temporal prediction benchmarks. RealTime QA and LiveBench regularly add new questions so that their answers do not become stale [22,36]. ForecastQA and Autocast use historical cut-off dates, whereas ForecastBench collects predictions for events whose out- comes are still unknown [20,21,41]. WorldCupArena follows the last approach. A prediction is recorded before kickoff and scored after the match. When one competition ends, a future league or cup supplies the next set of matches, so the benchmark does not depend on repeatedly testing the completed 2026 World Cup. Statistical sports forecasting. Classical football models often esti- mate how many goals each team will score. Early work uses Poisson models, later work accounts for recent form and low-scoring depen- dence, and Bayesian models represent uncertainty in team strength [2,9,27]. Elo ratings offer another way to turn team strength into result probabilities [17]. We also use betting odds as baselines be- cause they combine the judgments of many market participants and are often difficult to beat [12]. These methods remain strong references for predicting the result. Our benchmark asks whether general-purpose models can also predict the details. 3 Benchmark Construction WorldCupArena follows the same four steps for every match: regis- ter a future fixture, collect information until the prediction dead- line, save each model’s forecast, and score it after the official match record is available. Applying the benchmark to another competition changes the schedule and competition rules, but not these four steps or the match-level metrics. Figures 1 and 2 show what is predicted and how one match moves through the benchmark. 3.1 Benchmark Scope and Protocol What models predict. The benchmark asks for two kinds of predic- tion. For each match, a model predicts the result and score, players, events, and match statistics. It also predicts the competition as a whole. Match predictions have the same form in any football com- petition. Competition predictions need a small set of rules that describes what can be decided: for a league, this includes the final table, qualification, relegation, and champion; for a cup, it includes group order, advancing teams, knockout pairings, and champion. Only these competition rules change when a new league or cup is added. This paper evaluates the 2026 World Cup. Models first predict all 72 group matches. The program then calculates each group table and the eight best third-place teams from those predicted scores. Finally, the models predict the 32 knockout matches produced by their own group forecasts [10,11]. The same procedure can be configured before a future domestic league, continental cup, or World Cup begins. Five evaluation layers. For brevity, we group the predictions into five layers: match result and score (T1), players and lineups (T2), match events (T3), tactics and statistics (T4), and the whole competition (T5). Table 1 summarizes their targets, metrics, and weights. The next section explains how the layers are scored and combined. Temporal protocol. We save the evidence and prediction 24 hours before kickoff, so every system is judged using information available at the same time. For agents that search the web, we also save the returned URLs and timestamps. A prediction is excluded if it contains information published after the deadline or reveals the final result. WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football ForecastingWorldCupArena, 2026 1 Competition schedule • Match list • Dates & times • Venues • Competition format 2 Collect pre- match information Squads Recent form News Odds Statistics 3 Save model prediction S1 Common evidence S2 Self-directed search Saved 24h before kickoff 4 Official records Score · events Lineups · cards Standings · Pairings· Official statistics 5 Five evaluation layers T1 Result & score T2 Players & lineups T3 Events & timeline T4 Tactics & statistics T5 Competition outcome 6 Public leaderboard and records Overall scores Layer scores Saved predictions Evidence snapshot Scoring record Audit records BEFORE KICKOFF Schedule, information, and predictions are saved. AFTER KICKOFF Official records are added. SCORING & RECORDS T1-T5 scoring and public records. Competition records Match records Figure 2: Overview of WorldCupArena. Match information and model predictions are saved before kickoff; the final match and competition records are added afterward for scoring. 3.2 Predictions and Ground Truth Evidence settings. We test two ways of obtaining pre-match in- formation. In S1, every model receives the same package of squads, recent form, statistics, news, and available odds, and cannot use tools. In S2, the agent receives only the fixture and must search for its own evidence. The comparison shows how the complete products behave in these two settings; it is not a controlled com- parison of model architecture because providers may use different underlying models and search systems. Prediction format. Each model returns the same JSON fields for result probabilities, predicted score, players, events, and match statistics, plus a written explanation. The score entered by the model is the score used in evaluation; the program does not replace it with a score derived from other fields. In-play records. For a subset of matches, we repeatedly give the current minute, score, events, and available match statistics to three models and ask them to update the final-result probabilities and predicted final score. Every checkpoint stores the observed match state and submission time. These in-play forecasts form a separate exploratory track and do not enter the main leaderboard, because they cover fewer models and their polling times are not perfectly regular. Forecast rationale. The written explanation covers the expected lineup, tactics, recent form and past meetings, important player matchups, injuries, and possible paths to a win, draw, or upset. It should also explain why the predicted result and score are consistent with the predicted players, events, and statistics. We publish this explanation so readers can inspect the forecast, but do not assign it a score. Only claims that can be checked against the match record are graded. Before accepting a response, we check that it can be parsed, contains the required fields, and has no basic contradictions. If a repair attempt still fails, the response is kept in the record but counted as an invalid prediction. Truth acquisition and quality control. After the match, we collect the final score, lineups, events, and statistics independently of the model prediction. Team and player identifiers are checked before scoring. If part of the official record is missing or fails a basic consistency check, that field is left out of the evaluation instead of being filled with a default value. 4 Evaluation Method 4.1 Match-Level Metrics Overview. All metrics are defined so that a larger value is better: • Composite is the overall score that combines the five groups of predictions in Table 1, with more weight on the match result and score. •Result accuracy is the percentage of matches for which the system correctly predicts home win, draw, or away win. •Exact-score accuracy is the percentage for which it predicts both teams’ goal totals exactly. •Scoreline score also rewards close misses. For example, predict- ing 2–1 when the match ends 3–1 receives credit for the correct winner and a similar number of goals, but less than an exact prediction. Result and scoreline metrics. Result accuracy checks whether the most probable home/draw/away outcome is correct. We also use the standard three-way Brier score to reward well-calibrated proba- bilities and penalize misplaced confidence [5]. Exact-score accuracy is simply the percentage of matches for which both predicted goal totals equal the final score. Exact accuracy treats every miss equally, so we additionally score how close the predicted score is. An exact prediction receives 100; otherwise, 푆 score = 45퐼 푟 + 25 h 1− 푒 푑 5 i + + 20 h 1− 푒 푡 6 i + + 10 h 1− 푒 푡푒푎푚 8 i + (1) Here퐼 푟 marks a correct result class, while푒 푑 ,푒 푡 , and푒 푡푒푎푚 measure errors in goal difference, total goals, and team-wise goals. The WorldCupArena, 2026Zhaokai Wang, Tianlin Gui, Jiayuan Rao, Shangzhe Di, Yihong Tang, and Dingli Liang Table 1: Evaluation taxonomy, metrics, and fixed layer and task weights. LayerRepresentative targetsMetricsLayer weight Task weights T1 (Core result)1X2 probability, exact score, score closeness, goal difference Brier, exact, scoreline score 0.401X2 0.35; exact 0.20; scoreline 0.30; goal diff. 0.10; qualification 0.05 T2 (Player-level)starters, formation, scorers, assists, player of match Jaccard, F1, nDCG0.20starting eleven 0.30; formation 0.10; scorers 0.35; assists 0.15; player of match 0.10 T3 (Event-level)goal time, substitutions, cards, penalties, own goals matched MAE, event F10.15goal time 0.45; subs 0.20; cards 0.25; penalties 0.05; own goals 0.05 T4 (Tactics/statistics)possession, shots, corners, passing, fouls, saves sMAPE0.15formation 0.15; eight statistics 0.85 T5 (Competition-level)group order, bracket, champion, scorers, awards Kendall, bracket, top-10.10groups 0.25; bracket 0.35; champion 0.20; scorer 0.10; awards 0.10 weights make the result the largest single component and prevent large misses from producing negative scores. For display, the mean Scoreline value uses the same fixed mono- tonic calibration described below, with center 70 and temperature 5. It preserves model order and the 0/100 endpoints; both raw and displayed values remain public. In-play aggregation. Each valid in-play checkpoint reuses result accuracy, the three-way Brier score, exact-score accuracy, and the raw Scoreline metric above. We first average checkpoints within a match and then average across matches, so a match with more suc- cessful polls does not receive more weight. To show how forecasts change during play, we report result accuracy for minutes 0–30, 31–60, 61–90, and 91 or later. We also pair consecutive checkpoints whenever the observed score changes and compare the predictions immediately before and after that change. We do not interpolate missing checkpoints or assume that the polling interval was fixed. Player, event, and statistic metrics. For lineups and scorers, we measure how many predicted players appear in the match and reward a correct player more when the model ranks that player highly [19]. A predicted goal, card, or substitution is matched to the closest real event, with penalties for the wrong player or minute [25]. Counts such as shots and corners are scored by percentage error using sMAPE [18]. 4.2 Aggregation, Weights, and Calibration Availability-aware aggregation. Some match records omit fields such as player of the match or formation. We exclude unavailable truth instead of scoring it as zero. At both the task-to-layer and layer-to-composite levels, the score is the weighted mean over the available components 퐴: 푆 퐴 = Í 푗∈퐴 푤 푗 푆 푗 Í 푗∈퐴 푤 푗 .(2) There is an important difference between “none happened” and “no data.” A confirmed match with no red card can support a correct prediction of no red card; a missing card record is simply excluded. Fixed contrast calibration. The raw overall scores cluster in a nar- row range around 50, which makes small but consistent differences difficult to see in a leaderboard. We therefore use the following fixed S-shaped transformation for display: 퐶(푅)= 100 휎((푅− 50)/5)− 휎(−10) 휎(10)− 휎(−10) .(3) This transformation changes the spacing between displayed scores, not the ranking: if model A has a higher raw score than model B, it also has a higher displayed score. It also leaves 0 and 100 unchanged and does not depend on the current set of models. Around the middle of the scale, it turns raw scores of 44 and 47 into 23.15 and 35.43. We use this stronger setting only for the overall score; individual layer scores use a milder transformation. 4.3 Competition-Level Evaluation A full-competition prediction is checked in four ways. First,퐺mea- sures how close the predicted order of each group is to the real order, using Kendall’s휏[23]. Second,퐴 푟 measures how many of the correct teams reach knockout round푟. Third,푀 푟 measures how many exact matchups are correct in that round. Fourth,퐻checks whether the predicted champion is correct. A model therefore re- ceives some credit for selecting the right teams even if it places them on the wrong sides of the bracket. Later rounds receive weights 1,2,4,8,16 from R32 to the final. Let퐴and푀be the resulting round-weighted advancement and exact-pairing scores, and let퐻be 100 when the predicted champion is Spain and 0 otherwise. The completed competition score is 푇 5 complete = 0.25퐺 + 0.35(0.7퐴+ 0.3푀)+ 0.20퐻 0.80 .(4) The completed evaluation includes every group table, all knockout matchups, and champion Spain. Top-scorer and award components remain outside T5 because this snapshot does not yet provide their normalized competition-level truth; they are omitted rather than counted as model errors. 4.4 Across-Match Aggregation and Coverage Each metric is averaged over the matches for which that model has both a valid prediction and the required truth, with every match receiving equal weight. We report these match counts beside the scores. This makes clear when a high value is based on only a small number of matches. For the overall leaderboard, we average the raw match scores first and apply the display transformation퐶only once to that WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football ForecastingWorldCupArena, 2026 Table 2: Results on all 104 matches of the 2026 World Cup.푁is the number of valid predictions and differs when a provider call fails. S1 (standardized context) and S2 (self-directed search) denote the two evidence settings. ModelSetting 푁CompositeResultExactScoreline Claude Opus 4.7 (Thinking) [1]S19533.7668.415.863.16 Claude Opus 4.7 (Thinking + Search) [1]S25831.2670.7 17.268.49 GPT-5.4 [30]S110229.9165.711.856.05 GLM-5.1 [40]S110328.5968.013.662.27 GPT-5.4 (Search) [30]S29728.1468.010.358.78 Kimi K2.6 [24]S110127.4968.312.962.65 MiniMax M2.7 [29]S110126.5365.312.957.35 DeepSeek V4 Pro [7]S110326.3168.011.761.02 Gemini 3.1 Pro Preview (Thinking + Search) [14]S28526.1163.511.854.54 Gemini Deep Research [15]S25825.4958.612.141.36 Doubao Seed 2.0 Lite [6]S110224.0667.610.858.16 Gemini 3.1 Pro Preview (Thinking) [14]S110223.6959.812.736.95 Qwen3.7 Max [32]S110422.7861.510.639.85 Polymarket [31]–104–65.48.7– BetVictor [35]–104–68.316.353.35 Human fans–94–69.715.653.40 mean. Transforming each match before averaging could change the ranking, so we do not do it. The leaderboard updates whenever another match is completed. A model is ranked only if it covers at least half as many matches as the most complete model; in this paper, that means at least 52 matches. We also compare models on shared matches when possible. Failed provider calls and invalid responses reduce coverage instead of being treated as football predictions. 5 Experiments In this section, we answer six research questions. RQ1: Do detailed predictions reveal differences that result accuracy misses? RQ2: What do we learn by reporting exact scores and close score predic- tions separately? RQ3: Does web search improve forecasting? RQ4: How well do models predict the full competition, from group tables through knockout matchups to the champion? RQ5: Do language models improve on betting-market and human-fan baselines? RQ6: How does forecast quality change as a match unfolds and after the score changes? 5.1 Models and Baselines We evaluate 13 systems: nine receive the prepared S1 evidence pack- age, three are search-enabled S2 systems, and one is a deep-research agent. Every valid response is converted to the same prediction format before scoring. The evaluated set contains all 104 matches of the 2026 World Cup: 72 group matches, 16 Round-of-32 matches, eight Round-of-16 matches, four quarterfinals, two semifinals, the third-place playoff, and the final. All 13 systems meet the minimum-coverage rule, with between 58 and 104 valid predictions [10]. Additionally, we consider three baselines: Polymarket baseline. We take the last available home/draw/ away price in the window[푇 −24h,푇), where푇is kickoff [31]. The associated exact-score child event supplies 16 numeric scorelines and an Any Other Score category. For the strict numeric exact-score metric, Any Other Score is not counted as a hit; it is the top category in 39 of 104 fixtures. The baseline achieves 65.4% result accuracy and 8.7% strict exact-score accuracy over all 104 fixtures, while scoreline partial credit is not reported because the top selection is non-numeric in some fixtures. BetVictor baseline. This baseline uses BetVictor’s most likely match result and exact score, retrieved from Sportmonks [35] for all 104 matches. It reaches 68.3% result accuracy, 16.3% exact-score accuracy and 53.35 Scoreline. Sportmonks provides the latest stored pre-match odds rather than a guaranteed value from exactly 24 hours before kickoff. For matches that go to extra time, we follow the existing convention and include extra-time goals. Human-fan baseline. Human predictions come from an internal football-fan group of 152 users. For each match, we use the most common predicted result and score. If several answers are tied for most common, each receives an equal share of one prediction. This baseline covers 94 matches, with 69.7% result accuracy, 15.6% exact-score accuracy, and a Scoreline value of 53.4. 5.2 Match-Level Results Table 2 shows why the metrics should be read together. Claude Opus 4.7 (Thinking) has the highest overall score, while its search version has the best result and exact-score accuracy. Several other systems predict about 68% of results correctly, so result accuracy alone pro- vides little separation. (RQ2) Exact scores are much harder: systems are correct on only 10.3–17.2% of matches. The Scoreline column gives additional information about the remaining matches by dis- tinguishing close predictions from clearly wrong ones; because this column includes partial credit, its numerical value should not be read as an accuracy percentage. WorldCupArena, 2026Zhaokai Wang, Tianlin Gui, Jiayuan Rao, Shangzhe Di, Yihong Tang, and Dingli Liang Table 3: Full-competition predictions after the final. Group measures predicted group order; Adv. measures whether the correct teams reach each round; Pair measures exact knockout matchups; Champ. is champion accuracy. T5 combines these four parts. ModelSettingGroupAdv.PairChamp.T5 Claude Opus 4.7 (Thinking + Search) [1]S287.587.0 71.0 100.0 88.3 Claude Opus 4.7 (Thinking) [1]S184.7 87.070.4100.087.4 GPT-5.4 [30]S186.160.819.4 100.073.1 Doubao Seed 2.0 Lite [6]S188.958.715.3 100.072.8 DeepSeek V4 Pro [7]S188.960.720.20.049.0 GLM-5.1 [40]S186.160.919.20.048.1 MiniMax M2.7 [29]S186.160.419.00.047.9 GPT-5.4 (Search) [30]S284.761.1 19.40.047.7 Kimi K2.6 [24]S184.760.718.30.047.5 Qwen3.7 Max [32]S184.759.918.30.047.2 Gemini 3.1 Pro Preview (Thinking + Search) [14]S286.159.115.30.047.0 Gemini Deep Research [15]S287.5 35.319.20.040.7 Gemini 3.1 Pro Preview (Thinking) [14]S183.334.917.70.039.1 T1 Match 74.1 75.0 76.0 T2 Players 14.2 15.8 17.3 T3 Events 55.1 57.8 60.5 T4 Tactics / Stats 36.4 37.3 38.2 T5 Competition 47 68 88 Leaderboard systems Claude Opus 4.7 (Thinking) 76.0 / 17.3 / 59.9 / 38.2 / 87.4 Claude Opus 4.7 (Thinking + Search) 75.9 / 15.1 / 60.5 / 37.6 / 88.3 GPT-5.4 75.0 / 17.1 / 57.4 / 37.5 / 73.1 GLM-5.1 75.1 / 15.5 / 58.1 / 36.9 / 48.1 GPT-5.4 (Search) 74.6 / 15.2 / 58.6 / 37.5 / 47.7 Kimi K2.6 75.5 / 15.1 / 56.2 / 36.6 / 47.5 MiniMax M2.7 74.6 / 14.2 / 58.6 / 36.4 / 47.9 DeepSeek V4 Pro 74.8 / 15.5 / 55.1 / 36.8 / 49.0 Gemini 3.1 Pro Preview (Thinking + Search) 74.1 / 15.3 / 56.8 / 36.9 / 47.0 Values listed as T1 / T2 / T3 / T4 / T5 Figure 3: Scores of leading systems for result and score (T1), players and lineups (T2), events and timeline (T3), tactics and statistics (T4), and competition outcome (T5). Each axis has its own labeled range so that differences remain visible even when one task has much lower scores than another. (RQ5) The strongest model advantage over betting-market and human-fan baselines depends on the metric. Claude Opus 4.7 (Think- ing + Search) reaches 70.7% result accuracy, only 1.0 percentage point above the human-fan baseline, 2.4 points above BetVictor, and 5.3 points above Polymarket. Its 17.2% exact-score accuracy is 0.9 points above BetVictor, the strongest exact-score baseline. The difference is larger for near misses: its Scoreline of 68.49 exceeds BetVictor by 15.14 display points and the human-fan baseline by 15.09. Thus, the clearest observed gain is not a large increase in strict hits, but a greater tendency to predict a score close to the final score. These comparisons are descriptive rather than paired: the Claude search system covers 58 matches, the fan baseline covers 94, and the market baselines cover all 104. (RQ1) Figure 3 gives the clearest answer. Claude Opus 4.7 (Think- ing) leads T1 and T2, its search version leads T3, and Gemini Deep Research leads T4. No system leads every task, and models with similar result accuracy can still differ sharply on players, events, and statistics. Missing match fields are omitted from these scores rather than replaced with zeros. Figure 4 shows how one saved prediction is compared with the match record. (RQ3) For all three models tested both with and without search, the no-search version scores higher on their shared matches. Adding WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football ForecastingWorldCupArena, 2026 Table 4: In-play forecasting results over 100 World Cup matches. Metrics are averaged within each match and then across matches. Gap is the median wall-clock interval in minutes with the interquartile range in brackets. In panel (b), Before and After compare consecutive checkpoints whose observed score differs. (a) Overall in-play performance ModelMatches CheckpointsGapResult Brier Exact Scoreline Claude Opus 4.7 (Thinking) [1]95909 7.0 [6.2, 13.4] 70.5 79.8 30.776.9 GPT-5.4 [30]971,022 6.7 [6.1, 11.8]64.378.527.472.8 Gemini 3.1 Pro Preview (Thinking) [14]971,026 6.7 [6.2, 11.8]69.979.430.577.2 (b) Result accuracy by match state Model0–30 31–60 61–9091+ Goal pairs Before AfterΔ Claude Opus 4.7 (Thinking) [1]66.1 64.475.1 80.2208 66.9 78.9 +11.9 GPT-5.4 [30]52.958.574.4 67.722755.477.4 +22.0 Gemini 3.1 Pro Preview (Thinking) [14]68.364.1 75.8 73.822964.678.5 +13.9 Table 5: Difficulty-stratified results: means over 12 coverage-eligible systems and displayed-composite leaders; Result and Exact are percentages. (a) Eligible-model means SliceFixtures Comp. Result Exact Score Balanced (푝 max ≤ 0.50)2321.555.414.043.1 Heavy favorite (푝 max ≥ 0.65)3429.275.59.670.3 Knockout3234.970.718.472.2 Low scoring (outcome ≤ 1 goal)2314.057.74.631.2 (b) Composite leaders SliceModelComp. BalancedClaude Opus 4.7 (Thinking)28.7 Heavy favorite Claude Opus 4.7 (Thinking)38.2 KnockoutClaude Opus 4.7 (Thinking)46.1 Low scoringGemini 3.1 Pro Preview (Thinking + Search)17.9 search changes the overall score by−0.26 for Claude (푁=50),−4.26 for GPT-5.4 (푁=95), and−1.03 for Gemini (푁=82). These results cover commercial products rather than identical base models, so they show the observed result but do not isolate its cause. All 13 systems predicted France, Spain, Brazil, and Argentina as semifinalists, so every system missed England and incorrectly re- tained Brazil. Four systems predicted champion Spain: both Claude configurations, GPT-5.4, and Doubao Seed 2.0 Lite. Only the two Claude configurations also predicted the exact Spain–Argentina final. With champion credit included, search changes T5 by+0.9 for Claude,−25.3 for GPT-5.4, and+8.0 for Gemini; the large GPT-5.4 gap occurs because only its no-search forecast selected Spain as champion. Figure 5 summarizes these results. 5.3 Competition-Level Results (RQ4) Table 3 separates four competition-level signals. Models predict the broad group order more consistently than the exact knockout route: group scores occupy a narrow 83.3–88.9 range, while pairing scores range from 15.3 to 71.0. The two Claude models lead with T5 scores of 88.3 and 87.4 because they predict the Spain– Argentina final and champion Spain. GPT-5.4 and Doubao also select Spain and score 73.1 and 72.8, whereas systems that miss the champion score 39.1–49.0. Evaluating the complete route therefore distinguishes a correct champion chosen through the right bracket from a correct champion reached through the wrong path. 5.4 In-Play Results The in-play track contains 2,957 valid checkpoints from 100 World Cup matches. Claude Opus 4.7 (Thinking) covers 95 matches, while GPT-5.4 and Gemini 3.1 Pro Preview (Thinking) each cover 97. The logs do not support describing this as a strict ten-minute schedule. Across models, the median wall-clock gap is 6.8 minutes (interquar- tile range 6.2–12.4); among pairs for which the provider’s match clock advances, the median change is 7 minutes (interquartile range 6–10). A sampled group match was usually updated every five to six minutes apart from halftime, whereas several knockout matches contain much longer gaps or repeated provider minutes. We there- fore evaluate the checkpoints that were actually recorded. (RQ6) Table 4 shows that the three systems are close overall but reach that level differently. Claude Opus 4.7 (Thinking) has the best result accuracy (70.5%), Brier score (79.8), and exact-score accuracy (30.7%), while Gemini has the highest raw Scoreline score (77.2). Accuracy is not monotonic in every bin, but all three systems are strongest or close to strongest during minutes 61–90, where their result accuracies reach 74.4–75.8%. The 91+ column covers only eight or nine matches per model and should not be treated as a stable ranking. Score changes provide a more direct test of updating. Across 208–229 paired changes per model, result accuracy rises from 55.4– 66.9% before the observed change to 77.4–78.9% afterward. The corresponding Brier score, not shown in panel (b), improves by 8.4 points for Claude, 11.2 for GPT-5.4, and 12.1 for Gemini. GPT-5.4 makes the largest categorical correction but remains the weakest overall, suggesting that it reacts strongly to score changes after less accurate early predictions. These gains measure adaptation to newly observed match state, not independent evidence that the model anticipated the goal. WorldCupArena, 2026Zhaokai Wang, Tianlin Gui, Jiayuan Rao, Shangzhe Di, Yihong Tang, and Dingli Liang ArgentinavsSwitzerland· Quarterfinal · 12 July 2026 SAVED PREDICTION · Opus 4.7 (Thinking) · S1 OFFICIAL MATCH RECORD 2 - 0 Win probability 60% home 25% draw · 15% away 3 - 1 After extra time 4 goals 10′ · 67′ · 112′ · 120+1′ PREDICTED FORMATIONS Argentina 4-3-3 · Switzerland 4-2-3-1 ACTUAL FORMATIONS Argentina 4-3-3 · Switzerland 4-2-3-1 PREDICTED EVENTS & TIMELINE Messi · Lautaro · Álvarez ACTUAL EVENTS & TIMELINE Mac Allister · Embolo · Álvarez · Martínez Predicted score, players, events, and statistics are saved before kickoff.Official records are normalized before scoring. PREDICTED VS ACTUAL STATISTICS · Argentina / Switzerland PredictedActualΔ Possession60% / 40%59% / 41%-1 / +1 Shots17 / 922 / 11+5 / +2 On target6 / 37 / 5+1 / +2 Corners7 / 38 / 2+1 / -1 Figure 4: Claude Opus 4.7 (Thinking)’s prediction for Argentina–Switzerland beside the recorded result. The example shows the predicted score, players, events, and statistics used in T1–T4. Table 6: Consensus-failure case studies. Modal score counts are taken from the locked headline forecasts; Wrong 1X2 counts systems whose highest-probability result class disagrees with the final result. FixtureActualGradedPredicted headline scoresWrong 1X2Failure pattern Spain–Cape Verde0–0137× 4–0; 6× 3–013/13Extreme favorite held to a scoreless draw Netherlands–Japan2–21312× 2–1; 1× 1–013/13Concentrated favorite-win and scoreline consensus France–England (3rd)4–6126× 2–1; 3× 3–2; 3× 3–112/12Unanimous favorite pick reversed in a ten-goal match 6 Discussion What the benchmark measures. Each score describes the com- plete system that produced the forecast. A good result may come from the language model, the search tool, better probability esti- mates, or simply more reliable output formatting. The benchmark compares systems as users receive them; it cannot by itself prove which internal component caused the difference. Why we report several metrics. Result accuracy treats a cautious 45% prediction and a confident 90% prediction in the same way once both choose the same outcome. The Brier score distinguishes them and penalizes misplaced confidence [5,13]. Exact-score accuracy answers a different question from the Scoreline metric, and the player, event, and statistic scores reveal different strengths again. We therefore report all these values instead of asking one overall number to explain every aspect of a forecast. Sample size and fixture difficulty. The main leaderboard aver- ages many different kinds of match. Table 5 checks whether the result changes for four easier-to-understand groups: • Balanced: 23 matches in which the betting market gives no out- come more than 50% probability after removing the bookmaker’s margin. •Heavy favorite: 34 matches in which one outcome has at least 65% probability. •Knockout: all 32 elimination matches, including the third-place playoff and final. •Low scoring: 23 matches that finish with zero or one goal. This group is defined after the match and is used only to study a difficult pattern. The first two groups use the 86 matches with valid pre-match odds. As in the main table, a system must cover at least half of the largest available match count. Panel (a) shows the average system perfor- mance for each group; panel (b) shows the best individual system. For example, 21.5 is the mean overall score on balanced matches, while 28.7 is the score of the best system on those matches. Table 6 shows matches in which every evaluated system chose the wrong result. The clearest example is Spain–Cape Verde. The market gave Spain an 89.1% chance after removing the bookmaker’s margin, and all 13 systems predicted either 3–0 or 4–0; the match WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football ForecastingWorldCupArena, 2026 Match-level comparison over 104 fixturesCompetition-level evaluation after the final CompositeResultExactScoreline Claude Opus 4.7 (Thinking) 33.868.415.863.2 Claude Opus 4.7 (Thinking + Search) 31.370.717.268.5 GPT-5.4 29.965.711.856.1 GLM-5.1 28.668.013.662.3 GPT-5.4 (Search) 28.168.010.358.8 Kimi K2.6 27.568.312.962.7 Different metrics produce different leaders; exact-score accuracy remains below 18%. GroupAdvancePairingChampionT5 Claude Opus 4.7 (Thinking + Search) 87.587.071.0100.088.3 Claude Opus 4.7 (Thinking) 84.787.070.4100.087.4 GPT-5.4 86.160.819.4100.073.1 Doubao Seed 2.0 Lite 88.958.715.3100.072.8 DeepSeek V4 Pro 88.960.720.20.049.0 GLM-5.1 86.160.919.20.048.1 Completed final Spain 1-0 Argentina (AET) Champion: Spain Four systems predicted the champion Two also predicted the exact final pairing Figure 5: Summary of match and competition-level results over all 104 World Cup matches. The right panel includes the completed Spain–Argentina final, champion accuracy, and the final T5 ranking. ended 0–0. The completed tournament adds a different failure: all 12 available systems chose France in the third-place playoff and clustered around 2–1, 3–1, or 3–2, before England won 6–4. These examples show that different models can follow the same favorite and fail together, even when the match develops into an unusual scoring pattern. The groups reveal different weaknesses. Heavy favorites make the winner easier to predict but do not make the exact score easier. Low-scoring matches are hardest overall: average result accuracy is 57.7% and exact-score accuracy is 4.6%. Claude Opus 4.7 (Think- ing) leads the first three groups, whereas Gemini 3.1 Pro Preview (Thinking + Search) leads the low-scoring group. These groups overlap, so they should be read as descriptive examples rather than a controlled test of match difficulty. Leakage and changing model versions. We save prediction times, sources, and model identifiers so that a result can be checked later. This record is not perfect: web pages may have unreliable publication dates, and a provider may silently update a model be- hind the same public name. Important search-based comparisons therefore still require manual checks. Beyond text-only prediction. The current evaluation uses text and structured statistics. Future versions could give models video clips, press conferences, tracking data, or formation graphics. These sources should be added as clearly defined input settings so that any improvement can be attributed to the new evidence. Using the benchmark after the World Cup. A model released after the 2026 World Cup cannot make a genuine before-match prediction for that tournament [10]. We will therefore keep the World Cup results as a completed record and evaluate new models on future leagues and cups. The teams and competition format will change, but predictions will still be saved before kickoff and scored with the same match-level rules. In this sense, the benchmark grows through new matches rather than by asking new models to pretend that an old result is unknown. 7 Conclusion We introduced WorldCupArena for large language models and agents to evaluate football predictions made before the result is known. The benchmark checks not only the winner and score, but also players, events, statistics, and the outcome of the competition. Across all 104 World Cup matches, detailed scoring reveals differ- ences that result accuracy alone misses; web search does not consis- tently help match prediction, and models sometimes fail together by choosing the same favorite. Competition evaluation adds another distinction: four systems predicted champion Spain, but only two also predicted the exact final pairing. The separate exact-score and scoreline metrics distinguish a correct score from a reasonable near miss. The 2026 World Cup is the first use of the benchmark, not its endpoint. Future work may extend to other leagues and cups and even other sports following the same before-match schedule, allow- ing newly released models to be evaluated on genuinely unknown outcomes. WorldCupArena, 2026Zhaokai Wang, Tianlin Gui, Jiayuan Rao, Shangzhe Di, Yihong Tang, and Dingli Liang References [1]Anthropic. 2026. Claude Opus 4.7. https://w.anthropic.com/news/claude- opus-4-7. Accessed 2026-07-16. [2]Gianluca Baio and Marta Blangiardo. 2010. Bayesian Hierarchical Model for the Prediction of Football Results. Journal of Applied Statistics 37, 2 (2010), 253–264. doi:10.1080/02664760802684177 [3] Ryan Beal, Stuart E. Middleton, Timothy J. Norman, and Sarvapali D. Ramchurn. 2021. Combining Machine Learning and Human Experts to Predict Match Out- comes in Football: A Baseline Model. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 15447–15451. doi:10.1609/aaai.v35i17.17815 [4]Nikos I. Bosse, Jon Evans, Robert G. Gambee, Daniel Hnyk, Peter Mühlbacher, Lawrence Phillips, Dan Schwarz, and Jack Wildman. 2025. Deep Research Bench: Evaluating AI Web Research Agents. arXiv:2506.06287 https://arxiv.org/abs/ 2506.06287 [5]Glenn W. Brier. 1950. Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review 78, 1 (1950), 1–3. doi:10.1175/1520-0493(1950)078<0001: VOFEIT>2.0.CO;2 [6]ByteDance Seed. 2026. Seed 2.0. https://seed.bytedance.com/en/seed2. Accessed 2026-07-16. [7]DeepSeek-AI et al.2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv preprint arXiv:2606.19348 (2026). doi:10.48550/arXiv. 2606.19348 [8]Adrien Deliège, Anthony Cioppa, Silvio Giancola, Meisam J. Seikavandi, Jacob V. Dueholm, Kamal Nasrollahi, Bernard Ghanem, Thomas B. Moeslund, and Marc Van Droogenbroeck. 2021. SoccerNet-v2: A Dataset and Benchmarks for Holis- tic Understanding of Broadcast Soccer Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 4503–4514. doi:10.1109/CVPRW53098.2021.00508 [9]Mark J. Dixon and Stuart G. Coles. 1997. Modelling Association Football Scores and Inefficiencies in the Football Betting Market. Journal of the Royal Statistical Society: Series C (Applied Statistics) 46, 2 (1997), 265–280. doi:10.1111/1467-9876. 00065 [10]FIFA. 2026.World Cup 2026 Match Schedule, Fixtures, Results, Teams and Stadiums.https://w.fifa.com/en/tournaments/mens/worldcup/ canadamexicousa2026/articles/match-schedule-fixtures-results-teams- stadiums. Accessed 2026-07-16. [11]FIFA Council. 2023.FIFA Council Approves International Match Cal- endars.https://inside.fifa.com/organisation/fifa-council/media-releases/fifa- council-approves-international-match-calendars. [12]David Forrest, John Goddard, and Robert Simmons. 2005. Odds-Setters as Fore- casters: The Case of English Football. International Journal of Forecasting 21, 3 (2005), 551–564. doi:10.1016/j.ijforecast.2005.03.003 [13] Tilmann Gneiting and Adrian E. Raftery. 2007. Strictly Proper Scoring Rules, Prediction, and Estimation. J. Amer. Statist. Assoc. 102, 477 (2007), 359–378. doi:10.1198/016214506000001437 [14]Google. 2026. Gemini 3.1 Pro. https://blog.google/innovation-and-ai/models- and-research/gemini-models/gemini-3-1-pro/. Accessed 2026-07-16. [15] Google. 2026. Gemini Deep Research. https://gemini.google/overview/deep- research/. Accessed 2026-07-16. [16] Andreas Groll, Christophe Ley, Gunther Schauberger, and Hans Van Eetvelde. 2019. A Hybrid Random Forest to Predict Soccer Matches in International Tournaments. Journal of Quantitative Analysis in Sports 15, 4 (2019), 271–287. doi:10.1515/jqas-2018-0060 [17] Lars Magnus Hvattum and Halvard Arntzen. 2010. Using ELO Ratings for Match Result Prediction in Association Football. International Journal of Forecasting 26, 3 (2010), 460–470. doi:10.1016/j.ijforecast.2009.10.002 [18]Rob J. Hyndman and Anne B. Koehler. 2006. Another Look at Measures of Forecast Accuracy. International Journal of Forecasting 22, 4 (2006), 679–688. doi:10.1016/j.ijforecast.2006.03.001 [19]Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated Gain-Based Evaluation of IR Techniques. ACM Transactions on Information Systems 20, 4 (2002), 422–446. doi:10.1145/582415.582418 [20] Woojeong Jin, Rahul Khanna, Suji Kim, Dong-Ho Lee, Fred Morstatter, Aram Galstyan, and Xiang Ren. 2021. ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data. In Proceedings of ACL-IJCNLP. 4636–4650. doi:10.18653/v1/2021.acl-long.357 [21]Ezra Karger, Houtan Bastani, Yueh-Han Chen, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E. Tetlock. 2025. ForecastBench: A Dynamic Bench- mark of AI Forecasting Capabilities. In International Conference on Learning Representations. arXiv:2409.19839 https://arxiv.org/abs/2409.19839 [22]Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2023. RealTime QA: What’s the Answer Right Now?. In Advances in Neural Information Processing Systems. [23] Maurice G. Kendall. 1938. A New Measure of Rank Correlation. Biometrika 30, 1–2 (1938), 81–93. doi:10.1093/biomet/30.1-2.81 [24]Kimi. 2026. Kimi K2.6. https://w.kimi.com/ai-models/kimi-k2-6. Accessed 2026-07-16. [25]Harold W. Kuhn. 1955. The Hungarian Method for the Assignment Problem. Naval Research Logistics Quarterly 2 (1955), 83–97. doi:10.1002/nav.3800020109 [26]Haopeng Li, Andong Deng, Jun Liu, Hossein Rahmani, Yulan Guo, Bernt Schiele, Mohammed Bennamoun, and Qiuhong Ke. 2026. Sports-QA: A Large-Scale Video Question Answering Benchmark for Complex and Professional Sports. International Journal of Computer Vision 134, 5 (2026), 196. doi:10.1007/s11263- 026-02734-1 [27]M. J. Maher. 1982. Modelling Association Football Scores. Statistica Neerlandica 36, 3 (1982), 109–118. doi:10.1111/j.1467-9574.1982.tb00782.x [28]Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: A Benchmark for General AI Assistants. In International Conference on Learning Representations. https://proceedings.iclr. c/paper_files/paper/2024/hash/25ae35b5b1738d80f1f03a8713e405ec-Abstract- Conference.html [29]MiniMax. 2026. MiniMax M2.7. https://w.minimax.io/news/minimax-m27-en. Accessed 2026-07-16. [30]OpenAI. 2026. Introducing GPT-5.4. https://openai.com/zh-Hans-CN/index/ introducing-gpt-5-4/. Accessed 2026-07-16. [31]Polymarket. 2026. Market Data Overview and Historical Prices API. https: //docs.polymarket.com/market-data/overview. Accessed 2026-07-16. Historical prices: https://docs.polymarket.com/api-reference/markets/get-prices-history. [32]Qwen Team. 2026. Qwen3.7. https://qwen.ai/blog?id=qwen3.7. Accessed 2026- 07-16. [33] Jiayuan Rao, Haoning Wu, Hao Jiang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2025. Towards Universal Soccer Video Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8384–8394. https://cvpr.thecvf.com/virtual/2025/poster/32544 [34] Jiayuan Rao, Haoning Wu, Chang Liu, Yanfeng Wang, and Weidi Xie. 2024. MatchTime: Towards Automatic Soccer Game Commentary Generation. In Pro- ceedings of EMNLP. 1671–1685. doi:10.18653/v1/2024.emnlp-main.99 [35]Sportmonks. 2026.Football API 3.0: Standard Pre-match Odds Feed. https://docs.sportmonks.com/v3/endpoints-and-entities/endpoints/standard- odds-feed/pre-match-odds/get-all-odds. Accessed 2026-07-16. [36]Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Sid- dhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2025. LiveBench: A Challenging, Contamination-Limited LLM Benchmark. In International Confer- ence on Learning Representations. https://proceedings.iclr.c/paper_files/paper/ 2025/hash/e4a46394ba5378b3f9a186a5b4c650d1-Abstract-Conference.html [37]Haotian Xia, Zhengbang Yang, Junbo Zou, Rhys Tracy, Yuqing Wang, Chi Lu, Christopher Lai, Yanjun He, Xun Shao, Zhuoqing Xie, Yuan-fang Wang, Weining Shen, and Hanjie Chen. 2025. SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models. In International Conference on Learning Representations. https://proceedings.iclr.c/paper_files/paper/2025/ hash/49c1879ae366644ce2c17fb39ddea982-Abstract-Conference.html [38] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Mod- els. In International Conference on Learning Representations. https://openreview. net/forum?id=WE_vluYUL-X [39]Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. 2024. AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?. In Proceedings of EMNLP. 8938–8968. doi:10.18653/v1/ 2024.emnlp-main.505 [40]Z.ai. 2026. GLM-5.1: From Vibe Coding to Agentic Engineering. https://z.ai/blog/ glm-5.1. Accessed 2026-07-16. [41]Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. 2022. Forecasting Future World Events with Neural Networks. In Advances in Neural Information Processing Systems. https://proceedings.neurips.c/paper_files/paper/2022/hash/ aec870a6772336c15dac992c16f2e7c9-Abstract-Datasets_and_Benchmarks.html WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football ForecastingWorldCupArena, 2026 A Implementation Details A.1 Artifact Layout and Lifecycle Each fixture passes through five idempotent phases. Ingest creates the fixture snapshot; populate attaches squads, form, news, sta- tistics, and odds. Lock-and-predict freezes the snapshot hash and writes one record per model and setting. Live updates are stored separately and never enter pre-match grading. After completion, truth ingest creates the normalized truth artifact; grading writes ver- sioned result files; and the site builder emits static language-specific payloads. The scheduler uses windows rather than exact instants. In the default configuration, fixtures are ingested up to seven days ahead, evidence is populated during the day before the prediction lock, predictions are frozen 24 hours before kickoff, live state is polled around the match, and truth is graded after completion. Repeated scheduler ticks are no-ops when current artifacts already exist. A changed scoring version automatically refreshes old result files from saved predictions and truth without rerunning a model. A.2 Full-Tournament Generation Tournament prediction uses two model calls. The first predicts all group matches. Code computes points, goal difference, goals scored, and deterministic tie-break fallbacks, then resolves best third-place teams into Round-of-32 slots. The second call receives the model’s own standings and fixed knockout fixtures, and predicts matches 73–104 under the official format [11]. Champion, runner-up, third place, and top scorers are derived from the saved match list. rather than accepted as unconstrained independent claims. load adds result and exact-score hit counts and denominators, mean scoreline quality, and provisional tournament evaluation. This redundancy is intentional: a leaderboard is easy to consume, but a research artifact must also support independent recomputation. The tournament truth is kept in a dated configuration file so the same prediction can be evaluated through the semifinal, final, and completed-tournament stages without hindsight changes to the prediction. A.3 Prediction Record Contract A saved prediction record separates execution metadata from the parsed football forecast. Execution metadata includes model ID, setting, submission time, latency, token counts, estimated cost, tool- call count, snapshot hash, raw response, validation status, and error summary. The parsed object contains only schema-governed benchmark fields. Keeping both forms is important because a parser improvement can be tested against the original response without claiming that the model produced different text. The semantic validator operates after JSON Schema. Probability vectors are normalized only within a small tolerance; larger er- rors trigger repair. The declared result must agree with the largest 1X2 probability, and the headline score must have the same result class. Starting lineups must contain 11 players per side, and ev- ery requested statistic must contain home and away values. Repair prompts quote the failed constraints and request a complete replace- ment object. They do not reveal any post-lock truth. The number of repair attempts and all rejected outputs remain in the record so format reliability can later be reported as a model capability. A.4 Failure States and Recovery The pipeline distinguishes skipped, failed, invalid, leaked, and suc- cessfully graded runs. A skipped run already has a current artifact and makes no provider call. A failed run records transport, authen- tication, timeout, or provider errors. An invalid run reached the model but did not satisfy the contract after repair. A leaked run con- tains outcome information that was unavailable at the lock. Only the last state is excluded for benchmark-integrity reasons; the other states remain visible when computing coverage and operational reliability. Recovery is artifact driven. A failed model can be retried before the deadline without rerunning successful peers. A truth-ingest fail- ure leaves predictions untouched and is retried by a later scheduler tick. A new scoring version refreshes results from stored prediction and truth files. Site generation writes complete payloads only after in-memory construction succeeds, limiting the chance that readers receive a mixture of old and new metric fields. In deployment, the generated files are published together by the Pages workflow. A.5 Current Implementation Boundaries The present truth adapter depends mainly on a football data provider and does not yet cross-check every player and event field against a second official source. Search-source publication times are also im- perfect, so automatic leakage checks need periodic manual review. Team aliases and player transliterations are normalized pragmati- cally rather than through a global sports ontology. These limitations are recorded because they affect fine-grained layers more than the final score.