Paper deep dive

InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information

Jiaze Li, Aocheng Shen, Bing Liu, Boyu Zhang, Xiaoxuan Fan, Qiankun Zhang, Xianjun Deng

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 157

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 9/2/2026, 3:51:46 AM

Summary

The paper introduces InteractBench, a benchmark designed to evaluate Large Language Models (LLMs) on competitive programming interactive problems, where information is revealed dynamically through queries rather than provided upfront. It comprises 322 problems from major platforms (Codeforces, AtCoder, IOI, ICPC) with executable local interactors for offline evaluation. The study highlights a significant 'interaction gap' where even advanced models struggle with protocol compliance and query budget management, proposing a fine-grained failure taxonomy to diagnose these issues.

Entities (12)

InteractBench · benchmark · 100%AtCoder · platform · 95%Codeforces · platform · 95%ICPC · competition · 95%Interactive Problems · problem-type · 95%IOI · competition · 95%Batch-style Problems · problem-type · 90%Huazhong University of Science and Technology · organization · 90%LLMs · model-class · 90%Qiankun Zhang · person · 90%Query Budget · constraint · 90%Protocol Violations · failure-mode · 85%

Relation Signals (10)

InteractBench → containsproblemsfrom → Codeforces

confidence 95% · curated from Codeforces, AtCoder, IOI, and ICPC

InteractBench → containsproblemsfrom → AtCoder

confidence 95% · curated from Codeforces, AtCoder, IOI, and ICPC

InteractBench → containsproblemsfrom → IOI

confidence 95% · curated from Codeforces, AtCoder, IOI, and ICPC

InteractBench → containsproblemsfrom → ICPC

confidence 95% · curated from Codeforces, AtCoder, IOI, and ICPC

InteractBench → evaluates → LLMs

confidence 95% · Benchmarking LLMs on Competitive Programming

InteractBench → focuseson → Interactive Problems

confidence 95% · we introduce InteractBench, a benchmark comprising 322 high-quality interactive problems

Qiankun Zhang → affiliatedwith → Huazhong University of Science and Technology

confidence 90% · Correspondence to: Qiankun Zhang... Huazhong University of Science and Technology

Interactive Problems → contrastswith → Batch-style Problems

confidence 90% · Batch-style and interactive problems... The batch-style problem has all the input provided upfront

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Competitive programming is increasingly being used to evaluate the algorithmic reasoning capabilities of large language models (LLMs). However, existing benchmarks primarily focus on full-information tasks where all problem inputs are provided upfront. This overlooks a critical dimension of algorithmic reasoning: the ability of generated programs to operate when key information is not revealed upfront. Interactive problems, a distinctive component of competitive programming, embody this challenge. These problems require programs to engage in multi-round interaction with an interactor (a judge program) under strict protocol constraints and limited query budgets, with new information revealed only in response to queries. To address this gap, we introduce InteractBench, a benchmark comprising 322 high-quality interactive problems curated from Codeforces, AtCoder, IOI, and ICPC. Each problem is packaged with executable local interactors, enabling fully offline evaluation. Unlike existing benchmarks, InteractBench assesses whether model-generated code can acquire information and track state dynamically. Our evaluation reveals a significant interaction gap: even the most advanced reasoning models achieve limited success on interactive problems. Beyond success rates, we propose a fine-grained failure taxonomy to diagnose the root causes of these deficiencies. Although algorithmic logic errors remain dominant, protocol violations and query-budget overruns are frequent. Code is available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

153,754 characters extracted from source content.

Expand or collapse full text

InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Jiaze Li 1 2 3 Aocheng Shen 1 Bing Liu 1 Boyu Zhang 1 Xiaoxuan Fan 1 Qiankun Zhang 1 2 3 Xianjun Deng 1 3 Abstract Competitive programming is increasingly being used to evaluate the algorithmic reasoning capa- bilities of large language models (LLMs). How- ever, existing benchmarks primarily focus on full- information tasks where all problem inputs are provided upfront. This overlooks a critical di- mension of algorithmic reasoning: the ability of generated programs to operate when key informa- tion is not revealed upfront. Interactive problems, a distinctive component of competitive program- ming, embody this challenge. These problems require programs to engage in multi-round inter- action with an interactor (a judge program) un- der strict protocol constraints and limited query budgets, with new information revealed only in response to queries. To address this gap, we intro- duce InteractBench, a benchmark comprising 322 high-quality interactive problems curated from Codeforces, AtCoder, IOI, and ICPC. Each prob- lem is packaged with executable local interactors, enabling fully offline evaluation. Unlike exist- ing benchmarks, InteractBench assesses whether model-generated code can acquire information and track state dynamically. Our evaluation re- veals a significant interaction gap: even the most advanced reasoning models achieve limited suc- cess on interactive problems. Beyond success rates, we propose a fine-grained failure taxonomy to diagnose the root causes of these deficiencies. Although algorithmic logic errors remain domi- nant, protocol violations and query-budget over- runs are frequent. Code is available athttps:// github.com/kmsgk0/InteractBench. 1 School of Cyber Science and Engineering, Huazhong Uni- versity of Science and Technology, Wuhan, China 2 Key Labora- tory of Cyberspace Security, Ministry of Education, Zhengzhou, China 3 Hubei Key Laboratory of Distributed System Secu- rity, Wuhan, China.Correspondence to: Qiankun Zhang <qiankun@hust.edu.cn>. Proceedings of the43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s). 1. Introduction Frontier large language models (LLMs) have made rapid progress in code generation and algorithmic reasoning. On widely used benchmarks such as HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021), performance has approached saturation, making it difficult to meaningfully differentiate top models using a single overall success rate. Competitive programming pairs formally specified algo- rithmic tasks with automated judging, providing reliable correctness verdicts under strict time and memory limits, and has been argued to serve as an effective evaluator for LLMs (Huang et al., 2024). This has motivated multiple competitive-programming benchmarks, including CodeCon- tests (Li et al., 2022), LiveCodeBench (Jain et al., 2024), and USACO (Shi et al., 2024). More recently, bench- marks such as CodeElo (Quan et al., 2025), HLCE (Li et al., 2025), ICPC-Eval (Xu et al., 2025), LiveCodeBench Pro (Zheng et al., 2025), and LiveOIBench (Zou et al., 2025) have emerged. These benchmarks suggest progress on harder competitive-programming problems, particularly with reasoning-focused models. Yet most adopt a full- information evaluation protocol: the solver reads a complete, static input at the start of execution; such tasks are referred to as batch-style problems. As a result, it remains unclear how well model-generated programs can solve interactive problems when key information is not revealed upfront and must be acquired through runtime queries. Batch-style and interactive problems. The batch-style problem has all the input provided upfront, and the in- put does not depend on the solver’s behavior (Verhoeff, 2009).In an interactive problem, key information is held by an interactor, the program responsible for answer- ing queries (Mirzayanov, 2016; CMS Development Team, 2017). This information is revealed only in response to the solver’s queries, subject to a strict protocol and a problem- specific query budget. 1 Success therefore depends on an online querying strategy in addition to producing a correct final answer. Figure 1 illustrates this distinction with an example problem. 1 This differs from HLCE (Li et al., 2025)’s use of “interactive- based” to denote function-signature evaluation. 1 arXiv:2608.29632v1 [cs.SE] 30 Aug 2026 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Problem: Find the K-thZero Find the smallest index xsuch that A[1..x] contains kzeros. Constraints: n = 8, k = 3. BatchInteractive Input 10110101 A = Output: 7 Interactor (Hidden Array) ???????? A = Query Budget: 3 / 7 Index: 12345678 Ask: Sum(A[1...4]) # Round 1 Sum(A[1...4]) = 3 ????01?? Sum(A[5...6]) = 1 1011???? ??????0? Sum(A[7...7]) = 0 Ask: Sum(A[5...6]) Ask: Sum(A[7...7]) Answer: Index 7 Response: 3 Response: 1 Response: 0 All input data is fixedat the beginning of the run. # Round 2 # Round 3 Interaction Protocol: Submit Answer Query Range Sum Figure 1. An illustration of the difference between batch-style and interactive problems. We use “Find the K-th Zero” as an example. In the batch-style setting (left), the program receives the complete input array at the start of execution and outputs the answer in a single shot. In the interactive setting (right), the array is hidden and the program must interact with an interactor (judge program) under a strict protocol and a limited query budget to gather information. The program can make several queries that ask the sum of all elements in a specific interval, and receive the answer from the interactor. Each query consumes part of the budget, and the interactor’s response depends on the query and the hidden input. The program reasons over multiple rounds before submitting the final answer. These interactive settings introduce additional evaluation challenges; in particular, the following limitations and chal- lenges in existing benchmarks need to be addressed: • Interactive coverage. Many benchmarks focus on batch- style evaluation (Jain et al., 2024; Shi et al., 2024; Xu et al., 2025), and interactive problems are rare or explicitly excluded to simplify evaluation infrastructure. •Offline evaluation barriers. Some evaluations, such as CodeElo (Quan et al., 2025) and HLCE (Li et al., 2025), depend on remote submissions to external online judges, which hinders fully offline evaluation and large-scale ex- perimentation. •Limited outcome analysis. Many studies primarily re- port an overall success metric (e.g.,pass@k) and provide limited diagnostics for distinguishing algorithmic errors from interaction-specific failures (Quan et al., 2025; Li et al., 2025; Zheng et al., 2025; Zou et al., 2025). To address these limitations, we introduce InteractBench, a benchmark of competitive-programming interactive prob- lems, together with an evaluation harness to run these inter- actions reliably offline. Our contributions are as follows: Contribution #1: a benchmark of interactive problems. We curate interactive problems from Codeforces (Code- forces, 2026), AtCoder (AtCoder Inc., 2026), IOI (Interna- tional Olympiad in Informatics, 2026), and ICPC (ICPC Foundation, 2026). Each problem is annotated with consol- idated categories and difficulty tiers. We release Interact- Bench as versioned snapshots with semi-annual refreshes to maintain freshness, and report a time-split temporal contam- ination diagnostic in Appendix C.3. Contribution #2: a self-contained evaluation harness. Each problem is packaged with an executable local interac- tor, enabling fully offline evaluation without external judge submission and supporting repeatable experimentation. Contribution #3: interaction-aware diagnostics. Be- yondpass@k, our harness checks protocol compliance, en- forces query budgets, and reports a failure taxonomy that distinguishes algorithmic errors such as wrong answers from interaction-specific failures such as protocol violations, query-budget overruns, and deadlocks or timeouts. 2. Related Work We discuss related work along two axes: general code bench- marks and competitive programming benchmarks. Table 1 compares representative benchmarks. 2 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information General code benchmarks.Early execution-based bench- marks such as HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) focus on short, self-contained function synthesis, where performance has become near- saturated for state-of-the-art models. Recent benchmarks broaden to harder tasks (Hendrycks et al., 2021; Jain et al., 2024) and software engineering (Jimenez et al., 2023). Other lines of work strengthen test suites (Liu et al., 2023a), cover complex instructions and diverse function calls (Zhuo et al., 2024), use explicit test-case generation (He et al., 2025; Wang et al., 2025b), and expand to domain-specific and repository-level benchmarks such as DS-1000 (Lai et al., 2023), RepoBench (Liu et al., 2023b), and Cross- CodeEval (Ding et al., 2023). Despite these advances, most benchmarks still score one-shot submissions on fixed tests. Recent agent-style programming benchmarks, such as SWE- bench (Jimenez et al., 2023), AgentBench (Liu et al., 2023c), InterCode (Yang et al., 2023), and CodeAct (Wang et al., 2024), emphasize the development loop, where a model iter- atively edits code with execution feedback (e.g., via tools and tests). However, few of these benchmarks cover interac- tive problems that require a strict multi-round protocol and an explicit query budget. InteractBench targets a different setting: the LLM generates a self-contained program once, and success depends on whether the generated program implements a sound interactive strategy during program– interactor interaction, rather than on iterative debugging. Competitive programming benchmarks. Competitive programming tasks provide a demanding test for algorith- mic reasoning under strict time and memory constraints. Representative benchmarks include AlphaCode (Li et al., 2022), ICPC-Eval (Xu et al., 2025), LiveCodeBench (Jain et al., 2024), LiveCodeBench Pro (Zheng et al., 2025), and LiveOIBench (Zou et al., 2025). Many focus on batch-style problems where the full input is provided upfront (Shi et al., 2024; Xu et al., 2025; Jain et al., 2024). Some rely on sub- missions to external online judges (Quan et al., 2025; Li et al., 2025), limiting fully offline evaluation. Even when interactive problems are included, they typically constitute a small subset (Zheng et al., 2025; Zou et al., 2025), and most suites provide limited protocol-level diagnostics be- yond pass/fail outcomes. Other suites and analyses include OJBench (Wang et al., 2025c) and AetherCode (Wang et al., 2025a). InteractBench addresses these limitations by provid- ing offline-executable local interactors and protocol-aware diagnostics for interactive problems. 3. InteractBench Construction Overview. Unlike in batch-style evaluation, interactive problems require multi-round interaction under strict proto- cols and explicit query budgets. To enable self-contained offline evaluation, we package each task with an executable local interactor and a sandboxed harness. The harness records interaction transcripts and structured termination reasons, enabling interaction-aware diagnosis. We begin with problem definition and task selection, then describe an execution-driven propose–validate–adjudicate loop for con- structing offline evaluator artifacts, and finally the post-hoc online agreement audit (Figure 2). Appendix D.3 provides an end-to-end packaged task example. 3.1. Problem Definition Following standard interactive judging conventions (Mirza- yanov, 2016; CMS Development Team, 2017), we formalize the competitive-programming interactive tasks considered in InteractBench as follows. LetSbe the solver,Jbe the interactor,xbe the hidden test case,Πbe the interaction protocol, andBbe the query budget. For an execution that terminates afterTrounds underΠ, letq t andr t denote the solver’s query and the interactor’s response at roundt, respectively. We useτ t to denote the transcript after the firsttrounds, starting from an initial public transcriptτ 0 (empty if no public data is revealed), and writeτ <t := τ t−1 for the transcript prefix before round t. An execution then unfolds as q t = S qry (τ <t ), r t = J (x,τ <t ,q t ), τ t = τ <t ◦ (q t ,r t ), t = 1,...,T, τ = τ T , a = S out (τ ), v = Eval(a,τ,x; Π,B)∈Accepted, Rejected. Here◦denotes transcript concatenation,S qry is the query- generation component ofS, andS out is its final-output component. Thusτis the complete interaction transcript. The protocolΠspecifies message formats and termina- tion conditions, and induces a budgeted countQ Π (τ )(e.g., queries, rounds, or grader calls). The evaluatorEvalre- turnsAcceptedif and only ifτconforms toΠ,Q Π (τ )≤ B, and the final outputais correct forx; otherwise it returns Rejected. 3.2. Task Selection We curate candidates from Codeforces, AtCoder, IOI, and ICPC. A task is included only if it satisfies all of the follow- ing criteria: • Multi-round interaction. The solution must interact with an external interactor for multiple rounds rather than producing a one-shot output. • Protocol constraints. The interaction protocol is suffi- ciently well-defined (e.g., it specifies message formats 3 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Table 1. Comparison of representative code and competitive programming benchmarks. Columns report the number of problems; whether the benchmark has interactive support (with counts when available); interface type (stdio or grader-linked (CMS Development Team, 2017)); and whether it provides self-contained evaluation, as well as tags and difficulty. Interactive counts may vary for continuously updated benchmarks. Benchmark# ProblemsInteractiveInterfaceSelf-containedTagsDifficulty HumanEval (Chen et al., 2021)164 ✗ – ✓✗ MBPP (Austin et al., 2021)974 ✗ – ✓✗ APPS (Hendrycks et al., 2021)10,000 ✗ – ✓✗✓ CodeContests (Li et al., 2022)13,328 ✗ – ✓ LiveCodeBench (Jain et al., 2024)511 ✗ – ✓✗✓ USACO (Shi et al., 2024)307 ✗ – ✓✗✓ ICPC-Eval (Xu et al., 2025)118 ✗ – ✓✗ HLCE (Li et al., 2025)235 ✓ grader ✗ LiveCodeBench Pro (Zheng et al., 2025)584 ✓ (21)stdio ✓ LiveOIBench (Zou et al., 2025)403 ✓ (23)stdio+grader ✓ CodeElo (Quan et al., 2025)408 ✓ (17)stdio ✗✓ InteractBench (Ours)322 ✓ (322)stdio+grader ✓ and termination conditions) to allow offline execution and compliance checking. •Query budget. The task has an explicit or enforceable budget, such as the number of queries, rounds, or calls, that can be monitored and validated during execution. InteractBench covers both stdio and grader-linked interfaces. For the latter, we retain only multi-query settings with an explicit budget and exclude essentially batch-style single- call interfaces. Difficulty and categories. Each task is annotated with a difficulty tier (Easy/Medium/Hard) and one or more cate- gory labels. Experts assign difficulty based on the problem statement and expected solution complexity, consulting con- test statistics when available. Category labeling starts from platform metadata, is consolidated into seven categories, and is verified by experts. When platform labels are missing or inconsistent, model-assisted suggestions provide candi- date labels before expert finalization. The seven categories are Graph, Search, Greedy, Bit, Data Structures, Math, and Game. Graph refers to graph or tree structure, Search to feasible-set localization, Greedy to committed step-wise construction, and Bit to Boolean/parity-style feedback or compact encodings. Data Structures marks tasks driven by structured state, such as orders, partitions, or reconstruction invariants. Math covers arithmetic, algebraic, or geomet- ric deduction, while Game denotes minimax or worst-case strategic reasoning. Appendix B.1 provides more detailed category definitions. 3.3. Construction Pipeline We construct offline evaluator artifacts via an execution- driven propose–validate–adjudicate loop. Proposer models draft candidate generators and interactors from the problem statement. Candidates are then validated against a per-task verification pool of submissions with known official verdicts. Failed candidates receive trace-grounded feedback from an adjudicator model and re-enter the loop; remaining cases are escalated to human experts after a fixed iteration cap. Propose. Constructing reliable interactive evaluators re- quires jointly designing a test case generator and a judge- side interactor. We adopt an execution-driven propose– validate–adjudicate loop with two proposer models, GPT- 5.2 (OpenAI, 2025a) and Gemini-3-Pro-Preview (Deep- Mind, 2025); we use GPT-5.2 as a code-oriented adjudicator model, while acceptance is determined by execution-based validation. First, the proposer models draft candidate gener- ators, and the adjudicator model selects one that satisfies the contract. We then generate a fixed, seeded suite of100test cases per task. Based on the selected generator and input schema, the proposer models draft candidate interactors. Validate. We validate candidate evaluator artifacts for fi- delity to official verdicts and strict enforcement of the pro- tocol and budget constraints. Our offline harness addition- ally records a structured rejection reason from a unified failure taxonomy (Section 4.2) for diagnosis and analysis. Interactor mode is determined by the original problem state- ment. We implement an adaptive interactor only when the statement permits history-dependent choices by the interac- tor. In such tasks, the interactor may select a hidden value consistent with the transcript and choose among multiple statement-permitted replies to make identification harder; see Appendix D.4 for a concrete example. Otherwise, we use a non-adaptive interactor where the hidden test case is fixed before interaction begins. 4 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information 2. Construction Pipeline Step 1: Propose 1. Task Selection Selected Problems (322 Tasks) Filter ➢Multi-round interaction. ➢Protocol constraints. ➢Query budget. Step 2: Validate Submission Replay in Offline Harness 15 AC + 30 RE / Task 3. Post-hoc Audit Sampled Solutions (5 Sols / Task) Official Judge (Online) Offline Harness (Ours) Agreement Step 3: Adjudicate Results Comparison Proposer Models Test Case Generators & Interactors ≤ 3 Iterations Human Expert Intervention No Human Submissions Accepted Rejected Step 2: Validate Submission Replay in Offline Harness Result Consistent? Feedback Loop Yes No Figure 2. An overview of the InteractBench construction pipeline. We select interactive tasks with multi-round protocols and explicit query budgets and construct offline evaluator artifacts (generator and local interactor) via a propose–validate–adjudicate loop validated on a per-task verification pool. Finally, where permitted, we post-hoc audit agreement by submitting a small sample of solutions to the official evaluation interface and comparing Accepted/Rejected outcomes with our offline harness. We build a per-task verification pool with15accepted sub- missions and30rejected submissions. The rejected side includes15rejected human submissions with known official judge outcomes and15mutation-derived rejected probes obtained by lightly mutating accepted or rejected submis- sions. These mutations target common failure surfaces (e.g., protocol/format deviations, query-budget violations, and minor logic perturbations) that a faithful evaluator should reject. A candidate is accepted only if our offline harness matches the official judge outcomes on all human submis- sions and rejects all mutation-derived probes. Appendix D.5 details the probe construction. When official evaluation packages are publicly available (e.g., IOI test suites and checkers/graders/managers), we adapt them to our harness. Validation is performed fully offline: the harness runs the solver and a local interactor as two isolated processes con- nected via standard streams. It enforces protocol and budget constraints via explicit counters and logs the interaction transcript and termination signals for diagnosis. Both the solver and the interactor are sandboxed with explicit per- process time and memory budgets; by default, we follow the original task resource budgets. This two-process interface makes evaluation portable across machines and, for stdio tasks, across programming lan- guages under the same packaged interactor, hidden test cases, and sandbox configuration. Portability supports re- peatable experimentation without dependence on external judging infrastructure. In particular, cross-language exe- cution requires only that the solver follows the protocol, including explicit flushing after each query (Mirzayanov, 2016). Adjudicate.Candidates that fail validation proceed to ad- judication: we provide trace-grounded feedback and run another iteration. We cap the loop at three iterations before escalating remaining ambiguous cases to human experts for final auditing and correction. After reaching the iteration cap,31tasks required expert intervention. Appendix D.6 provides a representative expert-intervention case study. Our human experts are experienced competitive programmers with strong backgrounds in algorithms and interactive judg- ing, and they finalize the evaluator by revising the generator and interactor based on trace evidence. 3.4. Post-hoc Audit As a supplementary sanity check, we conduct a one-time post-hoc online agreement audit by manually submitting sampled model-generated solutions to the official evaluation interface, where permitted. Direct submission-based eval- uation does not scale well: platforms such as Codeforces 5 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information prohibit automated submissions and impose strict rate lim- its (Zou et al., 2025; Xu et al., 2025). We therefore restrict this audit to5submissions per task and verify consistent Accepted/Rejected outcomes between the official evaluation interface and our offline harness. Submissions are made conservatively in non-competitive settings. 4. Experiments 4.1. Experimental Setup Overview. We study interactive problem solving under unrevealed information at execution time. Solvers must ac- quire information through multi-round queries under strict protocol constraints and query budgets. We conduct three analyses. In Experiment #1, we report success rates strat- ified by difficulty (Table 2). In Experiment #2, we break down performance across categories that require informa- tion acquisition and dynamic state tracking under partial observability (Table 3). In Experiment #3, we diagnose whether failures stem from algorithmic logic errors (WA) or interaction-specific breakdowns such as protocol violations and query-budget overruns. Tasks. We report results on InteractBench. Each task is evaluated locally under our harness with its executable local interactor (Section 3). For analysis in Experiments #1–2, tasks are annotated with difficulty tiers and categories; dataset statistics are provided in Appendix B.1. Models and outputs.We evaluate a mix of closed-source and open-source LLMs that generate self-contained pro- grams. Our suite includes reasoning configurations (e.g., DeepSeek-V3.2-Thinking (Liu et al., 2025), GPT-5.2 (Ope- nAI, 2025a), Gemini-3-Pro-Preview (DeepMind, 2025), Gemini-2.5-Pro (Google DeepMind, 2025b), Gemini-2.5- Flash (Google DeepMind, 2025a), Claude Opus 4.5 (An- thropic, 2025), and GPT-OSS (Agarwal et al., 2025)) as well as standard non-reasoning baselines (DeepSeek-V3.2, GPT-4.1 (OpenAI, 2025b), Qwen3-32B-NonThinking, and Qwen3-14B-NonThinking). In total, we evaluate 16 model configurations (12 reasoning and 4 non-reasoning). Detailed model information is provided in Appendix B.2. Since some model families have both a reasoning and a non- reasoning variant (notably DeepSeek-V3.2, Qwen3-32B, and Qwen3-14B), we report each variant as a separate con- figuration and use within-family comparisons to isolate the effect of explicit reasoning. While the harness is language- agnostic, we report results for C++ submissions in this pa- per to match common competitive-programming practice and prior benchmarks (Zou et al., 2025; Xu et al., 2025) (Appendix C.4 reports cross-language and template compar- isons). Prompting and sampling. We use a standardized zero- shot prompt consisting of the problem statement plus ex- plicit instructions for interactive I/O (e.g., adhere to the pro- tocol and flush after each query for stdio tasks). For model access, we query closed-source models via provider APIs. For open-source checkpoints, we self-host those that fit on our inference server (8 NVIDIA A100 GPUs); otherwise, we access them via provider APIs. For each problem, we samplen = 10independent solutions with temperature 0.8 to reduce the variance ofpass@k estimates. Appendix B.3 lists the evaluation-time prompt wrapper. Execution environment.We compile and execute all sub- missions in an isolated sandbox (Docker with cgroups) on Ubuntu 22.04 with an Intel Xeon Platinum 8470Q CPU. We enforce per-run limits of 2 CPU cores and 4 GB RAM. Solutions are compiled withg++11.4.0 (C++17,-O2); we use Python 3.12, Java 21, and Go 1.21 for cross-language evaluation (Appendix C.4). We enforce time limits using CPU time (to align with competitive-programming judging), together with a conservative wall-clock cap to terminate hanging runs such as blocked I/O or deadlocks. 4.2. Metrics pass@k. We reportpass@1 andpass@5. Givennsam- pled solutions for a problem, letcdenote the number of solutions accepted on the full hidden test suite under our harness. Following HumanEval (Chen et al., 2021), we estimate pass@k using the unbiased estimator: pass@k = 1− n−c k n k . We then average pass@k over the evaluation set. Failure taxonomy. To support interaction-aware diagno- sis, we assign each unsuccessful execution a primary failure label (mutually exclusive): IDLE (blocked I/O or dead- lock terminated by the wall-clock cap), PE (protocol/format violations), QLE (query-budget overrun), CE (compile er- ror), RE (runtime error), TLE (time limit exceeded), MLE (memory limit exceeded), and WA (protocol-compliant but incorrect answer). If several conditions could apply, we keep compilation, runtime, resource, or idle failures as the primary cause; among the remaining rejected runs, pro- tocol violations are labeled before query-budget overruns and incorrect final answers. For discussion, we refer to WA as an algorithmic logic error and PE/QLE/IDLE as interaction-specific failures, while CE/RE/TLE/MLE cap- ture compilation-, runtime-, and resource-limit failures. We provide representative examples for each category in Ap- pendix D.2. 6 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Table 2.pass@1 andpass@5 (higher is better) stratified by difficulty on InteractBench. We samplen = 10solutions per task. Configurations are grouped into reasoning and non-reasoning. ModelEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Reasoning Configurations Gemini-3-Pro-Preview0.8030.9610.5120.7160.3670.5970.5540.752 GPT-5.20.7420.9210.5300.6900.3400.4920.5420.705 DeepSeek-V3.2-Thinking0.6450.8160.2760.4840.1040.2390.3320.513 Gemini-2.5-Pro0.5870.7630.2590.4520.0920.2240.3050.480 GPT-OSS-120B0.5210.7500.1990.3550.0780.1790.2540.416 Claude-Opus-4.5 0.3180.5920.1410.2840.0210.0600.1590.312 Gemini-2.5-Flash0.3320.5660.1440.2900.0090.0450.1620.305 GPT-OSS-20B0.3130.5790.0960.2260.0210.0750.1340.282 Qwen3-30B-A3B-Thinking0.1900.4080.1040.2130.0120.0600.1050.228 Qwen3-32B0.2370.4210.0650.1230.0090.0450.0960.181 Qwen3-14B0.1470.2630.0400.1030.0000.0000.0580.121 DeepSeek-R1-Distill-Llama-70B0.1050.2240.0350.0770.0000.0000.0450.097 Non-Reasoning Configurations DeepSeek-V3.20.3080.5000.1230.2320.0240.0600.1480.262 GPT-4.1 0.1630.3420.0390.0900.0030.0150.0620.138 Qwen3-32B-NonThinking0.0320.0920.0170.0320.0000.0000.0170.040 Qwen3-14B-NonThinking 0.0260.0920.0100.0260.0000.0000.0120.037 4.3. Experimental Results Experiment #1: performance across difficulty tiers.We begin by examining how interactive performance degrades with problem difficulty. Interactive tasks with unrevealed in- formation at execution time remain far from solved even for frontier reasoning models. We reportpass@1 andpass@5 stratified by easy/medium/hard to reveal the scaling curve that can be obscured by a single aggregate score. Table 2 shows a steep scaling curve with difficulty. We highlight four findings: • Hard interactive tasks remain unsolved and strongly dis- criminative. With resampling, Easy problems approach saturation for frontier models. For example, Gemini-3- Pro-Preview reachespass@5 of 0.961. In contrast, Hard problems remain far from solved: the best-performing model reaches 0.597 on hardpass@5. Only two config- urations exceed 0.400 on hardpass@5 (Gemini-3-Pro- Preview and GPT-5.2), yielding a clear separation be- tween frontier configurations and the rest of the suite. •Gaps betweenpass@1 andpass@5 quantify the benefit of resampling. Even on Easy,pass@1 can lag meaning- fully behindpass@5, indicating that additional attempts often recover a correct solution by exploring alternative approaches and interaction strategies. In interactive set- tings, some of this variance also comes from brittle ex- ecution details (e.g., formatting, flushing, and protocol adherence), rather than always missing high-level ideas. Appendix D.1 provides a representative same-task case study where a small robustness fix flips a protocol failure into a successful run. •Resampling yields uneven gains across models and dif- ficulty tiers. Some frontier configurations see larger im- provements on Hard under resampling, suggesting that correct solutions are often attainable but sensitive to small implementation details. For example, on Hard, Gemini- 3-Pro-Preview rises from 0.367 inpass@1 to 0.597 in pass@5, whereas GPT-5.2 rises from 0.340 to 0.492. Oth- ers show diminishing gains on Hard, suggesting a more fundamental capability gap. • Explicit reasoning improves pass rates, but does not remove the Hard barrier. Within-family comparisons show large improvements from reasoning variants (e.g., DeepSeek-V3.2-Thinking attains an overallpass@5 of 0.513, compared with 0.262 for DeepSeek-V3.2). How- ever, smaller models remain bottlenecked on Hard even with reasoning (e.g., Qwen3-14B achieves 0.000 hard pass@5), suggesting that explicit reasoning is helpful but insufficient without strong base capability. Experiment #2: breakdown by category.We next local- ize the interaction gap across categories to identify which categories most stress information acquisition and dynamic state tracking under unrevealed information. Table 3 highlights where interactive capability breaks down, and reveals four patterns: •Graph is the dominant bottleneck. Graph is consistently challenging: even the best-performing model stays be- low 0.500 inpass@1. This reflects the need to acquire information from an initially unrevealed structure under strict query budgets, and to track state under partial ob- servability. In contrast, tasks in the Bit group are compar- atively easier across the suite; for example, Gemini-3-Pro- Preview attains apass@1 of 0.613 on Bit compared with 7 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Table 3. pass@1 and pass@5 by category. Configurations are grouped into reasoning and non-reasoning. ModelGraphSearchGreedyBitData StructuresMathGame pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Reasoning Configurations Gemini-3-Pro-Preview0.4750.7050.5610.7700.4700.7250.6130.7110.4870.7500.5890.7440.4800.743 GPT-5.2 0.4500.5910.5470.7080.5300.7250.5690.7110.5000.7110.5630.7120.5660.800 DeepSeek-V3.2-Thinking0.2950.5110.3400.5040.3450.5800.3380.4670.2950.5130.3140.4720.3770.657 Gemini-2.5-Pro0.2550.4430.3110.4690.2930.4780.3420.4440.2470.4470.3070.4720.3090.600 GPT-OSS-120B 0.2160.3860.2690.4070.2260.3620.2800.5110.1840.3160.2540.4240.2910.457 Claude-Opus-4.50.1390.2610.1520.2740.1480.3190.1200.2890.1290.3030.1650.3040.1490.314 Gemini-2.5-Flash0.1140.2040.1700.3100.1710.3910.1510.2890.1290.3030.1660.3120.1830.429 GPT-OSS-20B0.1200.2500.1360.2660.1450.3190.0980.2440.0680.2100.1410.3040.1490.343 Qwen3-30B-A3B-Thinking0.1020.1930.1150.2570.1540.3480.0840.1780.0950.2100.0820.2080.1370.314 Qwen3-32B0.1110.1820.1010.1950.1130.2320.0980.1560.0740.1450.0560.1280.1200.229 Qwen3-14B0.0610.1360.0730.1240.0670.1590.0490.0670.0530.1320.0400.1040.0740.200 DeepSeek-R1-Distill-Llama-70B0.0390.1140.0730.1330.0520.1450.0180.0440.0370.0920.0370.0640.0570.171 Non-Reasoning Configurations DeepSeek-V3.20.1390.2610.1650.2830.1590.2900.1380.2440.0950.2370.1310.2160.1310.229 GPT-4.10.0570.1590.0640.1150.0780.1740.0360.0670.0530.1320.0420.1120.0570.229 Qwen3-32B-NonThinking 0.0140.0340.0120.0440.0140.0580.0220.0220.0080.0260.0110.0240.0170.029 Qwen3-14B-NonThinking0.0210.0450.0120.0530.0150.0430.0000.0000.0110.0400.0050.0160.0170.057 0.475 on Graph. This is consistent with more templated information extraction and less complex dynamic state tracking. •Frontier models exhibit complementary strengths across categories. Gemini-3-Pro-Preview is especially strong on Graph, whereas GPT-5.2 is strongest on Game, suggest- ing that interactive performance decomposes into distinct skills and that aggregate scores can hide specialization. •Stateful maintenance remains a recurring source of diffi- culty. Data Structures tasks in our taxonomy often behave like online reconstruction and maintenance: the solver must maintain a structured internal state (e.g., refining an order/permutation/partition from relational tests) and update it correctly after each reply while preserving in- variants. This long-horizon bookkeeping amplifies small update mistakes over many rounds. • Reasoning lifts performance broadly across categories. Within-family comparisons show consistent gains across columns; for example, DeepSeek-V3.2-Thinking im- proves over DeepSeek-V3.2 on Graph, raisingpass@5 from 0.261 to 0.511, with similar improvements on Search, Greedy, Math, and Game. Experiment #3: failure-mode decomposition. Finally, we diagnose failure causes: whether the bottleneck is al- gorithmic logic errors (WA) or interaction-specific break- downs.We decompose failed executions using our interaction-aware taxonomy (Section 4.2) and report the fraction attributable to each primary failure type. Table 4 yields three insights into failure causes in interactive runs: •Algorithmic logic errors (WA) dominate, but interaction- specific failures occur at different stages. In Table 4, WA is the most common failure for many configurations, indicating that interactive tasks remain bottlenecked by algorithmic reasoning and state tracking under partial observability. The interaction-specific labels further dis- Table 4. Failure composition among unsuccessful executions (higher means more common). We condition on failed runs and report the fraction of failures attributable to each primary failure type for each model (i.e., normalized by that model’s failed runs; Section 4.2). Configurations are grouped into reasoning and non- reasoning. ModelIDLEPECETLEMLEREWAQLE Reasoning Configurations Gemini-3-Pro-Preview0.0180.1200.0270.0780.0030.0110.6240.119 GPT-5.20.0190.1580.0310.0640.0000.0150.6120.101 DeepSeek-V3.2-Thinking0.0240.1320.0630.0450.0020.0120.6470.074 Gemini-2.5-Pro0.0460.1570.0240.0350.0020.0110.6580.068 GPT-OSS-120B0.4010.1050.0130.0130.0020.0100.4240.031 Claude-Opus-4.50.0120.1240.2010.0670.0060.0030.5050.082 Gemini-2.5-Flash0.0350.1410.0410.0280.0040.0130.7050.033 GPT-OSS-20B 0.0940.1250.0740.0160.0020.0090.6530.027 Qwen3-30B-A3B-Thinking0.0510.1180.0230.0200.0020.0200.7300.036 Qwen3-32B0.0390.1600.0340.0280.0010.0110.6830.044 Qwen3-14B0.0520.1490.0470.0240.0010.0140.6860.028 DeepSeek-R1-Distill-Llama-70B0.0710.1880.1060.0250.0020.0150.5640.028 Non-Reasoning Configurations DeepSeek-V3.20.0400.1710.0590.0390.0040.0110.6120.063 GPT-4.10.0340.2000.0690.0420.0020.0100.5790.064 Qwen3-32B-NonThinking0.0480.1370.1210.0230.0010.0140.6230.035 Qwen3-14B-NonThinking0.0790.1870.1390.0310.0020.0210.5240.018 tinguish where execution breaks: PE and IDLE usually indicate that execution breaks before a stable query loop is established, whereas QLE indicates that the solver en- ters the protocol but exhausts the query budget. Some configurations still break down at the earlier stage, no- tably elevated IDLE for GPT-OSS-120B; qualitative cases suggest it may misinterpret interactive tasks as ordinary input-output problems and never enter a valid query loop. Appendix D.1 provides representative cases. •Query efficiency limits frontier models as well. QLE is more prevalent among frontier models (e.g., 0.119 for Gemini-3-Pro-Preview and 0.101 for GPT-5.2) than among weaker ones, suggesting that stronger models can follow the protocol but still fail to manage the query bud- get. •Runtime and resource-limit failures are present but sec- ondary. TLE, MLE, and RE appear across the suite, but are generally secondary to algorithmic logic errors and interaction-specific failures. 8 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information We provide representative qualitative examples for each fail- ure type in Appendix D.2. Appendix C.1 complements this analysis with a batch-vs-interactive comparison: revealing the hidden test data improvespass@1 and reduces proto- col errors and query-budget overruns, while wrong answers remain the main residual failure type. Appendix C.2 then compares independent resampling, few-shot prompting, and iterative refinement. Few-shot prompting and independent resampling perform similarly overall, whereas iterative re- finement is more capability-dependent and mainly repairs execution-level failures. 5. Conclusion InteractBench targets competitive-programming settings where key information is unrevealed at execution time and must be acquired through multi-round queries under strict protocols and limited query budgets. We curate 322 interac- tive tasks from Codeforces, AtCoder, IOI, and ICPC, and package each task with an executable local interactor, en- abling fully offline and reproducible evaluation. Beyond pass@k, our harness surfaces interaction-specific failure modes via protocol compliance checks, query-budget en- forcement, and a fine-grained taxonomy that separates algo- rithmic errors from interaction breakdowns. Across 16 model configurations, we observe consistent degradation as problem difficulty increases, highlighting a clear gap between standard (full-information) coding bench- marks and interactive problem solving. While failures are dominated by algorithmic logic errors, protocol violations and query-budget overruns remain frequent even for frontier reasoning models. Looking forward, we plan to expand coverage and main- tain freshness through versioned releases, and to develop prompting and training strategies that directly target interac- tive querying, state maintenance, and protocol adherence. 6. Limitations Scale remains the main practical limitation of InteractBench. Interactive tasks are much less common than batch-style problems on competitive-programming platforms, and of- fline evaluation requires a reliable local interactor. Future versions will expand coverage by collecting more eligible tasks and constructing new ones with verified interactors. In addition, our experiments use zero-shot model-generated solvers, and follow-up studies can examine prompting and training strategies for interactive querying, state mainte- nance, and protocol adherence. Impact Statement InteractBench is a benchmark for evaluating program synthe- sis under unrevealed information, where a generated solver must follow a strict interaction protocol and acquire infor- mation through multi-round queries. We expect it to support research on reliable code generation, state tracking, and pro- tocol adherence, and to enable fine-grained, reproducible evaluation of LLM-based coding systems. More broadly, this paper presents work whose goal is to advance the field of machine learning by providing an evaluation methodol- ogy for model capabilities. We encourage responsible use in accordance with applicable rules and norms. Acknowledgments Qiankun Zhang is supported by the National Natural Science Foundation of China (Grant 62302183), Open Foundation of Key Laboratory of Cyberspace Security, Ministry of Edu- cation of China (Grant KLCS20240401), and CCF-DiDi GAIA Collaborative Research Funds (Grant CCF-DiDi GAIA 202522). Xianjun Deng is supported by the National Key R&D Program of China (Grant 2022YFE0138600), and the National Natural Science Foundation of China (Grant U24B20153). References Agarwal, S., Ahmad, L., Ai, J., Altman, S., Applebaum, A., Arbus, E., Arora, R. K., Bai, Y., Baker, B., Bao, H., et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. Anthropic. Introducing claude opus 4.5, November 2025. URLhttps://w.anthropic.com/news/c laude-opus-4-5. AtCoder Inc. Atcoder online judge, 2026. URLhttps: //atcoder.jp/. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. CMS Development Team. Cms documentation: Task types (batch, outputonly, communication, twosteps), 2017. URLhttps://cms.readthedocs.io/en/v1 .3/Task%20types.html. Codeforces. Codeforces online judge, 2026. URLhttps: //codeforces.com/. 9 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information DeepMind, G. A new era of intelligence with gemini 3, November 2025. URLhttps://blog.google/pr oducts-and-platforms/products/gemini /gemini-3/. Ding, Y., Wang, Z., Ahmad, W., Ding, H., Tan, M., Jain, N., Ramanathan, M. K., Nallapati, R., Bhatia, P., Roth, D., et al. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information Processing Systems, 36:46701– 46723, 2023. Google DeepMind. Gemini 2.5 flash is now in preview, 2025a. URLhttps://blog.google/products /gemini/gemini-2-5-flash-preview/. Google DeepMind. Gemini 2.5: Our newest gemini model with thinking, 2025b. URLhttps://blog.googl e/technology/google-deepmind/gemini-m odel-thinking-updates-march-2025/. He, Z., Choi, Y. M., Zhang, K., Ji, J., Zhou, J., Xu, D., Bercovich, I., Zhang, A., and Li, L. Hardtests: Synthesiz- ing high-quality test cases for llm coding. arXiv preprint arXiv:2505.24098, 2025. Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938, 2021. Huang, Y., Lin, Z., Liu, X., Gong, Y., Lu, S., Lei, F., Liang, Y., Shen, Y., Lin, C., Duan, N., and Chen, W. Competition-level problems are effective llm evalua- tors. In Findings of the Association for Computational Linguistics: ACL 2024, p. 13526–13544, 2024. doi: 10.18653/v1/2024.findings-acl.803. ICPC Foundation. International collegiate programming contest, 2026. URL https://icpc.global/. International Olympiad in Informatics.International olympiad in informatics, 2026. URLhttps://io informatics.org/. Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free eval- uation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language mod- els resolve real-world github issues?arXiv preprint arXiv:2310.06770, 2023. Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettle- moyer, L., Yih, W.-t., Fried, D., Wang, S., and Yu, T. Ds-1000: A natural and reliable benchmark for data sci- ence code generation. In International Conference on Machine Learning, p. 18319–18345. PMLR, 2023. Li, X., Li, X., Dong, K., Zhang, Q., Ruan, R., Dai, X., Liu, X., Xu, S., Wang, Y., and Tang, R. Humanity’s last code exam: Can advanced llms conquer human’s hardest code competition? arXiv preprint arXiv:2506.12713, 2025. Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., et al. Competition-level code generation with AlphaCode. Science, 378(6624):1092–1097, 2022. doi: 10.1126/scie nce.abq1158. Liu, A., Mei, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558– 21572, 2023a. Liu, T., Xu, C., and McAuley, J. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091, 2023b. Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. Agentbench: Evalu- ating llms as agents. arXiv preprint arXiv:2308.03688, 2023c. Mirzayanov, M. Interactive problems: Guide for partici- pants, June 2016. URLhttps://codeforces.com /blog/entry/45307. OpenAI. Update to gpt-5 system card: Gpt-5.2, December 2025a. URLhttps://openai.com/index/gpt -5-system-card-update-gpt-5-2/. OpenAI. Model documentation: Gpt-4.1, 2025b. URL https://platform.openai.com/docs/mod els/gpt-4.1. Quan, S., Yang, J., Yu, B., Zheng, B., Liu, D., Yang, A., Ren, X., Gao, B., Miao, Y., Feng, Y., et al. Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings.arXiv preprint arXiv:2501.01257, 2025. Shi, Q., Tang, M., Narasimhan, K., and Yao, S. Can lan- guage models solve olympiad programming?arXiv preprint arXiv:2404.10952, 2024. Verhoeff, T. 20 years of ioi competition tasks. Olympiads in Informatics, 3:149–166, 2009. URLhttps://io informatics.org/journal/INFOL047.pdf. 10 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., and Ji, H. Executable code actions elicit better llm agents. In Forty-first International Conference on Machine Learn- ing, 2024. Wang, Z., Chen, J., Liu, Z., Mak, M., Du, Y., Moon, G., Xu, L., Tua, A., Peng, K., Lu, J., et al. Aethercode: Evaluating llms’ ability to win in premier programming competitions. arXiv preprint arXiv:2508.16402, 2025a. Wang, Z., Liu, S., Sun, Y., Li, H., and Shen, K. Codecon- tests+: High-quality test case generation for competitive programming. arXiv preprint arXiv:2506.05817, 2025b. Wang, Z., Liu, Y., Wang, Y., He, W., Gao, B., Diao, M., Chen, Y., Fu, K., Sung, F., Yang, Z., et al. Ojbench: A competition level code benchmark for large language models. arXiv preprint arXiv:2506.16395, 2025c. Xu, S., Hu, Y., Min, Y., Chen, Z., Zhao, W. X., and Wen, J.-R. Icpc-eval: Probing the frontiers of llm reasoning with competitive programming contests. arXiv preprint arXiv:2506.04894, 2025. Yang, J., Prabhakar, A., Narasimhan, K., and Yao, S. Inter- code: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36:23826–23854, 2023. Zheng, Z., Cheng, Z., Shen, Z., Zhou, S., Liu, K., He, H., Li, D., Wei, S., Hao, H., Yao, J., et al. Livecodebench pro: How do olympiad medalists judge llms in competitive programming? arXiv preprint arXiv:2506.11928, 2025. Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. Big- codebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877, 2024. Zou, K., Xiong, A., Zhang, Y., Zhang, F., Ren, Y., Yang, J., Lee, A., Bhushan, S., and Wang, L. Liveoibench: Can large language models outperform human con- testants in informatics olympiads?arXiv preprint arXiv:2510.09595, 2025. 11 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Appendix A InteractBench Specification13 A.1 Task Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 B Evaluation Setup13 B.1 Dataset Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 B.2 Model Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14 B.3 Evaluation Prompt for Code Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 C Supplementary Experiments15 C.1 Batch vs. Interactive Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 C.2 Resampling, Few-Shot Prompting, and Iterative Refinement . . . . . . . . . . . . . . . . . . . . . . . .16 C.3 Temporal Contamination Diagnostic via Time Splits . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 C.4 Multi-language and I/O Template Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 D Diagnostics and Artifacts18 D.1 Qualitative Evidence for Key Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 D.2 Error Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 D.3 End-to-End Task Showcase: cf2036G . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26 D.4 Adaptive Interactor Example: cf2096G . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37 D.5 Mutation-based Stress Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40 D.6 Expert Intervention Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41 D.7 Interactive I/O Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44 D.8 Prompt Templates for the Propose–Validate–Adjudicate Loop . . . . . . . . . . . . . . . . . . . . . . . .47 D.9 Model Performance by Category and Difficulty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52 12 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information A. InteractBench Specification This appendix collects benchmark specifications and supplementary materials for InteractBench. We organize the appendix into four parts: (i) specification, (i) evaluation setup, (i) supplementary experiments, and (iv) diagnostics and artifacts. A.1. Task Card Each InteractBench task is accompanied by a structured task card specifying the task interface required by our offline harness. We include the schema here to make the benchmark interface explicit and auditable. Table 5. Task card fields in InteractBench. FieldDescription problemidUnique identifier of the task. descProblem statement, including the stdin/stdout interaction protocol, termination rules, and con- straints (e.g., query limits). interactormodeInteractor mode metadata: non-adaptive, adaptive, or both. cputimelimitmsCPU time limit (in milliseconds). memorylimitmbMemory limit (in megabytes). testcasesHidden test cases used for offline evaluation and consumed by the interactor. interactorInteractor program that implements the protocol and checks the solver’s behavior. generatorOptional generator program used to produce hidden test cases. difficultyTask difficulty label. categorizationCategory labels assigned to the task (e.g., Graph, Search). B. Evaluation Setup B.1. Dataset Statistics Experiments #1–2 in Section 4 use InteractBench. Section 3 describes how difficulty and category labels are assigned. This appendix reports the source, difficulty, and category breakdowns and gives the full category definitions. Task sources and difficulty. InteractBench tasks are curated from four competitive-programming sources: Code- forces (Codeforces, 2026) and AtCoder (AtCoder Inc., 2026) (online judges), and the IOI (International Olympiad in Informatics, 2026) and ICPC (ICPC Foundation, 2026) series (competition series). Table 6 reports task counts by source and difficulty. Category breakdown. Table 7 reports task counts by category stratified by difficulty. Because category labels are multi-label, a task may contribute to more than one category. Category definitions. For label assignment and category-level analyses, we use the definitions below. •Graph. The hidden object is naturally modeled as a graph or tree, or a hidden target is defined on a known graph structure. Queries usually reveal relational or metric-like signals, such as connectivity, reachability, distances, neighborhoods, or objective values on graph elements. Solvers choose probes from the partial structure inferred so far. • Search. The solver localizes a hidden value, position, or configuration by shrinking a feasible set. Replies provide constraints such as membership, counts over a subset or region, monotone comparisons, or directional feedback. The solution maintains the feasible region across rounds and handles boundary cases at termination. • Greedy. The solver builds an answer through committed steps whose feedback constrains later moves. The key invariant is that the partial construction remains extendable; otherwise locally plausible decisions can lead to dead ends or waste interaction opportunities. 13 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information •Bit. Replies carry low-bandwidth information, often Boolean or parity-like. Solvers either design queries that collect independent constraints or decode a short transcript into the hidden object under the protocol restrictions. • Data Structures. These tasks require online maintenance of structured state across replies. Typical state includes orders, permutations, partitions, or reconstruction invariants, where bookkeeping errors affect later queries and the final answer. •Math. Replies follow arithmetic, algebraic, or geometric structure that supports deduction or constructive design from a small number of queries. The solver combines these relations to identify an unknown object or construct a valid witness, while handling uniqueness and degenerate cases. •Game. The solver must remain correct against worst-case replies allowed by the protocol. Solutions reason from what the transcript guarantees and often use minimax-style planning over the remaining knowledge states. Table 6. Task counts by difficulty, broken down by source. SourceEasyMediumHardTotal Codeforces5211553220 ICPC16331261 AtCoder87217 IOI97824 Total8516275322 Table 7. Task counts by category stratified by difficulty. A task may be assigned multiple categories. CategoryEasyMediumHardTotal Graph23452088 Search315824113 Greedy16421169 Bit12221145 Data Structures14511176 Math316034125 Game1118635 B.2. Model Card Table 8. Model card for models used in InteractBench. Models are grouped by provider and access mode (closed-source or open-source). We list parameter counts when publicly specified and provide official hyperlinks. ModelModeSizeHyperlink OpenAI (Closed-source) GPT-5.2Thinking– https://chatgpt.com GPT-4.1NonThinking– https://chatgpt.com OpenAI (Open-source) GPT-OSS-120BThinking120B https://huggingface.co/openai/gpt-oss-120b GPT-OSS-20BThinking20B https://huggingface.co/openai/gpt-oss-20b Google (Closed-source) Gemini-3-Pro-PreviewThinking– https://gemini.google.com Gemini-2.5-ProThinking– https://gemini.google.com Gemini-2.5-FlashThinking– https://gemini.google.com Anthropic (Closed-source) Claude-Opus-4.5Thinking– https://claude.ai DeepSeek (Open-source) DeepSeek-V3.2-ThinkingThinking685B https://huggingface.co/deepseek-ai/DeepSeek-V3.2 DeepSeek-V3.2NonThinking685B https://huggingface.co/deepseek-ai/DeepSeek-V3.2 DeepSeek-R1-Distill-Llama-70BThinking70B https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B Qwen/Alibaba (Open-source) Qwen3-32BThinking32B https://huggingface.co/Qwen/Qwen3-32B Qwen3-32B-NonThinkingNonThinking32B https://huggingface.co/Qwen/Qwen3-32B Qwen3-30B-A3B-ThinkingThinking30B-A3B https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507 Qwen3-14BThinking14B https://huggingface.co/Qwen/Qwen3-14B Qwen3-14B-NonThinkingNonThinking14B https://huggingface.co/Qwen/Qwen3-14B 14 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information B.3. Evaluation Prompt for Code Generation We evaluate models with a standardized zero-shot wrapper prompt. The wrapper is shown as a language-parameterized template (placeholders such asLANGUAGE/LANGUAGETAG) and instantiated with the target language in each experiment. The wrapper is intentionally minimal to reduce prompt-engineering confounds, while still enforcing interactive I/O hygiene (protocol compliance, flushing, and no extraneous output). Models are evaluated by compiling and executing the generated program against the task interactor; correctness is determined by the interactive transcript under the protocol. Evaluation prompt: code generation You are a competitive programming expert solving an INTERACTIVE problem. Implement a complete LANGUAGE solution that deduces hidden parameters strictly through the defined query protocol. Your solution will be compiled and executed against an interactive judge; only the program’s interactive behavior under the protocol is evaluated. Requirements: - Single-file program with a standard entry point (e.g., main / top-level), reading from stdin and writing to stdout. - Flush stdout after every query using the idiomatic mechanism in LANGUAGE (examples): - C++: ‘std::endl‘, ‘std::cout << std::flush‘, or ‘std::cout.flush()‘. - Python: ‘print(..., flush=True)‘ or ‘sys.stdout.flush()‘. - Java: ‘System.out.flush()‘. - Go: if using ‘bufio.Writer‘, call ‘w.Flush()‘ after each query. - Rust: ‘use std::io::Write; std::io::stdout().flush().unwrap();‘. - Terminate immediately if the judge returns an invalid response (e.g., -1). - No debug output to stdout. Output format: - Output a single fenced code block containing the complete program (use the correct language identifier, e.g., cpp/python/java), and nothing else: ‘LANGUAGE_TAG // code... ‘ C. Supplementary Experiments This section collects supplementary experiments that complement the main experiments in Section 4. C.1. Batch vs. Interactive Evaluation We compare a batch variant against the standard interactive evaluation reported in Section 4 on150matched problems. In the batch variant, the solver receives the hidden test data upfront through standard input and still has to produce the final answer under the same validation logic. This keeps the final validation unchanged while removing the need to acquire hidden information through queries. Table 9. Batch vs. interactive evaluation.pass@1 and failure composition on150matched problems. Failure-type fractions are computed over unsuccessful executions. ModelSettingpass@1WAPEQLEIDLECERETLEMLE GPT-5.2 Batch0.7670.8000.0860.0000.0290.0290.0000.0570.000 Interactive0.5110.6290.1640.0760.0050.0380.0080.0790.000 DeepSeek-V3.2-Thinking Batch0.5600.7880.1060.0150.0150.0300.0150.0300.000 Interactive0.3130.6920.1150.0590.0140.0600.0100.0470.004 DeepSeek-V3.2 Batch0.3670.6950.0840.0320.0210.1370.0110.0210.000 Interactive0.1470.6530.1780.0330.0280.0410.0170.0460.003 Qwen3-14B Batch 0.0930.7940.0880.0220.0150.0510.0150.0150.000 Interactive0.0560.6810.1500.0330.0520.0410.0170.0240.003 Qwen3-14B-NonThinking Batch0.1800.6910.1140.0160.0240.1300.0080.0160.000 Interactive0.0110.5350.1900.0140.0960.1150.0230.0270.001 15 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Table 9 shows that revealing the hidden test data improvespass@1 for every tested configuration. The gain is largest for GPT-5.2, DeepSeek-V3.2-Thinking, and DeepSeek-V3.2, while Qwen3-14B shows a smaller improvement. A rank reversal appears for the two Qwen3-14B variants: Qwen3-14B-NonThinking is higher in batch evaluation (0.180vs.0.093), but lower in interactive evaluation (0.011vs.0.056). Batch evaluation also substantially reduces protocol errors and query-budget overruns, but wrong answers remain the dominant failure type among unsuccessful runs. Overall, unrevealed information adds a distinct source of difficulty beyond the batch algorithmic core. At the same time, the remaining wrong answers indicate that revealing the hidden data upfront does not eliminate the underlying reasoning problem. C.2. Resampling, Few-Shot Prompting, and Iterative Refinement We further compare three evaluation-time strategies: independent resampling, few-shot prompting with a solved interactive example, and iterative refinement. Following ICPC-Eval (Xu et al., 2025), iterative refinement runs the solver for several rounds: after each failed attempt, the execution feedback is returned to the model, which then revises its program for the next round. refine@k is the cumulative solve rate within k refinement rounds. Table 10. Comparison of mitigation strategies. Independent resampling (pass@5), few-shot prompting, and iterative refinement (refine@5). Modelpass@5pass@5 (few-shot)refine@5 GPT-5.20.7050.7140.677 DeepSeek-V3.2-Thinking0.5130.5060.491 DeepSeek-V3.2 0.2620.2790.227 Qwen3-14B0.1210.1430.071 Qwen3-14B-NonThinking0.0370.0560.019 Table 10 shows that few-shot prompting is competitive with independent resampling and is slightly higher for most configurations. The clearest few-shot gains appear for weaker configurations, such as Qwen3-14B and Qwen3-14B- NonThinking. Iterative refinement improves over a single attempt for stronger models, but its final scores remain below those of independent resampling and few-shot prompting. Table 11. Per-round failure composition under iterative refinement. Error-type fractions among still-unsolved executions at rounds1, 3, and 5, together with the cumulative solve rate refine@k, across five representative model configurations. ModelRoundCEPEIDLEWAQLETLEMLERErefine@k GPT-5.2 r10.0400.1390.0200.6090.1130.0600.0070.0130.531 r30.0160.1070.0080.5900.1150.0490.0000.1150.621 r50.0100.0380.0100.6250.1350.0480.0000.1350.677 DeepSeek-V3.2-Thinking r1 0.0870.0970.0150.6550.0920.0390.0000.0150.360 r30.0340.0690.0060.6320.0980.0340.0060.1210.460 r50.0240.0300.0060.6460.0980.0300.0060.1590.491 DeepSeek-V3.2 r10.0540.1930.0320.5640.1000.0460.0000.0110.130 r30.0200.1410.0230.5740.1020.0390.0000.1020.205 r50.0080.1240.0120.5780.1000.0280.0040.1450.227 Qwen3-14B r1 0.0590.1470.0490.6570.0620.0160.0000.0100.050 r30.0300.1070.0330.6730.0630.0130.0030.0770.068 r5 0.0230.0900.0170.6720.0640.0100.0000.1240.071 Qwen3-14B-NonThinking r10.1440.1720.1100.5020.0280.0280.0000.0160.009 r30.0630.1420.0690.5710.0320.0250.0000.0980.016 r50.0250.1110.0410.5890.0320.0190.0030.1800.019 Table 11 further shows that refinement can reduce execution-level failures, especially compilation and protocol errors. For GPT-5.2, for example, CE decreases from0.040at round1to0.010at round5, and PE decreases from0.139to0.038. At the same time, WA remains the dominant residual failure type after refinement for all five configurations. This pattern suggests that feedback can repair some execution-level issues, but many remaining failures still require solving the underlying 16 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information 2019202020212022202320242025 Cut year (post set: year >= cut) 0 10 20 30 40 50 60 70 80 Mean pass@1 (%) Time split (post set): mean pass@1 vs cut year Qwen3-32B Qwen3-30B-A3B-Thinking Claude-Opus-4.5 DeepSeek-V3.2-Thinking Gemini-3-Pro-Preview GPT-5.2 Figure 3. Temporal contamination diagnostic via time splits. For each cut yearY, we compute the meanpass@1 over problems released in years≥ Y (the post set). partially observed interactive reasoning problem. C.3. Temporal Contamination Diagnostic via Time Splits We use a time-split diagnostic on InteractBench to probe potential temporal data contamination for tasks with known release years. For a cut yearY, we define a pre setP <Y (problems released in years< Y) and a post setP ≥Y (problems released in years≥ Y), and report the meanpass@1 onP ≥Y as a function ofY. Intuitively, if memorization or leakage concentrated around a particular time window dominates performance, we may observe a sharp change (a “cliff”) as the cut moves past that window; smoother curves suggest that no single sharp temporal boundary drives the score. Results. Figure 3 shows the post-set meanpass@1 curves for six representative model configurations spanning major providers and open weights: Claude-Opus-4.5, GPT-5.2, Gemini-3-Pro-Preview, DeepSeek-V3.2-Thinking, Qwen3-32B, and Qwen3-30B-A3B-Thinking. Across the sweep, we do not observe a cliff-like discontinuity as the cut year advances; instead, the curves change gradually for all models. C.4. Multi-language and I/O Template Evaluation We evaluate five representative model configurations on InteractBench across C++, Python, Java, and Go under the same packaged local interactors. For each language, we compare the standard zero-shot prompt with a prompt that includes the interactive I/O template in Appendix D.7. To account for language runtime overhead, we relax time limits for Python and Java. Because the main experiments use C++ without templates, this section reports whether the same model ordering appears under other language and template settings. 17 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Table 12. Multi-language and I/O template evaluation. pass@1 on interactive tasks executable across C++/Python/Java/Go. ModelSettingC++PythonJavaGo GPT-5.2 no template 0.5420.5500.5640.550 I/O template 0.5570.5440.5440.540 DeepSeek-V3.2-Thinking no template0.3320.3090.3120.242 I/O template0.3190.2890.2850.285 DeepSeek-V3.2 no template 0.1480.0600.1340.104 I/O template 0.1440.1310.1070.124 Qwen3-14B no template0.0580.0600.0670.027 I/O template 0.0600.0670.0770.013 Qwen3-14B-NonThinking no template 0.0120.0070.0100.000 I/O template 0.0170.0130.0130.003 Table 12 reports pass@1 for each configuration. The template effect is mixed rather than uniformly beneficial. For DeepSeek-V3.2 on Python, adding the template improves pass@1 from 0.060 to 0.131; for DeepSeek-V3.2-Thinking on Python, it slightly decreasespass@1 from 0.309 to 0.289. This pattern is consistent with the failure labels. Python without templates more often leads to idle or deadlocked runs, so the template mainly reduces low-level I/O failures rather than resolving the algorithmic reasoning errors. Java and Go introduce more compile failures, especially for weaker models. After these language-specific failures are separated, wrong answers remain the main source of unsuccessful runs, consistent with the failure analysis in Section 4. C++ shows the smallest difference between template and no-template settings, supporting our use of C++ without templates as the default evaluation configuration. D. Diagnostics and Artifacts D.1. Qualitative Evidence for Key Findings Section 4 reports aggregate pass rates and failure compositions for interactive problem solving. To make two of the most salient phenomena concrete, we manually inspected a small set of representative tasks and model-generated submissions. The examples below are illustrative rather than exhaustive: they highlight (i) how resampling can “rescue” runs by exploring alternative approaches and avoiding small robustness bugs, and (i) how some models exhibit disproportionately frequent interaction-level breakdowns that prevent meaningful progress (notably IDLE). Resampling can recover from protocol/implementation brittleness Observation. In Experiment #1,pass@1 can lag noticeably behindpass@5 even on easier tasks, suggesting that some failures are caused not by missing high-level ideas, but by small protocol or robustness issues that resampling may “rescue.” Source task. cf2066A.Model. DeepSeek-V3.2. Harness evidence (failing sample). The submission fails immediately with a format error before issuing any valid query: verdict=PE wrong output format Invalid query indices queries=0 query_limit=2 Key code difference. In the failing sample, the solver attempts to query an index pair that may remain uninitialized: int p = -1, q = -1; ... cout << "? " << p << " " << q << endl; A second sample for the same model follows the same high-level strategy but adds a simple fallback that guarantees valid indices before querying: if (a == -1) a = 1; b = 2; // fallback to a valid pair 18 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Takeaway. This pair illustrates a common “recoverable” pattern in interactive evaluation: a plausible strategy is present, but brittle edge handling causes a protocol failure. Resampling can help by producing a nearby implementation that respects the protocol. GPT-OSS-120B: frequent IDLE breakdowns Observation. In Experiment #3, GPT-OSS-120B shows an unusually high share of IDLE failures. Many of these stalls occur without issuing any query, consistent with raw generations in which the model misinterprets interactive tasks as ordinary offline input-output settings and blocks waiting for hidden inputs. This contrasts with frontier models that more often execute the protocol correctly and fail as WA. Case 1 (queries=0 IDLE from offline misinterpretation). On cf2181C, the raw generation explicitly assumes an offline version and reads an edge list, rather than issuing interactive queries. The harness reports: verdict=IDLE queries=0 query_limit=8 The saved .cpp begins by reading m edges: int m; cin >> m; for (int i = 1; i <= m; ++i) int u, v; cin >> u >> v; ... Under the true interactive protocol, no such edge list is ever provided, so the solver blocks immediately and is classified as IDLE. Case 2 (late IDLE from brittle sentinel handling). On cf1979F, a submission issues thousands of queries but stalls near the end of interaction. The harness reports: verdict=IDLE queries=2868 One plausible mechanism suggested by the saved code is brittle error handling around the termination sentinel. In this protocol, an invalid query or budget violation may cause the judge to return a single-1token, after which the solver should terminate immediately. However, the submission unconditionally reads two integers before checking whether the first one is -1: cout << "? " << d << endl; cout.flush(); int v, u; cin >> v >> u; if (v == -1) return 0; If the judge emits only-1(with no second integer), the second read can block, producing a “late” IDLE after many otherwise protocol-shaped queries. Takeaway. These examples illustrate why GPT-OSS-120B’s failure composition can differ qualitatively from models that reliably emit well-formed submissions and reach the WA stage: in interactive evaluation, a model may fail to enter a valid query loop at all (e.g., by assuming an ordinary offline setting), and output discipline and I/O protocol robustness can become the dominant bottleneck. D.2. Error Case Studies We provide one representative example for each error type in our interaction-aware failure taxonomy. Each example summarizes the interaction protocol, includes the relevant excerpt from the model-generated submission, and reports the key harness evidence together with a short diagnosis of the underlying failure mode. Legend. We use the following abbreviations:CE(compile error),PE(protocol/format error),IDLE(deadlock/idle timeout),QLE(query-budget overrun),RE(runtime error; often stream desynchronization),TLE(time limit exceeded),MLE 19 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information (memory limit exceeded), and WA (wrong answer under a protocol-correct run). CE: compile error caused by flushing the input stream Source task. 2025 ICPC Central Europe Regional Contest. Model. Qwen3-32B. Problem and protocol (paraphrased). • Hidden vector A of length N (fixed before interaction; N ≤ 60). • Query: print ? b1 b2 ... bN, then read an integer reply. • Final answer: print ! a1 a2 ... aN and terminate. • Flushing is required after each output line. Full statement (source). Interactive note. This task is interactive. After printing each line, flush the output buffer. For example:cout << flushorcout.flush()inC++,System.out.flush()in Java, and sys.stdout.flush() in Python. Problem. There is a hidden sequenceA = (a 1 ,...,a N )of lengthN, where eacha i is an integer in the range 0≤ a i < 2 N . Your task is to recover A using queries. In each query you provide a sequence B = (b 1 ,...,b N ) with 0≤ b i < 2 N . The reply is computed as follows: • Define C = (c 1 ,...,c N ) where c i = a i ⊕ b i and⊕ denotes bitwise XOR. • LetSbe the set of values obtainable by XOR-ing an arbitrary subset ofc 1 ,...,c N , including the empty subset whose XOR is 0. • The answer to the query is|S|. Example. IfA = (1, 4, 3)andB = (0, 4, 7)thenC = (1⊕ 0, 4⊕ 4, 3⊕ 7) = (1, 0, 4)andS =0, 1, 4, 5, hence the answer is 4. XOR reminder. The bitwise XOR of two numbersxandyhas bitiset iff exactly one ofxandyhas bitiset. For example, 5⊕ 3 = 6 since 101 2 ⊕ 011 2 = 110 2 . In C++ and Python, XOR is written as ˆ. Interaction protocol. The sequence A is fixed at the start and does not depend on the queries. • Read an integer N with 1≤ N ≤ 60. • To ask a query, print ? b1 b2 ... bN, flush, and read one integer reply. • When ready, print ! a1 a2 ... aN, flush, and terminate immediately. Local testing. The original statement provides a local grader for testing; its behavior may differ from the official interactor. Incorrect code excerpt (model-generated C++). int N; cin >> N; cin.flush(); // BUG: std::istream has no flush() Harness evidence. The compiler rejects the submission: error: ’std::istream’ has no member named ’flush’ cin.flush(); ˆ ̃ ̃ ̃ ̃ Why it fails. Interactive statements emphasize flushing after printing queries, but the submission attempts to flush the input stream. InteractBench classifies this as CE to separate compilation failures from interaction-logic errors. PE: protocol violation by emitting a query during the answer phase Source task. 2025 ICPC Asia Yokohama Regional Contest. Model. Gemini-2.5-Pro. Problem and protocol (paraphrased). • Input provides n stars and their squared distances to the origin. • Query up to a budget (≤ 300): print measure i j and read the squared distance between stars i and j. • After finishing measurements: print a lineanswer, then print the upper-triangular matrix of all pairwise squared distances. • Once answer starts, no further measure commands are allowed. Full statement (source). Problem. The positions ofnstars are lattice points inZ 3 under a Cartesian coordinate system whose origin is the Earth. You are given the squared distance from the origin to each star, but the directions are 20 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information unknown. In one measurement you specify two distinct stars, and the instrument returns their squared distance. Your task is to determine the squared distances between all pairs of stars within a limited number of measurements. Interaction. • Read an integer n (2≤ n≤ 100). The stars are numbered 1,...,n. • Readnintegers; thei-th is the squared distance from the origin to stari. It is guaranteed that all star coordinates are integers in [−4000, 4000], no two stars coincide, and no star is at the origin. • You may perform up to300measurements. To measure starsi ̸= j, outputmeasure i j, flush, and read one integer reply d i,j , the squared distance between stars i and j. • After determining all pairwise distances, output a line containing onlyanswer, followed byn− 1lines that list the upper-triangular distances: – Line 1: d 1,2 d 1,3 ... d 1,n – Line 2: d 2,3 ... d 2,n – . . . – Line n− 1: d n−1,n • After printing the answer block, terminate without extra output. Notes. If your output format is invalid, or if you exceed300measurements, the verdict is Wrong Answer. The coordinates of the stars are fixed before the interaction starts and do not change during the interaction. The original statement provides sample interactions, figures, and a local testing tool. Incorrect code excerpt (model-generated C++). long long ask_dist_sq(int i, int j) cout << "measure " <> d_sq; return d_sq; if (N == 2) cout << "answer" << endl; cout << ask_dist_sq(1, 2) << endl; // BUG: emits "measure" during answer phase return 0; Harness evidence. The interactor reports a format error while parsing the numeric answer payload: wrong output format Invalid number in answer: ’measure’. Why it fails. The submission mixes protocol phases. It begins the final-answer block, but then issues a new query. This is diagnosed asPEbecause the output stream no longer matches the grammar expected by the interactor at that stage. IDLE: deadlock from missing flush after printing the final answer Source task. 2025 ICPC Asia Pacific Championship. Model. GPT-5.2. Problem and protocol (paraphrased). • Multiple test cases. • Query types:type i(read a string reply) andmulti j(read an integer reply), with a strict per-test query budget (≤ 10 queries). • Final answer per test: print answer k and continue to the next test. • Flushing is required for both queries and answers. Full statement (source). Problem. There arenflowers arranged in a line and numbered1,...,nfrom left to right. Each flower is either a lily or a rose. For an integerjwith0≤ j ≤ n, letl j denote the number of lilies among the leftmost j flowers, and let r j denote the number of roses among the rightmost n− j flowers. Initially, onlynis provided; the flower types are hidden. Your task is to find an integerk(0 ≤ k ≤ n) such that l k = r k using a limited number of queries. At least one such k is guaranteed to exist. Queries. In one query you may perform one of the following: 21 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information • type i (1≤ i≤ n): the interactor returns lily or rose, the type of flower i. • multi j (0≤ j ≤ n): the interactor returns the integer value l j · r j . Interaction. • Read an integer t (1≤ t≤ 100), the number of test cases. • For each test case, read n (1≤ n≤ 100), then issue queries. • You may ask at most 10 queries per test case (Type and Multiply queries combined). • When you have found a validk, outputanswer k(this does not count as a query) and continue to the next test case. • After processing all test cases, terminate. Notes. The flower types are fixed before the interaction starts (non-adversarial / non-adaptive). Flush the output buffer after each query and each answer line. The original statement provides sample interactions, figures, and a local testing tool. Incorrect code excerpt (model-generated C++). ios::sync_with_stdio(false); cin.tie(nullptr); ... cout << "answer " << finalCandidates[0] << " "; // BUG: no flush // proceeds to read the next test case Harness evidence. The run stalls until the idle timeout with no CPU activity: queries=2 cpu_ms=0 Why it fails. Without flushing, the interactor never receives theanswerline and therefore does not advance to the next test case. Meanwhile, the solver blocks on reading the next input, creating a classic interactive deadlock. QLE: query-budget overrun by exhaustive enumeration Source task. 2025 Polish Collegiate Programming Contest, AMPPZ. Model. DeepSeek-V3.2. Problem and protocol (paraphrased). • Two-phase interaction: a large “training” phase followed by an “answering” phase. • Training query: print ? ver s and read an integer result, capped by a strict budget (≤ 300,000 queries). • Switch phase by printing !, then answer q offline queries (q ≤ 10,000) by printing integers. Full statement (source). Problem. The game “Plumber Mariusz” consists ofn + 1levels (numbered0ton), each withkversions (numbered0tok− 1). Each version of every level (except the last) ends with two pipes, left and right, and Mariusz must choose one. Each pipe contains some number of coins and leads to a version of the next level. Exactly two pipes lead into each version of each level (except level0); equivalently, each node in levels> 0has in-degree 2. A playthrough starts by choosing a start versionverat level0and then choosing a length-nstringsoverLandP (left/right). The result is the total number of coins collected along thenchosen pipes. The path itself is not revealed. Coins reset after each playthrough, so playthroughs are independent. You may perform up to300,000test playthroughs and observe their results. After that, you will receive up to 10,000queries about playthroughs; your task is to output their results. The test is fixed before the interaction begins (non-adaptive). Interaction protocol. • Read integers n and k (1≤ n,k ≤ 20). •Training phase (up to300,000times): output? ver swhere0≤ ver < kandsis a length-nstring overL/P; flush; read one integer reply x (the playthrough result). • End the training phase by outputting a single ! and flushing. • Read an integer q (1≤ q ≤ 10,000). • For each of the q queries, read ver and s (same format), output the integer result, and flush. Notes. Flush after each line and do not open files. The original statement provides examples, sample solutions, and local testing artifacts. 22 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Incorrect code excerpt (model-generated C++). void brute_force_all() for (int ver = 0; ver < k; ver++) for (int mask = 0; mask < (1 << n); mask++) string s = ...; // all 2ˆn strings over L,P cout << "? " << ver << " " << s << endl; int res; cin >> res; memo[ver, s] = res; Harness evidence. The run exceeds the training budget: queries=1048576 query_limit=300000 Why it fails. Enumerating all action strings scales ask· 2 n and quickly breaks the interaction budget. InteractBench surfaces this as QLE even when the per-query protocol is correct. RE: stream desynchronization from skipping judge feedback Source task. 2025 ICPC Chengdu Regional Contest. Model. Gemini-2.5-Flash. Problem and protocol (paraphrased). • Multiple test cases. • Query: print ? x y and read a numeric reply. •Final answer: print! d; the interactor then prints a one-word verdict (CorrectorWrong) before the next test begins. • The per-test query budget is small (≤ 5). Full statement (source). Problem. Panda is trapped by a regular polygon withNvertices numbered1,...,N counterclockwise. The polygon’s centroid must be destroyed. Panda does not know the polygon’s size or his own position. He may perform at most5measurements. In a measurement, Panda chooses two verticesxandyand receives the area of the triangle formed by his position and these vertices. Your task is to output the distance from Panda’s position to the centroid. Interaction protocol. • Read an integer T (1≤ T ≤ 10 4 ), the number of test cases. • For each test case, read an integerN(4 ≤ N ≤ 200). Hidden parameters remain fixed during the test case. The polygon’s circumradius is in [1, 100], and the answer does not exceed 10,000. • For a measurement, output? x ywith1≤ x,y ≤ Nandx̸= y, flush, and read a non-negative real number (the triangle area) printed with at least10valid digits. If you make more than5queries, the interactor outputs-1and terminates. •After at most5measurements, output! dwheredis a non-negative real number. Letpbe your output andqbe the judge answer. The answer is accepted if min p− q q , |p− q| ≤ 10 −4 . •After your answer, the interactor printsCorrectand continues to the next test case if accepted; otherwise it prints Wrong and terminates. Notes. Each output must end with a newline and be flushed (e.g.,cout.flush()inC++,System.out.flush() in Java). The original statement provides an example interaction. Incorrect code excerpt (model-generated C++). // prints the final answer but never reads the judge’s "Correct/Wrong" line cout << "! " << fixed << setprecision(10) << d << endl; return; // returns to the main loop 23 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information int T; cin >> T; while (T--) solve(); Harness evidence. The interactor terminates abnormally (broken pipe): exit_codes[solution]=0 exit_codes[interactor]=-13 Why it fails. Skipping theCorrect/Wrongline shifts the input stream. The next test case reads the wrong token, the solver exits early, and the interactor crashes when writing to a closed pipe, which is recorded as RE. TLE: unbounded pre-processing before issuing any query Source task. 2025 ICPC Asia Bangkok Regional Contest. Model. GPT-OSS-120B. Problem and protocol (paraphrased). • Hidden permutation P over N elements. • Query: print a permutation ? q1 q2 ... qN, then read N bits describing the post-meeting “sus” statuses. • Total number of meetings is capped by a strict budget (≤ 400). Full statement (source). Problem. There areNcrewmates. Each crewmate secretly suspects exactly one crewmate, and no crewmate is suspected by more than one person. Formally, there is a hidden permutationPof lengthNwhere P [i] is the crewmate suspected by crewmate i. Your goal is to determine P using at most 400 emergency meetings. In one emergency meeting you choose the order in which crewmates speak. When a crewmate speaks, they accuse the crewmate they suspect. If the speaker has not yet been marked sus, their accusation is processed and the accused crewmate becomes sus. If the speaker is already sus, their accusation is ignored. After the meeting, the ship reports the sus status of all crewmates as a binary array. Each meeting is independent: all crewmates start as not sus. Interaction protocol. • Read an integer N (2≤ N ≤ 100). • A query has the form? q1 q2 ... qN, where(q 1 ,...,q N )is a permutation of1,...,Nindicating the speaking order. •After each query, flush and read a line containingNintegerss 1 ,...,s N . Heres i = 1means crewmateiis sus after the meeting, and s i = 0 means not sus. • You may perform at most 400 queries. • When you have determined P , output ! p1 p2 ... pN and terminate immediately. Notes. The interactor is non-adaptive: the hidden permutation does not depend on your queries. Remember to flush after each query and after the final answer; otherwise you may receive an idleness verdict. The original statement provides example interactions and additional notes. Incorrect code excerpt (model-generated C++). mt19937 rng(...); while (true) // generate random permutations until a rare property holds for (int t = 0; t < K; ++t) vector<int> a(N); iota(a.begin(), a.end(), 1); shuffle(a.begin(), a.end(), rng); ... if (ok) break; // BUG: no iteration cap, no fallback // TLE happens before the first interactive query. Harness evidence. The solver times out without issuing any query: queries=0 cpu_ms=19990 Why it fails. The submission spends the entire time budget in uncontrolled randomized pre-processing. In interactive tasks, computation before the first query counts toward the same time limit, so such unbounded setup loops lead to 24 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information TLE. MLE: stack growth from a missing empty-partition check Source task. 2025 ICPC Asia Pacific Championship. Model. DeepSeek-R1-Distill-Llama-70B. Problem and protocol (paraphrased). • The judge holds a hidden expression over n occurrences of x using a binary operator (a− b) on bits. • Query: print query S where S is a binary string of length n, then read a reply bit. • The query budget is 500. • Final answer: print answer T and terminate. Full statement (source). Problem. The judge holds a hidden expression overnterminal symbolsxusing the binary operator(a− b)on bits:(a− b) = 1ifa = 1andb = 0, and(a− b) = 0otherwise. The expression is a binary tree whose leaves are occurrences of x. At the start you are given n; the expression is hidden. Queries. In one query, you provide a binary stringSof lengthn. The judge substitutes thei-th occurrence ofx(from the left) with S i and evaluates the expression, returning 0 or 1. Interaction protocol. • Read n (3≤ n≤ 200). • To ask a query, print query S, flush, and read one integer reply (0 or 1). • You may ask at most 500 queries. • When you have identified the expression, print answer T and terminate. Notes. The judge is non-adaptive: the hidden expression is fixed before the interaction starts. Flush the output buffer after each output line. Incorrect code excerpt (model-generated C++). // Partition x_list based on one-hot query replies. vector<int> left, right; for (int i = 0; i < (int)x_list.size(); ++i) if (results[i]) left.push_back(x_list[i]); else right.push_back(x_list[i]); // BUG: does not handle left.empty() or right.empty(). string left_expr = build_expression(left, n); string right_expr = build_expression(right, n); Harness evidence. The run exceeds the memory cap and crashes: queries=399 query_limit=500 memory_limit_mb=256 max_rss_solution_kb=265472 exit_codes[solution]=-11 stderr_interactor_tail=wrong answer Unexpected EOF (solution terminated without providing answer) Why it fails. When the partition produces an emptyleftorrightset, the recursion has no base case and keeps expanding, causing stack growth and eventually a crash. We classify such runs asMLEwhen peak RSS exceeds the task’s memory limit; the interactor may subsequently reportWAdue to premature termination (e.g.,Unexpected EOF). WA: algorithmic error despite protocol compliance Source task. 2025 ICPC Rocky Mountain Regional Contest. Model. GPT-4.1. Problem and protocol (paraphrased). • There are N applicants with a hidden strict total order. • Query: print ? a b and read the winner, capped by a linear budget (≤ 12N + 1000). • Output: print ! followed by a set of K applicants that must equal the true top-K. Full statement (source). Interactive note. This is an interactive problem: the input depends on your program’s output. 25 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Problem. There areNapplicants, each with a distinct skill level (no ties). You may compare two applicants by holding a match; the stronger one wins. Skill levels are transitive: if a defeats b and b defeats c, then a defeats c. Given K, your task is to determine the K strongest applicants using a limited number of matches. Interaction. • Read integers N and K (1≤ N ≤ 15000, 1≤ K ≤ min(N, 30)). • You may make at most 12N + 1000 queries. • A query has the form? a bwith1≤ a,b≤ Nanda̸= b. Flush and read the reply, eitheraorb, indicating the winner. • Once you have identified the top-K applicants, output ! x1 x2 ... xK (any order) and terminate. Non-adaptivity. The interactor is not adaptive: the skill levels are fixed at the start. Uniqueness requirement. In addition to being correct, your queries must establish that your output is the unique possible set of top-K applicants consistent with all observed comparisons; otherwise the verdict is Wrong Answer. Invalid input. If the interactor receives invalid input, it outputs-1and terminates; your program should exit cleanly. Local testing. The original statement provides a command-line tool for local testing. Incorrect code excerpt (model-generated C++). // If K > selected.size(), fill up with random (this should never happen). while ((int)selected.size() < K) for (int i = 1; i <= N && (int)selected.size() < K; ++i) selected.insert(i); cout << "! "; for (int x : selected) cout << x << " "; cout << endl; Harness evidence. The interactor rejects the final set: wrong answer Incorrect set of applicants (not the actual top K) Why it fails. The submission is protocol-correct and stays within the query budget, but its selection logic does not guarantee that the reported set equals the true top-K. This is a reasoning-level failure: interactive problems often require sufficient comparison evidence to uniquely certify the answer, not merely to “guess” a plausible set. D.3. End-to-End Task Showcase: cf2036G To make the offline evaluation pipeline concrete, we provide one end-to-end interactive task example usingcf2036G (Codeforces; non-adaptive). This showcase illustrates how a task is packaged (protocol, generator, local interactor, and offline hidden cases) and evaluated with recorded artifacts. Protocol excerpt (interaction-relevant clauses). We summarize the interaction rules from the original statement and omit narrative content: • Hidden multiset. Exactly three distinct valuesa,b,cappear once; every other value in[1,n]appears twice. The goal is to output (a,b,c) in any order. • Query. Output xor l r with 1≤ l≤ r ≤ n, flush, and read one integer reply (the XOR of all values in [l,r]). • Budget. At most 150 queries per test case; further queries return -1; the solver should terminate upon receiving -1. • Answer. Output ans a b c (any order), then proceed to the next test case. • Non-adaptive. The hidden input is fixed in advance and does not depend on the solver’s queries. Test-case generator (full listing). cf2036G generator (gencases.cpp) #include "testlib.h" #include <bits/stdc++.h> using namespace std; 26 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information static long long highestPow2LE(long long n) long long p = 1; while ((p << 1) > 0 && (p << 1) <= n) p <<= 1; return p; static long long pickNotIn(long long n, const vector<long long>& used) while (true) long long x = rnd.next(1L, n); bool ok = true; for (long long u : used) if (x == u) ok = false; break; if (ok) return x; static void ensureDistinctInRange(long long n, long long &a, long long &b, long long & c) auto inRange = [&](long long x) return 1L <= x && x <= n; ; if (!inRange(a)) a = rnd.next(1L, n); if (!inRange(b)) b = rnd.next(1L, n); if (!inRange(c)) c = rnd.next(1L, n); if (a == b) b = pickNotIn(n, a); if (a == c) c = pickNotIn(n, a, b); if (b == c) c = pickNotIn(n, a, b); static void genTriple(long long n, long long seed, long long &a, long long &b, long long &c) // Hand-crafted corner cases for small seeds. if (seed == 1) a = 1; b = 2; c = 3; // minimal n=3, and aˆbˆc = 0 return; if (seed == 2) a = 1; b = n - 1; c = n; // boundary-heavy, (n-1)ˆn = 1 => total xor = 0 return; if (seed == 3) long long x = (1L << 58); a = 1; b = x; c = x + 1; // xˆ(x+1)=1 => total xor = 0, mixes low/high bits if (c > n) // Fallback: still keep xor=0 if possible. a = n; b = n - 1; c = 1; return; // Random stress + adversarial structured patterns for other seeds. int type = rnd.next(0, 99); if (type < 50) // Adversarial: near power-of-two boundary with total xor = 0. long long k = highestPow2LE(n); a = k; b = k - 1; c = 1; 27 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information if (k <= 2) // For very small n, just pick distinct. a = 1; b = 2; c = 3; else if (b == c) c = 2; ensureDistinctInRange(n, a, b, c); return; if (type < 75) // Adversarial: force aˆbˆc = 0 if feasible. bool ok = false; for (int it = 0; it < 200; it++) long long x = rnd.next(1L, n); long long y = rnd.next(1L, n); if (x == y) continue; long long z = x ˆ y; if (1L <= z && z <= n && z != x && z != y) a = x; b = y; c = z; ok = true; break; if (ok) return; // Fallback to spread. // Spread across the range to maximize branching/partition uncertainty. long long third1 = max(1L, n / 3); long long third2 = max(third1 + 1, (2 * n) / 3); a = rnd.next(1L, third1); b = rnd.next(min(third1 + 1, n), min(third2, n)); c = rnd.next(min(third2 + 1, n), n); ensureDistinctInRange(n, a, b, c); int main(int argc, char ** argv) registerGen(argc, argv, 1); string mode = opt<string>("mode", "non"); long long fixedN = opt<long long>("n", 0L); long long nmin = opt<long long>("nmin", 3L); long long nmax = opt<long long>("nmax", 1000000000000000000L); ensuref(nmin >= 3, "nmin must be >= 3"); ensuref(nmax >= nmin, "nmax must be >= nmin"); long long seed = 1; if (argc > 1) seed = atoll(argv[1]); long long n; if (fixedN != 0) n = fixedN; else if (seed == 1) n = 3; else if (seed == 2) n = 1000000000000000000L; else if (seed == 3) n = 999999999999999937L; else 28 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information long long span = nmax - nmin; long long delta = 0; if (span > 0) delta = rnd.next(0L, min(1000000L, span)); n = nmax - delta; if (n < nmin) n = nmin; if (n > nmax) n = nmax; ensuref(3L <= n && n <= 1000000000000000000L, "n out of bounds"); if (mode == "non") long long a = 1, b = 2, c = 3; genTriple(n, seed, a, b, c); ensureDistinctInRange(n, a, b, c); ensuref(a != b && a != c && b != c, "a,b,c must be distinct"); cout << 1 << " "; cout << n << " "; cout << a << " " << b << " " << c << " "; return 0; if (mode == "adp") // Adaptive schema: only provide n; the interactor picks a,b,c adversarially within [1..n]. cout << 1 << " "; cout << n << " "; return 0; quitf(_fail, "Unknown --mode (expected non/adp): %s", mode.c_str()); Local interactor (full listing). cf2036G non-adaptive interactor (nonadaptive.cpp) #include "testlib.h" #include <bits/stdc++.h> using namespace std; static long long queries = 0; static long long query_limit = 0; static void log_metrics() try tout << "queries=" << queries << " "; tout << "query_limit=" << query_limit << " "; tout.flush(); catch (...) // best-effort static void finish(TResult verdict, const string& msg) log_metrics(); quitf(verdict, "%s", msg.c_str()); static bool readLongLongFromCin(long long &out) // Robust-ish: rely on operator>>; if it fails, caller handles. cin >> out; 29 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information return (bool)cin; int main(int argc, char ** argv) registerInteraction(argc, argv); atexit(log_metrics); long long t = 0; try t = inf.readLong(); catch (...) // In practice testlib terminates the process on read errors, but keep a fallback. query_limit = 0; log_metrics(); return 0; if (t < 1 || t > 300) query_limit = 0; log_metrics(); quitf(_fail, "Invalid t in hidden input: %lld", t); struct CaseData long long n, a, b, c; ; vector<CaseData> cases; cases.reserve((size_t)t); for (long long i = 0; i < t; i++) long long n = inf.readLong(); long long a = inf.readLong(); long long b = inf.readLong(); long long c = inf.readLong(); cases.push_back(n, a, b, c); query_limit = 150L * t; log_metrics(); // MUST happen before any blocking reads from contestant. const long long hard_cap = max(200000L, 10L * query_limit); cout << t << " "; cout.flush(); for (long long tc = 0; tc < t; tc++) const long long n = cases[tc].n; const long long A = cases[tc].a; const long long B = cases[tc].b; const long long C = cases[tc].c; if (!(3L <= n && n <= 1000000000000000000L)) finish(_fail, "Invalid n in hidden input"); auto inRange = [&](long long x) return 1L <= x && x <= n; ; if (!inRange(A) || !inRange(B) || !inRange(C) || A == B || A == C || B == C) finish(_fail, "Invalid (a,b,c) in hidden input"); cout << n << " "; cout.flush(); long long case_queries = 0; while (true) if (queries > hard_cap) finish(_pe, "Hard cap exceeded"); 30 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information string cmd; if (!(cin >> cmd)) finish(_pe, "Unexpected EOF from contestant"); if (cmd == "xor") long long l, r; if (!readLongLongFromCin(l) || !readLongLongFromCin(r)) finish(_pe, "Malformed xor query (expected two integers)"); if (!(1L <= l && l <= r && r <= n)) finish(_pe, "Invalid xor query bounds"); queries++; case_queries++; if (case_queries > 150) cout << -1 << " "; cout.flush(); continue; long long res = 0; if (l <= A && A <= r) res ˆ= A; if (l <= B && B <= r) res ˆ= B; if (l <= C && C <= r) res ˆ= C; cout << res << " "; cout.flush(); else if (cmd == "ans") long long x, y, z; if (!readLongLongFromCin(x) || !readLongLongFromCin(y) || ! readLongLongFromCin(z)) finish(_pe, "Malformed ans (expected three integers)"); if (!inRange(x) || !inRange(y) || !inRange(z) || x == y || x == z || y == z) finish(_wa, "Invalid answer values"); array<long long, 3> got = x, y, z; array<long long, 3> want = A, B, C; sort(got.begin(), got.end()); sort(want.begin(), want.end()); if (got != want) finish(_wa, "Wrong answer"); if (tc + 1 == t) finish(_ok, "OK"); break; // proceed to next test else finish(_pe, "Unknown command (expected ’xor’ or ’ans’)"); finish(_pe, "Contestant did not finish all testcases"); return 0; Hidden offline test case (example).The offline suite provides hidden inputs consumed by the local interactor (not by the solver). 31 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Hidden case example (cases/003.in) 1 999999999999999937 1 288230376151711744 288230376151711745 Representative passing solver (model-generated). Passing solver (C++; GPT-5.2) #include <bits/stdc++.h> using namespace std; using ull = unsigned long long; using l = long long; static ull N; static ull ask_xor(ull l, ull r) cout << "xor " << l << " " << r << endl; // endl flushes l resp; if (!(cin >> resp)) exit(0); if (resp == -1) exit(0); return (ull)resp; static ull prefix_xor(ull x) if (x == 0) return 0; return ask_xor(1, x); int main() ios::sync_with_stdio(false); cin.tie(nullptr); int t; cin >> t; while (t--) ull n; cin >> n; N = n; if (n == 3) cout << "ans 1 2 3" << endl; continue; ull total = ask_xor(1, n); if (total != 0) // Find smallest stolen number s1: first x where prefix_xor(x) != 0. ull lo = 0, hi = n; while (lo + 1 < hi) ull mid = lo + (hi - lo) / 2; ull v = prefix_xor(mid); if (v == 0) lo = mid; else hi = mid; ull s1 = hi; // Find largest stolen number s3: first x where prefix_xor(x) == total. 32 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information lo = s1; hi = n; while (lo + 1 < hi) ull mid = lo + (hi - lo) / 2; ull v = prefix_xor(mid); if (v == total) hi = mid; else lo = mid; ull s3 = hi; // Get prefix before s3 to recover s2. ull pref_before = prefix_xor(s3 - 1); // = s1 ˆ s2 ull s2 = s1 ˆ pref_before; cout << "ans " << s1 << " " << s2 << " " << s3 << endl; else // total == 0: find a stolen number by scanning MSB blocks [2ˆk, 2ˆk +1-1]. // In this case, not all three can share the same MSB, so some block has exactly one. ull known = 0; for (int k = 0; k <= 60; k++) ull l = 1ULL << k; if (l > n) break; ull r = min(n, (1ULL << (k + 1)) - 1ULL); ull v = ask_xor(l, r); if (v != 0 && (v & (1ULL << k)) != 0) known = v; // singleton in this MSB block -> v is that number break; if (known == 0) exit(0); // should be impossible; safety // Now the remaining two numbers y,z satisfy y ˆ z = known (since total=0) . // Remove ’known’ from any prefix query by XORing it back if it’s included . auto pref2 = [&](ull x) -> ull ull v = prefix_xor(x); if (known <= x) v ˆ= known; return v; // xor of remaining two within [1..x] ; // Find the smaller of the remaining two: first x where pref2(x) != 0. ull lo = 0, hi = n; while (lo + 1 < hi) ull mid = lo + (hi - lo) / 2; ull v = pref2(mid); if (v == 0) lo = mid; else hi = mid; ull y = hi; ull z = known ˆ y; cout << "ans " << known << " " << y << " " << z << endl; return 0; Failure counterpart (model-generated). 33 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Failure solver (C++; GPT-4.1) #include <iostream> #include <vector> #include <algorithm> using namespace std; // Helper: XOR from 1 to x long long xor1toN(long long n) long long res; if (n % 4 == 0) res = n; else if (n % 4 == 1) res = 1; else if (n % 4 == 2) res = n + 1; else res = 0; return res; int main() ios::sync_with_stdio(false); cin.tie(nullptr); int t; cin >> t; while (t--) long long n; cin >> n; // Query: "xor 1 n" to get the overall xor cout << "xor 1 " << n << endl; cout.flush(); long long S; cin >> S; if (S == -1) return 0; // Compute double xor of 1..n long long doubleXor = xor1toN(n) ˆ xor1toN(n); // always 0 // The result S = doubleXor ˆ (a ˆ b ˆ c) = 0 ˆ (a ˆ b ˆ c) // So S = a ˆ b ˆ c long long abcXor = S; // Now, we need to find a, b, c such that a ˆ b ˆ c = abcXor, and a, b, c are distinct in [1,n] // Let’s try to get more info. // Query: "xor 1 1", "xor 2 2", "xor 3 3" // To find which ones have only 1 copy (stolen ones will return their number, others will return 0) vector<long long> candidates; int cnt_queries = 1; for (long long i = 1; i <= min(50L, n); ++i) cout << "xor " <> ans; if (ans == -1) return 0; if (ans != 0) candidates.push_back(i); if (candidates.size() == 3) break; 34 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information if (candidates.size() == 3) cout << "ans " << candidates[0] << " " << candidates[1] << " " << candidates[2] << endl; cout.flush(); continue; // If n is very large, it’s likely that the missing ones are not in the first 50. // Let’s query last 50 as well for (long long i = max(1L, n - 49); i <= n; ++i) if (cnt_queries >= 150) break; cout << "xor " <> ans; if (ans == -1) return 0; if (ans != 0) // Don’t add duplicates if (find(candidates.begin(), candidates.end(), i) == candidates.end()) candidates.push_back(i); if (candidates.size() == 3) break; if (candidates.size() == 3) cout << "ans " << candidates[0] << " " << candidates[1] << " " << candidates[2] << endl; cout.flush(); continue; // If still not found, need to search more! // Try blocks: step by 1000000, query "xor i i" long long step = max(n / 100, 1L); for (long long i = 1; i <= n && cnt_queries < 150; i += step) cout << "xor " <> ans; if (ans == -1) return 0; if (ans != 0) if (find(candidates.begin(), candidates.end(), i) == candidates.end()) candidates.push_back(i); if (candidates.size() == 3) break; // If not found, try blocks in the middle if (candidates.size() < 3 && n > 100) long long mid = n / 2; for (long long off = -25; off <= 25 && cnt_queries < 150; ++off) long long idx = mid + off; if (idx < 1 || idx > n) continue; 35 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information cout << "xor " << idx << " " << idx << endl; cout.flush(); cnt_queries++; long long ans; cin >> ans; if (ans == -1) return 0; if (ans != 0) if (find(candidates.begin(), candidates.end(), idx) == candidates. end()) candidates.push_back(idx); if (candidates.size() == 3) break; // If still not found, as a last resort, brute force up to 150 queries for (long long i = 1; i <= n && cnt_queries < 150; ++i) cout << "xor " <> ans; if (ans == -1) return 0; if (ans != 0) if (find(candidates.begin(), candidates.end(), i) == candidates.end()) candidates.push_back(i); if (candidates.size() == 3) break; // Now, output the answer if (candidates.size() == 3) cout << "ans " << candidates[0] << " " << candidates[1] << " " << candidates[2] << endl; cout.flush(); else // If unable to find, output 1 2 3 (possible for very small n) cout << "ans 1 2 3" << endl; cout.flush(); return 0; Failure reason (statement-aligned).The failure solver relies heavily on point queries of the formxor i iand sparse scanning heuristics to locatea,b,c. This probing pattern can fail to locate values that lie far from the probed indices whenn is large. On the bundled hidden case (cases/003.in), two of the three target values are very large (around2 58 ) and lie far from the probed indices, so they are not discovered within the query budget, leading to an incorrect reconstructed triple and a WA verdict. Evaluation summary. We evaluate offline on the bundled 100-case suite (files undercases/) using the same two- process local runner as in the main experiments. In this bundle, the passing solver succeeds on all cases, while the failure solver produces aWAoncases/003.inafter exhausting the per-case query budget. For brevity, we omit raw logs and machine-readable result dumps here. 36 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information D.4. Adaptive Interactor Example: cf2096G To complement the non-adaptive end-to-end showcase in Appendix D.3, we provide an adaptive interactor example usingcf2096G(Codeforces). The goal is not to repeat the full packaging pipeline, but to concretize what we mean by history-dependent choices by the interactor and how such interactors are implemented in our offline harness. Task and protocol (paraphrased). The task hides an integerx ∈ [1,n]. The solver must first output the number of queriesq, then output exactlyqqueries. Each query is a linek a1 a2 ...akwherekis even and thea i are distinct in [1,n]: ifxlies in the first half, the reply isL; if it lies in the second half, the reply isR; otherwise, the reply isN. After allq queries are printed, the solver reads a length-qreply string overL,R,N,?, where exactly one position is?(an ignored query). The solver then outputs its final guessx. Importantly, the statement requires using exactly the minimal number of queries f (n); using more queries is rejected even if the final guess is correct. What “adaptive” means in this task.Unlike a non-adaptive interactor that fixesxbefore interaction,cf2096Gexplicitly allows the interactor to choose a consistent hidden value and which query to ignore after seeing the full set of queries, as long as at least onexremains consistent with the returned reply string. Therefore, a correct solver must ensure that its query design uniquely determinesxeven under statement-permitted history-dependent choices by the interactor (e.g., an adversarial choice of the ignored position). Hidden offline input (example). In our offline harness, the hidden input for this task contains only public parameters (e.g., n) and does not fix a concrete hidden value in advance. Hidden case example (cases/001.in) 1 2 Local adaptive interactor (full listing). Our local adaptive interactor instantiates the statement-permitted adaptivity via a deterministic worst-case policy: it selects the ignored query index that maximizes the remaining ambiguity among candidates and uses a fixed tie-breaking rule to keep evaluation reproducible. cf2096G adaptive interactor (interactoradaptive.cpp) #include "testlib.h" #include <bits/stdc++.h> using namespace std; static long long queries = 0; static long long query_limit = 0; static void log_metrics() try tout << "queries=" << queries << endl; tout << "query_limit=" << query_limit << endl; catch (...) // best-effort logging only static void finish(TResult verdict, const string& msg) log_metrics(); quitf(verdict, "%s", msg.c_str()); static bool readLongLong(long long& out) if (!(cin >> out)) return false; 37 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information return true; static int f_min_queries(int n) // Smallest q such that 3ˆ(q-1) >= n long long pw = 1; int q = 1; while (pw < n) pw * = 3; q++; if (q > 60) break; return q; static int symToDigit(char c) if (c == ’L’) return 0; if (c == ’R’) return 1; return 2; // ’N’ int main(int argc, char ** argv) registerInteraction(argc, argv); atexit(log_metrics); int t = 0; try t = inf.readInt(); catch (...) finish(_pe, "failed to read t from input"); if (t < 1 || t > 100000) finish(_pe, "t out of reasonable range"); query_limit = 20L * t; log_metrics(); // must happen before any blocking read from std::cin cout << t << " "; cout.flush(); const long long hard_cap = max(200000L, 10L * query_limit); for (int tc = 1; tc <= t; tc++) int n = 0; try n = inf.readInt(); catch (...) finish(_pe, "failed to read n from input"); if (n < 2 || n > 200000) finish(_pe, "n out of range"); cout << n << " "; cout.flush(); long long qll = 0; if (!readLongLong(qll)) finish(_pe, "failed to read q from contestant"); if (qll < 1 || qll > 20) finish(_pe, "q must be in [1,20]"); int q = (int)qll; int fq = f_min_queries(n); if (q != fq) finish(_wa, "wrong number of queries (must be minimal f(n))"); queries += q; if (queries > hard_cap) finish(_pe, "too many queries (hard cap exceeded)"); 38 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information vector<vector<char>> code(n + 1, vector<char>(q, ’N’)); for (int qi = 0; qi < q; qi++) long long kll = 0; if (!readLongLong(kll)) finish(_pe, "failed to read k"); if (kll < 2 || kll > n) finish(_pe, "k out of range"); int k = (int)kll; if (k % 2 != 0) finish(_pe, "k must be even"); vector<int> a(k); vector<int> seen(n + 1, 0); for (int i = 0; i < k; i++) long long xll = 0; if (!readLongLong(xll)) finish(_pe, "failed to read array element"); if (xll < 1 || xll > n) finish(_pe, "array element out of range"); int x = (int)xll; if (seen[x]) finish(_pe, "array elements must be distinct"); seen[x] = 1; a[i] = x; int h = k / 2; for (int i = 0; i < h; i++) code[a[i]][qi] = ’L’; for (int i = h; i < k; i++) code[a[i]][qi] = ’R’; int bestJ = 0; int bestRep = 1; int bestSize = -1; vector<pair<long long, int>> keys; keys.reserve(n); for (int j = 0; j < q; j++) keys.clear(); for (int x = 1; x <= n; x++) long long key = 0; for (int i = 0; i < q; i++) if (i == j) continue; key = key * 3 + symToDigit(code[x][i]); keys.emplace_back(key, x); sort(keys.begin(), keys.end(), [](const auto& p1, const auto& p2) if (p1.first != p2.first) return p1.first < p2.first; return p1.second < p2.second; ); for (int idx = 0; idx < n; ) int nxt = idx + 1; while (nxt < n && keys[nxt].first == keys[idx].first) nxt++; int sz = nxt - idx; int rep = keys[idx].second; // smallest x in this group due to sort if (sz > bestSize || (sz == bestSize && (j < bestJ || (j == bestJ && rep < bestRep)))) bestSize = sz; bestJ = j; bestRep = rep; idx = nxt; 39 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information string s(q, ’N’); for (int i = 0; i < q; i++) s[i] = (i == bestJ ? ’?’ : code[bestRep][i]); cout << s << " "; cout.flush(); long long xll = 0; if (!readLongLong(xll)) finish(_pe, "failed to read final answer x"); if (xll < 1 || xll > n) finish(_pe, "final answer x out of range"); int x = (int)xll; bool inGroup = true; for (int i = 0; i < q; i++) if (i == bestJ) continue; if (code[x][i] != code[bestRep][i]) inGroup = false; break; if (!(bestSize == 1 && inGroup)) finish(_wa, "answer not uniquely determined by the interaction"); finish(_ok, "OK"); return 0; D.5. Mutation-based Stress Tests Motivation.Section 3 introduces mutation-based stress tests—lightly mutated submissions used as an internal robustness check. Here we detail the construction procedure and mutation operators used in our construction loop. Two stress-test sets. We generate 15 mutated submissions per task and partition them into two sets. Wrong-answer, protocol-valid submissions. These mutated submissions follow the protocol and respect the budget, but violate task semantics so a statement-consistent evaluator should reject them. They test correctness checking under realistic interaction traces without relying on protocol violations. Protocol- or budget-invalid submissions. These mutated submissions intentionally violate protocol rules or exceed query budgets to exercise edge cases of parsing, accounting, and termination. They test whether the evaluator enforces the published interaction rules consistently. Submission sources. We apply lightweight source-level mutations to existing solutions. We prioritize verified accepted submissions as stable bases and optionally derive mutated submissions from representative rejected submissions to diversify failure modes. Mutation operators. We use a small and interpretable set of operators. Wrong-answer, protocol-valid. • State-update perturbations that preserve I/O format but break a key invariant, such as reversing an interval update direction or skipping a required update. • Early-stop behaviors that terminate interaction and output a guess while still following the protocol. • Response-interpretation perturbations that keep queries valid but mis-handle interactor feedback, such as flipping a binary response or shifting a comparison threshold. 40 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information •Query-strategy coarsening that keeps the protocol valid but drops a necessary class of queries, making the strategy incomplete. • Output perturbations for tasks with uniquely checkable answers or explicit validity constraints, where a small edit to the final output violates a requirement while keeping the protocol well-formed. Protocol- or budget-invalid. • Parameter boundary violations designed to test fencepost cases in the parser and checker. • Malformed tokens or unsupported commands that test strict parsing. • Budget fencepost violations that exceed the query limit by one step. • Invalid termination behaviors, such as continuing to send output after the interactor has terminated. Filtering and usage. Mutated submissions are used as rejected stress-test submissions during validation rather than as additional officially labeled data. We require each stress-test submission to be rejected; any stress-test submission that is mistakenly accepted triggers refinement of the generator, interactor, or checkers. After refinement, we re-validate on the officially labeled human subset of the verification pool to ensure the fix does not introduce regressions. D.6. Expert Intervention Case Study We include an expert-intervention case study to illustrate how a statement-level semantic mismatch in an interactor can lead to incorrect Accepted/Rejected decisions, and how expert review restores statement-consistent behavior. In our construction, 31tasks required such expert intervention after reaching the iteration cap (Section 3). We usecf1847Eto illustrate how a statement ambiguity can lead to a mismatched verdict in an initial interactor implementation, and how expert review resolves the discrepancy. cf1847E: Made in Heaven (problem statement) This is an interactive problem. Made in Heaven is a rather curious Stand. Of course, it is (arguably) the strongest Stand in existence, but it is also an ardent puzzle enjoyer. For example, it gave Qtaro the following problem recently. Made in Heaven hasnhidden integersa 1 ,a 2 ,...,a n (3≤ n≤ 5000,1≤ a i ≤ 4). Qtaro must determine all thea i by asking Made in Heaven some queries of the following form. • In one query, Qtaro gives three distinct indices i, j, and k. • If a i , a j , a k form the sides of a non-degenerate triangle, Made in Heaven responds with the area of this triangle. • Otherwise, Made in Heaven responds with 0. By asking at most5500such questions, Qtaro must either tell Made in Heaven all the values of thea i , or report that it is not possible to uniquely determine them. Unfortunately due to the universe reboot, Qtaro is not as smart as Jotaro. Please help Qtaro solve Made in Heaven’s problem. Non-degenerate triangle. Three positive integersa,b,care said to form the sides of a non-degenerate triangle if and only if all of the following three inequalities hold. • a + b > c, • b + c > a, • c + a > b. Interaction. The interaction begins with readingn(3≤ n≤ 5000), the number of hidden integers. To ask a question corresponding to a triple (i,j,k) with distinct indices, output ? i j k and then read a single integer s. • If s = 0, then a i , a j , and a k are not the sides of a non-degenerate triangle. •Otherwise,s = 16∆ 2 , where∆is the area of the triangle. The area is provided in this format for convenience so that only integer input is needed. If the numbersa i cannot be uniquely determined, output! -1. On the other hand, if you have determined all the values of a i , output ! a 1 a 2 ... a n on a single line. The interactor is non-adaptive. The hidden arraya 1 ,a 2 ,...,a n is fixed beforehand and is not changed during the interaction process. Flushing. After printing a query, do not forget to output the end of line and flush the output. Otherwise, you may get “Idleness limit exceeded”. To do this, use the following methods in different languages. 41 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information • fflush(stdout) or cout.flush() in C++; • System.out.flush() in Java; • flush(output) in Pascal; • stdout.flush() in Python; • see the documentation for other languages. Protocol. We focus on the statement clauses that matter for the ! -1 ambiguity verdict: • Hidden array. The interactor fixes hidden integers a 1 ,...,a n with 3≤ n≤ 5000 and a i ∈1, 2, 3, 4. • Query. The solver prints? i j kwith three distinct indices, flushes, and reads an integer replys. The reply is0when the three values do not form a non-degenerate triangle, and otherwise s = 16∆ 2 , where ∆ is the triangle area. • Budget. The solver may ask at most 5500 queries. • Final output. The solver prints either! a 1 a 2 ... a n or! -1when it is not possible to uniquely determine the hidden array. • Fixed hidden array during interaction. The hidden array remains unchanged throughout the dialogue. Ambiguity witness.The solver queries triples of indices and receives an oracle reply that depends only on the correspond- ing triple of hidden values; the hidden array is fixed before the interaction and does not change during the dialogue. The statement allows! -1whenever the hidden array is not uniquely identifiable under the protocol: it must be Accepted if there exists another hidden array that would produce identical oracle replies for all possible queries. Letreply(x,y,z)denote the interactor response to a triple of values(x,y,z). We call a hidden arrayaambiguous if there exists anotherb̸=a such that ∀i,j,k distinct,reply(a i ,a j ,a k ) = reply(b i ,b j ,b k ). In this case, output ! -1 is the only statement-consistent behavior. Minimal counterexample. For n = 4, consider the two hidden arrays⟨1, 1, 1, 2⟩ and⟨1, 1, 1, 4⟩. For any triple of indices, the oracle response depends on whether the three values form a non-degenerate triangle. In both arrays, the triple consisting of the three1s yields the same non-zero response, while any triple containing the fourth element yields0. Therefore, the replies to all allowed queries are identical, yet the hidden arrays differ. Consequently, the hidden array is not uniquely identifiable, and! -1must be Accepted under the statement. This ambiguity is not an artifact of a limited query budget: whenn = 4there are only four possible triples, so the solver can query all of them and still cannot distinguish the two arrays. Non-zero replies do not imply uniqueness. Ambiguity may persist even when the transcript contains non-zero replies. For example,⟨1, 2, 3, 4⟩and⟨1, 2, 4, 3⟩induce the same replies for all four triples whenn = 4: all triples containing1return0, and the remaining triple returns a fixed non-zero value. This shows why uniqueness cannot be inferred from the existence of any non-degenerate triangle. Why this affects correct solvers. This is not a contrived edge case of a trivial program. Standard solutions explicitly output ! -1whenever the interaction transcript admits multiple consistent hidden arrays. For example, a typical implementation enumerates candidates for small n and emits ! -1 when the solution is not unique: Solver excerpt: emitting ! -1 under ambiguity dfs(1); // enumerate hidden arrays consistent with replies if (cnt != 1) cout << "! -1" << endl; return 0; Pre-fix and post-fix. The pre-fix interactor incorrectly rejected! -1on a class of ambiguous arrays due to an overly strong uniqueness assumption. Concretely, it treated the existence of any non-degenerate triangle among the hidden values as sufficient evidence of uniqueness once n is large enough, which is not guaranteed by the statement. The expert revision 42 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information replaced this heuristic with an ambiguity check aligned with the statement, leveraging the small value domain to test whether an alternative hidden array can preserve all oracle replies. After the fix,! -1is Accepted on ambiguous arrays while remaining Rejected on uniquely identifiable arrays, demonstrating that the correction restores semantic fidelity rather than weakening the evaluator. Table 13. Effect of expert intervention on ! -1 verdicts for cf1847E. Array type with solver output ! -1Pre-fixPost-fix Ambiguous array (multiple consistent hidden arrays)RejectedAccepted Uniquely identifiable array (single consistent hidden array)RejectedRejected Validation.To avoid overfitting the fix, we validated the ambiguity checker by exhaustive enumeration forn≤ 9over the full value domain1, 2, 3, 4 n . Across all enumerated arrays, every ambiguous array admitted a local witness detectable by the post-fix check (single-position replacement or two-position swap), and we observed no counterexample. Table 14. Exhaustive enumeration validation of the ambiguity checker (n≤ 9). n4 n # ambiguous# missing witness 364600 4256840 510241350 640962280 7163843570 8655365280 92621447470 Interactor excerpt: pre-fix ambiguity check (incorrect) bool check_ambiguity() // Count frequencies of values in hidden array vector<int> cnt(5, 0); for (int x : a) cnt[x]++; bool possible_triangle = false; // Iterate all possible sorted triplets of values (u, v, w) for (int u = 1; u <= 4; u++) for (int v = u; v <= 4; v++) for (int w = v; w <= 4; w++) if (is_triangle(u, v, w)) // Check if hidden array has enough of these values to form this triangle int need[5] = 0; need[u]++; need[v]++; need[w]++; if (cnt[1] >= need[1] && cnt[2] >= need[2] && cnt[3] >= need[3] && cnt[4] >= need[4]) possible_triangle = true; // If no non-degenerate triangle can be formed, all queries return 0. if (!possible_triangle) return true; // Special case for N=3 if (n == 3) long long val = get_val(a[0], a[1], a[2]); if (val == 63) return true; 43 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information // Incorrect: assumes N>=4 and any triangle implies uniqueness. return false; Interactor excerpt: post-fix ambiguity check (statement-consistent) bool check_ambiguity() array<int, 5> cnt; for (int x : a) cnt[x]++; auto pair_exists = [&](const array<int, 5>& c, int u, int v) -> bool if (u == v) return c[u] >= 2; return c[u] >= 1 && c[v] >= 1; ; auto equal_on_present_pairs = [&](int x, int y, const array<int, 5>& rest) -> bool for (int u = 1; u <= 4; u++) for (int v = u; v <= 4; v++) if (!pair_exists(rest, u, v)) continue; if (get_val(x, u, v) != get_val(y, u, v)) return false; return true; ; // Single-position replacement: change one x into y without changing any answers. for (int x = 1; x <= 4; x++) if (cnt[x] == 0) continue; array<int, 5> rest = cnt; rest[x]--; for (int y = 1; y <= 4; y++) if (y == x) continue; if (equal_on_present_pairs(x, y, rest)) return true; // Two-position swap: swap one x with one y. for (int x = 1; x <= 4; x++) for (int y = x + 1; y <= 4; y++) if (cnt[x] == 0 || cnt[y] == 0) continue; array<int, 5> rest = cnt; rest[x]--; rest[y]--; if (equal_on_present_pairs(x, y, rest)) return true; return false; D.7. Interactive I/O Templates Interactive problems are sensitive to I/O synchronization and buffering. To reduce incidental failures unrelated to algorithmic reasoning, we provide optional language-specific I/O scaffolding for common languages. The templates emphasize explicit flushing after each query, graceful termination on EOF or invalid replies, and avoiding any non-protocol output onstdout. 44 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information C++ interactive template #include <bits/stdc++.h> using namespace std; // Interactive notes: // - Print a query, then flush (endl is fine). // - Read the judge reply immediately. // - Many judges return -1 on invalid query; just exit. // - Don’t print anything else to stdout. int main() ios::sync_with_stdio(false); cin.tie(nullptr); // TODO: implement // // cout << "? " << x << " " << flush; // int r; // if (!(cin >> r)) return 0; // if (r == -1) return 0; // // cout << "! " << ans << " " << flush; return 0; Python interactive template #!/usr/bin/env python3 import sys input = sys.stdin.readline # Interactive notes: # - Print a query, then flush=True. # - Read the judge reply immediately. # - Many judges return -1 on invalid query; just return. # - Don’t print anything else to stdout. def main() -> None: # TODO: implement # # print("?", x, flush=True) # s = input() # if not s: # return # s = s.strip() # if s == "-1": return # # print("!", ans, flush=True) return if __name__ == "__main__": main() 45 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Java interactive template import java.io. * ; import java.util. * ; // Interactive notes: // - Print a query, then flush(). // - Read the judge reply immediately. // - Many judges return -1 on invalid query; just return. // - Don’t print anything else to stdout. public class Main private static final BufferedReader br = new BufferedReader(new InputStreamReader( System.in)); private static StringTokenizer st; private static final PrintWriter out = new PrintWriter(new BufferedWriter(new OutputStreamWriter(System.out))); private static String next() throws IOException while (st == null || !st.hasMoreTokens()) String line = br.readLine(); if (line == null) System.exit(0); st = new StringTokenizer(line); return st.nextToken(); private static int nextInt() throws IOException return Integer.parseInt(next()); public static void main(String[] args) throws Exception // TODO: implement // // out.println("? " + x); // out.flush(); // Integer r = nextInt(); // if (r == -1) return; // // out.println("! " + ans); // out.flush(); Go interactive template package main import ( "bufio" "fmt" "os" ) // Interactive notes: // - Print a query, then Flush(). // - Read the judge reply immediately. // - Many judges return -1 on invalid query; just exit. // - Don’t print anything else to stdout. 46 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information func main() in := bufio.NewReader(os.Stdin) out := bufio.NewWriter(os.Stdout) defer out.Flush() // TODO: implement // // fmt.Fprintln(out, "?", x) // out.Flush() // var r int // fmt.Fscan(in, &r) // if _, err := fmt.Fscan(in, &r); err != nil return // // fmt.Fprintln(out, "!", ans) // out.Flush() D.8. Prompt Templates for the Propose–Validate–Adjudicate Loop This subsection lists the prompt templates used in our propose–validate–adjudicate loop (Section 3). We include them to improve auditability and transparency of the construction process. Prompt templates. • generator.txt: proposer prompt for generators. • interactor * .txt: proposer prompts for interactors. • mutated submissions.txt: proposer prompt for mutated submissions. • decision * .txt: adjudicator prompts for selecting candidates. Generator prompt: testlib.h generator You are an expert competitive programmer. Write a C++17 ‘testlib.h‘ generator that outputs EXACTLY ONE hidden test case for this interactive problem. Constraints (one-shot): - Do not rely on external tools, file access, or web browsing. - Do not propose running commands or fetching additional information. - Output a complete final C++17 program in one shot. IMPORTANT (what this generator does): - This generator is run OFFLINE to create ‘cases/ * .in‘. - It outputs the HIDDEN INPUT that the interactor reads from ‘inf‘. It is NOT an interaction transcript. - The solver cannot see these ‘.in‘ files; only the interactor reads them. Invocation (used in our pipeline): - ‘gen <seed> [--key value / --key=value / -k value ...]‘ - Use ‘registerGen(argc, argv, 1)‘ so ‘argv[1]‘ is the seed. - testlib’s ‘rnd‘ becomes deterministic w.r.t. the seed; do NOT seed manually. Requirements (strict): 1) Always include ‘#include "testlib.h"‘ FIRST, then ‘<bits/stdc++.h>‘, and use ‘using namespace std;‘. 2) Initialize with ‘registerGen(argc, argv, 1);‘. 3) Use testlib randomness (‘rnd. * ‘) only; do NOT use ‘rand()‘ and do NOT set seeds manually. 4) Parse args via ‘opt<>()‘ with sensible defaults; include ‘mode = opt<string>("mode ", "non")‘. 47 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information 5) Output exactly ONE test case to stdout in the EXACT hidden-input format the interactor expects for that mode. 6) Deterministic coverage: - small seeds (e.g., 1..3): hand-crafted corner cases (min, max, tricky). - other seeds: random stress + structured adversarial patterns. 7) Generate cases that stress query efficiency (hard instances near worst-case). Common testlib pitfalls (avoid): - Do NOT use ‘testlib::‘ namespace; symbols are global. - When using ‘rnd.next(l, r)‘, ensure ‘l‘ and ‘r‘ have the same type (both int or both long long). - Do NOT call non-existent methods like ‘rnd.sample()‘, ‘rnd.permutation()‘, ‘rnd. nextf()‘. Output ONLY the C++ code, no markdown, no explanations. Interactor prompt You are an expert competitive programmer. Write a NON-ADAPTIVE interactor (interactive judge) in C++17 using ‘testlib.h‘ for the given stdin/stdout interactive problem. Constraints (one-shot): - Do not rely on external tools, file access, or web browsing. - Do not ask for more context. - Output a complete final C++17 program in one shot. EXECUTION MODEL (evaluation harness): - Invoked as: ‘./interactor in.txt tout.txt‘ - You MUST call ‘registerInteraction(argc, argv);‘ - ‘inf‘ reads the hidden input from ‘argv[1]‘ (‘in.txt‘, copied from ‘cases/ * .in‘). - ‘tout‘ writes logs to ‘argv[2]‘ (‘tout.txt‘). - ‘std::cin‘ reads the solver’s outputs (pipe). - ‘std::cout‘ writes the interactor’s replies to the solver (pipe). HIDDEN INPUT RULE (non-adaptive): - The hidden test case is FIXED and comes ONLY from ‘inf‘. - Do NOT generate new hidden data inside the interactor (no randomness, no sampling). - The solver CANNOT read ‘in.txt‘; any initial public parameters must be printed by the interactor to stdout as part of the protocol. LOGGING CONTRACT (STRICT; parsed by the harness): - Maintain TWO integer metrics (digits only): 1) ‘queries‘ = budget used ‘Q‘ (usually query count; if the problem uses " energy/cost", log the consumed cost instead). 2) ‘query_limit‘ = budget limit ‘B‘ from the statement (integer; may depend on parameters read from ‘inf‘). - You MUST write BOTH of the following lines to ‘tout‘ on EVERY termination path (OK/ WA/PE/RE/early exit): - ‘queries=<int>‘ - ‘query_limit=<int>‘ - Lowercase keys exactly. No extra text on those lines. - The runner uses regex ‘queries * = * ( +)‘ and ‘query_limit * = * ( +)‘, and takes the LAST match. You may print multiple times defensively, but the final values must be correct. - Use an ‘atexit(...)‘ guard PLUS a ‘finish(...)‘ wrapper that logs before calling ‘ quitf(...)‘. - IMPORTANT: the harness may terminate stalled processes, which can bypass ‘atexit‘. To avoid ‘queries=None‘ / ‘query_limit=None‘ in such failures, you MUST: - call ‘log_metrics()‘ once right after setting ‘query_limit‘ (before any blocking read from ‘std::cin‘). 48 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information BUDGET ENFORCEMENT (important): - Do NOT enforce the budget limit as WA/PE. Just count ‘queries‘ and report ‘ query_limit‘; the harness validates budget adherence using these counters. - To prevent infinite loops / output spam, enforce a HARD CAP and terminate with PE if exceeded. - Recommended: ‘hard_cap = max(200000, 10 * query_limit)‘ (or a fixed large constant if needed). ROBUST I/O (important for correctness + logging): - Avoid ‘ouf.readInt()/ouf.readToken()/...‘ because malformed output may trigger internal ‘quitf‘ and skip your logging. - Read solver output via ‘std::cin‘, validate it manually, and decide verdict via your own ‘finish(...)‘. - After EVERY output to the solver (including initial parameters and every query reply ), flush. (‘cout << endl;‘ is fine, or ‘cout.flush()‘). - When reading hidden input via testlib ‘inf‘, DO NOT use ‘operator>>‘ (e.g. ‘inf >> x ‘) -- ‘InStream‘ does not implement it. Always use ‘inf.readInt() / inf.readLong() / inf.readToken()‘ (and validate ranges). testlib notes: - Always ‘#include "testlib.h"‘ first (then ‘<bits/stdc++.h>‘). - All testlib symbols are in the global namespace (do NOT use ‘testlib::‘). - Verdicts: ‘_ok‘ (0), ‘_wa‘ (1), ‘_pe‘ (2). Minimal skeleton (fill in protocol-specific logic; print ONLY protocol outputs to stdout): #include "testlib.h" #include <bits/stdc++.h> using namespace std; static long long queries = 0; static long long query_limit = 0; // must be set based on the statement / inf parameters static void log_metrics() try tout << "queries=" << queries << endl; tout << "query_limit=" << query_limit << endl; catch (...) // best-effort logging only static void finish(TResult verdict, const string& msg) log_metrics(); quitf(verdict, "%s", msg.c_str()); int main(int argc, char ** argv) registerInteraction(argc, argv); atexit(log_metrics); // TODO: read hidden input from ‘inf‘ // TODO: set query_limit (integer budget B) // TODO: implement protocol loop using std::cin/std::cout Output ONLY the C++ code, no markdown, no explanations. 49 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Mutated-submission prompt You are an expert competitive programmer and evaluator engineer. Your task is to write ONE mutated submission for a stdin/stdout interactive problem by lightly mutating a given baseline solver. You will be given: 1) The problem statement and protocol specification (including the exact query/output formats and termination rules). 2) The query budget and any parameter constraints. 3) ONE baseline solver program that is Accepted by the official judge (source code). Optionally, you may also be given a small set of representative Rejected submissions. You will also be given a required category. Goal: This mutated submission is used as an internal adversarial stress test. It is NOT for increasing the quantity of rejected submissions. It is used to expose evaluator weaknesses that are hard to cover naturally in a small human verification pool: 1) The evaluator/interactor is too permissive: semantically incorrect behavior is mistakenly accepted. 2) Protocol/budget enforcement has gaps: invalid interaction is not consistently rejected. Constraints (one-shot): - Do not rely on external tools, file access, or web browsing. - Do not ask for more context. - Output a complete final program in one shot. Output requirements: - Output exactly ONE mutated submission program in the SAME language as the baseline solver. - Keep changes minimal and localized. Do NOT rewrite the entire solver from scratch. - The mutated submission should be plausible and close to correct behavior, not obviously sabotaged. Required category: You MUST follow the provided category, which is exactly one of the following. Category A: SEMANTIC_WRONG_PROTOCOL_VALID - The submission MUST follow the protocol and respect the query budget. - The submission MUST flush after every query. - The submission MUST not print anything to stdout outside the protocol. - The submission MUST terminate normally. - The submission MUST implement a subtle semantic bug so it is incorrect on some test cases. Prefer interaction-specific bugs such as: - incorrect state updates that break an invariant - early stopping with a plausible but incorrect final answer - misinterpreting judge feedback (e.g., flipping a binary reply, shifting a threshold) - dropping a necessary class of queries while keeping the protocol valid - small perturbations to the final output when answers are uniquely checkable Category B: PROTOCOL_OR_BUDGET_WRONG - The submission MUST compile and run, but MUST violate exactly one protocol or budget rule in a minimal, boundary way. Prefer fencepost violations that test completeness: - out-of-range query argument by 1 - malformed token count or unsupported command word - exceeding the query limit by exactly 1 - invalid termination behavior, such as printing after completion - Do NOT combine multiple violations in one submission unless required by the protocol 50 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information structure. Output format (STRICT): - Output ONLY code, no markdown and no explanations. - Begin the program with a single-line comment header: // MUTATED_SUBMISSION | <CATEGORY> | <one short intent> where <CATEGORY> is exactly the provided category. Adjudicator prompt: select generator You are a judge selecting the best test case generator (‘gen_cases.cpp‘) candidate. You will be given: 1) The problem statement and meta.json 2) Hard constraints from ‘INTERACTOR_GENERATOR_PLAN.md‘ 3) Candidate generator codes for a small set of models (exact names provided) STRICT RULES: - You MUST choose one model name from "Allowed models" - You MUST output JSON only (single line), no explanations: "pick": "<model_name>" - Do NOT invent or modify model names SELECTION PRIORITY (in order): 1) Contract compliance: uses testlib generator pattern (‘registerGen‘), prints exactly one case to stdout, deterministic given seed, no hidden randomness. 2) Compatibility with the harness: supports the required mode(s) indicated by meta. json (e.g., ‘--mode non‘ or ‘--mode adp‘ when applicable). 3) Likely strength/diversity of generated cases and reasonable schema design. 4) Robustness and simplicity as tie-breaker. Adjudicator prompt: select interactor You are a judge selecting the best interactor candidate from multiple LLM-generated candidates. You will be given: 1) The problem statement and meta.json 2) The selected generator code (‘generator/gen_cases.cpp‘) to ensure schema coupling 3) Candidate interactor codes (non_adaptive.cpp and possibly adaptive.cpp) 4) Evaluation results from the harness for each candidate: - std-check: runs reference-correct solver programs in ‘codes/std/ * .cpp‘ if present . PASS means all cases are Accepted. NOTE: compile errors (CE) are ignored; if all std are CE (or the folder is empty) , std-check is SKIPPED. - std_wa-check: runs representative incorrect solver programs in ‘codes/std_wa/ * . cpp‘ if present. FAIL if any std_wa passes all cases. - If a check is marked SKIPPED, you MUST ignore that check (do not assume anything) . STRICT RULES: - You MUST choose one model name from "Allowed models" - You MUST output JSON only (single line), no explanations: "pick": "<model_name>" - Do NOT invent or modify model names SELECTION PRIORITY (in order): 51 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information 1) Correctness: if std-check exists, the interactor MUST PASS std-check. 2) Strength: if std_wa-check exists, the interactor MUST NOT allow any std_wa to pass all cases. 3) Query efficiency: prefer fewer queries if query stats are available. 4) Robustness and contract compliance (queries/query_limit logging, safe parsing, deterministic behavior). Adjudicator prompt: select solver and interactor You are a judge for selecting the best solver and interactor combination. In the inputs, solver candidates are labeled as "std". You will be given: 1. Tournament test results (PASS/FAIL for each std x interactor pair) 2. Valid model names for std (solver) and interactor 3. Code snippets of passing candidates (truncated) STRICT RULES: - You MUST choose a combination marked PASS in Tournament Results - You MUST use exact model names from "Valid std models" and "Valid interactor models" - Do NOT invent or modify model names SELECTION PRIORITY (in order): 1. Correctness: must be PASS 2. Query efficiency: prefer solutions likely to use FEWER queries 3. Code clarity and robustness as tie-breaker Output format (JSON only, single line, no explanation): "std": "<model_name>", "interactor": "<model_name>" D.9. Model Performance by Category and Difficulty We provide per-model category-wise results stratified by difficulty tiers (Easy/Medium/Hard). These tables complement Table 3 by reporting mean pass@1 and pass@5 for each category and difficulty tier, as well as overall. 52 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Table 15. Performance by category and difficulty: Gemini-3-Pro-Preview. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.7130.8700.4670.7110.2200.5000.4750.705 Search 0.7940.9680.5310.7240.3330.6250.5610.770 Greedy0.7380.9380.3900.6670.3820.6360.4700.725 Bit0.8831.0000.6450.7270.2550.3640.6130.711 Data Structures 0.8431.0000.4160.6860.3640.7270.4870.750 Math0.7230.9680.6130.7330.4240.5590.5890.744 Game0.7641.0000.3110.6110.4670.6670.4800.743 Table 16. Performance by category and difficulty: GPT-5.2. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.7570.8700.4090.5560.1900.3500.4500.591 Search0.7350.9680.5450.6900.3080.4170.5470.708 Greedy0.6130.8120.5290.7380.4180.5450.5300.725 Bit 0.8501.0000.5450.7270.3090.3640.5690.711 Data Structures0.7430.9290.4750.7060.3090.4550.5000.711 Math0.7350.9030.5800.7000.3760.5590.5630.712 Game0.6910.9090.5330.7220.4330.8330.5660.800 Table 17. Performance by category and difficulty: DeepSeek-V3.2-Thinking. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.6260.7830.2360.5110.0500.2000.2950.511 Search0.6390.8060.2860.4660.0830.2080.3400.504 Greedy 0.6500.8750.2950.5480.0910.2730.3450.580 Bit 0.7670.8330.1730.3640.2000.2730.3380.467 Data Structures 0.6000.7860.2710.5290.0180.0910.2950.513 Math0.5550.7740.3130.4830.0940.1760.3140.472 Game0.6910.9090.2560.6110.1670.3330.3770.657 53 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Table 18. Performance by category and difficulty: Gemini-2.5-Pro. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.5300.7390.2180.4440.0200.1000.2550.443 Search 0.5870.7420.2660.4480.0670.1670.3120.469 Greedy0.5500.6880.2520.4760.0730.1820.2930.478 Bit0.6500.7500.2730.3640.1450.2730.3420.444 Data Structures 0.5290.7140.2080.4510.0730.0910.2470.447 Math0.5030.7100.3200.4830.1060.2350.3070.472 Game0.4550.5450.2780.6670.1330.5000.3090.600 Table 19. Performance by category and difficulty: GPT-OSS-120B. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.5220.7830.1510.3330.0100.0500.2160.386 Search0.5350.6770.1970.3450.1000.2080.2690.407 Greedy0.5250.7500.1710.3100.0000.0000.2260.362 Bit 0.5830.9170.2270.4090.0550.2730.2800.511 Data Structures0.4140.6430.1570.2750.0180.0910.1840.316 Math0.4320.7100.2600.4000.0820.2060.2540.424 Game0.5640.7270.1560.3330.2000.3330.2910.457 Table 20. Performance by category and difficulty: Claude-Opus-4.5. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.2960.5650.1200.2220.0000.0000.1390.261 Search0.2710.4840.1480.2590.0080.0420.1520.274 Greedy 0.3750.6880.0950.2380.0180.0910.1480.319 Bit 0.3170.6670.0450.1820.0550.0910.1200.289 Data Structures 0.3430.7140.0980.2550.0000.0000.1290.303 Math0.2580.4840.1930.3500.0290.0590.1650.304 Game0.3450.6360.0780.2220.0000.0000.1490.314 54 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Table 21. Performance by category and difficulty: Gemini-2.5-Flash. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.2870.4780.0760.1560.0000.0000.1140.205 Search 0.3100.4840.1620.3280.0080.0420.1700.310 Greedy0.3120.6880.1570.3570.0180.0910.1710.391 Bit0.4000.6670.0820.1820.0180.0910.1510.289 Data Structures 0.1860.4290.1410.3330.0000.0000.1290.303 Math0.2710.5480.2000.3330.0120.0590.1660.312 Game0.3820.8180.1220.3330.0000.0000.1830.429 Table 22. Performance by category and difficulty: GPT-OSS-20B. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.3300.6090.0670.1780.0000.0000.1200.250 Search0.3100.5160.0900.2070.0250.0830.1360.265 Greedy0.4250.7500.0760.2380.0000.0000.1450.319 Bit 0.2330.5000.0550.1360.0360.1820.0980.244 Data Structures0.2000.5710.0470.1570.0000.0000.0680.211 Math0.2390.5160.1570.3170.0240.0880.1410.304 Game0.3640.7270.0670.2220.0000.0000.1490.343 Table 23. Performance by category and difficulty: Qwen3-30B-A3B-Thinking. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.1740.3480.1070.1780.0100.0500.1020.193 Search0.2320.4520.0970.2410.0080.0420.1150.257 Greedy 0.2000.6250.1670.2860.0360.1820.1540.348 Bit 0.2170.5000.0550.0910.0000.0000.0840.178 Data Structures 0.1710.4290.0940.1960.0000.0000.0950.211 Math0.0710.2260.1230.2670.0180.0880.0820.208 Game0.2180.6360.1220.1670.0330.1670.1370.314 55 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Table 24. Performance by category and difficulty: Qwen3-32B. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.2700.4350.0760.1110.0100.0500.1110.182 Search 0.2450.4190.0620.1380.0080.0420.1010.195 Greedy0.3000.6250.0670.1190.0180.0910.1130.232 Bit0.2670.4170.0550.0910.0000.0000.0980.156 Data Structures 0.2710.5000.0350.0780.0000.0000.0740.145 Math0.0650.1940.0770.1330.0120.0590.0560.128 Game0.3270.6360.0330.0560.0000.0000.1200.229 Table 25. Performance by category and difficulty: Qwen3-14B. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.1390.2610.0490.1330.0000.0000.0610.136 Search0.1870.2900.0410.0860.0000.0000.0730.124 Greedy0.2120.4380.0290.0950.0000.0000.0670.159 Bit 0.1330.1670.0270.0450.0000.0000.0490.067 Data Structures0.2140.4290.0200.0780.0000.0000.0530.132 Math0.0320.0970.0670.1670.0000.0000.0400.104 Game0.1640.4550.0440.1110.0000.0000.0740.200 Table 26. Performance by category and difficulty: DeepSeek-R1-Distill-Llama-70B. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.0700.2170.0400.1110.0000.0000.0390.114 Search0.1810.3230.0450.0860.0000.0000.0730.133 Greedy 0.1620.4380.0240.0710.0000.0000.0520.145 Bit 0.0170.0830.0270.0450.0000.0000.0180.044 Data Structures 0.1570.2860.0120.0590.0000.0000.0370.092 Math0.0390.0650.0570.1000.0000.0000.0370.064 Game0.1090.3640.0440.1110.0000.0000.0570.171 56 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Table 27. Performance by category and difficulty: DeepSeek-V3.2. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.3040.4780.1070.2440.0200.0500.1390.261 Search 0.3230.5480.1380.2240.0250.0830.1650.283 Greedy0.3620.5620.1240.2620.0000.0000.1590.290 Bit0.3500.5000.0820.1820.0180.0910.1380.244 Data Structures 0.3290.6430.0510.1760.0000.0000.0950.237 Math0.1610.3230.1730.2500.0290.0590.1310.216 Game0.2730.4550.0890.1670.0000.0000.1310.229 Table 28. Performance by category and difficulty: GPT-4.1. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.1300.3910.0440.1110.0000.0000.0570.159 Search0.1870.2900.0240.0690.0000.0000.0640.115 Greedy0.2120.4380.0480.1190.0000.0000.0780.174 Bit 0.1330.2500.0000.0000.0000.0000.0360.067 Data Structures0.1290.2860.0430.1180.0000.0000.0530.132 Math0.0900.2580.0370.0830.0060.0290.0420.112 Game0.1450.5450.0110.0560.0330.1670.0570.229 Table 29. Performance by category and difficulty: Qwen3-32B-NonThinking. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.0260.0870.0130.0220.0000.0000.0140.034 Search0.0390.1290.0030.0170.0000.0000.0120.044 Greedy 0.0500.1880.0050.0240.0000.0000.0140.058 Bit 0.0000.0000.0450.0450.0000.0000.0220.022 Data Structures 0.0430.1430.0000.0000.0000.0000.0080.026 Math0.0260.0650.0100.0170.0000.0000.0110.024 Game0.0550.0910.0000.0000.0000.0000.0170.029 57 InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information Table 30. Performance by category and difficulty: Qwen3-14B-NonThinking. CategoryEasyMediumHardOverall pass@1pass@5pass@1pass@5pass@1pass@5pass@1pass@5 Graph0.0260.0870.0270.0440.0000.0000.0200.045 Search0.0320.1290.0070.0340.0000.0000.0120.053 Greedy0.0500.1250.0050.0240.0000.0000.0140.043 Bit 0.0000.0000.0000.0000.0000.0000.0000.000 Data Structures0.0430.1430.0040.0200.0000.0000.0110.039 Math0.0130.0320.0030.0170.0000.0000.0050.016 Game0.0550.1820.0000.0000.0000.0000.0170.057 58

Instant research discovery

InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information

Intelligence

Summary

Entities (12)

Relation Signals (10)

Abstract

Tags

Links

PDF

Full Text

Paper	Published	Area	Tags	Intel	Citations