Paper deep dive

Don't Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development

Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, Pinjia He

Year: 2026Venue: arXiv preprintArea: cs.SEType: PreprintEmbeddings: 89

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 10/12/2026, 6:47:47 AM

Summary

The paper introduces ReqClarifyBench, an interactive requirement clarification benchmark for repository-level software development, containing 341 tasks across game, miniapp, and web domains. It evaluates six LLMs using LLM-based user simulators with diverse personas to assess how well models clarify vague user intents into clear requirement documents. Key findings include that stronger clarification effectiveness does not guarantee higher document quality, performance varies significantly across user personas, and coding-agent harnesses do not improve requirement document quality compared to base LLMs.

Entities (10)

ReqClarifyBench · benchmark · 98%GameCraft-Bench · benchmark · 95%Mingyu Chen · person · 95%MiniAppBench · benchmark · 95%Pinjia He · person · 95%WebGen-Bench · benchmark · 95%ClarifyCodeBench · benchmark · 90%LLM · model · 90%Requirement Document · artifact · 90%SWE-chat · dataset · 90%

Relation Signals (9)

Pinjia He → affiliatedwith → The Chinese University of Hong Kong, Shenzhen

confidence 95% · PINJIA HE ∗ , The Chinese University of Hong Kong, Shenzhen, China

Mingyu Chen → affiliatedwith → The Chinese University of Hong Kong, Shenzhen

confidence 95% · MINGYU CHEN, The Chinese University of Hong Kong, Shenzhen, China

ReqClarifyBench → containstasksfrom → WebGen-Bench

confidence 95% · We choose GameCraft-Bench [38], MiniAppBench [57], and WebGen-Bench [36]...

ReqClarifyBench → containstasksfrom → GameCraft-Bench

confidence 95% · We choose GameCraft-Bench [38], MiniAppBench [57], and WebGen-Bench [36]... Based on these criteria, we choose...

ReqClarifyBench → containstasksfrom → MiniAppBench

confidence 95% · We choose GameCraft-Bench [38], MiniAppBench [57], and WebGen-Bench [36]...

ReqClarifyBench → focuseson → repository_level_development

confidence 95% · ReqClarifyBench, an interactive requirement clarification benchmark for repository-level software development.

ReqClarifyBench → evaluates → LLM

confidence 90% · We evaluate six LLMs in terms of clarification effectiveness and generated requirement document quality.

ClarifyCodeBench → focuseson →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want. Users often begin with vague intent, leaving important requirements to be clarified through interaction. In repository-level development, misunderstandings can propagate and cause costly rework. However, existing benchmarks typically evaluate implementation from complete specifications or underspecified requests, overlooking pre-coding requirement clarification. Prior clarification studies mainly focus on function-level tasks. It therefore remains unclear how well LLMs can transform vague intent into clear requirements through interaction and how this capability affects subsequent code generation. We introduce ReqClarifyBench, an interactive requirement clarification benchmark for repository-level software development. It contains 341 tasks spanning game, miniapp, and web development, each with a vague user intent, a human-validated requirement document, and code evaluation artifacts. We employ LLM-based user simulators with diverse personas derived from real-world conversations. We evaluate six LLMs in terms of clarification effectiveness and generated requirement document quality. Results show that stronger clarification effectiveness does not necessarily translate into higher document quality, and performance varies substantially across user personas. We also find that coding-agent harnesses do not further improve requirement document quality compared with their base LLMs. These findings offer insights for building LLM-based end-to-end software development systems.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

85,469 characters extracted from source content.

Expand or collapse full text

Don’t Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development MINGYU CHEN, The Chinese University of Hong Kong, Shenzhen, China ZHIQING ZHONG ∗ , The Chinese University of Hong Kong, Shenzhen, China MANYI WANG, The Chinese University of Hong Kong, Shenzhen, China LINXI LIANG, The Chinese University of Hong Kong, Shenzhen, China ROUYU CHEN, The Chinese University of Hong Kong, Shenzhen, China PINJIA HE ∗ , The Chinese University of Hong Kong, Shenzhen, China With the rapid advancement of Large Language Models (LLMs), software development is gradually moving toward a paradigm where users describe what they want and LLMs carry out the implementation. As LLMs’ im- plementation capabilities improve, an increasingly critical bottleneck emerges: whether LLMs truly understand what users want. In practice, users often begin with vague intent, leaving many important requirements to be progressively clarified through interaction. This is especially consequential in repository-level development, where an incorrect LLM’s guess about user intent can propagate through subsequent development phases, causing costly rework in long-horizon tasks. However, existing repository-level code generation benchmarks typically evaluate code generation directly from either complete specifications or underspecified requests, overlooking the importance of pre-coding requirement clarification. Meanwhile, prior requirement clarification studies mainly focus on function-level tasks, failing to reflect the complexity of real-world software develop- ment scenarios. Thus, it remains unclear how well LLMs can turn vague user intent into clear requirements through interactive clarification and how such clarification capability affects subsequent code generation. To bridge this gap, we introduce ReqClarifyBench, an interactive requirement clarification benchmark for repository-level software development. ReqClarifyBench contains 341 tasks spanning game, miniapp, and web development. Each task includes a high-level vague user intent, a detailed requirement document validated by human, and the corresponding code evaluation artifacts. To better simulate a realistic interactive clarification process, we further employ LLM-based user simulators with diverse personas derived from real- world conversation history. Based on ReqClarifyBench, we conduct a systematic empirical study. Specifically, we evaluate six LLMs’ requirement clarification capabilities from two complementary perspectives: how effectively they ask clarification questions and the quality of the generated requirement documents. The experimental results demonstrate that stronger clarification effectiveness does not necessarily translate into higher requirement document quality, and that LLMs exhibit substantial performance variations across different user personas. We also find that, compared with standalone LLMs, coding-agent harnesses do not further improve the ability to transform vague user intent into requirement documents. Based on these findings, we provide valuable insights for building LLM-based end-to-end software development systems. ∗ Corresponding author. Authors’ Contact Information: Mingyu Chen, mingyuchen1@link.cuhk.edu.cn, The Chinese University of Hong Kong, Shenzhen, Guangdong, China; Zhiqing Zhong, zhiqingzhong@link.cuhk.edu.cn, The Chinese University of Hong Kong, Shenzhen, Guangdong, China; Manyi Wang, manyiwang@link.cuhk.edu.cn, The Chinese University of Hong Kong, Shenzhen, Guangdong, China; Linxi Liang, linxiliang@link.cuhk.edu.cn, The Chinese University of Hong Kong, Shenzhen, Guangdong, China; Rouyu Chen, rouyuchen@link.cuhk.edu.cn, The Chinese University of Hong Kong, Shenzhen, Guangdong, China; Pinjia He, hepinjia@cuhk.edu.cn, The Chinese University of Hong Kong, Shenzhen, Guangdong, China. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM X-X/2026/10-ART https://doi.org/10.1145/n.n , Vol. 1, No. 1, Article . Publication date: October 2026. arXiv:2610.13022v1 [cs.SE] 9 Oct 2026 2Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, and Pinjia He CCS Concepts:• Software and its engineering→Software development techniques;• Computing method- ologies→ Artificial intelligence. Additional Key Words and Phrases: Large Language Models, Interactive Requirement Clarification, Benchmark ACM Reference Format: Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, and Pinjia He. 2026. Don’t Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development. 1, 1 (October 2026), 22 pages. https://doi.org/10.1145/n.n 1 Introduction Large Language Models (LLMs) are becoming increasingly capable of handling long-horizon software development tasks [17,18]. This progress is paving the way for a requirement-driven development paradigm, where users describe what they want and LLMs carry out the long-horizon implementation [13,35,47,49]. However, the success of this paradigm depends on clear and com- plete requirements, while real-world users often begin with vague descriptions of their intents that need to be progressively clarified through further interactions [22,48]. Neglecting requirement clarification may lead to incorrect assumptions, which will propagate to subsequent design, imple- mentation, and testing phases, causing the implementation to drift away from the user’s intent and increasing the need for human oversight and rework in long-horizon tasks [11, 24]. Figure 1 presents two representative real-world user complaints that illustrate a shared failure mode. When a request is underspecified, LLMs may make a guess about what the user wants without asking for clarification, wasting time and tokens without producing a satisfactory outcome. Although LLMs’ implementation capabilities continue to improve, as reflected in rising benchmark scores [5,15,17,18], they may still spend substantial time and tokens acting on a misunderstanding of the user’s request. Turning vague user intents into clear requirements through interactive clarification is therefore becoming an increasingly critical bottleneck. Existing benchmarks [19,36,38,57,59] have demonstrated progress in the end-to-end trans- formation of Natural Language Requirements (NLRs) into repository-level code. However, they largely overlook the importance of requirement clarification in this process, evaluating code gener- ation from either well-specified requirement documents or vague user requests without multi-turn requirement clarification. Meanwhile, the Requirements Engineering (RE) community has explored LLM-based requirement clarification. Nevertheless, existing studies [21,39,53] mainly focus on function-level tasks, failing to reflect the complexity of real-world software projects. Moreover, these studies often employ simulated users that follow a fixed interaction pattern, failing to capture diverse real-user behaviors [32]. To address these limitations, we introduce ReqClarifyBench, an interactive requirement clar- ification benchmark for repository-level software development. ReqClarifyBench aims to evaluate LLMs’ ability to transform vague user intents into clear requirement documents through multi-turn requirement clarification. ReqClarifyBench contains 341 tasks spanning game, miniapp, and web development. It is constructed through a semi-automated pipeline. Each task includes a high-level vague user intent, a detailed requirement document validated by humans, and the corresponding code evaluation artifacts. To support realistic interactions, we derive three representative user personas from SWE-chat [7], a large-scale dataset of multi-turn conversations between software developers and coding agents. These personas are used to guide the LLM-based user simulator throughout the clarification process. In addition, we introduce three new metrics to assess LLMs’ requirement clarification capabilities from two perspectives: how effectively LLMs ask clarification questions and the quality of the generated requirement documents. , Vol. 1, No. 1, Article . Publication date: October 2026. Don’t Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development3 Misunderstandings Lead to Token & Time WastePremature Implementation under Vague Intent Underspecified User Intent Long-horizon Implementation Late Discovery of Misunderstanding Rework Leads to Substantial Time and Token Waste 657 101 When prompts are unclearor variables are missing, Claude tends to make its own assumptions and execute anyway, usually leading to bad outcomes. LLMdidn’t understand my prompt correctly and started going in the wrong direction.Letting it finish would cost at least double in tokensand time. Fig. 1. One complaint describes incorrect assumptions leading to an undesirable implementation; the other describes time and tokens wasted as the model continues along a misunderstood path. Based on ReqClarifyBench, we conduct a systematic empirical study focusing on the following four dimensions. (1) Overall performance across LLMs and user personas. The results show that stronger clarification effectiveness does not necessarily lead to higher requirement document quality, and that user interaction styles can substantially affect model performance. (2) LLMs’ clarification behavior. The results reveal three distinct interaction patterns: clarification with expert users is typically short and efficient, interactions with vague users are sustained but low- yield, and new requirement information often emerges in later rounds when users change their requirements. (3) The impact of clarification on requirement document quality and code generation. Our experiments show that interactive clarification consistently improves requirement document quality, with particularly substantial gains in fine-grained requirement alignment, and further benefits subsequent implementation. (4) Effect of coding-agent harness. We compare standalone LLMs with the same LLMs running within coding-agent harnesses. The results show that coding-agent harnesses improve clarification effectiveness but do not further enhance the quality of derived requirement documents. Overall, these findings suggest that requirement clarification goes beyond simply asking more questions. Requirement clarification requires the evaluated LLM to elicit key information, understand and integrate requirements expressed during multi-turn interaction, and adapt its clarification strategy to different user personas. These capabilities together determine whether an LLM can transform a vague user intent into reliable software requirements, supporting subsequent code generation. The main contributions of our work are summarized as follows: •ReqClarifyBench: We introduce ReqClarifyBench, the first interactive requirement clarification benchmark for repository-level software development, aiming to evaluate LLMs’ ability to transform vague user intents into clear and complete requirement documents through multi-turn interaction. In addition, we design three new evaluation metrics to assess LLMs’ interactive requirement clarification capabilities. •Realistic User Simulation: We construct three representative user personas based on a large-scale dataset of multi-turn conversations between software developers and coding agents, aiming to support realistic multi-turn requirement clarification. • Empirical Study: We conduct an empirical study to evaluate the requirement clarification capabilities of different LLMs and their impact on subsequent code generation, providing a systematic understanding of both current model capabilities and the role of requirement clarification in LLM-based software development. • Data Availability: We open-source the code and data at [2]. , Vol. 1, No. 1, Article . Publication date: October 2026. 4Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, and Pinjia He Table 1. Comparison of ReqClarifyBench with representative code generation benchmarks in terms of publication year, task granularity, dataset size, requirement characteristics, and support for interactive requirement clarification. “R.” denotes “Requirement” and Character. denotes “characteristic”. BenchmarkTimeGranularity#TasksR. Character.R. Clarification HumanEval [12]2021Function-level164Clear&Complete✗ MBPP [6]2021Function-level974Clear&Complete✗ SWE-bench [31]2023Pull Request-level2,294Clear&Complete✗ LiveCodeBench [30]2024Function-level400+Clear&Complete✗ WebGen-Bench [36]2025Repository-level101Underspecified✗ FeatureBench [59]2026Repository-level200Clear&Complete✗ NL2RepoBench [19]2026Repository-level104Clear&Complete✗ MiniAppBench [57]2026Repository-level500Underspecified✗ GameCraft-Bench [38]2026Repository-level140Clear&Complete✗ ClarifyCodeBench [21]2026Function-level419Underspecified✓ ReqClarifyBench2026Repository-level341Underspecified✓ 2 Background & Related Work 2.1 LLM-based Repository-level Code Generation LLM-based code generation has advanced from function-level programming to end-to-end repository-level software engineering [6,12,29,31,37,44,54]. Several approaches are proposed to facilitate this progress. MetaGPT [29] employs agents with different roles to collaboratively perform requirement analysis, system design, and code implementation. ChatDev [44] coordinates role-playing agents through communication across the design, coding, and testing stages. RPG [37] uses a structured Repository Planning Graph to support planning and repository generation. However, these approaches typically treat the initial user intent as a fixed task objective. Although requirement analysis is included, missing information is generally inferred by LLMs rather than clarified through interaction with the user. Consequently, the resulting software may deviate from the user’s actual intent. The conversational capabilities of LLMs enable them to proactively ask clarification questions and construct a clear and complete requirement document based on user feedback. Nevertheless, interactive requirement clarification remains underexplored in repository- level software development. In this paper, we introduce ReqClarifyBench and conduct an empirical study to investigate the role of requirement clarification in repository-level software development. 2.2 Code Generation Benchmarks As shown in Table 1, early code generation benchmarks primarily focus on function-level tasks. HumanEval [12], MBPP [6], and LiveCodeBench [30] require LLMs to generate correct programs or functions from natural language problem descriptions. SWE-bench [31] subsequently extends evaluation to real-world code repositories. Its tasks are derived from GitHub issues and their corresponding pull requests, requiring LLMs to understand existing repositories and generate code modifications that resolve the reported issues. Recent benchmarks have further shifted toward end-to-end repository-level software development. FeatureBench [59] evaluates the implementation of complex features in existing repositories. WebGen-Bench [36] and NL2Repo-Bench [19] assess the generation of multi-file websites and complete code repositories from natural language require- ments, respectively. MiniAppBench [57] and GameCraft-Bench [38] further extend evaluation to the end-to-end generation of applications and playable games. , Vol. 1, No. 1, Article . Publication date: October 2026. Don’t Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development5 Despite the growing task complexity, these benchmarks typically evaluate code generation from fixed requirements. In real-world development, however, users often begin with a vague intent, leaving many important requirements to be progressively clarified through interactions. Benchmarks that provide complete and clear descriptions [6,12,19,30,31,38,59] overlook the requirement clarification process, failing to reflect real-world development scenarios. Conversely, benchmarks using vague requests as inputs [36,57] require LLMs to infer missing information without user feedback, making it difficult to determine whether failures stem from limited code generation capabilities or requirement misunderstandings. This highlights the need to systematically evaluate LLMs’ ability to transform vague and incomplete user intents into clear requirement documents through interactions. 2.3 Interactive Requirement Clarification Requirement clarification is a central problem in Requirements Engineering (RE) [8,53]. It typically involves iterative interaction with stakeholders to clarify their goals, constraints, and unstated requirements [23]. Recent studies have explored requirement clarification for LLM-based code generation from different perspectives. ClarifyGPT [39] detects ambiguous requirements, generates targeted clarification questions, and refines the requirements based on user responses before generating code. ClarifyCoder [50] fine-tunes LLMs to recognize underspecified problems and request missing information. ClarifyCodeBench [21] introduces an interactive benchmark containing underspecified programming problems, annotated clarification questions and answers, and executable test cases. However, these studies primarily focus on simple function-level problems, failing to reflect real-world complex software development. ReqElicitGym [32] models requirement clarification as a multi-turn elicitation process, but its simulated user is modeled as a single oracle user following predefined interaction rules, which fails to capture the diversity of interaction styles and behavioral patterns among real users. Moreover, ReqElicitGym does not further investigate the impact of requirement clarification on subsequent code generation. In contrast, our proposed ReqClarifyBench evaluates interactive requirement clarification for repository-level software development using LLM-based user simulators with diverse personas. 3 ReqClarifyBench In this section, we introduce ReqClarifyBench, the first interactive requirement clarification benchmark for repository-level software development, aiming to evaluate LLMs’ ability to transform vague user intents into clear requirement documents through multi-turn interaction. We first describe the construction procedure and then present the characteristics of ReqClarifyBench. 3.1 Benchmark Construction As illustrated in Figure 2, ReqClarifyBench is constructed through a semi-automated human- in-the-loop pipeline, which consists of three stages: source benchmark curation, requirement reconstruction and intent abstraction, and human validation. The pipeline is designed to enable adaptation to new datasets. Each task instance ultimately contains three core components: a high- level vague user intent, a complete and clear requirement document, and the corresponding code evaluation artifacts for assessing the generated code. Stage I: Source benchmark curation. Following [28,45,53,55], we construct ReqClarify- Bench based on existing code generation benchmarks rather than designing tasks from scratch. Existing benchmarks provide high-quality software development scenarios and reliable code eval- uation protocols. Building on existing benchmarks allows us to focus on data construction and evaluation design for the requirement clarification stage. Meanwhile, we can reuse their code eval- uation protocols to measure the impact of requirement clarification on subsequent implementation. , Vol. 1, No. 1, Article . Publication date: October 2026. 6Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, and Pinjia He Stage I: Source Benchmark Curation Stage I: Requirement Reconstruction and Intent Abstraction Stage I: HumanValidation ExistingBenchmarks Criteria ①Real-world Requirements ②Behavioral Evaluation ③Repository-level Complexity GameCraft-Bench WebGen-Bench MiniAppBench Benchmark Artifacts Project Specification Evaluation Rubrics Reference Code Traceability-Grounded Requirement Document High-Level User Intent ... ... ① ② ③ ④ Req. Req. Req. Req. LLM LLM Spec. Rubric Code Spec.+Code < > IntentAbstraction Validation Please help me design a web-based Excel formula generator. Pass？ ReqClarifyBench RequirementDocumentValidation Correctness Unambiguity Consistency Completeness Fig. 2. The overview of ReqClarifyBench construction pipeline. For source Benchmark Curation, we consider three main criteria. 1) Real-world Requirements. The tasks and development scenarios should be grounded in real-world requirements, rather than artificially constructed programming problems. 2) Repository-level Complexity. The tasks should target repository-level code generation rather than simple function completion, reflecting the scale and complexity of real-world software development. 3) Behavioral Evaluation. The benchmark should provide reliable behavioral evaluation. We avoid evaluations that heavily rely on unit tests with rigid interface assumptions, as they may penalize functionally correct implementations that do not conform to predefined APIs or code structures. Based on these criteria, we choose GameCraft-Bench [38], MiniAppBench [57], and WebGen- Bench [36], covering game, application, and web development, respectively. These benchmarks span diverse software domains and provide high-quality repository-level tasks. As shown in Figure 3, they also adopt behavioral evaluation protocols. Specifically, GameCraft-Bench evaluates recorded gameplay videos and frames using a multimodal rubric-based judge; MiniAppBench combines static code analysis with dynamic browser-based evidence collected through Playwright; and WebGen- Bench employs a UI agent to interact with the generated website and evaluates both test-case accuracy and visual appearance. These protocols assess generated software through observable behavior, allowing us to reliably measure the impact of clarification on subsequent phases. Stage I: Requirement Reconstruction and Intent Abstraction. Different benchmarks provide diverse artifacts, including project descriptions, evaluation rubrics, reference code, test cases, etc. To derive a unified requirement representation, we adopt the requirement document as our target format, because structured requirement documents are widely established artifacts in real- world development practices [25, 58]. We leverage an LLM to reconstruct structured requirement documents from these artifacts. This process aims to collect implicit requirements distributed across different artifacts and represent them in an explicit requirement format. To improve the reliability of the reconstructed requirements, we further introduce a require- ment–artifact traceability mechanism. For each reconstructed requirement, the LLM is required to , Vol. 1, No. 1, Article . Publication date: October 2026. Don’t Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development7 GameCraftBenchMiniAppBenchWebGenBench BuildGame Replay MultimodalRubricJudge Video & Frames Instructions DOM Static Dynamic Code Query Eval-Ref Playwright LLM-as-Judge Website AppearanceScore UI Agent Screenshots Testcase Accuracy browse Fig. 3. Behavioral evaluation protocols of the three selected source benchmarks. associate it with the corresponding artifact and record the specific evidence source. This mechanism helps prevent the LLM from introducing unsupported requirements or hallucinated details during reconstruction, while providing explicit evidence for subsequent human validation. After obtaining the detailed requirement documents, we use an LLM to perform intent abstraction. By removing implementation details and preserving core software objectives, we derive high-level user intents that simulate initial user requests in real-world software development scenarios. The high-level user intents serve as the starting point for requirement clarification. Stage I: Human validation. To ensure the reliability of the constructed benchmark, we further conduct human validation. The validation is conducted by two authors. Guided by the requirement quality criteria in ISO/IEC/IEEE 29148:2018 [1], the reconstructed requirement documents are examined for Correctness, Unambiguity, Completeness, and Consistency. Specifically, Correctness assesses whether the requirements accurately capture the intended software functionality and be- havior. Unambiguity checks whether each requirement is clearly stated. Completeness examines whether essential functions, constraints, and behaviors are sufficiently covered. Consistency checks for conflicts across requirements. Requirement–artifact traceability facilitates human validation by allowing annotators to inspect the supporting evidence for each requirement. Requirements found to be incorrect, ambiguous, incomplete, or inconsistent are regenerated and then re-validated. After validating the reconstructed requirement documents, we further examine the abstraction validity of the high-level user intents, ensuring that they preserve the core software objectives. Only samples that pass both validation stages are included in ReqClarifyBench. 3.2 Benchmark Statistics. Table 2 presents the statistics of ReqClarifyBench across its three source benchmarks. In total, ReqClarifyBench contains 341 tasks, comprising 100 tasks from MiniAppBench, 140 from GameCraft-Bench, and 101 from WebGen-Bench. Each task is paired with a human-validated detailed requirement document and its corresponding high-level user intent. Each detailed require- ment document contains 193 atomic requirements and 2,380 tokens on average. In contrast, the corresponding abstracted user intents contain only 13 tokens on average. Notably, this obvious length gap is consistent across all three sources, indicating that intent abstraction removes many details and leaves substantial room for subsequent requirement clarification. 4 Experiment Design 4.1 Interactive Requirement Clarification As illustrated in Figure 4, we formulate requirement clarification as a multi-turn interaction between an LLM and a user. For each ReqClarifyBench task, the LLM initially receives only the user’s high-level intent and proactively asks clarification questions. Based on the interaction history, it decides what to clarify next and when to stop asking questions. It then produces a detailed requirement document based on the information gathered during the interaction. The , Vol. 1, No. 1, Article . Publication date: October 2026. 8Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, and Pinjia He Table 2. Statistics of ReqClarifyBench across its three source benchmarks, including the number of tasks, the average number of atomic requirements per task, and the average token lengths of detailed requirement documents and high-level user intents. “#” denotes the number of items. Source Benchmark # Tasks Avg. # Atomic Req. Avg. Doc Tokens Avg. Intent Tokens MiniAppBench [57]1001571,78110 GameCraft-Bench [38]1402403,52612 WebGen-Bench [36]1011651,38317 ReqClarifyBench3411932,38013 complete interaction, including the clarification questions, user responses, and the final requirement document, is recorded automatically. These records are used to evaluate the LLM’s ability to produce a complete requirement document through interactive clarification. 4.2 LLM-based User Simulator Following prior work [32,39,48,60], we employ an LLM to act as the user to engage in multi- turn requirement clarification with the evaluated LLM. Compared with involving human users in interactions, an LLM-based simulator enables large-scale reproducible experiments with substan- tially lower effort and time costs, while approximating basic user behaviors and producing natural responses under controlled settings [20,41,51,61]. For each task, the LLM-based user simulator is provided with a human-validated detailed requirement document as the ground-truth requirements and answers the clarification questions accordingly. However, ensuring that the LLM-based user simulator’s responses are consistent with the ground- truth requirements is insufficient to simulate realistic user interactions. Real users differ in how they express their needs, disclose information, and revise their requirements [9,34]. To simulate realistic interactions, we introduce representative user personas based on real-world interaction data. Specifically, we build on SWE-chat [7], a large-scale dataset of multi-turn interactions between real software developers and coding agents. SWE-chat categorizes users into four types: Expert Nitpicker, Vague Requester, Mind Changer, and Other. We adopt the first three personas, which together account for approximately 80% of SWE-chat. To better capture their distinct interaction patterns, we further refine each persona based on corresponding real-world interaction trajectories. Expert Nitpickers provide accurate and detailed responses while keeping their goals stable. Vague Requesters describe their needs in broad terms and leave many details unspecified. Mind Changers revise their requirements as the interaction progresses. These personas are then used to guide the LLM-based user simulator to respond throughout the clarification process, producing more diverse and realistic user interactions. To assess whether the user simulator exhibits distinct behavioral patterns corresponding to the intended personas, we conduct a blind human evaluation with two annotators. Specifically, the annotators are unaware of the specific persona assigned and independently classify each complete interaction trajectory as Expert Nitpicker, Vague Requester, Mind Changer, or Other. We randomly sample 60 tasks and obtain interaction trajectories for all three personas, resulting in 180 cases. Following [10,32], we report the average classification accuracy across annotators relative to the assigned personas, and Cohen’s휅[33] for inter-annotator agreement. The resulting accuracy is 95.8% and Cohen’s휅is 0.892, suggesting that the user simulator exhibits distinct behavioral patterns corresponding to the intended personas. , Vol. 1, No. 1, Article . Publication date: October 2026. Don’t Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development9 InteractiveRequirementClarification LLM LLM-based UserSimulator lRequirementDocument Expert Nitpicker Vague Requester Mind Changer Evaluation OK! Please answer these questions: ... Build a task management app. Multi-turn Clarification Vague User intent ... ... Tasks should support due dates and priorities, and overdue items must be highlighted. A simple way to keep track of tasks would be great. Actually, a shared task list would be better than personal reminders. Requirement Document Quality Clarification Effectiveness Requirement Alignment (ReqAlign) BERTScore Effective Question Rate (EQR) AvgClarified Fig. 4. Interactive requirement clarification and evaluation in ReqClarifyBench. 4.3 Evaluation Metrics To comprehensively evaluate the performance of LLMs in interactive requirement clarification, we focus on two complementary dimensions: how effectively LLMs ask clarification questions and requirement document quality. The former focuses on the effectiveness of the interaction process, whereas the latter focuses on the final outcome of clarification. 4.3.1 Clarification Effectiveness. We evaluate how effectively an LLM obtains new requirement information through interactive clarification using the following two metrics: Effective Question Rate (EQR) measures the proportion of questions that clarify at least one ground-truth atomic requirement: EQR= 1 푇 푇 ∑︁ 푡=1 I(푐 푡 > 0),(1) where푇is the total number of clarification questions andI(·)is the indicator function.푐 푡 denotes the number of atomic requirements clarified in the 푡 -th question-response pair. AvgClarified measures the average number of atomic requirements clarified per question: AvgClarified= 1 푇 푇 ∑︁ 푡=1 푐 푡 .(2) 4.3.2Requirement Document Quality. The preceding two metrics evaluate the effectiveness of the interactive clarification process, but do not assess how well the LLM translates the information obtained through clarification into an accurate and complete requirement document. We therefore use two complementary metrics to evaluate requirement document quality: BERTScore is a widely used automatic metric for text generation [56]. It matches tokens in the generated and reference texts using contextualized embeddings and computes semantic precision and recall through bidirectional matching. Following prior work [14], we report BERTScore F1, which combines semantic precision and recall to measure the overall semantic similarity between the generated and ground-truth requirement documents. Requirement Alignment (ReqAlign) measures how well the generated document aligns with the ground-truth requirements at the atomic requirement level. Specifically, we decompose the ground-truth requirement document into a set of atomic requirementsR= 푟 1 ,푟 2 , . . . ,푟 푁 . For each atomic requirement푟 푖 , we use an LLM-as-a-Judge to determine whether it is correctly covered, omitted, or violated in the generated requirement document. We assign a score푠 푖 to each atomic , Vol. 1, No. 1, Article . Publication date: October 2026. 10Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, and Pinjia He Table 3. Studied Large Language Models TypeModelPublisher Size Time Window Length Proprietary GPT-5.6-Sol [43]OpenAI–2026-071.05M Claude-Opus-5 [4]Anthropic–2026-071M Gemini-3.1-Pro [27]Google–2026-021M Open Weight GLM-5.3 [26]Z.ai753B2026-081M DeepSeek-V4-Pro [16]DeepSeek1.6T2026-041M Qwen3-235B-A22B [52]Alibaba235B2025-04128K requirement as follows: 푠 푖 =         1, 푦 푖 = Covered, 0, 푦 푖 = Omitted, −1, 푦 푖 = Violated, (3) where푦 푖 denotes the label assigned to requirement푟 푖 . ReqAlign is then computed by averaging the scores over all atomic requirements: ReqAlign= 1 푁 푁 ∑︁ 푖=1 푠 푖 ,(4) where푁denotes the total number of atomic requirements. ReqAlign ranges from−1 to 1, with a higher score indicating better alignment with the ground-truth requirements. 4.4 Studied LLMs We evaluate six representative LLMs on ReqClarifyBench, including three proprietary models (GPT-5.6-Sol [43], Claude Opus 5 [4], and Gemini-3.1-Pro [27]) and three open-weight models (GLM-5.3 [26], DeepSeek-V4-Pro [16] and Qwen3-235B-A22B [52]). Table 3 summarizes their basic information. To ensure a fair comparison, we vary only the evaluated LLM while keeping all other experimental settings fixed throughout all experiments, including the LLM-based user simulator, task inputs, persona settings, and interaction protocol. 4.5 Implementation Details We use GPT-5.6-Sol as the foundation model of user simulator and set its temperature to 0.7 to allow natural variation in simulated responses following prior works [32,40]. For all evaluated LLMs, we set the temperature to 0.0 to minimize sampling randomness and ensure a fair and stable comparison across models. Similarly, all LLM-as-a-Judge evaluations are conducted with a temper- ature of 0.0 to improve the consistency of automatic judgments, using DeepSeek-V4.1-Flash [15]. Following prior works [32, 46], each requirement clarification session is limited to a maximum of 20 interaction rounds to provide sufficient room for clarification while preventing excessively long interactions. We automatically record the complete interaction history for subsequent analysis and validation. All experimental configurations and parameters will be released together with our source code to facilitate reproducibility and future benchmarking. The choices of models for both the LLM-based user simulator and LLM-as-a-Judge are empirically validated. As described in Section 4.2, human annotators achieve an average persona classification accuracy of 95.8%, with an inter-annotator Cohen’s휅of 0.892, indicating that the GPT-5.6-Sol- based user simulator exhibits distinct behavioral patterns corresponding to the intended personas. For LLM-as-a-Judge, we compare GPT-5.6-Sol, DeepSeek-V4-Pro, and DeepSeek-V4.1-Flash on , Vol. 1, No. 1, Article . Publication date: October 2026. Don’t Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development 11 Table 4. Overall performance of studied LLMs on ReqClarifyBench. Model Clarification EffectivenessRequirement Document Quality EQRAvgClarifiedBERTScore F1ReqAlign EN.VR.MC.Avg.EN.VR.MC.Avg.EN.VR.MC.Avg.EN.VR.MC.Avg. GPT-5.6-Sol0.359 0.080 0.1500.1961.396 0.199 0.3560.6500.786 0.768 0.7680.7740.842 0.681 0.6590.727 Claude Opus 50.463 0.153 0.2160.2772.013 0.432 0.5761.0070.7430.705 0.7070.7180.7200.5040.5070.577 Gemini-3.1-Pro0.517 0.255 0.2290.3343.510 0.842 0.7521.7020.722 0.693 0.6950.7030.442 0.313 0.3170.357 DeepSeek-V4-Pro0.423 0.169 0.2170.2702.022 0.485 0.6491.0520.740 0.707 0.7110.7190.610 0.434 0.4500.498 Qwen3-235B-A22B0.5010.2330.2340.3233.3310.8410.7441.6390.662 0.623 0.6270.6370.424 0.197 0.2260.282 GLM-5.30.425 0.169 0.2030.2662.033 0.497 0.5851.0390.737 0.7080.7070.7170.618 0.471 0.4740.521 * EN., VR., and MC. denote the Expert Nitpicker, Vague Requester, and Mind Changer personas, respectively. Avg. denotes the average performance across the three personas. EQR (Effective Question Rate) is the proportion of questions that clarify at least one ground-truth atomic requirement. AvgClarified is the average number of atomic requirements clarified per question. ReqAlign (Requirement Alignment) measures how well the generated document aligns with ground-truth requirements at the atomic-requirement level. Bold and underlinedvalues indicate the best and second-best results. 200 manually annotated cases. Their judgments achieve accuracies of 92.5%, 90.0%, and 91.5%, respectively, showing comparable performance on this evaluation task. Given its competitive accuracy and lower inference cost, we select DeepSeek-V4.1-Flash as the LLM-as-a-Judge. 5 Evaluation We conduct a comprehensive empirical study on ReqClarifyBench and mainly investigate the following five research questions: •RQ1: Overall performance. We evaluate the overall requirement clarification performance of the studied LLMs on ReqClarifyBench across three representative user personas. • RQ2: LLMs’ clarification behavior. We analyze how LLMs differ in stopping behavior and clarification dynamics across interaction rounds under different user personas. •RQ3: Impact of Clarification on Requirement Document Quality. We quantify the impact of requirement clarification on requirement document quality by comparing documents generated with and without interactive requirement clarification. • RQ4: Effect of coding-agent harness. To assess how coding-agent harnesses affect requirement clarification, we compare standalone LLMs with the same LLMs running within coding-agent harness, while keeping other experimental conditions fixed. •RQ5: Impact on Code Generation. We investigate the impact of clarification and document generation on code implementation, based on the initial high-level intent, directly generated requirement documents, and interactively clarified requirement documents, respectively. 5.1 RQ1: Overall Performance Table 4 presents the overall performance of the six studied LLMs on ReqClarifyBench across three representative personas. Based on the results, we have the following observations. (1) Clarification effectiveness does not necessarily translate into requirement document quality. Different models exhibit inconsistent performance between clarification effectiveness and final requirement document quality. GPT-5.6-Sol achieves the best requirement document quality, with an average BERTScore F1 of 0.774 and an average ReqAlign of 0.727, outperforming the second- best model (Claude Opus 5) by 26.0% in ReqAlign. However, it exhibits the lowest clarification effectiveness, with an average EQR of 0.196 and AvgClarified of 0.650. In contrast, Gemini-3.1-Pro , Vol. 1, No. 1, Article . Publication date: October 2026. 12Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, and Pinjia He performs best during the clarification process, achieving the highest average EQR of 0.334 and AvgClarified of 1.702 among all models. However, its average ReqAlign is only 0.357, substantially lower than the 0.727 achieved by GPT-5.6-Sol. Therefore, clarification effectiveness and requirement document quality reflect two related but distinct capabilities. Effective clarification helps the model elicit important requirement information, whereas producing a high-quality document further requires the model to correctly understand and integrate the information gathered across multiple turns. This process may further depend on the model’s ability to leverage interaction context, domain knowledge, and reasoning. Finding 1: Effective clarification does not necessarily lead to high-quality requirement docu- ments. This finding indicates that asking effective clarification questions is only one important part of requirement clarification, while correctly understanding and integrating information across multiple turns is equally critical. (2) User personas substantially affect requirement clarification performance. All models perform best under the Expert Nitpicker persona, while both requirement document quality and clarification effectiveness decline noticeably when interacting with Vague Requesters and Mind Changers. For ReqAlign, the average score across the six models is approximately 0.609 under Expert Nitpicker, but decreases to 0.433 and 0.439 under Vague Requester and Mind Changer, respectively. In contrast, BERTScore F1 varies much less, decreasing only from 0.732 under Expert Nitpicker to approximately 0.702 under the other two personas. This suggests that user personas have a particularly pronounced impact on fine-grained requirement alignment. The differences are even more evident in clarification effectiveness. Among the six models, the average EQR is approximately 0.448 under Expert Nitpicker, compared with only 0.177 and 0.208 under Vague Requester and Mind Changer, respectively. AvgClarified similarly decreases from 2.384 to 0.549 and 0.610. These results suggest that LLMs remain highly sensitive to user interaction styles. They perform substantially better when users provide explicit and detailed responses, but degrade noticeably when user feedback is vague or when requirements change during the interaction. Finding 2: User personas have a substantial impact on requirement clarification performance, and current LLMs still exhibit noticeable performance degradation when faced with vague feedback or changing requirements. (3) Models exhibit different levels of robustness across user personas. In terms of final requirement document quality, GPT-5.6-Sol shows relatively strong robustness to persona changes. Its ReqAlign decreases from 0.842 under Expert Nitpicker to 0.681 and 0.659 under Vague Requester and Mind Changer, retaining approximately 81% and 78% of its performance, respectively. GLM- 5.3 also shows a relatively stable trend, with ReqAlign decreasing from 0.618 to 0.471 and 0.474, retaining approximately 76% and 77%. In contrast, Qwen3-235B-A22B is the most sensitive to persona changes, with ReqAlign dropping from 0.424 under Expert Nitpicker to 0.197 and 0.226, retaining only approximately 46% and 53% of its performance. Claude Opus 5, Gemini-3.1-Pro, and DeepSeek-V4-Pro fall between these two extremes. These results indicate that models differ substantially in their ability to maintain requirement understanding and integration when user feedback becomes less informative or requirements evolve during the interaction. Finding 3: Models show varying stability across user personas, suggesting that persona robust- ness is an important dimension for distinguishing their requirement clarification capabilities. , Vol. 1, No. 1, Article . Publication date: October 2026. Don’t Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development 13 Expert Nitpicker (EN.)Mind Changer (MC.)Vague Requester (VR.) Mean 05101520 VR. MC. EN. Claude-Opus-5 05101520 GPT-5.6-Sol 0510 Gemini-3.1-Pro 05101520 VR. MC. EN. End Round GLM-5.3 05101520 End Round DeepSeek-V4-Pro 0510 End Round Qwen3-235B-A22B Fig. 5. Distribution of stopping rounds across the six evaluated LLMs under three user personas. Boxes show the interquartile range, center lines denote the median, and diamonds indicate the mean stopping round. 5.2 RQ2: LLMs’ Clarification Behavior To delve into the behavioral patterns of LLMs during interactive requirement clarification, we further analyze their stopping behavior and clarification dynamics across interaction rounds under different user personas. 5.2.1 Stopping Behavior. Figure 5 illustrates the stopping behavior of the evaluated LLMs under the three personas (i.e., Expert Nitpicker, Mind Changer and Vague Requester). For each task, we record the number of interaction rounds and visualize the distributions using box plots. The left and right boundaries of each box correspond to the first quartile (Q1) and third quartile (Q3), respectively, while the vertical line inside the box denotes the median. The box covers the middle 50% of the distribution and the diamonds denote the mean number of interaction rounds. These distributions allow us to compare how stopping behavior varies across both personas and models. Overall, the stopping rounds exhibit a relatively consistent trend across the three personas. Interactions with the Expert Nitpicker tend to end earliest, followed by the Mind Changer, whereas interactions with the Vague Requester generally continue for more rounds. This suggests that user interaction style affects more than the quality of the requirement document and clarification effectiveness. It also influences when a model considers the requirements sufficiently clarified and terminates the interaction. However, the extent of variation across personas differs among LLMs. This variation is particularly pronounced for GPT-5.6-Sol and Claude-Opus-5. GPT-5.6-Sol typically terminates after only about 2–3 clarification rounds under the Expert Nitpicker, but increases to around 6 rounds under the Mind Changer and 14–15 rounds under the Vague Requester, with some interactions approaching the maximum round limit. Claude-Opus-5 exhibits a similarly pronounced pattern. DeepSeek-V4-Pro, GLM-5.3, and Qwen3-235B-A22B also exhibit the same general ordering across personas, but with more moderate differences. In contrast, Gemini-3.1-Pro remains concentrated at around 2–3 rounds across all three personas, showing the least variation. Overall, these results show that LLMs differ not only in their general stopping tendencies but also in how strongly they adjust the duration of clarification across user interaction styles. , Vol. 1, No. 1, Article . Publication date: October 2026. 14Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, and Pinjia He Expert Nitpicker (EN.)Vague Requester (VR.)Mind Changer (MC.) 0 0.2 0.4 0.6 EQR Claude-Opus-5 DeepSeek-V4-ProGemini-3.1-ProGLM-5.3GPT-5.6-SolQwen3-235B-A22B 14711 0 2 4 Round AvgClarified 1357 Round 123 Round 136 Round 161118 Round 136 Round Fig. 6. Clarification dynamics across interaction rounds for different models and user personas. For each model–persona pair, EQR and AvgClarified are reported only up to the third quartile (Q3) of the stopping- round distribution to avoid sparsely populated tail rounds. Finding 4: Across models, interactions tend to end after the fewest turns with the Expert Nitpicker and the most turns with the Vague Requester, although the size of this difference varies substantially by model. 5.2.2Clarification Dynamics. As shown in Figure 6, we further analyze how clarification effective- ness evolves across interaction rounds for different models and user personas. Unlike Section 5.2.1, which focuses on when models terminate clarification, this analysis investigates how clarifica- tion effectiveness changes round by round before termination. Specifically, we compute EQR and AvgClarified at each interaction round. Because tasks terminate at different rounds, the number of active tasks decreases as the interaction progresses, making statistics at very late rounds less representative. To mitigate this issue, for each model–persona pair, we report round-wise results only up to the third quartile (Q3) of its stopping-round distribution. This avoids drawing conclusions from sparsely populated tail rounds while retaining all tasks in the analysis. The six evaluated models generally exhibit similar trends across the three personas. For the Expert Nitpicker, most requirement information is obtained within the first few rounds. EQR and AvgClarified are relatively high early in the interaction, and the models typically stop after only a few rounds. This can be characterized as a short but information-intensive clarification process. In contrast, interactions with the Mind Changer tend to last longer, with EQR and AvgClarified often showing a noticeable increase after several rounds. This suggests that new requirement information is not concentrated at the beginning, but often emerges later as the user revises or supplements the requirements. For the Vague Requester, EQR and AvgClarified generally remain low across interaction rounds, and the models tend to continue asking clarification questions for more rounds. This pattern can be characterized as a low-yield but sustained clarification process, in which requirement information accumulates gradually over multiple interactions. Under the Vague Requester, the models broadly exhibit three representative clarification dynamics. Gemini-3.1-Pro and Qwen3-235B-A22B achieve relatively high per-round clarification gains, with both EQR and AvgClarified generally higher than those of the other models. Meanwhile, their interactions also terminate relatively early, particularly for Gemini-3.1-Pro, whose stopping- round Q3 is only 3. In contrast, Claude-Opus-5, DeepSeek-V4-Pro, and GLM-5.3 maintain relatively , Vol. 1, No. 1, Article . Publication date: October 2026. Don’t Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development 15 GPT-5.6-Sol Claude Opus 5 Gemini-3.1-Pro DeepSeek-V4-Pro Qwen3-235B-A22B GLM-5.3 0 0.5 1 Performance BERTScore F1: w/o Clarification BERTScore F1: Clarification Gain ReqAlign: w/o Clarification ReqAlign: Clarification Gain Fig. 7. Effect of clarification on BERTScore F1 and ReqAlign across different models. low values on both metrics, exhibiting a comparatively stable low-gain clarification process, with stopping-round Q3 values ranging from approximately 6 to 11. GPT-5.6-Sol also remains in a low-gain state for much of the interaction, but unlike the other low-gain models, it continues clarification for substantially more rounds, with a stopping-round Q3 of 18. These results indicate that models differ not only in their ability to obtain useful requirement information within each round, but also in their stopping behavior under sustained low-gain interactions. Finding 5: Different personas lead to distinct clarification dynamics that are generally consistent across models. Expert Nitpicker interactions are short but information-intensive, Mind Changer interactions tend to reveal new information later, and Vague Requester interactions accumulate information gradually over more rounds. Under vague requests, models further differ in their per-round information gains and stopping behavior. 5.3 RQ3: Impact of Clarification on Requirement Document Quality As shown in Figure 7, we investigate the impact of interactive clarification on final requirement document quality by comparing two settings: directly generating requirement documents from initial user intents and generating them after multi-turn requirement clarification. Overall, all models benefit from requirement clarification, producing more complete and clear requirement documents from vague user intents. Specifically, the average BERTScore F1 across the six models increases from 0.664 to 0.712, with a 7.2% relative improvement. Additionally, the improvement in fine-grained requirement alignment is substantially more pronounced. The average ReqAlign score rises from 0.212 to 0.494 with a relative improvement of 133.1%. This trend is also consistent across different models, with relative improvements ranging from 83.8% (GLM-5.3) to 241.3% (Qwen3-235B-A22B). The substantially larger improvement in ReqAlign suggests that clarification particularly helps models capture fine-grained requirements that are missing from the initial user intent. Notably, GPT-5.6-Sol achieves the highest ReqAlign both with and without interactive clarification, demonstrating its consistently strong requirement document generation capability across the two settings. Finding 6: Requirement clarification consistently improves requirement document quality across all models, with substantially larger gains in fine-grained requirement alignment than in overall semantic similarity. GPT-5.6-Sol further maintains the strongest ReqAlign performance both with and without clarification. 5.4 RQ4: Effect of Coding-Agent Harness As shown in Figure 8, we further compare Claude-Opus-5 and GPT-5.6-Sol with their corre- sponding agent-harness counterparts (i.e., Claude Code [3] and Codex [42]), in terms of both , Vol. 1, No. 1, Article . Publication date: October 2026. 16Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, and Pinjia He Claude-Opus-5Claude-Code-Opus-5 EN. VR. MC. 0 0.2 0.4 w/o clari. (a) EQR EN. VR. MC. 0 1 2 3 w/o clari. (b) AvgClarified EN. VR. MC. w/o clari. 0 0.2 0.4 0.6 0.8 1 w/o clari. (c) BERTScore F1 EN. VR. MC. w/o clari. 0 0.2 0.4 0.6 0.8 1 w/o clari. (d) ReqAlign GPT-5.6-SolCodex-GPT-5.6-Sol EN. VR. MC. 0 0.2 0.4 w/o clari. (e) EQR EN. VR. MC. 0 1 2 3 w/o clari. (f ) AvgClarified EN. VR. MC. w/o clari. 0 0.2 0.4 0.6 0.8 1 w/o clari. (g) BERTScore F1 EN. VR. MC. w/o clari. 0 0.2 0.4 0.6 0.8 1 w/o clari. (h) ReqAlign Fig. 8. Comparison of requirement-document quality and clarification effectiveness between base models and their coding-agent harness counterparts. EN., VR., and MC. denote Expert Nitpicker, Vague Requester, and Mind Changer, respectively. w/o clari. denotes directly generating the requirement document without interactive requirement clarification. clarification effectiveness and final requirement-document quality. In practical repository-level software development, developers increasingly interact with coding agents such as Claude Code and Codex rather than directly prompting base LLMs. These agent harnesses introduce additional scaffolding for task execution and software development workflows, which may also affect how models clarify and organize user requirements before implementation. Overall, agent harnesses improve clarification effectiveness, but this improvement does not translate into higher-quality requirement documents. Across the three personas, Claude Code and Codex consistently achieve higher EQR and AvgClarified than their corresponding base models. This indicates that agent harnesses help models ask more effective clarification questions through each interaction. However, the opposite trend is observed in the final requirement documents. Claude-Opus-5 and GPT-5.6-Sol generally achieve higher BERTScore F1 and ReqAlign than their agent-harness counterparts, with the gap being particularly pronounced for ReqAlign. For example, under the Expert Nitpicker, ReqAlign decreases by 15.2% for Claude and 24.0% for GPT when using coding-agent harnesses. The same trend is observed for Vague Requester and Mind Changer. These results reveal a trade-off introduced by coding-oriented agent harnesses. While they facilitate more effective requirement clarification, their scaffolding may compromise some of the base model’s capabilities needed to produce complete and accurate requirement documents. Finding 7: Coding-oriented agent harnesses improve clarification effectiveness, but this gain is accompanied by lower requirement-document quality than their corresponding base models, suggesting a trade-off between interactive clarification and final document generation. , Vol. 1, No. 1, Article . Publication date: October 2026. Don’t Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development 17 Table 5. Code-generation performance across three source benchmarks under three input settings: Intent (the initial user intent), Direct Doc. (a requirement document generated directly from the initial intent), and Clarified Doc. (a requirement document generated after interactive clarification). Benchmark Score Intent Direct Doc. Clarified Doc. GameCraft-Bench0.0120.2730.418 MiniAppBench0.4510.5430.943 WebGen-Bench 0.2710.3840.659 5.5 RQ5: Impact on Code Generation To further investigate the impact of requirement clarification on subsequent code generation, we conduct code-generation experiments on the three source benchmarks of ReqClarifyBench: GameCraft-Bench, MiniAppBench, and WebGen-Bench. We follow the original evaluation protocols of each benchmark. We use Codex with GPT-5.6-Sol to generate code under three requirement settings: the initial intent, the directly generated requirement document, and the interactively clarified requirement document. Both types of requirement documents are generated by GPT-5.6- Sol, and scores are averaged across the Expert, Vague, and Mind Changer personas. We compare performance across these settings to assess the effects of requirement documentation and interactive clarification on code generation. Table 5 summarizes the score results across the three benchmarks. Overall, requirement documentation itself improves code-generation performance, while interactive clarification provides further gains on top of it. Across all three source benchmarks, code-generation performance consistently improves when moving from the initial intent to a directly generated requirement document and then to an interactively clarified require- ment document. Even without any additional user feedback, guiding the LLM to first transform the initial intent into a requirement document and then implement based on that document is more effective than generating code directly from the initial intent. The score increases from 0.012 to 0.273 on GameCraft-Bench, from 0.451 to 0.543 on MiniAppBench, and from 0.271 to 0.384 on WebGen-Bench. Furthermore, interactive clarification leads to additional improvements on all three benchmarks. This consistent trend shows that the value of interactive clarification extends beyond improving requirement-document quality. The requirement information clarified through multi-turn interaction can further translate into tangible gains in subsequent code implementation. Finding 8: Requirement documentation improves code-generation performance even without additional user feedback, while interactive requirement clarification provides further gains across all three source benchmarks. 6 Discussion 6.1 Practical Implications (1) LLMs should proactively clarify requirements before implementation. In practice, developers often provide high-level and underspecified requests. Rather than proceeding directly to implementation and filling in missing details based on their own assumptions, LLMs should first identify unclear or missing requirements and actively ask developers for clarification. Our study shows that such interactive clarification can elicit additional requirement information that ultimately improves subsequent code-generation performance. This suggests that requirement clarification should be treated as an explicit capability of LLMs, rather than leaving the burden of fully specifying requirements to developers in the initial request. , Vol. 1, No. 1, Article . Publication date: October 2026. 18Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, and Pinjia He (2) Requirement documentation is beneficial even without additional user feedback. We find that guiding the LLM to first transform the initial intent into an explicit requirement document, and then generate code based on that document, consistently outperforms direct code generation from the initial intent. This suggests that a requirement document is not merely a record of clarification outcomes, but also serves as an important intermediate representation between user intent and code implementation. In practice, even when further clarification with the developer is unavailable, introducing an explicit requirement-document generation stage before coding can still improve subsequent implementation compared with directly using the original request. (3) Requirement engineering and code implementation may be better handled by differ- ent forms of model systems. We observe that coding-oriented agent harnesses such as Claude Code and Codex ask more effective clarification questions, yet produce lower-quality final require- ment documents than their corresponding base models. This suggests that scaffolding designed for code execution may not be equally well suited to requirement-document generation. A more suitable workflow may therefore decouple the requirement and implementation stages. Developers can first use a base model for requirement analysis, integration of clarified information, and final requirement-document generation, and then pass the resulting document to coding-agent harnesses such as Claude Code or Codex for repository-level implementation. Such a staged workflow can leverage the respective strengths of base models in requirement documentation and coding-agent harnesses in code execution and software-development tooling, rather than relying on a single agent to handle the entire process from underspecified requests to final implementation. 6.2 Threats to Validity External Validity. A primary threat to external validity is the reliability of user simulation. Real developers vary substantially in how they express requirements, provide information, and revise their needs, and an LLM-based user simulator may not fully capture this diversity. To mitigate this threat, we derive three representative user personas from SWE-chat [7] to guide the LLM-based user simulator. SWE-chat is a large-scale dataset of multi-turn conversations between software developers and coding agents, reflecting diverse interaction behaviors in real-world development settings. We also conduct human validation to verify that the user simulator exhibits distinct behavioral patterns corresponding to the intended personas. Internal Validity. A potential threat to internal validity arises from possible errors in the interaction protocol, prompt design, model configuration, and experimental execution. Such errors may affect the observed clarification behavior or subsequent code-generation evaluation. To reduce this risk, we use a consistent interaction protocol and standardized prompt templates across all evaluated models and settings, and keep the experimental configurations fixed whenever possible. In addition, we have open-sourced our implementation, experimental data, and all prompt templates, enabling external validation and community feedback for continuous improvement. Construct Validity. A primary threat to construct validity is the quality of the ground-truth requirement documents in ReqClarifyBench. To mitigate this threat, two authors manually validate the reconstructed requirement documents by examining their correctness, unambiguity, completeness, and consistency. Requirement–artifact traceability further helps the two validators verify the implementation evidence supporting each requirement. Requirements that fail the validation are regenerated and re-examined. We also validate the abstraction validity of the high- level user intents to ensure that they preserve the core software objectives. Only samples that pass both stages of human validation are included in ReqClarifyBench. , Vol. 1, No. 1, Article . Publication date: October 2026. Don’t Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development 19 7 Conclusion In this paper, we propose ReqClarifyBench, the first interactive requirement clarification benchmark for repository-level software development, which aims to evaluate LLMs’ ability to transform vague user intents into clear and complete requirement documents through multi-turn interactions. To better simulate realistic interactions, we further employ LLM-based user simulators with three representative personas. Based on ReqClarifyBench, we conduct a systematic empirical study. The results reveal that clarification effectiveness does not necessarily lead to high-quality requirement documents, and that model performance varies substantially across different user interaction styles. We also find that coding-agent harnesses do not further enhance the quality of derived requirement documents. These findings offer practical implications for developers using LLMs for software development. More broadly, our work highlights that software development agents’ autonomy should encompass understanding user intent through proactive interaction and aligning development goals accordingly. This offers a user-centered perspective on the design and evaluation of future autonomous development systems. Data Availability The code and data are available at [2]. References [1]2018. ISO/IEC/IEEE International Standard - Systems and software engineering – Life cycle processes – Requirements engineering. ISO/IEC/IEEE 29148:2018(E) (2018), 1–104. doi:10.1109/IEEESTD.2018.8559686 [2] 2026. https://zenodo.org/records/23031774. [3] Anthropic. [n. d.]. Claude Code. https://code.claude.com/docs/en/overview. [4] Anthropic. 2026. Claude Opus 5 System Card. https://w.anthropic.com/system-cards [5] Anthropic. 2026. Claude Opus 5.5 System Card. Technical Report. Anthropic. https://anthropic.com/claude-opus-5-5- system-card [6]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. doi:10.48550/arXiv.2108.07732 [7] Joachim Baumann, Vishakh Padmakumar, Xiang Li, John Yang, Diyi Yang, and Sanmi Koyejo. 2026. SWE-chat: Coding Agent Interactions From Real Users in the Wild. arXiv:2604.20779 [cs.AI] doi:10.48550/arXiv.2604.20779 [8]Brooks. 1987. No Silver Bullet Essence and Accidents of Software Engineering. Computer 20, 4 (1987), 10–19. doi:10.1109/MC.1987.1663532 [9] Corentin Burnay, Ivan J. Jureta, and Stéphane Faulkner. 2014. What stakeholders will or will not say: A theoretical and empirical study of topic importance in Requirements Engineering elicitation interviews. Information Systems 46 (2014), 61–81. doi:10.1016/j.is.2014.05.006 [10] Louis Castricato, Nathan Lile, Rafael Rafailov, Jan-Philipp Fränken, and Chelsea Finn. 2025. PERSONA: A Reproducible Testbed for Pluralistic Alignment. In Proceedings of the 31st International Conference on Computational Linguistics, Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (Eds.). Association for Computational Linguistics, Abu Dhabi, UAE, 11348–11368. https://aclanthology.org/2025.coling- main.752/ [11]Kaushal Chari and Manish Agrawal. 2018. Impact of incorrect and new requirements on waterfall software project outcomes. Empirical Software Engineering 23 (02 2018). doi:10.1007/s10664-017-9506-4 [12] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG] doi:10.48550/arXiv.2107.03374 , Vol. 1, No. 1, Article . Publication date: October 2026. 20Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, and Pinjia He [13]Mingyu Chen, Yakun Zhang, Zihao Xie, Yixing Luo, Jinrui Xu, Cuiyun Gao, Kaiqi Zhao, and Yunming Ye. 2026. TraceDev: A Traceability-Driven Multi-agent Framework for Requirement-to-Code Development. Proceedings of the ACM on Software Engineering 3, ISSTA (2026). doi:10.1145/3832171 [14]Gabriele De Vito, Sergio Di Martino, Filomena Ferrucci, Carmine Gravino, and Fabio Palomba. 2025. LLM-Based Automation of COSMIC Functional Size Measurement From Use Cases. IEEE Trans. Softw. Eng. 51, 5 (May 2025), 1500–1523. doi:10.1109/TSE.2025.3554562 [15] DeepSeek-AI. 2026. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression. arXiv:2609.19969 https: //arxiv.org/abs/2609.19969 [16]DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, et al.2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv preprint arXiv:2606.19348 (2026). doi:10.48550/arXiv.2606.19348 [17]Gangda Deng, Zhaoling Chen, Zhongming Yu, Haoyang Fan, Yuhong Liu, Yuxin Yang, Dhruv Parikh, Rajgopal Kannan, Le Cong, Mengdi Wang, Qian Zhang, Viktor Prasanna, Xiangru Tang, and Xingyao Wang. 2026. SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution. In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=HeRPNf Jaod [18]Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa R Kundurthy, Sean M. Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. 2026. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?. In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=uEVTdoAbnK [19]Jingzhe Ding, Shengda Long, Changxin PU, Ge Zhang, zhou huan, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, FEI HU, Zhaojian Li, Weiran Shi, Zaiyuan Wang, Daoguang Zan, Chenchen Zhang, Xiaoxu Zhang, Chen Qizhi, Xianfu Cheng, Bo Deng, Qingshui Gu, Kai Hua, Juntao Lin, Pai Liu, Mingchen Li, Minghao Li, Xuanguang Pan, Zifan Peng, Yujia Qin, Yong Shan, Zhewen Tan, Haoran Wang, Zihan Wang, Weihao Xie, Yishuo Yuan, Jiayu Zhang, Yunfei Zhao, He Zhu, LIYA ZHU, Chenyang Zou, Ming Ding, Jianpeng Jiao, Jiaheng Liu, Minghao Liu, Qian Liu, Chongyang Tao, Jian Yang, Tong Yang, Zhaoxiang Zhang, Xinjie Chen, and Wenhao Huang. 2026. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents. In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=wqQam1muOQ [20] Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, and Jianfeng Gao. 2025. SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 35212–35290. doi:10.18653/v1/2025.emnlp-main.1786 [21]Zheng Fang, Dongming Jin, Yihong Dong, Yongmin Li, Kechi Zhang, Zhi Jin, and Ge Li. 2026. ClarifyCodeBench: Evaluating LLMs on Clarifying Ambiguous Requirements for Code Generation. doi:10.48550/arXiv.2607.00711 [22]Alessio Ferrari, Paola Spoletini, and Sourav Debnath. 2022. How do requirements evolve during elicitation? An empirical study combining interviews and app store analysis. Requirements Engineering 27, 4 (2022), 489–519. doi:10.1007/s00766- 022-00383-7 [23]Jannik Fischbach, Julian Frattini, Daniel Méndez Fernández, Michael Unterkalmsteiner, Henning Femmer, and Andreas Vogelsang. 2021. How Do Practitioners Interpret Conditionals in Requirements? doi:10.48550/arXiv.2109.02063 [24] Julian Frattini, Lloyd Montgomery, Jannik Fischbach, Daniel Mendez, Davide Fucci, and Michael Unterkalmsteiner. 2023. Requirements quality research: a harmonized theory, evaluation, and roadmap. Requirements engineering 28, 4 (2023), 507–520. doi:10.1007/s00766-023-00405-y [25] Lingyue Fu, Bolun Zhang, Hao Guan, Yaoming Zhu, Lin Qiu, Weiwen Liu, Xuezhi Cao, Xunliang Cai, Weinan Zhang, and Yong Yu. [n. d.]. Automatically Benchmarking LLM Code Agents through Agent-driven Annotation and Evaluation. In Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS). International Foundation for Autonomous Agents and Multiagent Systems. doi:10.65109/hjfb4234 [26]GLM-5-Team, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, et al.2026. GLM-5: from Vibe Coding to Agentic Engineering. arXiv preprint arXiv:2602.15763 (2026). doi:10.48550/arXiv.2602.15763 [27] Google DeepMind. 2026. Gemini 3.1 Pro: Model Card. https://deepmind.google/models/model-cards/gemini-3-1-pro/ [28]Hojae Han, Jaejin Kim, Jaeseok Yoo, Youngwon Lee, and Seung-won Hwang. 2024. ArchCode: Incorporating Software Requirements in Code Generation with Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 13520–13552. doi:10.18653/v1/2024.acl-long.730 [29]Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VtmBAGCN7o , Vol. 1, No. 1, Article . Publication date: October 2026. Don’t Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development 21 [30]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id= chf JJYC3iL [31]Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66 [32]Dongming Jin, Zhi Jin, Zheng Fang, Linyu Li, XiaoTian Yang, Yuanpeng He, and Xiaohong Chen. 2026. Reqelicitgym: An evaluation environment for interview competence in conversational requirements elicitation. arXiv preprint arXiv:2602.18306 (2026). doi:10.48550/arXiv.2602.18306 [33] Jacob Kohen. 1960. A coefficient of agreement for nominal scale. Educ Psychol Meas 20 (1960), 37–46. [34] Viviane Laporti, Marcos R. S. Borges, and Vanessa Braganholo. 2009. Athena: A collaborative approach to requirements elicitation. Computers in Industry 60, 6 (2009), 367–380. doi:10.1016/j.compind.2009.02.011 [35]Xu Lu, Weisong Sun, Yiran Zhang, Ming Hu, Cong Tian, Zhi Jin, and Yang Liu. 2025. Requirements Development and Formalization for Reliable Code Generation: A Multi-Agent Vision. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). 3932–3937. doi:10.1109/ASE63991.2025.00345 [36] Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2026. WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=q2VpjD7k1V [37]Jane Luo, Xin Zhang, Steven Liu, Jie Wu, Jianfeng Liu, Yiming Huang, Yangyu Huang, Chengyu Yin, Ying Xin, Yuefeng Zhan, Hao Sun, Qi Chen, Scarlett Li, and Mao Yang. 2026. RPG: A Repository Planning Graph for Unified and Scalable Codebase Generation. In The Fourteenth International Conference on Learning Representations. https: //openreview.net/forum?id=VAQq3Y8tIF [38]Tongxu Luo, Rongsheng Wang, Jiaxi Bi, Chenming Xu, Zhengyang Tang, Jianlong Chen, Juhao Liang, Ke Ji, Shuqi Guo, Yuhao Du, Fan Bu, Wenyu Du, Xiaotong Zhang, Kyle Li, Shaobo Wang, Linfeng Zhang, Yuxuan Liu, Xin Lai, Chenxin Li, Yiduo Guo, Zhexin Zhang, Xinyuan Wang, Tianyi Bai, Ziniu Li, and Benyou Wang. 2026. GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine? arXiv:2606.17861 [cs.CL] doi:10.48550/arXiv.2606.17861 [39] Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. Clarifygpt: A framework for enhancing llm-based code generation via requirements clarification. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2332–2354. doi:10.1145/3660810 [40]Itay Nakash, George Kour, and Ateret Anaby Tavor. 2026. Efficient Agent Evaluation via Diversity-Guided User Simulation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), Yunyao Li, Georg Rehm, and Mei Tu (Eds.). Association for Computational Linguistics, San Diego, California, USA, 1627–1648. doi:10.18653/v1/2026.acl-industry.112 [41]Bo Ni, Yu Wang, Leyao Wang, Branislav Kveton, Franck Dernoncourt, Yu Xia, Hongjie Chen, Reuben Luera, Samyadeep Basu, Subhojyoti Mukherjee, Puneet Mathur, Nesreen K. Ahmed, Junda Wu, Li Li, Huixin Zhang, Ruiyi Zhang, Tong Yu, Sungchul Kim, Jiuxiang Gu, Zhengzhong Tu, Alexa Siu, Zichao Wang, Seunghyun Yoon, Nedim Lipka, Namyong Park, Zihao Lin, Trung Bui, Yue Zhao, Tyler Derr, and Ryan A. Rossi. 2026. A Survey on LLM-based Conversational User Simulation. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Vera Demberg, Kentaro Inui, and Lluís Marquez (Eds.). Association for Computational Linguistics, Rabat, Morocco, 4266–4301. doi:10.18653/v1/2026.eacl-long.200 [42] OpenAI. [n. d.]. Codex CLI. https://developers.openai.com/codex/cli. [43] OpenAI. 2026. GPT-5.6 System Card. https://deploymentsafety.openai.com/gpt-5-6 [44] Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ChatDev: Communicative Agents for Software Development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 15174–15186. doi:10.18653/v1/2024.acl-long.810 [45]Ruizhong Qiu, Weiliang Will Zeng, James Ezick, Christopher Lott, and Hanghang Tong. 2025. How efficient is LLM-generated code? A rigorous & high-standard benchmark. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=suz4utPr9Y [46]Tejas Srinivasan, Shikib Mehri, Nandita Shankar Naik, Anirban Das, William M. Campbell, and Jesse Thomason. 2026. AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics. arXiv:2608.27818 [cs.AI] https://arxiv.org/abs/2608.27818 [47]Jonathan Ullrich, Matthias Koch, and Andreas Vogelsang. 2025. From Requirements to Code: Understanding Developer Practices in LLM-Assisted Software Engineering. In 2025 IEEE 33rd International Requirements Engineering Conference , Vol. 1, No. 1, Article . Publication date: October 2026. 22Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, and Pinjia He (RE). 257–266. doi:10.1109/RE63999.2025.00032 [48]Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, and Graham Neubig. 2026. Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=X2yzXtH4wp [49]Bingyang Wei. 2024. Requirements are All You Need: From Requirements to Code with LLMs. In 2024 IEEE 32nd International Requirements Engineering Conference (RE). 416–422. doi:10.1109/RE59067.2024.00049 [50] Jie JW Wu, Manav Chaudhary, Davit Abrahamyan, Arhaan Khaku, Anjiang Wei, and Fatemeh H. Fard. 2025. Can Code Language Models Learn Clarification-Seeking Behaviors? arXiv:2504.16331 [cs.SE] doi:10.48550/arXiv.2504.16331 [51]Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. 2025. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. arXiv:2412.14161 [cs.CL] doi:10.48550/arXiv.2412.14161 [52] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388 (2025). doi:10.48550/arXiv.2505.09388 [53]Di Yang, Xinou Xie, Xiuwen Yang, Ming Hu, Yihao Huang, Yueling Zhang, Weikai Miao, Ting Su, Chengcheng Wan, and Geguang Pu. 2026. Assessing the Impact of Requirement Ambiguity on LLM-based Function-Level Code Generation. arXiv preprint arXiv:2604.21505 (2026). doi:10.48550/arXiv.2604.21505 [54] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-Agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Neural Information Processing Systems. doi:10.52202/079017-1601 [55]John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, Diyi Yang, Sida Wang, and Ofir Press. 2025. SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=riTiq3i21b [56]Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations. https://mlanthology.org/iclr/2020/ zhang2020iclr-bertscore/ [57] Zuhao zhang, Chengyue Yu, Yuante Li, Chenyi Zhuang, Linjian Mo, and Shuai Li. 2026. MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistants. In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=pwbLmew1aq [58]Enyu Zhou, Zhiheng Xi, Long Ma, Zhihao Zhang, Shihan Dou, Zhikai Lei, Guoteng Wang, Rui Zheng, Hang Yan, Tao Gui, Qi Zhang, and Xuanjing Huang. 2026. Steering LLMs via Scalable Interactive Oversight. arXiv:2602.04210 [cs.AI] doi:10.48550/arXiv.2602.04210 [59]Qixing Zhou, JiaCheng Zhang, Haiyang Wang, Rui Hao, Jiahe Wang, Minghao Han, Yuxue Yang, Shuzhe Wu, Feiyang Pan, Lue Fan, Dandan Tu, and Zhaoxiang Zhang. 2026. FeatureBench: Benchmarking Agentic Coding for Complex Feature Development. In The Fourteenth International Conference on Learning Representations. https://openreview.net/ forum?id=41xrZ3uGuI [60] Xuhui Zhou, Valerie Chen, Zora Zhiruo Wang, Graham Neubig, Maarten Sap, and Xingyao Wang. 2026. TOM-SWE: User Mental Modeling For Software Engineering Agents. In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=HeLa76rJuD [61] Xuhui Zhou, Hyunwoo Kim, Faeze Brahman, Liwei Jiang, Hao Zhu, Ximing Lu, Frank Xu, Bill Yuchen Lin, Yejin Choi, Niloofar Mireshghallah, Ronan Le Bras, and Maarten Sap. 2025. HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactions. arXiv:2409.16427 [cs.AI] doi:10.48550/arXiv.2409.16427 , Vol. 1, No. 1, Article . Publication date: October 2026.

Instant research discovery