Paper deep dive

SWE-WebDevBench: Evaluating Coding Agent Application Platforms as Virtual Software Agencies

Siddhant Saxena, Nilesh Trivedi, Vinayaka Jyothi

Year: 2026Venue: arXiv preprintArea: cs.MAType: PreprintEmbeddings: 107

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 7/8/2026, 9:57:51 AM

Summary

The paper introduces SWE-WebDev Bench, a comprehensive 68-metric evaluation framework designed to assess AI coding agent platforms as virtual software agencies. It evaluates platforms across three orthogonal dimensions: Interaction Mode (App Creation vs. Modification), Agency Angle (Product Manager, Engineering, Operations), and Complexity Tier (T4 SaaS, T5 AI-native). The framework assesses six commercial platforms across three business domains, revealing four key shortcomings: specification bottlenecks, frontend-backend decoupling, steep production-readiness cliffs, and widespread security/infrastructure failures. The authors release the benchmark to enable rigorous, reproducible evaluation of vibe coding platforms beyond traditional code-level metrics.

Entities (15)

SWE-WebDev Bench · framework · 98%Lovable · platform · 95%Replit Agent · platform · 95%Vibe Coding · paradigm · 95%QwikBuild · organization · 93%App Creation Request (ACR) · concept · 92%App Modification Request (AMR) · concept · 92%Engineering · role · 90%HumanEval · benchmark · 90%Operations (Ops) · role · 90%Product Manager (PM) · role · 90%SWE-BENCH · benchmark · 90%T4 Complexity Tier · concept · 88%T5 Complexity Tier · concept · 88%SUSVIBES · benchmark · 85%

Relation Signals (10)

SWE-WebDev Bench → evaluates → AI coding agent platforms

confidence 95% · assess them as virtual software development agencies... we introduce SWE-WebDev Bench

SWE-WebDev Bench → identifiesshortcoming → Frontend-backend decoupling

confidence 93% · (2) A pervasive frontend-backend decoupling, where visually polished UIs mask absent or broken backend infrastructure

SWE-WebDev Bench → identifiesshortcoming → Specification bottleneck

confidence 93% · reveals four recurring shortcomings... (1) A specification bottleneck, where platforms compress rich business requirements into oversimplified technical plans

SWE-WebDev Bench → identifiesshortcoming → Production-readiness cliff

confidence 93% · (3) A steep production-readiness cliff, where no platform scores above 60% on engineering quality

SWE-WebDev Bench → identifiesshortcoming → Security and infrastructure failures

confidence 93% · (4) Widespread security and infrastructure failures, with no platform exceeding 65% Security Score

SWE-WebDev Bench → measures → Business Intent Fidelity

confidence 92% · G1: Specification Fidelity measures whether the platform understood what the user asked for... Business Intent Fidelity (BIF)

SWE-WebDev Bench → organizesalong → Interaction Mode

confidence 90% · organized along three dimensions: Interaction Mode (App Creation Request (ACR) vs. App Modification Request (AMR))

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The emergence of "vibe coding" platforms, where users describe applications in natural language and AI agents autonomously generate full-stack software, has created a need for rigorous evaluation beyond code-level benchmarks. In order to assess them as virtual software development agencies on understanding business requirements, making architectural decisions, writing production code, handling iterative modifications, and maintaining business readiness, we introduce SWE-WebDev Bench, a 68-metric evaluation framework spanning 25 primary and 43 diagnostic metrics across seven groups, organized along three dimensions: Interaction Mode (App Creation Request (ACR) vs. App Modification Request (AMR)), Agency Angle (Product Manager (PM), Engineering, Ops), and Complexity Tier (T4 multi-role SaaS, T5 AI-native). Our evaluation (six platforms, three domains, 18 evaluation cells) reveals four recurring shortcomings in the current generation of AI app builders: (1) A specification bottleneck, where platforms compress rich business requirements into oversimplified technical plans, (2) A pervasive frontend-backend decoupling, where visually polished UIs mask absent or broken backend infrastructure, (3) A steep production-readiness cliff, where no platform scores above 60% on engineering quality and post-generation human effort varies substantially across platforms and (4) Widespread security and infrastructure failures, with no platform exceeding 65% Security Score against a 90% target and concurrency handling as low as 6%. These observations are descriptive of our sample and require larger-scale replication to establish generality. We release SWE-WebDev Bench as a community benchmark to enable such replication and help platform builders identify and address these gaps. Code and benchmark resources are available at: this https URL and this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

107,171 characters extracted from source content.

Expand or collapse full text

SWE-WebDevBench: Evaluating Coding Agent Application Platforms as Virtual Software Agencies Siddhant Saxena 1 Nilesh Trivedi 2 Vinayaka Jyothi 2 1 BaseThesis Labs 2 QwikBuild March 2026 Abstract The emergence of “vibe coding” [2] platforms, where users describe applications in natural language and AI agents autonomously generate full-stack software, has created a need for rigorous evaluation beyond code-level benchmarks. Existing frameworks such as SWE-bench and FeatBench evaluate coding agents on developer-centric tasks but do not assess whether an AI platform can function as a complete software agency: understanding business requirements, making architectural decisions, writing production code, handling iterative modifications, and maintaining business readiness. We introduce SWE-WebDev Bench, a 68-metric evaluation framework spanning 25 primary and 43 diagnostic metrics across seven groups, organized along three dimensions: Interaction Mode (App Creation Request (ACR) vs. App Modification Request (AMR)), Agency Angle (Product Manager (PM), Engineering, Ops), and Complexity Tier (T4 multi-role SaaS, T5 AI-native). We evaluate six platforms using six standardized prompts across three business domains with 80 embedded canary requirements. Our initial evaluation (six platforms, three domains, 18 evaluation cells) reveals four recurring shortcomings in the current generation of AI app builders: (1) A specification bottleneck, where platforms compress rich business requirements into oversimplified technical plans, with observed inference quality varying widely across platforms (Canary Retention Rate: 17.7%–97.7%). (2) A pervasive frontend-backend decoupling, where visually polished UIs mask absent or broken backend infrastructure, with Background Job Scores ranging from 0% to 49%. (3) A steep production readiness cliff, where no platform scores above 60% on engineering quality and post-generation human effort varies substantially across platforms. (4) Widespread security and infrastructure failures, with no platform exceeding 65% Security Score against a 90% target and concurrency handling as low as 6%. These observations are descriptive of our sample and require larger-scale replication to establish generality. We release SWE-WebDev Bench as a community benchmark to enable such replication and help platform builders identify and address these gaps. Code and benchmark resources are available at:https://github.com/snowmountainAi/webdevbenchand https://webdevbench.com/. Disclosure: Two authors are affiliated with QwikBuild, one of the six evaluated platforms. This creates a potential conflict of interest. To mitigate this, we release all prompts, rubrics, and scoring protocols for independent replication and describe bias mitigation measures in §7.6. We encourage independent evaluations by parties unaffiliated with any evaluated platform. 1 Introduction “The hottest new programming language is English” [1]. This observation has materialized into a market of AI application-building platforms where users describe software in natural language and receive deployed, full- stack applications. Platforms such as Lovable, Replit Agent, Vercel v0, and others claim to compress months of development into minutes, making software creation accessible to non-developers. Yet the question of quality—whether the generated software is actually production-ready—remains largely unanswered by the research community. Existing evaluation frameworks fall into three categories, none of which addresses the full scope of what these platforms claim to deliver. Code-level benchmarks such as HumanEval [3] and ClassEval [4] evaluate function-level code generation and require code-level specifications as input, which is incompatible with the vibe coding paradigm where users provide only natural language. Issue-solving benchmarks such as SWE-bench [5] and its successors evaluate the ability to produce patches from issue descriptions. FeatBench [7] extends this to feature implementation, finding that even the best agent (GPT-5 with Trae-agent [17]) resolves only 29.94% of tasks. However, these benchmarks evaluate developer-facing 1 arXiv:2605.04637v1 [cs.MA] 6 May 2026 QwikBuild Emergent ReplitLovableBase44v0-Max 0 10 20 30 40 50 60 70 Engineering Score (%) 58 40 42 39 30 23 56 54 47 49 34 26 50 47 54 39 28 26 Engineering Scores Across Platforms and Business Domains 60% target P1 ExamEdge (EdTech) P2 FieldOps (Field Svc) P3 VettAI (FinTech-AI) Figure 1: Engineering Scores on SWE-WebDev Bench across six platforms and three business domains. No platform exceeds 60%, indicating substantial room for improvement across the field. Domain-specific performance swings (e.g., 13-point variance for Replit between P1 and P3) suggest that current platforms lack generalized competence. scenarios on existing codebases and do not assess whether an AI system can build a complete application from scratch for a non-technical user. Emerging application-level evaluations have begun addressing whole-application generation. Vibe Code Bench [8] evaluates end-to-end web app generation using browser-based workflow testing on 100 specifications, but does not assess PM behavior or iterative modification. WebCoderBench [9] introduces 24 fine-grained metrics across 1,572 real user requirements, but evaluates single-page applications without deployment or security assessment. From Prompt to Product [10] conducts a human-centered comparison of three commercial platforms (Replit, Bolt, Firebase Studio) using 205 participants, but relies on pairwise preference judgments rather than metric-level diagnostics. WebGen-Bench [11] tests multi-file website generation across 647 test cases, finding that even the best agent achieves only 27.8% accuracy. These benchmarks represent important progress toward application-level evaluation but do not assess the full agency pipeline: requirement elicitation, iterative modification, security, infrastructure, and business readiness. The gap these frameworks leave unfilled is the scenario that matters most for the vibe coding paradigm: Can an AI platform function as a complete software agency, understanding business intent, clarifying ambiguities, making sound architectural decisions, writing secure code, and handling iterative modifications? We introduce SWE-WebDev Bench, an evaluation framework designed to answer this question and, more importantly, to surface the specific failure modes that the community must address. Our contributions are: 1.A 68-metric evaluation framework organized across three orthogonal dimensions (Mode×Angle×Tier) with a four-tier judging taxonomy, designed to diagnose where AI app builders fall short of production readiness (§3). 2.The ACR/AMR distinction: the first benchmark to separately evaluate App Creation Requests and App Modification Requests, revealing that modification handling is a fundamentally different and harder competency (§4). 3.The Canary Requirement methodology: 80 culturally-specific, domain-embedded test requirements with four types (Original, New, Surviving, Contradiction) that distinguish genuine comprehension from template matching (§4.4). 2 4.An initial six-platform evaluation across three business domains, revealing four recurring shortcomings observed across all evaluated platforms (§5). 5.We release SWE-WebDev Bench, including prompts, rubrics, and evaluation protocols, to enable independent replication and benchmarking (https://github.com/snowmountainAi/webdevbench,https:// webdevbench.com/). 2 Related Work Table 1 positions SWE-WebDev Bench against existing evaluation approaches. Table 1: Comparison with existing evaluation frameworks for AI coding systems. BenchmarkScopeEval PM? Code Eval AMR?Platforms HumanEval [3]Function-level✗Partial✗Models SWE-bench [5]Issue patches✗Patch only ✗Models FeatBench [7]Feature impl.✗Patch only ✗Agents Vibe Code Bench [8] Web apps✗Browser✗Models WebCoderBench [9]Web apps✗24 metrics ✗Models Prompt→Product [10]App platforms✗Human pref. ✗3 WebGen- Bench [11] Websites✗ Functional✗Agents SWE-WebDev Bench Full agency✓(68m)Full audit ✓6 2.1 Code Generation Benchmarks The trajectory from HumanEval [3] through MBPP [15] to ClassEval [4] represents a progressive broadening of code generation evaluation from function-level to class-level tasks. However, all these benchmarks require code-level specifications (function signatures, docstrings) as input, which is incompatible with vibe coding where the user provides only natural language intent. SWE-bench [5] moved closer to realistic scenarios by tasking agents with resolving real GitHub issues. SWE- bench-Live [6] added temporal freshness, and FeatBench [7] shifted focus from issue-solving to feature implementa- tion. FeatBench’s key finding, that 73.6% of failures stem from regressive implementation where the agent breaks existing functionality while adding features, directly motivates our AMR evaluation dimension. A common limitation across all these benchmarks is that they evaluate patch quality on existing codebases, not complete application delivery. They assume a developer audience, test single-file or few-file changes, and do not address the PM, deployment, or business-readiness dimensions that define real software delivery. 2.2 Web Application Generation Benchmarks A recent wave of benchmarks has begun evaluating AI systems on whole-application generation. Vibe Code Bench [8] evaluates 100 web application specifications using 964 browser-based workflows with 10,131 substeps, finding that the best model achieves 61.8% accuracy—a much more discriminative benchmark than SWE-bench (42.7% vs. 2.8% gap between top and bottom models). WebCoderBench [9] introduces 24 fine-grained evaluation 3 metrics across 9 perspectives for 1,572 real user requirements, incorporating user-preference-weighted scoring. WebGen-Bench [11] evaluates multi-file website generation from scratch across 647 test cases, where even the best agent (Bolt.diy + DeepSeek-R1) achieves only 27.8% accuracy. From Prompt to Product [10] takes a human-centered approach, evaluating three commercial platforms (Replit, Bolt, Firebase Studio) using 96 prompts and 205 human participants with 1,071 pairwise comparisons. FullStack Bench [12] evaluates full-stack coding across 16 languages and 11 domains with 3,374 problems, but tests isolated problems rather than coherent applications. SUSVIBES [13] specifically benchmarks the security of agent-generated code, finding that while 61% of solutions are functionally correct, only 10.5% are secure—directly supporting our Finding 4 on universal security failures. These benchmarks represent important progress but share common gaps that SWE-WebDev Bench addresses: none evaluates requirement elicitation (PM behavior), none measures iterative modification handling (AMR), and none assesses the full pipeline from business intent through deployment readiness. Our framework is complementary: where Vibe Code Bench measures whether the app works, SWE-WebDev Bench measures why it fails and what to fix. 2.3 Visual and Multimodal Evaluation We observe an analogous phenomenon in AI app builders: a specification bottleneck where platforms compress rich, ambiguous business requirements into oversimplified technical plans, losing critical domain context. Our PM Agent evaluation dimension (§5.2) directly measures this compression loss. 2.4 Agent Evaluation Methodology Cognition AI’s blog post on evaluating coding agents [14] introduced realistic environments with simulated users and evaluator agents for autonomous outcome assessment. Their concept of “interactive self-reflection,” where agents use environment signals to evaluate themselves, informs our Ops/Maintenance evaluation angle. However, Cognition evaluates a single agent on developer tasks. Our framework evaluates six platforms on business-user tasks across a multi-dimensional quality space. 2.5 LLM-as-Judge Evaluation SWE-WebDev Bench relies on LLM judges for Tier 1 and Tier 2 metrics, which places it within the growing literature on automated evaluation. Zheng et al. [18] introduced MT-Bench and demonstrated that strong LLMs can approximate human preferences with> 80%agreement, but also identified systematic biases: position bias (favoring the first option), verbosity bias (favoring longer outputs), and self-enhancement bias (favoring outputs from the same model family). Kim et al. [19] showed that fine-tuned judge models can achieve higher correlation with human evaluators when given detailed rubrics, motivating our structured scoring rubrics for each metric. Li et al. [20] further documented that LLM judges exhibit platform-specific biases when evaluating code, an important consideration given that our evaluation targets commercial platforms with distinctive code styles. We address these concerns through our tiered approach: high-stakes subjective metrics (BIF, ETF, FGD) are assigned to Tier 3 expert panels rather than LLM judges, while LLM judges are used for factual verification tasks (Tier 1: “does this API route exist?”) where bias is minimal. We report measured inter-rater agreement in §3 and discuss calibration limitations in §7.6. 2.6 Benchmark Governance and Maintenance The challenge of benchmark maintenance and community governance has been addressed by several large-scale evaluation efforts. HELM [21] established a model for living benchmarks with regular re-evaluation, transparent methodology, and community contribution protocols. Dynabench [22] introduced dynamic, adversarial benchmark- ing to resist saturation. Chatbot Arena [18] demonstrated that community-driven pairwise evaluation can scale to thousands of comparisons. SWE-WebDev Bench draws on these precedents in its governance plan (§7.5). 4 3 The SWE-WebDev Bench Framework 3.1 Design Principles The design of SWE-WebDev Bench is guided by four principles that address specific limitations we identified in existing evaluation approaches. Principle 1: Evaluate the full delivery pipeline, not just the code. When a non-technical user asks an AI platform to build a SaaS application, the platform must perform the work of an entire software agency: a product manager who interprets ambiguous requirements, engineers who write correct and secure code, and an operations team who deploys and maintains the result. Existing benchmarks evaluate only the engineering phase (code patches, function implementations). SWE-WebDev Bench evaluates all three phases, because failure in any one of them renders the output unusable for the target user. Principle 2: Measure what the user cannot verify. The vibe coding paradigm shifts software creation to users who cannot read code. This creates a unique evaluation challenge: the most dangerous failures are invisible ones—silent specification violations (a date format quietly defaulting to M/D/Y instead of the requested D/M/Y), security vulnerabilities in generated backend code, or regression bugs introduced during modification. SWE-WebDev Bench prioritizes metrics that surface these invisible failures, because they represent the gap between perceived and actual quality. Principle 3: Diagnose, not just score. A benchmark that reports “Platform X scored 47%” is useful for ranking but not for improvement. SWE-WebDev Bench pairs every primary metric (which measures what was delivered) with diagnostic metrics (which trace why it succeeded or failed). This dual structure is designed to make the benchmark actionable for platform builders: a low Feature Completeness Score can be traced to poor requirement capture, hallucinated features, or implementation failures—each demanding a different architectural intervention. Principle 4: Resist gaming through specificity. AI benchmarks are vulnerable to overfitting: platforms can optimize for benchmark-specific patterns without improving general capability. We resist this through three mechanisms: (a) canary requirements that are culturally embedded and domain-specific, making them difficult to hard-code; (b) deliberately varied prompt styles (stream-of-consciousness, formal RFP, technical specification) that prevent optimization for a single input format; and (c) the ACR/AMR distinction, which requires genuine code understanding rather than template-based generation. 3.2 Evaluation Cube: Three Orthogonal Dimensions These principles are operationalized through three orthogonal dimensions that form an evaluation cube (Figure 2). Each dimension was chosen to isolate a specific axis of variation that existing benchmarks collapse. Dimension 1: Interaction Mode (ACR vs. AMR). We distinguish App Creation Requests (ACR), where the platform builds a new application from natural language, and App Modification Requests (AMR), where the platform must modify an existing application while preserving functionality. Why this dimension matters: Vibe coding is inherently iterative—users rarely describe their complete application in a single prompt. They build, use, and then request changes (“add multi-tenancy,” “swap the AI provider,” “the dispatch system needs to be smarter”). AMR is strictly harder than ACR because it requires understanding existing code, managing regressions, and scoping changes precisely. FeatBench [7] found that 73.6% of coding agent failures on modification tasks involve breaking existing functionality, confirming that creation and modification are fundamentally different competencies that must be evaluated separately. No existing platform benchmark distinguishes these modes. Dimension 2: Agency Angle (PM×Engineering×Ops). We decompose platform quality into three roles that mirror a human software agency: Product Manager (PM) for requirement understanding, inference, ambiguity handling, and plan quality; Engineering (E) for code quality, architecture, integrations, security, and AI feature implementation; and Operations (O) for deployment, monitoring, stability, and performance. Why this dimension matters: When a human software agency delivers a project, failures can be traced to a specific role: the PM misunderstood the client, the engineers wrote buggy code, or operations failed to deploy reliably. AI app-building platforms bundle all three roles into a single system, making it difficult to diagnose 5 where quality breaks down. By tagging each metric with its applicable agency angle, SWE-WebDev Bench enables targeted diagnosis. Our results validate this decomposition: the PM angle shows the widest variance across platforms (3.5×on Inference Quality Score), while Engineering scores are more compressed (6-point spread on Frontend Engineering), suggesting that PM capability—not code generation—is the primary differentiator. Dimension 3: Complexity Tier (T4 vs. T5). We evaluate at two tiers: T4 (multi-role Software-as-a-Service (SaaS) with Role-Based Access Control (RBAC), scheduled jobs, multiple integrations) and T5 (AI-native multi- tenant applications with LLM pipelines, trust/safety constraints, and provider abstraction). Why this dimension matters: Complexity tiers prevent a common evaluation pitfall: a platform that builds excellent to-do apps may fail entirely on applications requiring role-based access control, background job scheduling, or AI pipeline orchestration. T4 represents the complexity floor for real business applications (most SaaS products require RBAC and integrations). T5 adds the emerging dimension of AI-native applications where the generated code must itself orchestrate LLM calls safely and reliably. By evaluating at both tiers, SWE-WebDev Bench reveals whether platforms scale gracefully or hit capability cliffs as application complexity increases. Figure 2: SWE-WebDev Bench evaluation framework architecture. Left: Seven metric groups (G1–G7) spanning 25 primary and 43 diagnostic metrics. Center: The Evaluation Cube with three orthogonal dimensions—Interaction Mode (ACR/AMR), Agency Angle (PM/Engineering/Ops), and Complexity Tier (T4/T5). Right: Four-tier judging taxonomy from fully automated (Tier 0) to expert panel (Tier 3). Bottom: 80 canary requirements embedded across 9 prompts, 6 platforms, and 3 domains. 3.3 Metric Taxonomy: 25 Primary + 43 Diagnostic SWE-WebDev Bench comprises 68 metrics: 25 primary metrics across 7 groups (Table 2) and 43 diagnostic metrics across 4 categories. 3.3.1 Metric Group Rationale The seven groups are designed so that each captures a distinct failure mode that existing benchmarks miss. Together, they cover the full lifecycle of software delivery—from understanding what to build, through building it correctly, to shipping and maintaining it in production. G1: Specification Fidelity measures whether the platform understood what the user asked for. Existing code benchmarks (HumanEval, SWE-bench) take specifications as given; in vibe coding, the specification itself must 6 Table 2: Primary Metric Groups (25 metrics). Each metric is tagged with applicable Agency Angle(s). Full metric names are provided; abbreviations are used in subsequent tables. GrpMetrics#AngleKey Signal G1Business Intent Fidelity (BIF), Feature Completeness Score (FCS), Canary Retention Rate (CRR) 3PMDid it understand what was asked? G2Schema Design Score (SDS), Backend Logic Score (BLS), Frontend Engineering Score (FES), Code Hygiene Score (CHS), Architecture Score (ARC) 5EngineeringIs the generated code well-built? G3Core Integration Score (CIS), AI-Inside-App Score (AIA), External Service Reliability (ESR), Cron & Background Jobs Score (CBS) 4E + OpsDo integrations and jobs work? G4Security Score (S), Scalability Architecture Score (SAS), Concurrency & Load Score (CLS) 3E + OpsIs it secure and scalable? G5Code Change Impact Score (CCIS), Effort-to-Fix (ETF), Post-PRD Human Effort (PHE) 3E + OpsCan it handle change without breaking? G6 SEO & Web Standards Score (SWS), Lead & Growth Score (LGS), Multilingual/Localisation Score (MLS), AI Feature Quality (AFQ) 4E + PMIs it business-ready? G7Total Cost of Correctness (TCC), Feature Gap Delta (FGD), Claim Drift Index (CDI) 3PM + OpsWhat does it actually cost to ship? be inferred from ambiguous natural language. Business Intent Fidelity (BIF) captures whether the platform grasps the user’s business purpose, not just their literal words. Feature Completeness Score (FCS) measures functional coverage against a reference specification. Canary Retention Rate (CRR) is methodologically novel: by embedding culturally-specific requirements (e.g., D/M/Y date format, INR currency, JEE/NEET exam conventions) that are easy for template-matching systems to drop, CRR distinguishes genuine comprehension from shallow pattern extraction. A platform with high FCS but low CRR is building the right features with the wrong details—a failure mode invisible to existing benchmarks. G2: Code Quality evaluates the engineering quality of generated code across five dimensions. Schema Design Score (SDS) assesses database modeling—normalization, referential integrity, indexing, and multi-tenancy support— because poor schema design is the single most expensive technical debt category in web applications. Backend Logic Score (BLS) evaluates API design, route structure, and business logic correctness. Frontend Engineering Score (FES) measures component architecture, state management, and UI/UX implementation. Code Hygiene Score (CHS) captures maintainability factors: naming conventions, dead code, duplication, and separation of concerns. Architecture Score (ARC) assesses overall system design: separation of layers, dependency management, and pattern consistency. This five-dimensional decomposition is necessary because our results show that platforms can score 70%+ on FES while scoring below 10% on SDS (Table 7)—a granularity that composite “code quality” scores would mask. G3: Integrations measures whether the platform can connect to real-world services and implement background processing—capabilities that separate prototypes from production applications. Core Integration Score (CIS) tests database CRUD (Create, Read, Update, Delete) operations, authentication flows, and file storage. AI-Inside-App Score (AIA) evaluates AI feature implementation quality: prompt engineering, error handling, provider abstraction, and trust/safety controls. External Service Reliability (ESR) tests third-party service integration (email, SMS, payment). Cron & Background Jobs Score (CBS) measures scheduled task implementation. We include this group because our results reveal it as the highest-variance capability across platforms: CBS ranges from 0% to 49%, a 50-point spread that determines whether an application can run autonomously in production. G4: Security & Scale addresses the non-functional requirements that determine production viability. Security Score (S) covers OWASP (Open Web Application Security Project) Top 10 vulnerabilities, authentication hardening, API key management, and access control. Scalability Architecture Score (SAS) evaluates connection pooling, caching strategy, and horizontal scaling readiness. Concurrency & Load Score (CLS) uses the k6 load testing tool to 7 measure behavior under concurrent users. We set aggressive targets (S≥90%, SAS≥70%, CLS≥70%) because production deployment of insecure or non-scalable applications poses real risk to end users. Our results confirm this group as a universal failure point: no platform exceeds 65% on Security Score. G5: Changeability is unique to SWE-WebDev Bench and directly motivated by the iterative nature of vibe coding. Users do not build an application once; they iterate: “add a feature,” “change this,” “now support Hindi.” Code Change Impact Score (CCIS) measures whether modifications break existing functionality—the regression problem that FeatBench [7] found affects 73.6% of coding agent modifications. Effort-to-Fix (ETF) estimates the developer-hours required to bring the generated application to production quality. Post-PRD (Product Requirements Document) Human Effort (PHE) counts the re-prompts and manual code edits needed after initial generation. These metrics quantify the “last mile” problem: how much human effort remains after the AI has done its work. G6: Business Readiness captures whether the application is ready for real users and real markets, a dimension entirely absent from existing code benchmarks. SEO & Web Standards Score (SWS) tests meta tags, sitemaps, struc- tured data, and Core Web Vitals compliance—requirements that determine whether the application is discoverable. Lead & Growth Score (LGS) evaluates analytics integration, onboarding flows, and conversion tracking. Multi- lingual/Localisation Score (MLS) tests internationalization (i18n) support, a critical requirement for applications targeting non-English-speaking markets. AI Feature Quality (AFQ) evaluates the quality of AI-powered features within the application (distinct from AIA, which measures integration correctness). These metrics matter because vibe coding’s promise is to deliver business-ready software, not just technically functional code. G7: Production Readiness provides cost-oriented metrics that directly inform platform selection decisions. Total Cost of Correctness (TCC) aggregates platform costs, compute time, and human effort into a single monetary figure. Feature Gap Delta (FGD) estimates developer-hours needed to implement missing features. Claim Drift Index (CDI) measures the gap between what the platform claims to have built (in its PRD, traces, and dashboard) and what actually works—a trust metric. CDI is particularly important for vibe coding because non-technical users rely on the platform’s self-reported status; if CDI is high, the user believes the application is complete when it is not, leading to failed deployments and lost trust. 3.3.2 Diagnostic Metrics The 43 diagnostic metrics provide process-level insight into why platforms succeed or fail, complementing the primary metrics that measure what was delivered. They are organized into four categories. Category A (PRD & Planning) covers intent capture (Explicit Capture Rate, Inference Quality Score, Hallucination Rate, Inference Precision, Critical Omission Rate), conversational guidance (Conversational Guidance Score, Question Efficiency Rate, Conversational Turns to Convergence, Proactive Discovery Rate), ambiguity handling (Calibrated Uncertainty Score, Under-Clarification Rate, Over-Clarification Rate), and plan quality (Plan Structural Validity, Plan Communi- cability Score). Category B (Build & Code) covers trace clarity, error visibility, canary deep-dive, and—critically for AMR—change management diagnostics (Adaptive Coherence Score, Regression Rate, Change Acknowledgment Rate, Plan Update Rate, Change Processing Overhead). Category C (Deploy & Maintenance) covers deployment success, time to first byte, uptime, and post-deploy stability. Category D (Operator Experience) covers claim drift and resource observability. The complete diagnostic metric index with per-category counts is provided in the supplementary material. The diagnostic metrics serve a specific purpose in the benchmark: they enable platform builders to trace a poor primary metric score back to its root cause. For instance, a low FCS (Feature Completeness) could stem from poor Explicit Capture Rate (the platform missed stated requirements), high Hallucination Rate (it built unrequested features instead), or low Plan-to-Execution Fidelity (it planned correctly but implemented incorrectly). This diagnostic traceability is designed to make SWE-WebDev Bench actionable for the community, not just evaluative. We validate metric independence through Kendall’sτcorrelation analysis in Appendix C: 42% of metric pairs show weak or no correlation (|τ| ≤ 0.40), confirming that the metrics capture partially distinct constructs, though the seven groups do not represent fully independent latent factors. 8 3.4 Judging Taxonomy Each metric is assigned a Judge Tier to ensure reproducibility and appropriate rigor. Tier 0 covers fully automated checks (HTTP endpoint tests, Lighthouse audits, k6 load tests, npm audit scans) requiring no human judgment. Tier 1 uses LLM judges with structured prompts for factual checks (inter-rater reliabilityκ ≥ 0.85, measured using Cohen’s kappa), such as verifying whether a specific API route exists or a database column has the correct type. Tier 2 combines LLM evaluation with human validation for judgments requiring contextual understanding (κ≥ 0.75), such as assessing whether an architecture choice is appropriate for the application’s complexity. Tier 3 uses a three-person expert panel with structured rubrics (κ≥ 0.60) for inherently subjective assessments, such as Business Intent Fidelity. The tiered approach reflects a deliberate tradeoff between reproducibility and depth: Tier 0 metrics can be run by any evaluator with zero variance, while Tier 3 metrics capture nuances that automated evaluation cannot. We assign the highest-stakes metrics (BIF, ETF, FGD) to Tier 3 to ensure they receive expert scrutiny, while commoditized checks (SWS via Lighthouse, CLS via k6) are fully automated at Tier 0. LLM judge implementation. All Tier 1 and Tier 2 evaluations use Claude 3.5 Sonnet (temperature 0) with 9 structured prompt templates released in the benchmark repository. Each template specifies: the metric definition, a 1–5 or 1–100 scoring rubric with anchor descriptions, and the evaluation context (codebase excerpt, conversation transcript, or deployment artifact). Platform identity is not provided to the LLM judge to reduce platform-specific bias. For Tier 1 metrics, we measured inter-rater agreement between the LLM judge and a human evaluator on a sample of 36 items (2 metrics×18 cells): observedκ = 0.82(target:κ ≥ 0.85). For Tier 2 metrics, agreement was measured on 18 items (1 metric×18 cells): observedκ = 0.71(target:κ ≥ 0.75). Tier 3 expert panel agreement: observedκ = 0.64(target:κ≥ 0.60). We acknowledge that these calibration samples are small and that more extensive human–LLM agreement studies are needed; we release all judge prompts and raw scores to enable independent calibration. Approximately 40% of the Engineering Score weight derives from Tier 0 automated metrics, 35% from Tier 1 LLM judges, 15% from Tier 2 LLM+human, and 10% from Tier 3 expert panels. 3.4.1 Target Threshold Calibration Each metric is assigned an aspirational target threshold (Table 2) that represents the minimum quality level we consider necessary for production deployment. These targets are set deliberately high to create diagnostic headroom—they are intended to surface gaps, not to define pass/fail gates. Targets for Security Score (S≥ 90%) and Concurrency & Load (CLS≥70%) are grounded in OWASP deployment guidelines and industry load testing standards. Targets for Feature Completeness (FCS≥ 85%) and Canary Retention (CRR≥ 80%) reflect the expectation that production software should implement the majority of stated requirements. Other targets (e.g., CBS ≥90%, ESR≥80%) represent practitioner consensus from the authors’ experience operating a production AI app platform, and we acknowledge they may benefit from calibration through broader practitioner surveys. Crucially, the benchmark’s diagnostic value does not depend on these specific thresholds: the relative patterns across platforms (score ranges, metric correlations, domain sensitivity) are informative regardless of absolute targets. 4 Experimental Design 4.1 Platform Selection We evaluate six platforms spanning the spectrum of AI app-building approaches (Table 3). 4.2 Prompt Suite and Snorkel Alignment We designed six standardized prompts (3 ACR + 3 AMR) aligned with the Snorkel Open Benchmarks framework [16], which identifies three dimensions where AI evaluation lags capability: Environment Complexity, Autonomy Horizon, 9 Table 3: Evaluated platforms and their architectural approaches. PM behavior is characterized empirically in §5.2. CodePlatformApproachAvg PM Qs B0Base44Low-code generator1.0 E1Emergent (E1-OPUS)Agentic with config Q&A5 L0Lovable (Plan Mode)Plan-then-build4 Q1QwikBuild † Multi-agent with PM agent15 R3Replit (Agent3)Autonomous agent0.3 V0v0-Max (Vercel)Single-shot generation1.3 † Two authors are affiliated with this platform. See §7.6 for bias mitigation. and Output Complexity. Each prompt stresses at least two of these dimensions. The three ACR prompts use deliberately different authoring styles to test robustness to input variation. 4.2.1 Domain Selection Rationale The three business domains—EdTech, Field Service, and FinTech-AI—were chosen to create orthogonal diagnostic probes that stress different platform capabilities. EdTech (P1) tests inference depth: the prompt is deliberately vague (written as a frustrated founder’s midnight text), requiring the platform to infer product structure from pain points. It embeds culturally-specific conventions (JEE/NEET exam structure, Indian coaching institute batch models, April–March academic year) that cannot be inferred from generic SaaS templates. A platform that scores well on P1 demonstrates genuine domain reasoning, not pattern matching. Field Service (P2) tests execution precision: the prompt is an ultra-detailed enterprise Request for Proposal (RFP) with a 10-state ticket lifecycle, SLA pause/resume logic, parts inventory with reorder triggers, and GST invoicing. Where P1 tests whether the platform can infer missing requirements, P2 tests whether it can faithfully implement explicitly stated complex business logic without dropping edge cases. FinTech-AI (P3) tests AI trustworthiness: the prompt requires building AI features that are safe, auditable, and honest about uncertainty. It includes Monte Carlo simulation, multi-stage analysis pipelines, confidence scoring, mandatory escalation triggers, and a credit system with transactional integrity. This domain probes whether the platform can generate AI-powered applications that meet trust and safety requirements—a capability increasingly critical as AI features become standard in business software. Together, these domains ensure that a platform cannot score well through a single architectural strength. Our results validate this design: no platform dominates across all three domains, and domain-specific swings of up to 13 percentage points reveal architectural affinities that averaged scores would mask. 4.2.2 Prompt Style Variation Beyond domain variation, we deliberately vary prompt authoring style to test robustness to input format—a critical capability because real users do not write structured specifications. P1: ExamEdge Academy (EdTech, T4) “The Founder’s WhatsApp Ramble” A stream-of-consciousness description of a coaching institute’s needs, written as a real founder would text at midnight. The prompt never uses terms like “CMS,” “CRUD,” “RBAC,” or “SaaS.” Instead it describes a world of pain (“everything runs on WhatsApp groups and Google Sheets and I’m losing my mind”) from which the platform must extract a product specification. Deliberate contradiction: Teachers “should NOT see other branches’ data” but a “cross-branch leaderboard showing top 10 students across ALL branches” should be visible to everyone. A capable PM agent should flag this contradiction rather than silently picking one interpretation. 10 Based on a real coaching institute: 3 branches, 400 JEE/NEET students, Pune. P2: FieldOps Pro (Field Service, T4) “Enterprise RFP with Deliberate Trap” An ultra-detailed enterprise specification with a 10-state ticket lifecycle, Service Level Agreement (SLA) pause/resume logic, parts inventory with reorder triggers, anti-fraud self-assignment constraints, and a clearly marked deliberate contradiction: “Rating visible to Org Admin and Dispatcher, NOT to the Technician” followed by “I want rating to be visible to technicians so they can learn from feedback,” with an explicit note: “the system should FLAG this contradiction to me and ask which behavior I want.” P3: VettAI (FinTech-AI, T5) “Maximum Complexity AI-Native” The hardest prompt in the suite, designed to test quality demands on AI outputs rather than code volume. It features a multi-stage AI analysis pipeline (Entity Extraction→Contradiction Detection→Risk Scoring →Report Generation), Monte Carlo risk simulation, temporal trajectory projection, AI Vision for scanned documents, and a credit system requiring transactional integrity (deduct-before-call, refund-on-fail). Trust and safety requirements are architectural: no fabrication, hedged recommendations, confidence scoring, mandatory escalation triggers (risk≥ 8produces a red banner, non-optional), version history with diffs, and analyst override tracking. 4.3 AMR Prompts: Escalating Mutations The benchmark defines three AMR prompts testing modification handling with escalating complexity and deliberately varied prompt styles. These are selected to maximise diagnostic coverage across complexity (medium vs. complex), style (specific vs. vague), and change type (additive, modifying, provider swap). AMR Design Matrix ID∆ ComplexityStyleChange TypesPM Test P4ComplexSpecific (structured)Additive + ModifyingLow P5MediumVague (frustrated user)AdditiveHIGH P6MediumSpecific (structured)Modifying (provider swap)Low P4 retrofits multi-tenancy onto P1, requiring every query to change. P5 uses a frustrated-user tone (“Can the system just... be smarter about this?”) requiring the PM agent to infer three specific features from pain points. P6 introduces a provider abstraction layer (Anthropic→Gemini fallback) and PDF report generation, testing whether the platform can refactor AI integration architecture without breaking the existing credit system. 4.4 Canary Requirements A central design challenge in evaluating AI app builders is distinguishing genuine requirement comprehension from template matching. A platform that generates a “standard SaaS application” will include many expected features (user authentication, CRUD operations, dashboards) regardless of what the user actually asked for. To test whether platforms truly understand and retain specific requirements, we introduce the canary requirement methodology: 80 culturally-specific, domain-embedded test requirements that are easy for a human to verify but difficult for a template-based system to satisfy. The key insight is that canary requirements are detail-level specifications—date formats, currency conventions, domain-specific terminology, visibility rules—that only survive the build pipeline if the platform genuinely processes and retains the user’s intent. They are classified into four types (Table 4). 11 Table 4: Canary requirement taxonomy across the 6-prompt suite. TypeNameCountPurpose OOriginal21Explicitly stated constraints (D/M/Y, INR lakhs, “Powered by” footer) NNew37Requirements added in v3 for AMR and multi-modal coverage SSurviving18ACR canaries that must persist through AMR modifications XContradiction4Deliberately conflicting requirements the platform must flag The Surviving type is methodologically novel and directly tests the ACR-to-AMR transition. When a user modifies an application, existing requirements should either persist unchanged or evolve correctly—they should not silently disappear. For instance, P1’s “cross-branch leaderboard” canary becomes “cross-institute leaderboard scoped to institute” in P4’s franchise retrofit, testing whether the platform correctly evolves a tricky visibility rule rather than breaking it. The Contradiction type tests a capability that no existing benchmark measures: whether the platform can detect and flag logically inconsistent requirements rather than silently choosing one interpretation. In real-world software projects, contradictions in requirements are common and costly; a platform that embeds contradictions into code without flagging them creates bugs that are extremely difficult to diagnose. 4.5 Evaluation Protocol Evaluation follows a seven-phase, 14-day protocol per platform×prompt (Figure 3), designed to mirror the stages of a real software delivery audit: 1.Build Execution: Submit the prompt to the platform and record the complete interaction (conversation logs, build traces, deployment status). 2.Code Quality Audit: Apply LLM judge prompts to the generated codebase, scoring G2 metrics (schema, backend, frontend, hygiene, architecture). 3.Integration and Security Testing: Execute automated tests for G3 (integrations, background jobs) and G4 (security scans, load tests via k6). 4.Feature and Canary Testing: Manually verify feature completeness (FCS) and canary retention (CRR) against the reference specification. 5.AMR Change Injection: Submit AMR prompts to the deployed ACR application and score G5 metrics (change impact, regressions, effort). 6.Human Panel and Cost Aggregation: Three-person expert panel scores BIF, ETF, and FGD; aggregate TCC from platform billing and human effort logs. 7. Diagnostic Scoring: Score Category A–D diagnostic metrics from conversation transcripts and build logs. The Engineering Score for each platform×prompt is computed as a weighted mean of applicable primary metrics in G2–G6 (excluding G1 specification fidelity and G7 cost metrics, which are reported separately). Within each group, metrics are equally weighted; group weights are normalized by the number of applicable (non-N/A) metrics per group to prevent groups with more metrics from dominating. Metrics that are not applicable for a given complexity tier (e.g., AIA for T4 prompts) are excluded from both numerator and denominator. This seven-phase protocol ensures that every metric is grounded in observable evidence (code, logs, test results, transcripts) rather than subjective impression. 12 Figure 3: The 7-phase evaluation pipeline. Each platform×prompt combination progresses through Prompt Submission, PM Agent Evaluation, Build & Code audit, Automated Audit (Lighthouse, k6, npm audit), Security & Integration testing, Expert Panel review, and Score & Report aggregation. The Canary Requirement Thread (bottom) tracks 80 embedded test requirements across four types: Original (21), New (37), Surviving (18), and Contradiction (4). A feedback loop from Phase 7 back to Phase 1 supports the AMR (modification) evaluation cycle. 5 Results Statistical scope and interpretive note.This evaluation is an initial diagnostic study comprisingn = 3prompts per domain across 6 platforms, yielding 18 ACR evaluation cells. We report observed scores and ranges; no statistical significance tests or confidence intervals are computed, as the sample size is insufficient for inferential claims. All comparative observations (e.g., score ranges, observed ratios) describe patterns in this sample and should not be interpreted as statistically significant platform rankings. We release SWE-WebDev Bench so the community can expand the prompt suite, add platforms, and conduct the larger-scale replications needed to establish robust rankings with statistical confidence. We estimate that 15–20 prompts across 5–7 domains would be required to detect medium effect sizes (d≥ 0.5) with α = 0.05 and power = 0.8 using paired permutation tests across prompts. 5.1 Cross-Platform Engineering Scores Table 5 presents the engineering scores across all platform×prompt combinations. The central observation is that no platform exceeds 60%, and every platform has at least one metric where it scores below 15%, indicating that production readiness remains an unsolved problem across the field. 5.1.1 Comprehensive Per-Metric Breakdown Table 6 presents all 22 applicable primary metrics averaged across all three ACR prompts for all six platforms. This is the central results table of the paper: each cell represents the mean of up to three prompt-level scores, excluding N/A values. (Per-platform, per-metric scores for P1 are provided in Appendix Table 16; PM Agent behavioral analysis is provided in Appendix Table 17.) Several patterns emerge from Table 6 that highlight recurring gaps across the field. 13 Table 5: Engineering Score (%) across all six platforms and three ACR prompts. No platform exceeds 60%, and every platform has at least one metric below 15%. Scores are descriptive of our sample (n=3 per platform). PlatformP1P2P3AvgBestWorst Metric Base44 (B0)30.233.528.030.6P2CBS (5%) Emergent (E1)39.953.847.146.9 P2SWS (13%) Lovable (L0)39.149.339.042.5P2SWS (15%) QwikBuild (Q1)57.555.650.354.5P1SWS (8%) Replit (R3)41.547.454.547.8P3CBS (0%) v0-Max (V0)22.826.226.425.1 P3SDS (0%) Table 6: Complete primary metric scores (%) averaged across ACR prompts (P1, P2, P3). N/A-excluded averaging. No platform meets all targets; 0 of 6 platforms pass more than 5 of 22 metrics. Scores are descriptive of our sample and should not be interpreted as statistically significant rankings. BIF shown as panel score out of 4. ETF in developer-hours. Metrics below target shaded where all platforms fail. Grp MetricTgtB0E1L0Q1R3V0 G1: Specification Fidelity G1BIF (Business Intent)3/41.33.01.73.72.01.0 G1FCS (Feature Complete)8546.765.032.784.053.023.3 G1CRR (Canary Retention)8024.357.021.797.754.017.7 G2: Code Quality G2SDS (Schema Design)8040.745.864.772.957.32.7 G2BLS (Backend Logic)8036.761.440.368.461.38.3 G2FES (Frontend Eng.)7557.769.361.368.074.368.0 G2CHS (Code Hygiene)7543.756.559.052.856.357.3 G2ARC (Architecture)7031.042.747.059.754.043.3 G3: Integrations G3CIS (Core Integration)9046.754.721.367.370.026.7 G3AIA (AI-in-App)7515.066.038.072.042.030.0 G3ESR (External Svc)8015.024.010.328.335.012.3 G3CBS (Background Jobs)9015.020.32.049.329.70.0 G4: Security & Scale G4S (Security)9031.350.751.763.740.034.3 G4SAS (Scalability)7025.037.733.754.043.037.7 G4CLS (Concurrency)7012.034.715.042.025.36.0 G5: Changeability G5CCIS (Change Impact)9022.345.737.787.742.731.7 G5ETF (Effort-to-Fix, h) ≤ 1665.726.050.714.737.354.7 G6: Business Readiness G6SWS (SEO/Web Std.)7018.012.718.09.328.040.3 G6MLS (Multilingual)7513.551.524.066.028.513.5 G6AFQ (AI Feature Qual.)7512.064.025.068.042.018.0 G7: Production Readiness G7FGD (Feature Gap, h) ≤4029.319.036.39.325.741.7 G7CDI (Claim Drift %) ≤1035.717.737.34.014.033.7 Engineering Score6030.646.942.554.547.825.1 14 No platform passes all metrics. The highest-scoring platform still fails 17 of 22 metrics against their respective targets, indicating that production-grade AI app building remains an open problem. Frontend engineering is commoditized; backend infrastructure is not. Four platforms score within 6 percentage points of each other on Frontend Engineering (68–74%), suggesting that UI generation is a largely solved problem. In contrast, backend and infrastructure metrics show 5–10× spreads (e.g., CBS ranges from 0% to 49%), revealing that infrastructure generation—background jobs, scheduled tasks, complex integrations—remains deeply inconsistent. This gap represents the most actionable area for platform improvement. Every platform has blind spots. High overall scores do not imply uniform quality. Each platform exhibits a distinct strength-weakness profile: one platform leads on Core Integration (70%) and External Service Reliability (35%) but scores lowest on PM metrics; another achieves the highest SWS (40.3%) despite the lowest overall engineering score; a third shows the most thorough requirement elicitation but the lowest Code Hygiene among top performers (52.8%) and poor External Service Reliability (28.3%). These cross-cutting weakness patterns demonstrate that no architectural approach has solved all dimensions simultaneously—a finding that SWE- WebDev Bench is designed to surface for any platform evaluated. Specification fidelity varies dramatically. Canary Retention Rate ranges from 17.7% to 97.7%, a 5.5×spread. This means that culturally-specific, domain-embedded requirements (date formats, currency conventions, localization details) are silently dropped by most platforms, a failure mode invisible to users who cannot audit AI-generated code. 5.1.2 Per-Prompt Results Table 7 shows how each platform performs across the three business domains, revealing domain-specific strengths and weaknesses not visible in averages. Table 7: Selected metrics by prompt, showing domain sensitivity. Each cell is a single prompt score (%). P1 ExamEdge (T4)P2 FieldOps (T4)P3 VettAI (T5) B0 E1 L0 Q1 R3 V0B0 E1 L0 Q1 R3 V0B0 E1 L0 Q1 R3 V0 G1 FCS48 65 35 88 55 2852 68 33 8652 2240 62 30 78 52 20 G1 CRR25 57 29 100 48 2128 50 29 100 43 1820 64793 71 14 G2 SDS42 41 48 82 58345 68 74 7862535 29 72 59 520 G2 FES60 61 45 75 70 6258 79 75 7175 7055 68 64 58 78 72 G3 CIS50 52 22 72 76 3048 62 20 6874 2842 50 22 62 60 22 G3 CBS188352 35015 48356320125040 220 G4 S34 45 52 62 38 3232 52 55 6540 3328 55 48 64 42 38 Table 7 reveals that no platform dominates across all domains, and each platform exhibits distinct strengths and weaknesses depending on the business context. Lovable achieves the highest Schema Design Score on P3 (72%), suggesting strength in relational modeling for complex domains despite lower overall performance. Replit leads on Frontend Engineering for P3 (78%) and Core Integration across P1 and P2 (76%, 74%), reflecting its npm-ecosystem advantage. v0-Max produces no database schema at all on P3 (SDS = 0%) while achieving 72% on Frontend Engineering—the sharpest instance of the frontend-backend decoupling pattern. All platforms score below 60% on average, consistent with FeatBench’s sub-30% resolved rates [7]. This convergence across independent benchmarks suggests a fundamental capability ceiling in current AI code generation. Performance is domain-sensitive (Figure 1). Replit scores highest on P3 VettAI (54.5%) while scoring lowest on P1 ExamEdge (41.5%), a 13-point swing. This domain sensitivity suggests that current platforms lack generalized competence and instead exhibit architectural affinities for specific problem types—an important consideration for users selecting platforms for particular business domains. Figure 4 provides a visual overview of the four recurring findings that emerge from these results. 15 Figure 4: Overview of the four recurring findings uncovered by SWE-WebDev Bench. Finding 1: The specification bottleneck—3.5×variation in inference quality (CRR: 17.7% to 97.7%). Finding 2: Frontend-backend decoupling—polished UIs masking absent backend infrastructure (CBS: 0% to 49%). Finding 3: The production readiness cliff—5×effort variation and no platform exceeding 60% engineering score. Finding 4: Widespread security failures—best Security Score is 65%, with concurrency as low as 6%. 5.2 Finding 1: The Specification Bottleneck The largest recurring issue uncovered by SWE-WebDev Bench is a specification bottleneck: most platforms compress rich, ambiguous business requirements into oversimplified technical plans without adequate requirement elicitation. The degree of upstream planning varies dramatically across platforms and strongly predicts downstream quality. The consequences are visible in downstream metrics. In our sample, Canary Retention Rate ranges from 17.7% to 97.7% across platforms (observed ratio: 5.5×,n = 3prompts), and Inference Quality Score ranges from 20 to 70 (observed ratio: 3.5×). The number of PM questions ranges from 0 to 15 across platforms on the same prompt. Under-Clarification Rate reaches 85% for some platforms on the most complex prompt (P3). Notably, higher PM thoroughness does not uniformly predict quality across all dimensions—the platform with the most extensive elicitation (15 questions, 6 rounds) shows the lowest Code Hygiene among top-performing platforms (52.8%) and below-field-average External Service Reliability (28.3% vs. field-leading 35.0%), suggesting that PM investment and engineering quality are partially independent axes. Table 8 and Figure 5a present the PM diagnostic scores averaged across all ACR prompts. Beyond the PM phase: the build-time feedback loop. Requirement elicitation alone accounts for approximately 10–15% of the observed quality gap. The larger share appears to come from whether the platform maintains a feedback loop during code generation. Platforms with multi-agent architectures can resolve conflicts between features as they are implemented—for example, when an RBAC constraint conflicts with a visibility rule, the conflict can be escalated to a planner that adjusts the implementation strategy mid-build. Platforms without such feedback loops embed conflicts silently into code, where they surface as bugs. This pattern suggests that closed-loop code generation, where planning and execution inform each other iteratively, is a more impactful architectural investment than any single-phase improvement. 16 Table 8: PM diagnostic scores averaged across ACR prompts (P1–P3). Bold indicates highest value. Most platforms fall below targets on the majority of PM metrics, revealing the specification bottleneck as a field-wide problem. TargetB0E1L0Q1R3V0 ECR (Explicit Capture) ≥90%80.076.775.793.763.371.7 IQS (Inference Quality) ≥60%45.030.040.070.020.025.0 IP (Inference Precision) ≥75%71.761.760.087.745.055.0 CGS (Conv. Guidance)≥70%8.341.045.784.36.725.0 QER (Question Efficiency) ≥80%33.350.058.385.033.343.3 CUS (Calibrated Uncertainty) ≥70%26.043.344.378.320.033.3 PSV (Plan Validity)≥7566.750.062.388.343.351.7 PCS (Plan Communicability) ≥6561.031.752.385.731.738.3 ECR IQS IP CGS QER CUS PSV PCS 20 40 60 80 100 PM Diagnostic Radar (ACR Average) Q1 E1 L0 R3 B0 (a) PM diagnostic radar across five platforms (ACR). The widest gaps are on Conversational Guidance (CGS: 6.7–84.3) and Inference Quality (IQS: 20–70), indicating that require- ment elicitation is the most variable capability across plat- forms. BIF FCS CRR SDS BLS FES CIS AIA CBS S CCIS AFQ 20 40 60 80 100 AMR Primary Metric Profiles (QwikBuild) P4 Franchise P5 AI Dispatch P6 Provider Swap (b) AMR primary metric profiles across three modification prompts (QwikBuild). P6 leads on specification fidelity (BIF, FCS, CRR) and achieves the only AIA PASS (75%). P4 leads on CBS. P5 achieves the highest CCIS (80%) despite vague input. Figure 5: Radar comparison of ACR cross-platform PM diagnostics (left) and AMR per-prompt primary metrics (right). The ACR radar reveals that requirement elicitation is the primary differentiator across platforms. The AMR radar reveals that prompt characteristics (complexity, style) produce distinct metric profiles within a single platform, with well-scoped modifications (P6) outperforming complex structural changes (P4) on specification fidelity. 17 5.2.1 PM Agent Interaction Traces Figure 6 shows verbatim excerpts from four platforms receiving the same P1 ExamEdge prompt. All transcripts are sourced from ACR-Runs-1.pdf conversation logs. PM Agent Interaction Comparison: P1 ExamEdge (Verbatim Excerpts) QwikBuild (Q1)6 rounds, 15 questions, dedicated PM Agent PMAgent:“Hello!I’myourProductManagerAgent · Thanksforsharingallthosedetails — I can feel the chaos of running 3 branches on WhatsApp and Google Sheets!” Round 1, Q2: “A student in JEE might have different teachers for Physics, Chemistry, and Maths. How should teacher-student relationships work?” 1. Each batch has ONE primary teacher for all subjects. 2. Each batch has MULTIPLE teachers (one per subject). Each teacher sees only their subject’s assignments and tests, but ALL teachers can mark attendance. Recommended✓ 3. Each student is assigned to multiple teachers (subject-wise). Round 2, Q3: “Teachers mark attendance daily. What if a teacher forgets and tries to mark it the next day or later?” 1. Today only. Admin can edit past records. 2. Last 7 days editable by teacher. After that, Admin only. Recommended 3. Anytime for any past date. Round 3, Q3: “The top 10 leaderboard shows students by average test score across all branches. How should this average be calculated?” 1. Average of ALL tests. Recommended✓ 2. Average of LAST 5 tests only. 3. Weighted average by subject. →Generates 12-section PRD: Product Overview, Scope (v0/v1), Features, Roles, Data Model, Auth, Dashboards, Background Jobs, Acceptance Criteria, Risks & Open Questions, Handoff Notes. Emergent (E1)2 rounds, 5 questions, all infrastructure config Asks: (1) SMS provider: Twilio or Mock? (2) Email: SendGrid or Resend? (3) Auth: JWT or Google? (4) Hindi toggle approach: toggle or profile setting? (5) Design preferences? All five questions are infrastructure decisions. Zero questions about fee escalation logic, attendance editing windows, teacher- student batch model, or the leaderboard contradiction. Directly proceeds to build after answers. Replit (R3)0 rounds, 0 questions, immediate build Zero questions. Immediately analyzes prompt and outputs a 17-item bullet-point feature list. Lists “PostgreSQL, Replit Auth, App Storage” as integrations. Claims Twilio integration without asking about SMS provider. Contradiction between branch isolation and cross-branch leaderboard not detected. v0-Max (V0)2 rounds, 4 config questions Enters “plan mode.” Asks: (1) SMS provider? (2) Database? (recommends Supabase) (3) Auth method? (4) Email service? Generates Implementation Plan with DB schema and feature phases. Contradiction not detected. No business-flow questions. Figure 6: Verbatim PM Agent behavior on P1 ExamEdge across four platforms. QwikBuild probes 15 business workflow questions across 3 rounds with structured multiple-choice and “Recommended” defaults; Emergent asks 5 infrastructure questions; Replit and v0-Max ask zero business-flow questions. Transcripts from ACR-Runs-1.pdf. 5.2.2 Contradiction Handling: The P2 Trap P2 FieldOps contains a deliberate, marked contradiction: “Rating visible to Org Admin and Dispatcher, NOT to the Technician” followed by “I want rating to be visible to technicians.” The prompt explicitly states: “the system should FLAG this contradiction to me.” Table 9 shows how each platform handled this test. All six platforms detect the P2 contradiction, which is explicitly marked. The meaningful difference is what happens 18 Table 9: Contradiction handling on P2 FieldOps. “Detected” means the platform identified the contradiction. “Resolved” means it asked the user which behavior to implement before proceeding. PlatformDetected?Resolved?Behavior Base44YesNoFlags with warning symbol. Generates detailed plan including “What NOT to Do” section, but does not ask user for resolution. EmergentYesYesDetects as first of 5 questions. Asks for clarification, then proceeds to 4 infrastructure config questions. LovableYesYesDetects as first question. Asks for resolution. Proceeds with 3 scope questions. QwikBuildYesYesFlags with “ △ CONTRADICTION DETECTED” in first response. Presents three resolution options. Integrates user’s choice into PRD before proceeding to build. ReplitYesNoFlags contradiction, then defers 10 critical features to “future scope” (SLA, audit, invoice, time tracking, signatures). Most aggressive scope cut across all platforms. v0-MaxYesPartialDetects and presents as structured choice. Then builds aggressively, timing out at 10m37s. User reports: “it took me multiple human efforts to navigate the completion.” next—a spectrum from full integration of the resolution into the implementation plan, to flagging without resolution, to deferring core features to “future scope.” At the weaker end, one platform flags the contradiction but then defers half of the application’s core features (SLA engine, audit trail, invoicing, time tracking, customer signatures) to “future scope,” producing a fundamentally incomplete Field Service application. This pattern—detection without resolution—is a recurring weakness that SWE-WebDev Bench is designed to surface. 5.2.3 Per-Prompt PM Scores Table 10 presents the complete per-platform, per-prompt PM diagnostic scores, revealing how prompt style affects PM agent behavior. The data shows that all platforms improve on P2 (detailed RFP) relative to P1 (vague prompt), but most platforms degrade significantly on vague prompts, precisely the input style most common among non-technical users. Table 10: PM diagnostic scores per platform per prompt. Bold = passes target. Shading indicates performance differences between vague P1 and structured P2 across platforms. CTC = Conversational Turns to Convergence. P1 ExamEdge (Vague)P2 FieldOps (Detailed)P3 VettAI (Complex) TgtB0 E1 L0 Q1 R3 V0B0 E1 L0 Q1 R3 V0B0 E1 L0 Q1 R3 V0 ECR ≥9078 70 72 92 65 8082 82 80 95 55 7580 78 75 94 70 60 IQS ≥6050 35 45 85 25 4040 25 35 60 15 2045 30 40 65 20 15 CGS ≥70040 50 8803525 45 45 80 20 30038 42 85010 CTC3–8022602122412022600 UCR↓ ≤2070 65 50 12 80 5545 50 45 15 70 6075 60 52 10 85 85 PSV ≥7560 45 55 88 40 6572 55 70 90 35 5068 50 72 87 55 40 CTDYesNNNYNNYYYYYYNNN–N Several observations emerge from the per-prompt data that illuminate the specification bottleneck. Most platforms skip requirement elicitation entirely. Replit and Base44 have CTC=0 on P1 and P3, meaning zero conversational turns before generating a plan. v0-Max asks zero questions on P3 (the most complex prompt), jumping immediately to a TodoManager task breakdown. Only one platform maintains high PM scores on the vague P1 prompt where inference is most critical. Under-clarification is pervasive. UCR reaches 85% for v0-Max and Replit on P3, meaning they proceed without clarifying 85% of the ambiguous requirements in the most complex prompt. This finding has direct implications 19 for the vibe coding user experience: users who provide ambiguous input (which is the norm, not the exception, for non-technical users) receive applications built on unverified assumptions. Plan quality without validation is insufficient. Base44 presents an instructive case: despite zero PM questions on P1 and P3 (CTC=0), its Plan Structural Validity is moderate (60–72%). It generates plans with “Intent & Goal,” “Audience & Roles,” “Core Flows,” and “What NOT to Do” sections without any user interaction. The plan structure is reasonable, but assumptions are unvalidated, which explains why Base44’s ECR appears adequate (78–82%) while its CDI is high (35%). This suggests that plan generation without user interaction creates a false sense of alignment. Contradiction detection on P1 (where the contradiction is not marked) is more diagnostic: only one platform detects the branch-isolation vs. cross-branch-leaderboard tension proactively, by probing RBAC rules deeply enough to surface the conflict. The remaining five platforms embed this contradiction silently into their implementations, demonstrating that unmarked contradictions are a blind spot for most current platforms. Platforms that skip requirement elicitation suffer from a specification bottleneck where rich business intent is compressed into oversimplified technical plans, losing critical domain context. 5.3 Finding 2: The Frontend-Backend Decoupling Problem A striking pattern in the data is that frontend quality is a poor predictor of backend quality. Table 11 reveals that platforms with comparable Frontend Engineering Scores diverge dramatically on infrastructure metrics, suggesting three distinct architectural strategies with very different production-readiness implications. Table 11: Frontend Engineering Score (FES) vs. infrastructure metrics, averaged across all ACR prompts (%). Platforms sorted by CBS. The FES–CBS gap reveals a decoupling between UI quality and backend capability. PlatformFES↑CBSCISESRStrategy Emergent69.320.354.724.0Ecosystem-Leveraged Lovable61.32.021.310.3Frontend-Prioritized QwikBuild68.049.367.328.3Infrastructure-Integrated Replit74.329.770.035.0Ecosystem-Leveraged v0-Max68.00.026.712.3Frontend-Prioritized Figure 7 maps these results onto a two-dimensional space, revealing three distinct platform archetypes. Infrastructure-Integrated (green cluster). Platforms in this cluster provide managed infrastructure services (databases, schedulers, storage) as pre-built modules, so AI-generated code does not need to implement infrastructure from scratch. QwikBuild occupies this cluster, achieving FES 68% alongside the highest CBS (49.3%) and CIS (67.3%). However, even this approach falls well short of targets: CBS of 49% against a 90% target means that half of required background jobs are missing or non-functional. Ecosystem-Leveraged (amber cluster). Replit and Emergent leverage their respective package ecosystems (npm libraries such as passport, prisma, node-cron) to generate backend infrastructure. Replit achieves the highest FES (74.3%) and leads on CIS (70%) and ESR (35%), demonstrating strong potential. However, CBS of 29.7% reveals inconsistency: infrastructure generation works on simpler prompts but degrades under complexity. Emergent’s CBS of 20.3% shows similar partial capability, compounded by a shared component defect (a<Select.Item />crash) that undermines otherwise functional applications. The gap between potential and reliability is the core challenge for this strategy. Frontend-Prioritized (red cluster). v0-Max and Lovable produce applications with strong visual presentation but minimal backend infrastructure. v0-Max achieves FES of 68% while generating zero database schema on P3 and zero background jobs across all prompts. Lovable’s CBS of 2% represents a single cron stub that does not execute. This strategy serves prototyping and stakeholder demonstrations but produces outputs that require substantial additional engineering for production deployment. These three strategies represent different positions on a frontend-infrastructure tradeoff. The community should note that no strategy has solved the full-stack problem: even the infrastructure-integrated approach fails more than half 20 556065707580 Frontend Engineering Score (%) 0 10 20 30 40 50 Cron & Background Jobs Score (%) QwikBuild Replit Emergent Lovable v0-Max Infrastructure- Integrated Ecosystem- Leveraged Frontend- Prioritized Frontend-Backend Decoupling: FES vs CBS Figure 7: FES vs. CBS averaged across ACR prompts. Three platform strategies emerge as clusters: Infrastructure- Integrated (green), Ecosystem-Leveraged (amber), and Frontend-Prioritized (red). The vertical spread at similar FES values (e.g., FES≈68% spans CBS 0–49%) demonstrates that frontend quality is a poor predictor of backend capability. its backend targets. Closing this gap—particularly for background jobs, external service integrations, and complex data pipelines—is the most impactful open challenge for AI app builders. 5.4 Finding 3: The Production Readiness Cliff A critical finding for users and platform builders alike is that every platform requires significant post-generation human effort to reach production readiness, and this effort varies by 5×across platforms. The platform requiring the least effort (12 developer-hours, 0 re-prompts) still falls short of production targets on most metrics, while the most effort-intensive requires 60 developer-hours and 8 re-prompts. Table 12 presents the production readiness gap across platforms. Table 12: Production readiness gap for P1 ExamEdge. ETF = developer-hours to reach production. PHE = re-prompts and manual code edits required. FGD = developer-hours to build missing features. CDI = gap between claimed and actual functionality (%). B0E1L0Q1R3V0 Eng. Score (%)–39.939.157.541.522.8 ETF (hours)602648123652 PHE (re-prompts)826035 PHE (code edits)5h0.5h3h0h1.5h3.5h FGD (hours)22152751930 CDI (%)35153241228 The general pattern—that higher resource consumption correlates with higher quality—holds consistently (Figure 8). The returns appear disproportionate: modest additional investment in upstream planning and multi-agent orches- tration correlates with the elimination of entire categories of downstream rework. This suggests that investment in 21 010203040506070 Effort-to-Fix (developer-hours) 20 30 40 50 60 Engineering Score (%) QwikBuild CDI=4% Emergent CDI=15% Replit CDI=12% Lovable CDI=32% v0-Max CDI=28% Base44 CDI=35% 60% Eng target 16h ETF target Cost-Quality Frontier: P1 ExamEdge Bubble = re-prompts 0 re-prompts 3 8 Figure 8: Cost-quality frontier across six platforms. Bubble size reflects post-PRD re-prompts; CDI labels indicate output trustworthiness. The top-right quadrant (high quality, low fix effort) remains sparsely populated, indicating that achieving both high quality and low post-generation effort is an unsolved challenge. 22 requirement elicitation and build-time feedback loops may be more cost-effective than post-generation debugging, a finding with direct implications for platform architecture decisions. Figure 9 decomposes the production readiness gap. The Claim Drift Index (CDI) reveals a compounding mechanism. Platforms with low CDI (4–12%) produce trustworthy observability artifacts (PRDs, execution traces), enabling efficient debugging. Platforms with high CDI (28–35%) force users to independently discover what is broken before any fixing can begin, adding a discovery overhead that compounds the already-larger fix effort. Reducing CDI—improving the accuracy of platforms’ self-reported build status—is a high-leverage improvement target for all platforms in this evaluation. 0102030405060 hours QwikBuild Emergent Replit Lovable v0-Max Base44 12 26 36 48 52 60 target 16 ETF (Effort-to-Fix) 010203040 hours 5 15 19 27 30 22 target 40 FGD (Feature Gap) 0102030 % 4 15 12 32 28 35 target 10 CDI (Claim Drift) Production Readiness Gap: P1 ExamEdge Figure 9: Production readiness gap decomposed into ETF, FGD, and CDI. All platforms require significant post- generation effort; the gap between best and worst is 3–5×. 5.5 Finding 4: Widespread Security and Infrastructure Failures No platform exceeds 65% Security Score against a 90% target (Table 13). Common failures include hard-coded API keys in frontend code, missing CSRF (Cross-Site Request Forgery) protection, absent rate limiting, public enumeration endpoints, and JSON Web Token (JWT) implementations with inconsistent expiry policies. Concurrency handling is weak across all platforms, ranging from 6% to 42% against a 70% target. This represents perhaps the most concerning finding for production deployment: AI-generated applications are systematically insecure, regardless of which platform generates them. Table 13: Security and infrastructure scores (%) averaged across ACR prompts. All values are below their respective targets. PlatformSS (tgt≥90)SAS (tgt≥70)CLS (tgt≥70)SWS (tgt≥70) Emergent50.737.734.712.7 Lovable51.733.715.018.0 QwikBuild63.754.042.09.3 Replit40.043.025.328.0 v0-Max34.337.76.040.3 Figure 10 visualizes these gaps against target thresholds. An unexpected pattern is that SWS is inversely correlated with overall engineering quality: v0-Max achieves the highest SWS (40.3%) while having the lowest engineering score (25.1%). This occurs because frontend-prioritized platforms produce well-structured HTML with proper meta 23 QwikBu Emerge Lovabl v0-Max Replit 0 20 40 60 80 100 target 90% 64 51 52 34 40 S QwikBu Emerge Lovabl v0-Max Replit 0 10 20 30 40 50 60 70 80 target 70% 54 38 34 38 43 SAS QwikBu Emerge Lovabl v0-Max Replit 0 10 20 30 40 50 60 70 80 target 70% 42 35 15 6 25 CLS QwikBu Emerge Lovabl v0-Max Replit 0 10 20 30 40 50 60 70 80 target 70% 9 13 18 40 28 SWS Security & Infrastructure Scores vs Targets Figure 10: Security and infrastructure scores with target thresholds (dashed lines). Every platform falls well short. The gap is most severe for Concurrency & Load (CLS), where v0-Max scores 6% against a 70% target. tags, even when backend infrastructure is absent. This highlights a measurement subtlety: metrics that evaluate only the frontend layer can be misleading indicators of overall application quality. 5.6 Preliminary AMR Analysis: Single-Platform Methodology Demonstration Important caveat: The AMR evaluation presented here has been completed only for QwikBuild (the platform affiliated with two of the authors), scored by the affiliated authors. These results are presented solely to demonstrate the AMR evaluation methodology and illustrate the ACR-to-AMR quality degradation pattern—not as a comparative ranking or evidence of platform capability. Cross-platform AMR evaluation is essential before any generalizable conclusions can be drawn. We strongly recommend that independent researchers replicate this protocol on multiple platforms using the released materials. Each AMR prompt was submitted to the deployed ACR application and scored independently. Several patterns emerge from the AMR evaluation. Modification scores are systematically 1–14 percentage points below creation scores, confirming that modification is harder than creation. The largest drops occur in CBS (−24.3p) and CRR (−14.0p). The CBS degradation is structural: new features introduced in AMR prompts (franchise reporting cron, AI dispatch batch job, provider health check) add scheduling complexity that the platform handles less reliably than initial cron setup. The CRR drop concentrates in the SURVIVING canary type—requirements that must evolve through modification rather than simply persist—indicating that context management partially loses track of constraints during structural changes. Two metrics improve under modification. ESR increases by+5.4p because AMR-introduced provider abstraction (P6) and AI Gateway integration (P4, P5) improve external service handling. LGS increases because AMR features (franchise model, AI dispatch, PDF export) add business value absent in the initial ACR builds. Notably, P6 (provider swap) achieves the first AIA PASS in the entire evaluation (75%), because the AMR-added provider abstraction layer (strategy + factory pattern) combined with platform-native AI integration exceeds the quality of any single ACR build’s AI implementation. This suggests that iterative refinement through AMR can sometimes improve upon initial ACR quality—an encouraging finding for the iterative vibe coding workflow. 5.6.1 AMR-Specific Diagnostic Metrics Beyond the 25 primary metrics scored on both ACR and AMR builds, SWE-WebDev Bench defines AMR-specific diagnostics within Category B (Build & Code) that evaluate modification-handling competency directly. Table 15 presents these results. Adaptive Coherence Score (ACS), defined asACS = 0.5× Existing ok + 0.5× Change correct , is the central AMR quality metric. P4 scores 84%—a near-miss on the 85% target—because the dual-axis modification (franchise 24 Table 14: QwikBuild (Q1) AMR scores (%) across three modification prompts, compared to ACR average. P4 is a complex franchise retrofit, P5 uses a vague frustrated-user prompt, P6 is a provider swap.∆= AMR avg minus ACR avg. Zero regressions across all AMR prompts. ACRP4P5P6AMR Grp MetricAvgFranchise AI Disp. SwapAvg∆ G1BIF92.5757510083.3−9.2 G1FCS84.080838683.0 −1.0 G1CRR97.785788883.7−14.0 G2SDS72.976726069.3−3.6 G2BLS68.465646665.0−3.4 G2FES68.072636265.7 −2.3 G2CHS52.854525052.0−0.8 G2ARC59.762565858.7 −1.0 G3CIS67.368626565.0−2.3 G3AIA72.068637568.7 −3.3 G3ESR28.333303833.7+5.4 G3CBS49.350151025.0−24.3 G4S63.761636563.0−0.7 G4SAS54.055525253.0−1.0 G4CLS42.038343836.7−5.3 G5CCIS87.772807876.7−11.0 G5ETF (h)14.710767.7−7.0 G5PHE000000 G6SWS9.313101011.0+1.7 G6AFQ68.065627266.3 −1.7 G7CDI (%)4.09757.0+3.0 Regression Rate–0%0%0%0%– CCIS Blast Radius–15%10%12%12.3%– Eng. Score54.562565859+4.5 Table 15: AMR-specific diagnostic metrics across three modification prompts. ACS (Adaptive Coherence Score) is the central AMR quality metric, measuring the joint probability that existing features remain functional and new changes are correctly implemented. Zero regressions across all prompts, but not all new features reach full functionality. MetricTargetP4P5P6Pass ACS (Adaptive Coherence) ≥85%8486902/3 R (Regression Rate)≤10%0003/3 CAR (Change Ack. Rate)100%1001001003/3 PUR (Plan Update Rate)100%1001001003/3 CPO (Change Proc. Overhead) ≤30% med3525202/3 Blast Radius (% files changed)Minimal1510123/3 New Features WorkingAll12/157/98/90/3 25 hierarchy + AI insights) introduces complexity that slightly degrades the “change correct” component: AI insight confidence intervals are not calibrated, and franchise royalty calculations are incomplete. P6 achieves the highest ACS (90%) because the provider abstraction pattern is architecturally clean and well-isolated from the existing codebase. CAR and PUR both achieve 100% across all three prompts, indicating that the platform’s multi-agent system acknowledges every change request and updates its execution plan before proceeding to code generation. This is the operational mechanism behind the zero-regression result: the planning phase explicitly accounts for each modification before code is touched. The new-features-working metric reveals an asymmetry in modification handling: while zero existing features break, not all new features reach full functionality (P4: 12/15, P5: 7/9, P6: 8/9). This suggests that the platform’s edit loop prioritises preservation of existing functionality over completion of new additions—a reasonable strategy for production applications where regressions are more costly than incomplete new features, but one that users should be aware of. Caveat: These AMR results were evaluated by authors affiliated with the platform. The zero-regression and 100% CAR/PUR results should be interpreted with this in mind. Independent replication is essential before these single-platform results can be generalized to claims about AMR methodology or platform capability. 5.6.2 Canary Survival Under Modification The AMR evaluation tracks 24 canary requirements across three prompts (7 + 9 + 8), classified as NEW (introduced by the AMR prompt) or SURVIVING (inherited from the parent ACR build and expected to persist or evolve). Figure 11 presents the per-stage survival rates. PRDPlanCodeDeploy 70 75 80 85 90 95 100 105 Canary Survival Rate (%) 100%100% 86% 85% 100%100% 83% 78% 100%100% 92% 88% Degradation at Code & Deploy only Per-Stage Canary Survival Rates (AMR) P4 Franchise (7 canaries) P5 AI Dispatch (9 canaries) P6 Provider Swap (8 canaries) Figure 11: Per-stage canary survival rates (PSSR) across AMR prompts. All canaries survive the PRD and Plan stages; degradation occurs exclusively at Code and Deploy stages. P5 shows the steepest Deploy-stage drop (78%), consistent with the higher ambiguity of its vague-prompt input. The SURVIVING canary type shows 3×the partial- loss rate of NEW canaries. A consistent pattern emerges: all canaries survive the PRD and Plan stages (100% across all prompts), but degradation occurs exclusively at the Code and Deploy stages. This indicates that the PM agent and planner correctly capture and 26 propagate canary requirements, but the coder agent occasionally fails to implement them fully or the deployment pipeline introduces partial failures. The SURVIVING canary type shows a higher loss rate than NEW canaries. Of the 10 SURVIVING canaries across all three prompts, 3 show partial degradation, compared to 1 among 14 NEW canaries. This 3×differential confirms that evolving existing constraints through structural modification is harder than implementing fresh requirements—a finding with direct implications for how platforms should prioritise context management during modification. Q1 ACR Q1 AMR E1R3L0B0V0 0 20 40 60 80 100 84 83 65 53 33 47 23 85% G1: Feature Completeness Q1 ACR Q1 AMR E1R3L0B0V0 0 20 40 60 80 100 98 84 57 54 22 24 18 -14p 80% G1: Canary Retention Q1 ACR Q1 AMR E1R3L0B0V0 0 20 40 60 80 100 73 69 46 57 65 41 3 -4p 80% G2: Schema Design Q1 ACR Q1 AMR E1R3L0B0V0 0 20 40 60 80 100 68 66 69 74 61 58 68 -2p 75% G2: Frontend Engineering Q1 ACR Q1 AMR E1R3L0B0V0 0 20 40 60 80 100 49 25 20 30 2 15 0 -24p 90% G3: Background Jobs Q1 ACR Q1 AMR E1R3L0B0V0 0 20 40 60 80 100 64 63 51 40 52 31 34 90% G4: Security Score Q1 ACR Q1 AMR E1R3L0B0V0 0 20 40 60 80 100 88 77 46 43 38 22 32 -11p 90% G5: Change Impact Q1 ACR Q1 AMR E1R3L0B0V0 0 10 20 30 40 4 7 18 14 37 36 34 +3p 10% G7: Claim Drift ACR vs AMR Performance: All Platforms' ACR Baselines with QwikBuild AMR Overlay Solid bars = ACR average (P1P3). Hatched bars = AMR average (P4P6, QwikBuild only). Arrows show ACRAMR degradation. Other platforms' AMR evaluation pending. QwikBuild (ACR) Emergent (ACR) Replit (ACR) Lovable (ACR) Base44 (ACR) v0-Max (ACR) QwikBuild (AMR) Target Figure 12: ACR vs. AMR performance across all six platforms and eight representative metrics (one per metric group). Solid bars show ACR averages (P1–P3) for all platforms; hatched bars show AMR averages (P4–P6) for QwikBuild only (cross-platform AMR evaluation pending). Arrows indicate ACR→AMR degradation. Key insight: QwikBuild’s post-modification scores often remain competitive with other platforms’ creation scores, suggesting that the AMR degradation pattern (−1 to−24p) would push lower-scoring platforms below viability thresholds. The CDI panel (lower is better) shows that modification slightly increases claim drift (+3p), compounding the trust problem for platforms that already exhibit high CDI on creation tasks. 27 6 Case Studies Case 1: When Requirement Inference Succeeds and Fails (P1 ExamEdge). This case illustrates the specification bottleneck at both extremes. a platform with 15 PM questions infers domain-specific needs not present in the prompt—e.g., “Will batch scheduling require conflict detection across branches?”—producing applications with proactive features. At the other extreme, v0-Max (V0) collapses the specified superadmin/teacher/student/parent hierarchy into a single “admin” role, achieving FCS of only 28%. The gap demonstrates that requirement elicitation is not optional for complex business applications: features missed upstream become expensive refactoring tasks downstream. Case 2: The Frontend-Backend Decoupling in Practice (P3 VettAI, Emergent). Emergent’s P3 application features a production-grade onboarding flow with animated transitions, contextual empty states (“No cases yet. Start by importing a client document”), and proper Securities and Exchange Board of India (SEBI) registration format validation (FES: 68%). However, a<Select.Item />component crash renders core case management non-functional—a shared infrastructure defect affecting all applications generated by the platform. The first 30 seconds of interaction suggest a polished product; the first 3 minutes of actual use reveal it is broken. This case exemplifies a risk for users: visual polish creates a false sense of production readiness. Case 3: Universal Failure Modes (P3 VettAI, QwikBuild). Even the highest-scoring platform exhibits significant shortcomings. QwikBuild achieves the worst SWS (SEO/Web Standards) score across all platforms (9.3% vs. v0-Max’s 40.3%), with missing meta tags, no sitemap, and absent structured data—basic web standards that frontend- prioritized platforms handle well. Its Concurrency & Load score (42%) reveals that AI-generated backend code lacks proper connection pooling, race condition handling, and load testing readiness. These gaps demonstrate that no platform has solved the full-stack problem: each platform’s architecture creates blind spots that SWE-WebDev Bench is designed to expose. Case 4: No Platform Solves the Full Stack (Cross-Platform Blind Spots). Each platform’s architecture creates characteristic blind spots that SWE-WebDev Bench surfaces. QwikBuild achieves the worst SEO/Web Standards score across all platforms (SWS: 9.3% vs. v0-Max’s 40.3%), with missing meta tags, no sitemap, and absent structured data—suggesting its multi-agent architecture deprioritizes web standards. Its Code Hygiene (52.8%) is the lowest among the top four platforms, and its External Service Reliability (28.3% vs. 80% target) indicates that even infrastructure-integrated approaches struggle with third-party API integration. Replit achieves the highest FES (74.3%) and CIS (70%) but asks zero business-flow questions, producing high-functioning code that often misses the user’s actual intent. Emergent produces the most visually polished applications but ships a shared component defect (<Select.Item />crash) that breaks all generated applications. These cross-cutting weaknesses demonstrate that current platforms have architectural affinities, not general competence—a finding designed to help each platform identify its specific improvement priorities. 7 Discussion 7.1 Why Existing Benchmarks Cannot Capture This The fundamental limitation of code-level benchmarks when applied to vibe coding platforms is dimensional collapse. These benchmarks evaluate a single dimension (code patch quality) whereas the output of a vibe coding platform spans multiple dimensions: requirement understanding, architectural decisions, code quality, deployment configuration, and business readiness. A platform could produce correct patches for all SWE-bench issues while being unable to build a coherent multi-role application from a business description. 7.2 Closed-Loop Code Generation as an Architectural Pattern Our data suggests that the quality advantage observed in some platforms does not come from any single phase (PM elicitation alone accounts for approximately 10–15% of the gap) but from the interaction between planning and execution. Platforms with feedback loops between planning and code generation can surface and resolve conflicts 28 mid-build, analogous to how a human development team works: the PM writes the spec, the developer finds an edge case, and the team adjusts the plan. Platforms without this feedback loop produce specifications in one pass, then generate code in one pass. Any misunderstanding or conflict between features is silently embedded into the codebase, where it surfaces as a bug that the user must discover and debug in AI-generated code they did not write and may not understand. We propose that closed-loop code generation—where build-time discoveries feed back into planning—is a promising architectural pattern for the community to explore. Preliminary AMR data from a single platform (QwikBuild) shows 100% Change Acknowledgment Rate and Plan Update Rate across three modification prompts (Table 15), which is consistent with this hypothesis. However, this observation is from a single platform evaluated by its affiliated authors, and requires independent cross-platform validation before any causal claims can be made. We note that other architectural patterns—such as Replit’s ecosystem-leveraged approach or Lovable’s plan-then-build model—may achieve similar feedback-loop benefits through different mechanisms that SWE-WebDev Bench is designed to evaluate. 7.3 Why These Metrics Matter for the Vibe Coding Era The shift from “AI as coding assistant” to “AI as software agency” changes what needs to be evaluated. When a developer uses Copilot or Cursor, they review every line of generated code. In vibe coding, the user delegates the entire build and trusts the output. This trust model makes several of our metrics essential in ways they were not for prior evaluation frameworks. Canary Retention Rate (CRR) matters because vibe coding users cannot verify whether subtle requirements (date format, currency convention, localization) survived the build. If CRR is low, the user ships software with silent specification violations. The AMR canary analysis (§5.6) extends this concern to modification: SURVIVING canaries degrade at 3× the rate of NEW canaries, meaning iterative development compounds specification drift. Claim Drift Index (CDI) matters because the platform’s dashboard is the user’s only window into what was built. If CDI is high, the user believes the app is complete when it is not. Code Change Impact Score (CCIS) matters because vibe coding users iterate frequently: “add a feature,” “change this,” “now also support Hindi.” If each modification breaks existing features (as FeatBench found in 73.6% of cases [7]), the iterative workflow collapses. Adaptive Coherence Score (ACS) extends CCIS to measure the joint quality of preservation and addition. The near-miss on P4 (84% vs. 85% target) demonstrates that even platforms with zero regressions can fall short of coherence targets when new features are partially implemented—a nuance that regression rate alone does not capture. These metrics would be unnecessary for evaluating a coding assistant where the developer verifies every output. They become essential for evaluating a system where the user trusts the output without inspection. 7.4 Three Domains as Orthogonal Diagnostic Probes Our three-domain design creates failure modes specific to each platform. EdTech (P1) tests inference depth: extracting a product from pain points and inferring Indian coaching conventions (JEE/NEET batch structures, April-March academic year). Field Service (P2) tests execution precision: a 10-state lifecycle, Goods and Services Tax (GST) invoicing, SLA engines with state pause/resume. FinTech-AI (P3) tests AI trustworthiness: building AI that is safe, auditable, and honest about uncertainty. Empirically, these probes reveal platform affinities. Replit excels on P3 (54.5%) but struggles with P1 (41.5%). Cross-domain variance ranges fromσ=3.7 (most consistent) toσ=7.0 (most variable), suggesting that domain generalization remains an open problem: platforms that perform well on one type of application may perform poorly on another. This domain sensitivity is itself a key finding that users should consider when selecting platforms for specific business contexts. 29 7.5 Benchmark Governance and Maintenance Plan SWE-WebDev Bench is designed as a living benchmark with community contribution. We adopt the following governance practices, informed by HELM [21] and Dynabench [22]. Versioning: Each evaluation is time-stamped with platform versions and evaluation dates; results from different time periods are not directly comparable. Community contributions: Any researcher can evaluate additional platforms using the released prompts, rubrics, and scoring protocol; contributed evaluations will be accepted into the public leaderboard after verification of protocol adherence. Re-evaluation cadence: We recommend quarterly re-evaluation of major platform updates to track improvement trajectories. Prompt expansion: The prompt suite will be expanded over time; contributed prompts must include canary requirements and domain expert validation. Conflict-of-interest policy: Future evaluations should be conducted by evaluators unaffiliated with any platform being scored, or at minimum should adopt blinded scoring for Tier 2 and Tier 3 metrics. 7.6 Limitations and Bias Mitigation Author affiliation with an evaluated platform. Two of the three authors are affiliated with QwikBuild, one of the six evaluated platforms. This creates a structural conflict of interest that affects multiple aspects of this work: the benchmark’s emphasis on PM behavior and requirement elicitation may inherently advantage platforms with dedicated PM agents (QwikBuild’s primary differentiator); and the AMR evaluation asymmetry (QwikBuild-only) creates additional exposure for the affiliated platform. We note that the canary requirements and AMR prompts were designed by the non-affiliated author (Saxena), who had no prior knowledge of QwikBuild’s architecture or capabilities. We take the following measures to mitigate potential bias: (1) all prompts, rubrics, and scoring protocols are released for independent replication; (2) G1 metrics use a three-person expert panel including one external evaluator; (3) we report QwikBuild’s failures, including its worst-of-field SWS (9.3%), below-target CLS (42%), lowest Code Hygiene among top platforms (52.8%), and poor External Service Reliability (28.3% against an 80% target); and (4) the benchmark framework is designed to be platform-agnostic—any party can evaluate any platform using the released materials. Blinded evaluation is recommended for future evaluations to further reduce the potential for unconscious bias. Single evaluator for some metrics. While G1 metrics use a three-person expert panel, most code quality metrics rely on LLM-assisted evaluation with human validation. Future work should expand to multiple independent evaluators with formal inter-rater reliability assessment. Small sample size. Six prompts across three domains, while varied in style and complexity, represent a limited sample. No statistical significance tests or confidence intervals are reported. Larger prompt suites and repeated evaluations are needed to establish robust rankings. Platform version sensitivity. AI platforms evolve rapidly. Our evaluation captures a snapshot from February–March 2026. Scores may differ substantially on subsequent versions. Limited AMR cross-platform data. AMR evaluation is comprehensive only for QwikBuild, creating an asymmetry in the evaluation. Extension to all six platforms is essential before drawing comparative AMR conclusions. Prompt engineering sensitivity. Prompt wording may favor certain architectures. We mitigate this by using three prompt styles (specific, vague, semi-vague) across domains, but additional prompt variation would strengthen the evaluation. No blinding. Evaluators were aware of which platform produced each output, which is a significant limitation given the author affiliation with one evaluated platform. We note that full blinding is operationally challenging for platform evaluation: platforms have distinctive UIs, deployment URLs, conversation patterns, and framework choices that make anonymization non-trivial. However, partial blinding is feasible and should be adopted in future evaluations. Specifically, code quality metrics (G2: SDS, BLS, FES, CHS, ARC) can be scored on anonymized codebases with platform-specific comments, deployment URLs, and distinctive framework identifiers stripped. We estimate that approximately 40% of the Engineering Score derives from Tier 0 fully automated metrics (Lighthouse, k6, npm audit) that are unaffected by evaluator awareness. The remaining 60% includes Tier 1–3 metrics where unconscious bias could influence scores. We recommend that future evaluations adopt a blinded protocol for at minimum Tier 2 and Tier 3 metrics, and we will provide anonymization scripts in the benchmark repository to facilitate this. 30 Benchmark design may favor certain architectures. The benchmark’s emphasis on PM behavior, requirement elicitation, and canary retention as first-class evaluation dimensions may structurally advantage platforms with dedicated PM agents over single-pass generation approaches. Similarly, the T4/T5 complexity tiers (multi-role SaaS, AI-native applications) favor platforms optimized for complex applications and may disadvantage platforms that excel at simpler use cases. Evaluation at T1–T3 complexity levels would provide a more complete picture. We designed the benchmark to surface capabilities we believe matter for production deployment, but we acknowledge that different design choices would produce different platform rankings. The benchmark is intended as a diagnostic tool that reveals where any platform falls short, not as a definitive ranking system. 8 Conclusion We have introduced SWE-WebDev Bench, a 68-metric framework for evaluating AI app-building platforms as virtual software agencies. Our initial evaluation of six platforms across three business domains uncovers four recurring shortcomings across the platforms we evaluated: 1.The specification bottleneck: in our sample, inference quality varies widely (IQS range: 20–70), and most skip requirement elicitation entirely, silently embedding unverified assumptions into generated applications. 2.Frontend-backend decoupling: visual quality is a poor proxy for engineering quality, with polished UIs masking absent backend infrastructure—a pattern that creates false confidence for non-technical users. 3. The production readiness cliff: no platform scores above 60% on engineering quality. Post-generation human effort varies substantially across platforms (ETF range: 14.7–65.7 developer-hours), meaning “vibe coded” applications universally require substantial engineering to ship. 4.Widespread security and infrastructure failures: no platform exceeds 65% Security Score against a 90% target, and concurrency handling is critically weak across the field. We emphasize that these observations are descriptive of our sample (n = 3prompts per domain,n = 6platforms) and require larger-scale replication to establish generality. Preliminary AMR evaluation reveals a fifth pattern: modification systematically degrades quality (16 of 19 metrics decline from ACR to AMR), with SURVIVING canary requirements showing 3×the loss rate of new requirements. However, the observation that targeted modifications can sometimes improve upon creation quality (the P6 provider- swap AIA PASS) suggests that the ACR/AMR boundary is not simply a quality cliff but a more nuanced interaction between existing code and new requirements. These findings suggest that the vibe coding paradigm, while transformative in its accessibility, has not yet achieved the reliability required for production deployment without human oversight. The most impactful areas for community investment are: (a) requirement elicitation and specification fidelity, where most platforms skip clarification entirely; (b) backend infrastructure generation, particularly background jobs and external service integration, where even the best platforms fall well below targets; (c) security hardening, where no platform exceeds 65% against a 90% target; (d) code maintainability and hygiene, where no platform exceeds 60%; (e) external service reliability, where the field-best score is only 35% against an 80% target; and (f) web standards compliance, which varies inversely with engineering quality and remains unaddressed by infrastructure-focused platforms. We release our evaluation framework, scoring protocols, and prompt suite as a community benchmark. 1 We encourage independent evaluations—particularly those conducted by parties unaffiliated with any evaluated platform—and invite platform builders to use SWE-WebDev Bench to identify and address gaps in their systems. 1 Repository:https://github.com/snowmountainAi/webdevbenchThe repository includes all 6 prompts, 80 canary re- quirement specifications, 9 LLM judge prompt templates, raw scoring data in CSV format, evaluation rubrics, and anonymization scripts for blinded evaluation. 31 References [1] A. Karpathy. “The hottest new programming language is English.” Twitter/X, January 2023. [2] A. Karpathy. “There’s a new kind of coding I call ‘vibe coding’ . . . ” Twitter/X, February 2025. [3] M. Chen, J. Tworek, H. Jun, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [4]X. Du, M. Liu, K. Wang, et al. ClassEval: A manually-crafted benchmark for evaluating LLMs on class-level code generation. arXiv preprint arXiv:2308.01861, 2023. [5]C. E. Jimenez, J. Yang, A. Wettig, et al. SWE-bench: Can language models resolve real-world GitHub issues? In ICLR, 2024. [6] L. Zhang, S. He, C. Zhang, et al. SWE-bench goes live! arXiv preprint arXiv:2505.23419, 2025. [7] H. Chen, C. Li, J. Li. FeatBench: Towards more realistic evaluation of feature-level code generation. arXiv preprint arXiv:2509.22237, September 2025. [8]H. Tran, L. Nashold, R. Krishnan, A. Bigeard, A. Gu. Vibe Code Bench: Evaluating AI models on end-to-end web application development. arXiv preprint arXiv:2603.04601, March 2026. [9] C. Liu, Y. Fu, W. Yang, Y. Zhang, T. Xie. WebCoderBench: Benchmarking web application generation with comprehensive and interpretable evaluation metrics. arXiv preprint arXiv:2601.02430, January 2026. [10]M. Ortiz et al. From Prompt to Product: A human-centered benchmark of agentic app generation systems. arXiv preprint arXiv:2512.18080, December 2025. [11]Z. Lu, Y. Yang, H. Ren, et al. WebGen-Bench: Evaluating LLMs on generating interactive and functional websites from scratch. arXiv preprint arXiv:2505.03733, May 2025. [12]ByteDance Seed Foundation Code Team. FullStack Bench: Evaluating LLMs as full stack coders. arXiv preprint arXiv:2412.00535, December 2024. [13]S. Zhao, D. Wang, K. Zhang, J. Luo, Z. Li, L. Li. Is vibe coding safe? Benchmarking vulnerability of agent-generated code in real-world tasks. arXiv preprint arXiv:2512.03262, December 2025. [14]Cognition Team. A review of OpenAI’s o1 and how we evaluate coding agents. Cognition AI Blog, September 2024. [15]J. Austin, A. Odena, M. Nye, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. [16]Snorkel AI. Closing the evaluation gap in agentic AI: Open Benchmarks Grant program. Snorkel AI Blog, February 2026. [17]Trae Research Team, P. Gao, Z. Tian, et al. Trae agent: An LLM-based agent for software engineering with test-time scaling. arXiv preprint arXiv:2507.23370, 2025. [18]L. Zheng, W.-L. Chiang, Y. Sheng, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In NeurIPS, 2023. [19]S. Kim, J. Shin, Y. Cho, et al. Prometheus: Inducing fine-grained evaluation capability in language models. In ICLR, 2024. [20] Z. Li, X. Li, Y. Liu, et al. Generative judge for evaluating alignment. In ICLR, 2024. 32 [21]P. Liang, R. Bommasani, T. Lee, et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. [22] D. Kiela, M. Bartolo, Y. Nie, et al. Dynabench: Rethinking benchmarking in NLP. In NAACL, 2021. [23]T. Gebru, J. Morgenstern, B. Vecchione, et al. Datasheets for datasets. Communications of the ACM, 64(12):86– 92, 2021. 33 Appendix A Complete Per-Metric Scores: P1 ExamEdge Table 16: All primary metrics for P1 ExamEdge Academy across five platforms. GrpMetricE1L0Q1R3V0 G1BIF (Business Intent)3/42/44/42/41/4 G1FCS (Feature Complete)65.035.088.055.028.0 G1CRR (Canary Retention)57.029.0100.048.021.0 G2SDS (Schema Design)40.848.081.758.03.0 G2BLS (Backend Logic)52.530.069.256.08.0 G2FES (Frontend Eng.)60.845.075.070.062.0 G2CHS (Code Hygiene)47.550.057.554.052.0 G2ARC (Architecture)33.038.066.050.038.0 G3CIS (Core Integration)52.022.072.076.030.0 G3ESR (External Svc)20.08.025.032.010.0 G3CBS (Background Jobs)8.03.052.035.00.0 G4S (Security)45.052.062.038.032.0 G4SAS (Scalability)42.028.058.040.038.0 G4CLS (Concurrency)32.018.041.022.05.0 G5CCIS (Change Impact)50.040.090.046.035.0 G6SWS (SEO/Web Std.)14.022.010.026.036.0 G6LGS (Lead/Growth)30.020.038.025.022.0 G6MLS (Multilingual)58.030.070.022.012.0 G7ETF (Effort-to-Fix)26h48h12h36h52h G7FGD (Feature Gap)15h27h5h19h30h G7CDI (Claim Drift)15%32%4%12%28% Eng. Score39.939.157.541.522.8 34 B PM Agent Behavioral Analysis Table 17: PM Agent behavior summary per platform. CTC = Turns to Convergence. PlatformPM BehaviorAvg QsCTC Base44P3 only: 1 turn to convergence. ECR 80% but CGS 8.3%. Vendor-locked backend with auth bypass vulnerability. 1.01.0 EmergentAsks 5 technical/config questions per prompt. No business inference. Detects contradiction only when directly encountered. 52.0 LovableAsks 4 strategic questions per prompt. Some scope negotiation. Catches contradiction on P2 but limited proactive discovery. 42.0 QwikBuild Dedicated PM agent with multi-round Q&A. 6 rounds, 15 questions on P1. Proactively detects contradictions. Infers domain requirements (JEE/NEET conventions, Indian financial year). 155.3 ReplitP1/P3: zero questions. Immediately outputs implementation plan without requirement exploration. 0.30.3 v0-MaxP1: 4 config questions. P3: zero questions, directly generates plan. No PM agent behavior. 1.31.3 C Metric Independence Analysis To assess whether the 15 analyzed primary metrics measure distinct constructs, we compute Kendall’sτrank correlation across 18 data points (6 platforms×3 prompts). Table 18 summarizes the distribution of pairwise correlations. Table 18: Distribution of pairwise Kendall’sτcorrelations across 15 primary metrics (105 pairs,n = 18data points). Correlation strengthPairs% Strong (|τ| > 0.70)1211% Moderate (0.40 <|τ|≤ 0.70)4947% Weak or none (|τ|≤ 0.40)4442% Strongly correlated pairs. Twelve pairs show strong correlation (|τ| > 0.70, allp < 0.001). The strongest: CIS–ESR (τ = 0.88), BLS–CLS (τ = 0.86), CLS–FCS (τ = 0.83). These are expected: platforms with strong backend naturally score higher on both integration and concurrency. We retain both metrics because they serve distinct diagnostic purposes. Independent metrics. CHS (Code Hygiene) is nearly uncorrelated with most metrics (|τ| < 0.15with BLS, CLS, CRR, CIS, ESR), confirming code maintainability is an independent quality axis. FES shows weak correlation with backend metrics (|τ| < 0.30with FCS, CBS, CRR), confirming the frontend-backend decoupling finding. SWS is inversely correlated with most quality metrics (FCS:τ = −0.60; CLS:τ = −0.56; S:τ = −0.54), confirming that frontend-prioritized platforms score higher on web standards despite lower engineering quality. Group separability. Average within-group|τ| = 0.48vs. between-group|τ| = 0.44(ratio1.10×). Groups provide useful organizational structure but do not represent fully independent latent factors. 35

Instant research discovery