Paper deep dive
BrickBench: Evaluating Agentic Brick Design
Peter Kulits, Yiqing Xu, R. Kenny Jones, Cordelia Schmid, Jiajun Wu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/11/2026, 2:27:11 AM
Summary
The paper introduces BrickBench, a benchmark for evaluating agentic text-conditioned LEGO-set design, and BrickAgent, an environment for constructing and validating these designs. The study evaluates eleven frontier coding agents and two data-driven baselines across three settings (Model, Set, Alt-Build) varying in scale and part availability. Results show that leading agents achieve high physical validity and semantic alignment but fall short of human design quality.
Entities (14)
Relation Signals (12)
BrickBench â hassetting â Model Setting
confidence 95% · BrickBench contains 300 prompts across three settings: Model (at most 400 parts)...
BrickBench â hassetting â Alt-Build Setting
confidence 95% · BrickBench contains 300 prompts across three settings: ... and Alt-Build (the part inventory of set 10698).
BrickBench â hassetting â Set Setting
confidence 95% · BrickBench contains 300 prompts across three settings: ... Set (400â4000 parts)...
BrickBench â uses â BrickAgent
confidence 95% · We provide BrickAgent, an environment for coding agents to construct, inspect, and validate their designs... To support complex construction, we introduce BrickAgent...
BrickAgent â uses â LDraw
confidence 93% · It contains the LDraw library, the BrickNet connector system, and tools for authoring, validating, and viewing an assembly programmatically.
BrickBench â evaluates â GPT-6 Astra
confidence 92% · We evaluate eleven frontier agents... GPT-6 Astra... With BrickAgent, leading agents produce valid assemblies... the best satisfies approximately 95% of semantic requirements.
BrickBench â usesmetric â VQA
confidence 92% · We score validity, alignment, and design... VQA measures bounded semantic alignment with the prompt...
BrickBench â â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We propose BrickBench, a benchmark for agentic text-conditioned LEGO-set design. Given a prompt, an agent is tasked with producing an assembly that not only satisfies semantic and design criteria, but that can also be physically built. To do so, it must select parts from a discrete library and reason jointly about local and global constraints. We score validity, alignment, and design across three settings that vary in scale and part availability. We provide BrickAgent, an environment for coding agents to construct, inspect, and validate their designs. We find that leading agents largely satisfy verifiable physical and semantic requirements, but fall short of human designs. We release our benchmark and environment at this http URL
Tags
Links
- Source: https://arxiv.org/abs/2610.12452v1
- Canonical: https://arxiv.org/abs/2610.12452v1
Trouble viewing inline? Open PDF directly â
Full Text
56,992 characters extracted from source content.
Expand or collapse full text
10-9-2026 BRICKBENCH: E VALUAT ING A G ENTIC BR IC K DES I GN Peter Kulits 1,2 Yiqing Xu 1 R. Kenny Jones 1 Cordelia Schmid 3 Jiajun Wu 1 1 Stanford University 2 Max Planck Institute for Intelligent Systems 3 Inria Model A pelican riding a bicycle. SetA brown dog with floppy black ears and a small brown hat sits at a table in a burning room. A white coffee mug rests in front of it. Flames climb the curtains and surround the chair, but the dog sits upright with its paws resting calmly on the tabletop. The front of the room is open to reveal the scene. Alt-BuildA cowboy in a wide-brimmed hat sits on a saddled horse beside a cactus with two raised arms. He holds a looped lasso above his head. The horseâs front legs stand on a low rock, while its rear legs remain on the sandy ground. GPT-6 AstraClaude Opus 5.5Qwen 3.8 FlashGemini 3.8 FlashMuse Spark 1.3 Figure 1: We evaluate coding agents on their ability to design LEGO assemblies from a text prompt. We introduce three settings with varying constraints:Model(â€400 parts),Set(400â 4000 parts), andAlt-Build(the part inventory of set 10698). Each assembly above can be built. A B S T R A C T We propose BrickBench, a benchmark for agentic text-conditioned LEGO-set design. Given a prompt, an agent is tasked with producing an assembly that not only satisfies semantic and design criteria, but that can also be physically built. To do so, it must select parts from a discrete library and reason jointly about local and global constraints. We score validity, alignment, and design across three settings that vary in scale and part availability. We provide BrickAgent, an environment for coding agents to construct, inspect, and validate their designs. We find that leading agents largely satisfy verifiable physical and semantic requirements, but fall short of human designs. We release our benchmark and environment athttp://w.brickben.ch arXiv:2610.12452v1 [cs.AI] 8 Oct 2026 1. Introduction Designing a LEGO assembly is deceptively difficult. A designer must select from thousands of parts, arrange them into a recognizable form, and ensure that the structure holds together. Computational generative approaches to brick assembly have relied on specialized models and carefully engineered representations to handle these challenges (Chung et al., 2021; Pun et al., 2025; Kulits & Schmid, 2026). But coding agents offer a different approach: they can write programs, execute them, inspect the results, and revise. This shift is already transforming software engineering (Jimenez et al., 2024), graphics (Gu et al., 2025), and CAD (Zhang et al., 2026). We find that a general coding agent, without task-specific training, substantially outper- forms specialized LEGO-generation models on the BrickNet benchmark (Kulits & Schmid, 2026), approaching the semantic alignment of its reference assemblies (Fig. 2). Yet does this mean agents have learned to design LEGO? Existing evaluations leave two im- portant challenges unexplored. First, physical complexity: BrickNet evaluates small objects of at most 100 parts, far from the hundreds or thousands of interconnected pieces and inventory constraints of a full LEGO set. Second, and more fundamentally, design quality: making an assembly that works is not the same as making an assembly worth building. The appeal of LEGO lies not merely in reproducing an object, but in the creativity of its construction: clever part usage, shapes, proportions, and details. Two builds can be equally valid and faithful to a description, yet differ dramatically in how well they are designed. Unlike physical validity or semantic alignment with a prompt, design quality has no definitive pass/fail test. We introduce BrickBench, a benchmark for agentic text-conditioned LEGO-set design (Fig. 1). BrickBench contains 300 prompts across three settings:Model(at most 400 parts),Set(400â 4000 parts), andAlt-Build(a fixed inventory of parts). To support complex construction, we introduce BrickAgent, an environment for agentic LEGO design including construction tools and interpretable validation of connectivity, collisions, and stability. Rather than prescribing how to design an assembly, BrickAgent provides the building blocks, constraints, and feedback, leaving the agent to devise its own solution. To evaluate what validity and semantic alignment miss, we measure design quality through pairwise judgments, validated against human preferences. We evaluate eleven frontier agents and two data-driven baselines. With BrickAgent, leading agents produce valid assemblies for nearly every prompt, and the best satisfies approximately 95% of semantic requirements. Removing the environment sharply reduces physical validity, showing that reliable construction depends on grounded tools and feedback. Design quality, however, separates agents far more strongly, and human raters identify human-designed assem- blies in 323 of 360 comparisons. These findings reveal a shift in the challenge of LEGO design: Agents are learning to build what works, but not yet what makes a great design. Our contributions are as follows. 1. BrickBench, a benchmark of 300 prompts in three settings for agentic brick design, with metrics for physical validity (connections, collisions, and stability under simulation), se- mantic alignment (VQA, over yes/no questions decomposed from each prompt), and design quality (ELO, from pairwise judgments by a VLM validated against human raters). 2.BrickAgent, an environment enabling coding agents to programmatically construct, in- spect, and validate complex LEGO assemblies, with part search, placement by connectors, subassemblies, rendering, and a validator that names parts at fault. Without it, the share of valid assemblies falls from 100% to 40% for GPT-6 Astra and to under 1% for GPT-5.6 Luna. 3.A systematic evaluation of eleven frontier agents and two data-driven baselines, with human studies that validate the design metric and measure the gap to human design. 2 2. Related Work Agentic Benchmarks. Benchmarks for coding agents validate an agentâs work by executing it. SWE-bench (Jimenez et al., 2024) runs a repositoryâs tests on a patch an agent supplied, and OSWorld (Xie et al., 2024) tracks state through execution. SceneCraft (Hu et al., 2024) writes Blender scripts, and CAD-Assistant (Mallis et al., 2025) writes CAD programs. Benchmarks then execute the program and score the result. BlenderGym (Gu et al., 2025) compares the result with a goal scene, and BenchCAD (Zhang et al., 2026) executes CadQuery programs for industrial parts. BuildArena (Xia et al., 2026) scores machines assembled from modular parts by their simulated behavior, and PhyBlock (Ma et al., 2025) scores block placements for plausibility in a physics simulator. BrickBench executes the agentâs program into an assembly of real LEGO parts and validates connections, collisions, and stability under simulation. LEGO and Evaluation. Prior work generates LEGO assemblies with learned models and uses LEGO tasks to evaluate others. BrickGPT (Pun et al., 2025) fine-tunes a language model to place eight brick types on a 20Ă20Ă20 grid from a caption. BrickNet (Kulits & Schmid, 2026), predicts connections rather than positions and autoregressively generates structures through typed connectors. LEGO-Maker (Ge et al., 2025) produces building facades conditioned on an image. Other work evaluates agents and multimodal models on LEGO tasks. Break and Make (Walsman et al., 2022) gives an agent an LDraw model in a simulator to test whether it can inspect it, take it apart, and rebuild it from scratch. LEGO-Puzzles (Tang et al., 2026) presents multimodal models spatial questions about assembly steps. BrickBench does both: general coding agents generate set-scale assemblies from a text prompt, while we score the assemblies on physical validity, prompt alignment, and design. 3. Preliminaries 3.1. LDraw LDraw is a community-maintained part library and standard for digital LEGO modeling 1 . Each library part is intended to mirror a real-world counter-part. Assemblies are represented as text files of instances defined by their part type, color, and 6D pose. LDraw does not itself, however, determine whether an assembly holds together or could be built. 3.2. Connectivity and Physical Validity BrickNet (Kulits & Schmid, 2026) annotates each part in the LDraw library with typed connectors. Each connector instance has a type, a subtype, a polarity, and a frame. The subtype names the connector âs geometry within its type and fixes what it mates with: a stud mates with a hole or a tube, and a bar with a clip. The polarity distinguishes the two halves of a hinge, ball, or fixed joint, which must be opposite to connect. The frame is the connectorâs position and orientation in the coordinates of its part, with one axis along the direction of mating. Two part instances are connected when a compatible pair of their connectors is aligned up to tolerance. There are five connector types. A stud connector allows rotation about the connection axis, a hinge allows rotation and a possible flip, and an axle additionally allows a slide along its axis. A ball allows free rotation, and a fixed-type connector allows no motion. The connections between instances of an assembly form a graph over parts. An assembly may have more than one component to represent a scene with multiple (disconnected) objects. We define an assembly as stable if it remains at rest under gravity. We treat connected components 1 https://w.ldraw.org 3 This is a small, blocky LEGO robot model, primarily constructed from yellow and black bricks, featuring articulated limbs with wheel-like hands and feet, and presented from multiple angles to showcase its design. This is a 3D model of a classic LEGO minifigure, featuring a yellow head and hands, a red baseball cap, a plain white torso, and blue legs, shown from multiple angles against a white background. GTGPT-5.6 Luna BrickNet-14BBrickGPT (a) MethodPEâ SigLIP 2â VQAScoreâ BrickGPT0.1570.0520.050 BrickNet-0.6B 0.2790.6030.557 BrickNet-1.7B 0.2820.6310.593 BrickNet-4B0.2830.6390.615 BrickNet-8B0.2840.6470.608 BrickNet-14B 0.2830.6250.602 GT0.3150.8260.748 GPT-5.6 Luna 0.313 0.8180.732 (b) Figure 2: BrickNet validation. We evaluate GPT-5.6 Luna on the BrickNet text-to-assembly validation set (Kulits & Schmid, 2026). While it was not specialized for this dataset, we find it exceeds task-specific baselines including BrickGPT (Pun et al., 2025) across metrics and nears the alignment of the ground-truth objects. (a) Sample inputs and model outputs. (b) Caption align- ment following BrickNetâs evaluation protocol. PE (Bolya et al., 2025) and SigLIP 2 (Tschannen et al., 2025) score the similarity between the embeddings of the renders and of the caption, and VQAScore (Lin et al., 2024) the probability that a VQA model answers yes when asked whether the renders show the caption. See Kulits & Schmid (2026) for details on metric computation. as rigid bodies of the assembly and simulate them as single rigid bodies of uniform density in PyBullet (Coumans & Bai, 2016â2021). We set a high coefficient of friction and no restitution, so that inter-component contacts neither slide nor bounce. A component then fails by falling when it is not supported. We consider an assembly stable if no point of any component moves more than three quarters the height of the stud of a brick, 3 LDraw Units (LDU). Effects such as the holding force of stud connections and the play of axles are not modeled, so an assembly that balances as a rigid body passes even when a built one might sag or fall apart. Evaluating this would require a structural simulation, with either a force limit at each connection or a finite-element model of the parts, which we discuss in Sec. 6. Two parts collide when their geometry overlaps by more than a tolerance. An assembly with collisions cannot be built, and is determined invalid prior to simulation. 3.3. Text-Conditioned Assembly In text-conditioned assembly, a system receives a text prompt and is tasked with producing an assembly that depicts it. The system must select part types from the library and assign each instance a color, position, and 3D rotation. Prior work BrickNet trains a language model for this task on human-designed assemblies collected from online. Its captions are generated by a visionâlanguage model over renders of existing assemblies. The validation set contains 512 such held-out captions from single objects of at most 100 parts each. We evaluate GPT-5.6 Luna, a general coding agent, on the BrickNet validation prompts (Fig. 2). The agent is given the environment described in Sec. 4.3. To match BrickNet, we limit the agent to 100 parts. An output is scored by the alignment between its renders and the prompt, measured with PE (Bolya et al., 2025), SigLIP 2 (Tschannen et al., 2025), and VQAScore (Lin et al., 2024). 4 Without having been trained on the dataset, the agent substantially outperforms task-specific baselines BrickNet and BrickGPT (Pun et al., 2025), scoring within 0.02 of the held-out (GT) assemblies on each metric and effectively saturating the evaluation. This motivates our design of a new benchmark. 4. BrickBench BrickBench is a benchmark of agentic text-conditioned LEGO-set design. Given a text prompt, an agent must produce an assembly that satisfies semantic and design criteria and that can be physically built. The benchmark consists of 300 prompts in three settings (Sec. 4.1), an environment in which a coding agent can build, and an evaluation protocol for validity, semantic alignment, and design quality (Sec. 4.4). Except for the ablation in Sec. 5.2 and the two data- driven baselines, the agents work inside of the BrickAgent environment described in Sec. 4.3. 4.1. Settings The settings differ in the parts that an agent builds with. In theModelsetting, an agent may use at most 400 parts, while theSetsetting requires 400â4000 parts be used. Both draw from the full LDraw library as supported by BrickNet, 14,441 part types. In theAlt-Buildsetting, an agent may use only the 783 pieces, of 104 part types, of the retail set 10698, 2 containing mostly basic bricks and plates with a few windows, doors, wheels, and decorative elements. An assembly that violates these limits fails the validity check. The settings are intended to evaluate different abilities.Modelserves as the base evaluation and tests the creation of assemblies consisting primarily of a single component.Set, whose size is chosen to mirror the requirements of the BrickLink Designer Program, 3 tests whether an agent can build an assembly on the scale of a full-size retail set.Alt-Buildevaluates agent ability to design given a scarce, fixed part inventory. 4.2. Prompts Each evaluation setting has 100 prompts, with ten in each of ten topic categories. They follow the BrickLink Designer Program, except that we split the Floral / Nature / Animal category into Floral / Nature and Animal. The categories are Medieval / Castle, Vehicle / Boat / Airplane, Train, Space / Sci-Fi / Fantasy, Art / Object, Building, Floral / Nature, Animal, Pirate, and History / Period. See Figs. 1 and 4 for prompt samples by setting. The prompts were constructed through prompting of an LLM and were manually validated for conformity. Prompts average 31 words in Alt-Build, 45 in Model, and 57 in Set. We decompose each generation prompt into a question graph in the style of DSG (Cho et al., 2024), which evaluates text-to-image generation by turning a prompt into atomic yes/no questions with an LLM and answering them with a VLM after organizing them by graph dependency. The questions cover the entities named by the prompt, their attributes, and the relations among them. With the graph we can avoid evaluating properties invalidated by other failures, such as checking the color of a knightâs armor when the presence of a knight was not yet established. See Fig. A1 for their distribution by type and Fig. A2 for a full example graph, both in Sec. A. 2 https://w.bricklink.com/v2/catalog/catalogitem.page?S=10698 3 https://w.bricklink.com/v3/designer-program/main.page 5 4.3. BrickAgent We introduce BrickAgent, an environment in which the agents build. It contains the LDraw library, the BrickNet connector system, and tools for authoring, validating, and viewing an assembly programmatically. The tools include functions to search the library for parts, to place parts by their positions or by connectors, to mirror and measure them, to define subassemblies and combine them, and to render an assembly from any viewpoint. The provided validator includes the connectivity, collision, and stability tests of Sec. 3.2 and interpretably reports each failure with the parts involved. This enables an agent to validate the physical requirements of an assembly prior to submission. The prompt explains the environment with worked examples and states the settingâs part requirements. It informs the agent that experienced LEGO designers will judge the assembly against others by the standards of a published set. In the ablation of Sec. 5.2 the agent operates in the same container, with the harness CLI, LDraw library, and task requirements, but without the tools of BrickAgent. We find that without these tools, frontier coding agents cannot reliably construct valid builds. 4.4. Metrics We score each system per setting, and additionally aggregate across all three (Overall). An assembly is judged as valid if it meets the part requirements of its setting and is both collision-free and stable under the tests of Sec. 3.2. Valid is a proportion and is scored on every prompt. VQA measures bounded semantic alignment with the prompt, following DSG. We render the assembly from eight views using BrickNetâs renderer, with consistent render settings. We then ask Gemma 4 31B (Gemma Team, 2026) each question of the graph in one call, with instructions to judge on what the views show and to answer yes or no. The score of an assembly is the fraction of its questions satisfied, and VQA is the mean over assemblies. ELO measures relative semantic alignment and design quality against other systems. For each prompt, we form every pairing of assemblies from different systems and show the judge four views of each. We ask two questions of each pair: 1) The alignment question gives the judge the prompt and asks which assembly matches it better. 2) The design question withholds the prompt and asks which assembly is better designed by the standard of an official set, judged on how the pieces are used rather than the subject. Each pair is asked in both orderings to eliminate position bias. Fig. A3 in Sec. B contains the judge prompts. We fit a BradleyâTerry model (Bradley & Terry, 1952) to the wins from each question and report them as ELO. We anchor to a fixed set of references as noted in Sec. 5.ELOis the average ofAlign ELOandDesign ELOafter rescaling to a common spread over the anchors. VQAandELOare scored on the assemblies generated by a system, without considering whether they areValid.Costis the list price of the tokens consumed by a system for an assembly in US dollars. For the two trained baselines it is the serving cost per assembly, at serverless token rates for the base model of BrickNet and as wall-clock time on a rented GPU instance for BrickGPT. 5. Evaluations We evaluate eleven agents and two data-driven baselines. The agents include GPT-6 Astra, GPT-6.1 Sol, GPT-5.6 Sol, GPT-5.6 Luna, Claude Opus 5.5, Claude Opus 5, Gemini 3.8 Flash, Qwen 3.8 Flash, DeepSeek V4.1 Flash, GLM 5.3 Flash, and Muse Spark 1.3. Opus 5 and Opus 5.5 run in Claude Code, and the others in Codex CLI using the vendorsâ respective APIs. Astra, both Opus models, and both Sol models are evaluated athighreasoning to bound cost while 6 Table 1: BrickBench. We report metrics for each of the three evaluation settings and in the aggregate (Overall).Validrequires that the assembly satisfies the part requirement of the setting and is both collision-free and stable.VQAis the proportion of VQA questions satisfied (Sec. 4.4).ELO,Align ELO, andDesign ELOare ratings estimated from pairwise VLM judg- ments (Sec. 4.4).Costis the API cost in US dollars, or its equivalent for BrickGPT (Pun et al., 2025), of producing one assembly. *TheValidmetric of BrickNet-14B (Kulits & Schmid, 2026) doesnât include stability as the model doesnât produce assemblies in a canonical frame and the direction of gravity is undefined. SystemValidâVQAâELOâAlign ELOâDesign ELOâCost ($)â Overall GPT-6 Astra1.000.9541297±221295±271299±274.06 GPT-6.1 Sol1.000.9451293±231301±271286±260.87 Claude Opus 5.50.990.9231249±231212±261287±276.33 Claude Opus 51.000.9081101±171095±201108±2016.47 Qwen 3.8 Flash0.900.8521048±171040±211057±201.75 GPT-5.6 Sol1.000.8591015±161041±19988±191.02 Gemini 3.8 Flash0.980.8561014±171005±211024±203.70 DeepSeek V4.1 Flash0.940.8131006±17998±201015±200.71 GPT-5.6 Luna1.000.792898±16917±19879±200.75 Muse Spark 1.30.800.718868±20859±24878±236.68 GLM 5.3 Flash0.920.590752±23750±25753±290.54 Model GPT-6 Astra1.000.9661266±381277±451254±454.08 GPT-6.1 Sol1.000.9691269±361272±451265±420.92 Claude Opus 5.51.000.9521236±401216±461252±425.90 Claude Opus 51.000.9171083±301075±361089±3116.72 Qwen 3.8 Flash0.950.9211012±271030±30996±321.59 GPT-5.6 Sol1.000.9081032±281051±341015±311.14 Gemini 3.8 Flash0.980.8941012±34995±401028±374.05 DeepSeek V4.1 Flash0.950.8571007±301013±331001±330.65 GPT-5.6 Luna1.000.821892±30897±34889±370.99 Muse Spark 1.30.860.811848±31845±38851±365.99 GLM 5.3 Flash0.900.658848±37816±42878±430.69 BrickNet-14B0.260.180731±53485±62952±500.0006 BrickGPT1.000.024338±45129±64528±600.04 Set GPT-6 Astra1.000.9331305±351291±421316±465.15 GPT-6.1 Sol1.000.9201295±411293±471292±451.05 Claude Opus 5.50.990.8921240±441174±451315±538.08 Claude Opus 51.000.8901100±271092±311109±3418.80 Qwen 3.8 Flash0.790.8011090±381054±441131±412.26 GPT-5.6 Sol1.000.832993±271019±32963±321.16 Gemini 3.8 Flash0.970.8061012±311010±361013±364.26 DeepSeek V4.1 Flash0.870.7861005±35979±391036±410.87 GPT-5.6 Luna1.000.758929±26952±31902±350.85 Muse Spark 1.30.680.661871±38864±46882±428.54 GLM 5.3 Flash0.890.523695±33738±41649±450.61 Alt-Build GPT-6 Astra1.000.9611325±371314±511336±442.95 GPT-6.1 Sol1.000.9461326±351339±431312±460.64 Claude Opus 5.50.990.9251277±371248±411307±455.01 Claude Opus 51.000.9171126±311120±361132±3613.90 Qwen 3.8 Flash0.970.8271051±271038±351064±361.39 GPT-5.6 Sol1.000.8361021±311054±36986±330.76 Gemini 3.8 Flash1.000.8671023±251010±311036±312.79 DeepSeek V4.1 Flash0.990.7971007±291002±331014±340.60 GPT-5.6 Luna1.000.797875±28903±39845±370.40 Muse Spark 1.30.860.678887±30871±36904±375.52 GLM 5.3 Flash0.970.591686±36688±47684±470.31 7 0.51 Valid â 0.51 VQA â 80010001200 ELO â 051015 Cost ($) â GPT-6 Astra GPT-6.1 Sol Claude Opus 5.5 Claude Opus 5 Qwen 3.8 Flash GPT-5.6 Sol Gemini 3.8 Flash DeepSeek V4.1 Flash GPT-5.6 Luna Muse Spark 1.3 GLM 5.3 Flash Figure 3: BrickBench summary. We report summary metrics averaged across the three evalua- tion settings. See Tab. 1 for disaggregated results. The ELO-plot lines represent 95% intervals. Luna runs atmaxand the others atxhigh. Each agent is given a budget of 300 turns per prompt. The baselines evaluated are BrickNet-14B (Kulits & Schmid, 2026) and BrickGPT (Pun et al., 2025), both restricted to theModelsetting which is the only one applicable. Nine of the agents form the reference set of the ELO scale (Sec. 4.4). The baselines and ablated runs of Sec. 5.2 are rated against them and do not affect the scale. Opus 5.5 and GPT-6.1 Sol were released after the human studies of Secs. 5.3 and 5.4 were run, so they are scored on the automated metrics only. Like the baselines, they are rated against the reference set, which leaves every other rating unchanged, and both fall within the range of ratings on which the judge is validated. 5.1. Results We report each metric, per setting and overall, in Tab. 1, and plot the overall columns in Fig. 3. We observe that validity is nearly solved when provided the environment. Five agents deliver a valid assembly for each prompt and none fall below aValidproportion of 0.80 overall. Of the 137 invalid assemblies across the reference set, 44 are prompts with no delivered assembly. The others fail on stability (41), collisions (31), and the part requirement (21). VQAseparates the reference set gently, from 0.59 for GLM to 0.95 for Astra.Design ELOdoes so more sharply, with 545 ELO points between GLM and Astra, a gap that can be interpreted as a judge preferring Astraâs assembly in around 96% of pairs. Within the reference set, Astra leads on every metric, and Opus 5 is ranked second while incurring four times the cost. Sol, Gemini, Qwen, and DeepSeek form a middle within 41 ELO points of one another. Of the two later agents, GPT-6.1 Sol matches Astra onELO(1293 against 1297, intervals overlapping) at a fifth of its cost, and Opus 5.5 rates 148 points above Opus 5 at 40% of its cost, inside Astraâs interval onDesign ELObut 83 points lower onAlign ELO. Alignment varies more with the kind of question than with the subject (Fig. A7 in Sec. C). No prompt category is consistently harder than the others, and the two top agents score at least 0.90 in every one. Every agent satisfies at least 91% of the questions about color, but texture is satisfied 49â83% of the time, and object type and state are the next hardest. The stronger agents also build larger and more varied assemblies (Tab. A1 in Sec. C). InModel, Astra averages 217 parts of 35 types in 9 colors, and GLM 27 parts of 10 types in 5 colors. InSet, Qwen, DeepSeek, Muse, and GLM average 23 to 40 connected components per assembly, against 2.2 to 4.2 for Astra and both Sol and Opus models, so their scenes consist of far more disconnected pieces. Figs. A4 and A5 in Sec. C plot Tabs. 1 and A1 by setting. The reference set spans a factor of 30 in cost, and rank does not follow price. Muse, for instance, costs nine times what DeepSeek costs and rates 138 points lower. See Fig. 5a for a plot ofELO againstCost. WhileAlign ELOandDesign ELOagree for most agents, Sol scores 53 points higher on alignment than design. Both trained baselines fall drastically below every agent. 8 ModelA green dumpster on four small black wheels has its lid pushed open by a tall cluster of red, orange, and yellow flames. A crumpled white sheet protrudes from one corner, and a tipped-over trash can lies beside it. ModelA cramped mailroom corner has a wall covered in white notices, envelopes, and photographs joined by tangled red strings. A wild-eyed, brown- haired man in a white shirt and loosened striped tie gestures toward the board. Cardboard boxes and overflowing stacks of mail crowd the floor beneath it. ModelA blue dolphin leaps above a brilliant turquoise sea, its pale belly turned away from the rainbow. A bright rainbow arches behind its back between soft white clouds. The dolphinâs tail curves back toward a little burst of white spray, while gentle blue waves fill the space beneath the rainbow. SetA masked swordsman dressed entirely in black sits opposite a small balding man at a rough stone table on a rocky hillside. Two silver goblets stand between them. Behind the balding man, a blindfolded blonde woman in a red dress sits with her hands bound. The swordsman rests one gloved hand near his goblet. SetA gigantic blue toy locomotive with a round gray smiling face bursts through the side of a two-story suburban house. The locomotive is nearly as tall as the house, and its red buffer beam projects over the front lawn. Broken wall panels lie below it. Two people beside a small parked car look up at the engine. SetA blonde woman in a black off-the-shoulder top leans across a restaurant table, shouting and pointing directly at a white cat seated opposite her. A dark-haired woman stands behind her, gripping her shoulders and trying to pull her back. The cat sits upright on a dining chair with its ears splayed and its mouth slightly open, looking back at the pointing woman. A plate of green salad sits directly in front of the cat. Drinking glasses, cutlery, and folded napkins cover the table, with upholstered chairs and a low restaurant partition surrounding the confrontation. SetA huge, roughly spherical ball is covered in household objects pointing in every direction: a chair, a television, an umbrella, a frying pan, a banana, and several books. A tiny green figure with a long cylindrical head pushes against its base. Ahead of the ball, a traffic cone and a teapot remain on the floor. Alt-BuildA green sea serpent rises from the water beside a small pirate ship. Its neck bends over the bow, while its tail emerges from the water behind the stern. Alt-BuildA tiny blue-armored knight braces behind a shield and points a lance at a snail twice his height. The snail has a large spiral shell and two raised eye stalks. They face each other across a grassy patch. Alt-Build A steam locomotive crossing a bridge over a river. The bridge has a single arch directly beneath the engine. GPT-6 Astra GPT-6.1 Sol Claude Opus 5.5 Claude Opus 5 Qwen 3.8 Flash GPT-5.6 Sol Gemini 3.8 Flash DeepSeek V4.1 Flash GPT-5.6 Luna Muse Spark 1.3 GLM 5.3 Flash Figure 4: BrickBench samples. We show one assembly from each of the eleven agents for three prompts in each setting. We observe the agents vary in their interpretation of the prompts. One assembly is missing for Muse because it did not deliver one within the turn budget. 9 $0.5$1$2$5$10$20 Cost per Build 800 1000 1200 ELO GPT-6 Astra Claude Opus 5 Claude Opus 5.5 GPT-5.6 Sol GPT-6.1 Sol Qwen 3.8 Flash Gemini 3.8 Flash DeepSeek V4.1 Flash GPT-5.6 Luna GLM 5.3 Flash Muse Spark 1.3 (a) ELO vs. Cost 800900100011001200 Human ELO 800 900 1000 1100 1200 Design ELO GPT-6 Astra Claude Opus 5 Qwen 3.8 Flash Gemini 3.8 Flash GPT-5.6 Sol GPT-5.6 Luna DeepSeek V4.1 Flash Muse Spark 1.3 GLM 5.3 Flash (b) Validation of Design ELO Figure 5: ELO. (a) We plotELOagainst mean assembly cost. (b) To validate our model-based Design ELOcomputation, we perform a perceptual study where participants are tasked with judging design quality between a pair of assemblies for a given prompt. The prompt isnât given to the judge, and they make their decision based only on the images. We find that the rankings agree on 34/36 agent pairs (Kendallí=0.89), suggesting the metric is an effective proxy for human judgment. The study covers the nine reference agents; Opus 5.5 and GPT-6.1 Sol were evaluated after it. BrickNet-14B is collision-free on only 26% of prompts, while BrickGPT, though valid on each prompt, satisfies 2% of the questions. See Fig. 4 for sample visuals, and Fig. A6 in Sec. C for ten more prompts. We observe that the settings differ in what breaks. In terms of validity,Set is the hardest, with validity dropping to 0.68 for Muse, 0.79 for Qwen, and 0.87 for DeepSeek. Alignment falls for every agent, and part counts crowd the 400-part floor (Fig. A8 in Sec. C), so many agents build the smallest scene allowed by the setting.Alt-Buildcosts the least and maintains the highest validity. Alignment is highest inModel, the only setting in which the baselines apply. 5.2. Environment Ablation We ablate the BrickAgent environment in Tab. 2, with the same agent CLI (Codex CLI), container, and task requirements, but with only the LDraw library and a primer on the LDraw file format in place of its tools. Astra remains valid on 40% of prompts while Luna produces only one valid build in 300. Their assemblies average 7 and 156 colliding pairs, respectively, and fall apart into 6 and 83 connected components against 2 and 5 with the environment. Alignment and design do not fail, however. AstraâsVQAdrops by 0.01 and itsELOmoves within the interval. Both of Lunaâs metrics rise, by 0.02 inVQAand by 62 ELO points. Without the restriction of buildability, Luna makes assemblies that depict the prompt as well and look better, but that cannot be built, demonstrating the importance of relevant verifiers. Doing so costs Astra nothing on alignment or design, but costs Luna a little of both. 10 Table 2: Environment ablation. We ablate the effect of the BrickAgent environment, pooled over the three evaluation settings (Overall; 300 prompts). While both agents consistently produce valid assemblies when given tools, they struggle without them to varying degrees: GPT-6 Astra is somewhat robust but its assemblies drop to being valid only 40% of the time while GPT-5.6 Lunaâs fall to less than a percent. The environment improves the VQA of Astra but lowers its ELOwithin noise. Without it, Luna improves on both, suggesting the buildability constraint limits expressivity. SystemValidâCollisionsâConn. ComponentsVQAâELOâCost ($) GPT-6 Astra1.000.002.210.9541297 ±224.06 without BrickAgent0.407.196.120.9401309 ±244.66 GPT-5.6 Luna1.000.004.600.792898 ±160.75 without BrickAgent <0.01155.5183.000.813960 ±200.60 0.1.2.3.4.5 Win Rate vs. Human no preference BrickGPT Muse Spark 1.3 DeepSeek V4.1 Flash GLM 5.3 Flash Claude Opus 5 BrickNet-14B GPT-5.6 Sol Gemini 3.8 Flash Qwen 3.8 Flash GPT-5.6 Luna GPT-6 Astra (a)(b) Figure 6: Human-design evaluation. We perform a perceptual study comparing agent designs with human designs. For each assembly in theModelandSetsettings, we select a random assembly from the BrickNet dataset with a comparable number of parts and ask a human rater to identify the one made by a person. We find that raters are able to distinguish between the agent-designed and human-designed assemblies (a); the study covers the nine reference agents and the two baselines. While agent-designed assemblies often satisfy the prompt criteria, there is a clear gap in fine-grained design to those produced by humans (b). 5.3. Validation of the Design Judge To confirm the validity ofDesign ELO, which rests on a VLM judge, we evaluate against human ratings. We recruit participants on Prolific and show each 40 pairs of same-prompt assemblies, with four views per assembly. We collect 4,080 judgments from 102 raters. We fit the human judgments with the same BradleyâTerry model asELO(Sec. 4.4), and compare the ratings against Design ELOin Fig. 5b. The rankings agree on 34 of the 36 pairs among the agents, with a Kendallíof 0.89. This follows findings from prior work which demonstrated alignment between VLM and human preferences on text-to-3D generation (Wu et al., 2024). The judge is more decisive than the rater pool, so we align the two by a constant factor. Opus 5.5 and GPT-6.1 Sol were released after this study and are not part of it. 5.4. Comparison with Human Designs We asked people familiar with LEGO design to tell agent builds from human ones (Fig. 6). For assemblies in theModelandSetsettings, we drew part-count-matched human-designed models from the BrickNet dataset, and asked which of the two a person made. The drawn samples are not prompt-matched. No human designs exist for the BrickBench prompts, so the 11 pairs are matched on part count alone, and Fig. 6b shows human models of other subjects. A prompt-matched comparison would require commissioning designs, which we leave to future work. As in Sec. 5.3, the study predates Opus 5.5 and GPT-6.1 Sol. Five raters provided 360 ratings altogether, and selected the human model in 323 of them. No agent was taken for a human designer more than one time in five (Fig. 6a), and every agentâs rate falls significantly below chance. GPT-6 Astra, the agent most often taken for a person, was chosen in 4 of its 21 pairs. We show reference examples of human-designed models in Fig. 6b. While agent assemblies satisfy the prompt well, they lack the part usage, proportion, and surface detail that mark human design. 6. Discussion and Limitations We report four findings from our evaluations. First, we find that data-driven models built for the task are subsumed by general agents. A general coding agent exceeds models on their own benchmark (Sec. 3.3) and on ours. Cost, however, remains an axis for future work. BrickNet-14B produces an assembly for six hundredths of a cent, while the Astra does so in $4.06. Second, prompt adherence saturates. The best-performing agent satisfies 95% of the VQA questions asked of it, and absolute alignment does not strongly separate the top performers. Headroom lies instead in design quality. Third, agent performance is dependent on the environment. The same model produces consistently buildable assemblies with BrickAgent, and unbuildable ones without it (Tab. 2). Fourth, a gap remains to humans. Participants familiar with LEGO design filtered agent-designed assemblies from human ones nine out of ten times (Fig. 6). We interpret this as a measure of design quality: agents satisfy requirements but fall short of the standards of a designed set. Our simulator of physical validity is a proxy and not a complete measure of structural stability. Connections are not tested for strength, so an assembly that balances passes even when a physical one might sag or come apart (Sec. 3.2). Some works exist that attempt to build force models over stud connections WaĂmann & Weicker (2012) to evaluate FEM (Pletz & Drvoderic, 2023). None, however, is general, and integrating such would severely limit the available part catalog. Until a general one exists, our bar for physical evaluation is necessary but not sufficient. 7. Conclusion We introduced BrickBench, a benchmark for agentic text-conditioned LEGO-set design. The domain gives the task a floor that a simulator can verify and an open ceiling on design quality. It includes 300 prompts in three settings with different constraints, a reference environment in which agents build, BrickAgent, and a set of metrics that score build validity, alignment, and design confirmed by human raters. Nine frontier agents build valid assemblies for most prompts and satisfy the majority of what each requests, displacing task-specific data-driven models at an increase of 900â6,800 times the cost. While doing so, we highlight the open challenge of narrowing the gap to human design. 12 References Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, Junke Wang, Marco Monteiro, Hu Xu, Shiyu Dong, Nikhila Ravi, Daniel Li, Piotr DollĂĄr, and Christoph Feichtenhofer. Perception encoder: The best visual embeddings are not at the output of the network. In Advances in Neural Information Processing Systems (NeurIPS), 2025. Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324â345, 1952. Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. In International Conference on Learning Representations (ICLR), 2024. Hyunsoo Chung, Jungtaek Kim, Boris Knyazev, Jinhwi Lee, Graham W. Taylor, Jaesik Park, and Minsu Cho. Brick-by-Brick: Combinatorial construction with deep reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021. Erwin Coumans and Yunfei Bai. PyBullet, a Python module for physics simulation for games, robotics and machine learning. http://pybullet.org, 2016â2021. Jiahao Ge, Mingjun Zhou, Hanyou Zheng, Hao Xu, and Chi-Wing Fu. LEGO-maker: Autoregressive image-conditioned LEGO model creation. ACM Transactions on Graphics (TOG), 44(6), 2025. Gemma Team. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026. Yunqi Gu, Ian Huang, Jihyeon Je, Guandao Yang, and Leonidas Guibas. BlenderGym: Benchmarking foundational model systems for graphics editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A. Ross, Cordelia Schmid, and Alireza Fathi. SceneCraft: An LLM agent for synthesizing 3D scenes as Blender code. In International Conference on Machine Learning (ICML), 2024. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), 2024. Peter Kulits and Cordelia Schmid. BrickNet: Graph-backed generative brick assembly. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 39252â39261, June 2026. Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision (ECCV), 2024. Liang Ma, Jiajun Wen, Min Lin, Rongtao Xu, Xiwen Liang, Bingqian Lin, Jun Ma, Yongxin Wang, Ziming Wei, Haokun Lin, Mingfei Han, Meng Cao, Bokui Chen, Ivan Laptev, and Xiaodan Liang. PhyBlock: A progressive benchmark for physical understanding and planning via 3D block assembly. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2025. Dimitrios Mallis, Ahmet Serdar Karadeniz, Sebastian Cavada, Danila Rukhovich, Niki Foteinopoulou, Kseniya Cherenkova, Anis Kacem, and Djamila Aouada. CAD-assistant: Tool-augmented VLLMs as generic CAD task solvers. In IEEE/CVF International Conference on Computer Vision (ICCV), 2025. Martin Pletz and Matthias Drvoderic. BrickFEM: An automated finite element model for static and dynamic simulations of simple Lego sets. engrXiv preprint, 2023. doi: 10.31224/2898. 13 Ava Pun, Kangle Deng, Ruixuan Liu, Deva Ramanan, Changliu Liu, and Jun-Yan Zhu. Generating physically stable and buildable brick structures from text. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 14798â14809, October 2025. Kexian Tang, Junyao Gao, Yanhong Zeng, Haodong Duan, Yanan Sun, Zhening Xing, Wenran Liu, Kai Chen, and Kaifeng Lyu. LEGO-puzzles: How good are MLLMs at multi-step spatial reasoning? In European Conference on Computer Vision (ECCV), 2026. Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier HĂ©naff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. Aaron Walsman, Muru Zhang, Klemen Kotar, Karthik Desingh, Ali Farhadi, and Dieter Fox. Break and make: Interactive structural understanding using LEGO bricks. In European Conference on Computer Vision (ECCV), 2022. Martin WaĂmann and Karsten Weicker. Maximum flow networks for stability analysis of LEGO structures. In European Symposium on Algorithms (ESA), 2012. Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. GPT-4V(ision) is a human-aligned evaluator for text-to-3D generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. Tian Xia, Tianrun Gao, Wenhao Deng, Long Wei, Xiaowei Qian, Chenglei Yu, and Tailin Wu. BuildArena: A physics-aligned interactive benchmark of LLMs for engineering construction. In International Conference on Machine Learning (ICML), 2026. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Jing Hua Toh, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Haozhe Zhang, Kaichen Liu, Miaomiao Chen, Lei Li, Shaojie Yang, Cheng Peng, and Hanjie Chen. BenchCAD: A comprehensive, industry-standard benchmark for programmatic CAD. arXiv preprint arXiv:2605.10865, 2026. 14 BRICKBENCH: E VALU AT ING A G ENTIC BR IC K D ES I GN Appendix A. Prompts and Question Graphs Fig. A1 shows the share of each settingâs questions by kind and subtype.Setgraphs are the largest, at 24 questions per prompt on average, because their prompts often describe scenes with multiple objects. 0.1.2.3 Share of the setting's questions Entity whole part Attribute color shape state type count size material texture Relation spatial action ModelSetAlt-Build Figure A1: VQA question types. We plot the share of each settingâs questions by DSG type. Fig. A2 shows oneSetgraph in full, with the prompt it was generated from. Its roots are the entities the prompt names, and every attribute and relation question hangs from the questions that establish what it refers to, so a missing entity fails every subsequent question about it. B. Judge Prompts Fig. A3 reproduces the three prompts of Sec. 4.4 as the judge receives them. The two pairwise prompts differ in what the judge is told. The alignment prompt contains the prompt and asks which assembly covers more of it, whatever its build quality. The design prompt withholds the prompt and asks which assembly is better designed, whatever it depicts. The VQA prompt lists every question of the graph, tells the judge to answer from the views alone, and requires a yes or a no on each. 15 SetA brown dog with floppy black ears and a small brown hat sits at a table in a burning room. A white coffee mug rests in front of it. Flames climb the curtains and surround the chair, but the dog sits upright with its paws resting calmly on the tabletop. The front of the room is open to reveal the scene. entityattributerelation Is there a recognizable dog? Is the dog brown? Are the dog's ears floppy? Are the dog's ears black? Is the dog wearing a hat? Is the dog sitting at the table? Is the mug in front of the dog? Is the dog sitting upright? Are the dog's paws resting on the tabletop? Is the hat brown? Is there a room? Are there flames in the room? Is the table in the room? Does the room have curtains? Is one side of the room open so that the scene inside is visible? Do the flames climb the curtains? Do the flames surround the chair? Is there a table? Is there a coffee mug? Is there a chair? Figure A2: Question-graph sample. We plot a sample question graph for a set prompt. An arrow from one question to another indicates the second is satisfied only if the judge answers yes to the first. C. Additional Results Tab. A1 reports build statistics per setting. Astra uses the most parts in every setting and Qwen the second most. InSet, Qwen, DeepSeek, Muse, and GLM build scenes of 20 to 40 disconnected components on average, while Astra, Opus, Gemini, and Sol keep theirs under five. Figs. A4 and A5 plot Tabs. 1 and A1 by setting in the layout of Fig. 3, and Fig. A6 shows one assembly per agent for further prompts in each setting. Fig. A7 breaks VQA down by build category and by attribute type. Fig. A8 plots the distribution of part counts inModelandSet. Most agents peak near the 400-part floor of Set, and Astra is the exception. 16 Table A1: Further assembly statistics. We report further statistics of the assemblies per setting. Partsis the average number of parts in an assembly.Uniqueis the (average) number of unique parts.Colorsis the number of colors.Collisionsis the number of parts that nonphysically collide (overlap) with one another.Stableis the proportion of assemblies that remain static under simulation, with every part moving under 3 LDU.Conn. Componentsis the number of graph components after BrickNet parsing; a valid assembly may have multiple components in order to represent multiple objects. SystemPartsUniqueColorsCollisionsâStableâConn. Components Model GPT-6 Astra21735.49.40.001.0001.6 GPT-6.1 Sol23036.49.10.001.0001.2 Claude Opus 5.518238.29.10.001.0001.3 Claude Opus 511027.77.70.001.0001.3 Qwen 3.8 Flash12523.88.30.100.9794.8 GPT-5.6 Sol8320.48.00.001.0001.2 Gemini 3.8 Flash9525.69.60.000.9901.2 DeepSeek V4.1 Flash8819.37.00.560.9693.1 GPT-5.6 Luna4114.86.80.001.0002.0 Muse Spark 1.35614.56.70.460.9252.0 GLM 5.3 Flash2710.34.90.860.9281.7 BrickNet-14B4014.54.824.80â1.0 BrickGPT1397.11.00.001.0002.4 Set GPT-6 Astra151146.817.00.001.0002.2 GPT-6.1 Sol162549.116.70.001.0003.5 Claude Opus 5.5111268.517.50.001.0003.8 Claude Opus 586548.614.60.001.0004.2 Qwen 3.8 Flash111838.817.40.020.92932.8 GPT-5.6 Sol69829.914.80.001.0003.3 Gemini 3.8 Flash58136.517.30.020.9801.7 DeepSeek V4.1 Flash72633.512.96.940.88836.7 GPT-5.6 Luna51126.213.20.001.00010.2 Muse Spark 1.350019.211.09.960.80022.6 GLM 5.3 Flash56816.59.00.800.90840.2 Alt-Build GPT-6 Astra11030.015.40.001.0002.9 GPT-6.1 Sol11030.815.40.001.0002.0 Claude Opus 5.510327.514.20.001.0003.3 Claude Opus 58625.112.90.001.0001.8 Qwen 3.8 Flash10523.316.00.001.0005.1 GPT-5.6 Sol5219.312.10.001.0001.4 Gemini 3.8 Flash6424.714.20.001.0001.4 DeepSeek V4.1 Flash7318.112.20.001.0004.7 GPT-5.6 Luna4017.911.20.001.0001.6 Muse Spark 1.34914.811.00.630.9452.0 GLM 5.3 Flash2511.19.40.010.9801.5 17 Design The first image shows every view of one build, tiled. The second image shows the other build the same way. You are shown two LEGO builds. The first 4 images are Build A, photographed from 4 angles. The next 4 images are Build B, the same 4 angles. Pick the build that looks better designed, judged against the build quality you would expect from an official LEGO set. Look at how well the pieces are used, not at the subject: what a build depicts carries no weight here, and neither does how appeal- ing you find it. Both builds were rendered by the same software with identical lighting and background, so image quality carries no informa- tion. Answer with exactly one character, A or B. No other text. Alignment The first image shows every view of one build, tiled. The second image shows the other build the same way. Two people were given the same description and each built it out of LEGO. Description: [the prompt] The first 4 images are Build A, photographed from 4 angles. The next 4 images are Build B, the same 4 angles. Pick the build that better matches the description. Work through the description point by point: the subject, the parts it says the build should have, their arrangement, and any colours, counts or relations it states. A build that covers more of what was asked for wins, whatever its build quality. Answer with exactly one character, A or B. No other text. VQA You are shown 8 views of a single LEGO build, photographed from 8 angles around it. It was built from this brief: [the prompt] Answer the numbered questions below about THIS build. Use the views together â a feature need not be visible in every view. Judge only what you can see: do not assume a detail is present because the brief asks for it, and do not infer hidden details from how such an object is usually made. Keep one consistent assignment of objects to roles across all the questions. Say yes if the views support it and no if they do not. You must choose one of the two for every question; there is no third option. Where the evidence is thin, decide on the balance of what the eight views show. [the questions, numbered] End your reply with exactly [the number of questions] lines, nothing after them, one per question: <number>: <yes|no> Figure A3: Judge prompts. The three prompts of Sec. 4.4 as the judge receives them. For the two pairwise questions, the four views of each assembly are tiled into one image and the two images precede the text, and the design question never contains the prompt. ForVQA, the eight views precede the text as separate images, and the bracketed fields are filled in for each assembly. 18 0.51 Valid â 0.51 VQA â 01020 Cost ($) â GPT-6 Astra GPT-6.1 Sol Claude Opus 5.5 Claude Opus 5 Qwen 3.8 Flash GPT-5.6 Sol Gemini 3.8 Flash DeepSeek V4.1 Flash GPT-5.6 Luna Muse Spark 1.3 GLM 5.3 Flash BrickNet-14B BrickGPT ModelSetAlt-Build (a) 4008001200 ELO â 4008001200 Align ELO â 4008001200 Design ELO â GPT-6 Astra GPT-6.1 Sol Claude Opus 5.5 Claude Opus 5 Qwen 3.8 Flash GPT-5.6 Sol Gemini 3.8 Flash DeepSeek V4.1 Flash GPT-5.6 Luna Muse Spark 1.3 GLM 5.3 Flash BrickNet-14B BrickGPT ModelSetAlt-Build (b) Figure A4: BrickBench. We visualize disaggregated BrickBench quantitative results. The base- lines BrickNet-14B (Kulits & Schmid, 2026) and BrickGPT (Pun et al., 2025) are only evaluated on theModelsetting and theValidmetric reported of BrickNet-14B does not include stability. 19 101001000 Parts 02550 Unique 01020 Colors GPT-6 Astra GPT-6.1 Sol Claude Opus 5.5 Claude Opus 5 Qwen 3.8 Flash GPT-5.6 Sol Gemini 3.8 Flash DeepSeek V4.1 Flash GPT-5.6 Luna Muse Spark 1.3 GLM 5.3 Flash BrickNet-14B BrickGPT ModelSetAlt-Build (a) 0.1110 Collisions â 0.51 Stable â 131030 Conn. Components GPT-6 Astra GPT-6.1 Sol Claude Opus 5.5 Claude Opus 5 Qwen 3.8 Flash GPT-5.6 Sol Gemini 3.8 Flash DeepSeek V4.1 Flash GPT-5.6 Luna Muse Spark 1.3 GLM 5.3 Flash BrickNet-14B BrickGPT ModelSetAlt-Build (b) Figure A5: Further assembly statistics. We plot the statistics of Tab. A1 by setting. 20 ModelA twisting gray tornado narrows to a thin point above a patch of churning water. Three small sharks protrude from different heights of the funnel, their bodies tilted in different directions. White bellies, fins, and open jaws break up the spiraling gray bands, beneath a broad cap of storm clouds. ModelA saber-toothed squirrel is trapped inside a clear blue block of ice. Its long snout, enormous eyes, and curled bushy tail remain visible through the ice. One paw reaches out through a thawed opening toward a brown acorn lying just beyond its fingertips. A small puddle surrounds the bottom of the block. ModelA battered red fox animatronic has an eyepatch, a long toothy snout, and one hand replaced by a metal hook. Torn fur exposes gray mechanical parts at its knees and chest. It wears ragged brown shorts, and its large metal feet stand apart. One ear is damaged, giving the already lopsided face an even more disreputable appearance. SetA cutaway medieval gatehouse contains a wooden catapult with a black-and-white cow sitting in its sling. A helmeted guard stands beside the release lever. Outside the closed gate, three knights look up from behind their shields. A second cow waits beside a hay bale in the courtyard. SetA rectangular glass fish tank with a black rim and colored gravel. Three orange fish and a striped angelfish swim among tall green plants and a stone arch. A snail clings to the inside glass, and a small filter occupies a corner across the tank from the stone arch. SetA barefoot martial artist wearing a white uniform, black belt, and red headband smashes a parked sedan on a waterfront platform. The carâs hood is crumpled, its windows are broken, and one door hangs crookedly from the body. The fighter has one fist driven into the damaged front end. Harbor water and mooring posts stand behind the wreck. SetFour tiny children gather around a cream-filled cookie taller than they are in a jungle of enormous grass blades. Two sit on the back of a giant brown ant while another breaks a crumb from the cookie. The fourth peers into the opening of a discarded soda-can tab. Dewdrops cling to the grass above them. Alt-Build A ferry carrying a car on its open deck. Its front ramp rests on the shore, and a small control cabin stands above the back of the deck. Alt-Build A pink pig riding in a gray minecart. Its front legs rest over the rim, and its snout projects beyond the cart. Alt-Build A tree forks into two branches, with a nest between them. Two small birds sit in the nest. A larger bird perches on one branch. GPT-6 Astra GPT-6.1 Sol Claude Opus 5.5 Claude Opus 5 Qwen 3.8 Flash GPT-5.6 Sol Gemini 3.8 Flash DeepSeek V4.1 Flash GPT-5.6 Luna Muse Spark 1.3 GLM 5.3 Flash Figure A6: Additional BrickBench samples. We show further assemblies from the eleven agents in the three settings. 21 GPT-6 Astra GPT-6.1 Sol Claude Opus 5.5 Claude Opus 5 Qwen 3.8 Flash GPT-5.6 Sol Gemini 3.8 Flash DeepSeek V4.1 Flash GPT-5.6 Luna Muse Spark 1.3 GLM 5.3 Flash Animal Art / Object Building Floral / Nature History / Period Medieval / Castle Pirate Space / Sci-Fi / Fantasy Train Vehicle / Boat / Airplane 0.75 1.00 (a) By prompt category Color Size Material Count Shape State Type Texture 0.50 0.75 1.00 (b) By attribute type Figure A7: VQA breakdowns. We report score by prompt category (a) and attribute type (b). GPT-6 Astra GPT-6.1 Sol Claude Opus 5.5 Claude Opus 5 Qwen 3.8 Flash GPT-5.6 Sol Gemini 3.8 Flash DeepSeek V4.1 Flash GPT-5.6 Luna Muse Spark 1.3 GLM 5.3 Flash 0100200300400 parts per build density (a) Model 4001000200030004000 parts per build density (b) Set Figure A8: Part-count distributions. We report the density of part counts in theModel(â€400) andSet(400â4000) settings for each agent. Most agents peak near 400 inSet. GPT-6 Astra uses considerably more parts on average in both settings. 22