Paper deep dive
MatReplace: A Reference-Free, Conditioning-Aligned Benchmark for Material Replacement in Interior Scenes
Mingzhe Du, Thong Thanh Nguyen, Nguyen Tran Cong Duy, See-Kiong Ng, Luu Anh Tuan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/26/2026, 5:39:28 AM
Summary
The paper introduces MatReplace, a reference-free benchmark for evaluating material replacement in interior scenes. It defines a task where a specific surface's material is changed while preserving geometry, surroundings, and illumination. The benchmark evaluates models across three conditioning tracks (instruction only, instruction plus mask, instruction plus reference image) using four verifiable dimensions: local material correctness, global lighting harmony, outside preservation, and inside structure. Results indicate that while named material rendering is solved by top closed-source models, grounding materials from visual references remains challenging.
Entities (16)
Relation Signals (14)
MatReplace → evaluates → Material Replacement
confidence 95% · We introduce MatReplace, a reference-free benchmark that evaluates edits along four verifiable dimensions
MatReplace → measures → Inside Structure
confidence 95% · evaluates edits along four verifiable dimensions: ... inside structure
MatReplace → measures → Local Material Correctness
confidence 95% · evaluates edits along four verifiable dimensions: local material correctness
MatReplace → measures → Global Lighting Harmony
confidence 95% · evaluates edits along four verifiable dimensions: ... global lighting harmony
MatReplace → measures → Outside Preservation
confidence 95% · evaluates edits along four verifiable dimensions: ... outside preservation
MatReplace → usesconditioning → Track B
confidence 95% · Track B (instruction++mask)
MatReplace → usesconditioning → Track C
confidence 95% · Track C (mask++reference)
MatReplace → usesconditioning → Track A
confidence 95% · Track A (instruction)
BAGEL-7B → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Material replacement is a common interior-design operation: changing the material of a selected surface while preserving its geometry, surroundings, and illumination. Despite its commercial relevance, no public benchmark isolates this task, and evaluating it is challenging. Reference-based metrics penalize valid outputs in this inherently one-to-many setting, favor the style of the reference generator, and cannot fairly compare editors that receive different forms of guidance. We introduce MatReplace, a reference-free benchmark that evaluates edits along four verifiable dimensions: local material correctness, global lighting harmony, outside preservation, and inside structure. It defines three tracks that vary one conditioning signal at a time: (A) instruction only, (B) instruction plus region mask, and (C) material reference image instead of instruction. Our results reveal a clear divide between naming and visually grounding materials. In Track A, leading closed-source editors achieve exemplar-level material rendering and surpass the exemplar anchor under our primary aggregate. In Track B, masks help only mask-compatible models with weak scene preservation, with task-paired, single-seed effects ranging from +0.137 to -0.090 across aligned model families. In Track C, reference-image conditioning degrades every family under both aggregates, by -0.031 to -0.508; in the worst cases, models repaint the reference image itself and perform worse than returning the input unchanged. Thus, named-material rendering is largely solved by the strongest closed editors on this distribution, but grounding materials from pixels remains an open challenge. Expert ratings validate our ranking (Kendall's tau = 0.68) and align with our aggregates more closely than GT-referenced or CLIP-based baselines.
Tags
Links
- Source: https://arxiv.org/abs/2608.24107v1
- Canonical: https://arxiv.org/abs/2608.24107v1
Trouble viewing inline? Open PDF directly →
Full Text
46,008 characters extracted from source content.
Expand or collapse full text
MatReplace: A Reference-Free, Conditioning-Aligned Benchmark for Material Replacement in Interior Scenes Mingzhe Du Thong Thanh Nguyen †thanks: Corresponding author. Nguyen Tran Cong Duy See-Kiong Ng Luu Anh Tuan Abstract Generative image editors are entering interior-design workflows, where the routine request is material replacement: repaint one marked surface with a new material while the object’s geometry, the surrounding scene, and the illumination stay fixed. The operation is precisely specified and commercially routine, yet no public benchmark isolates it, and scoring it is subtle. The task is one-to-many, so distance to a stored reference penalizes valid diversity; a single-generator source rewards imitating that generator’s style; and editors given different guidance cannot share one leaderboard fairly. We introduce MatReplace, a reference-free benchmark that scores each edit on four verifiable dimensions (local material correctness, global lighting harmony, outside preservation, inside structure), across three tracks that vary one conditioning signal at a time: (A) the instruction alone, (B) the instruction plus a region mask, and (C) a reference image of the material in place of the instruction. Track A shows that strong closed-source models already render the requested material at exemplar quality, clearing the exemplar anchor under our primary aggregate. Track B shows that masks help only models that preserve the scene poorly and can consume them: their task-paired, single-seed value spans +0.137+0.137 to −0.090-0.090 across aligned families. Track C degrades every family under both aggregates, by −0.031-0.031 to −0.508-0.508, at worst repainting the reference itself and dropping below the score of returning the input untouched. Rendering a named material is thus largely solved for the strongest closed editors on this distribution, while grounding a material shown as pixels is not. Expert human ratings reproduce our ranking (Kendall τ=0.68τ=0.68) and track our aggregates more closely than GT-referenced or CLIP-based baselines. 1National University of Singapore 2Nanyang Technological University 3VinUniversity mingzhe, thong.nguyen, seekiong@nus.edu.sg, nguyentr003, anhtuan.luu@ntu.edu.sg 1 Introduction Interior-design tools increasingly use generative editors to answer a concrete client question: what would this room look like if the headboard were navy suede, or the floor herringbone oak? The underlying operation, material replacement, is clearly specified. The material of one marked surface must change to a stated target while the object’s geometry, the rest of the scene, and the scene’s illumination remain fixed. Despite its practical importance and clear definition, no existing benchmark isolates this task. Instruction-editing suites (Zhang et al. 2023; Sheynin et al. 2024; Wang et al. 2023) fold it into broad edit categories, and their metrics score edits against a single stored reference image per task. Figure 1: MatReplace at a glance. Each column is one task, read from the middle out. Middle: the target region in color, outlined over a dimmed grayscale surround (mask m), reference r inset. Up (✓): a reference-quality edit (Nano-Banana-2-Lite, Track A). Material changes while geometry, scene, and illumination survive. Down (✗): one failure per mode. BAGEL-7B (C) repaints the reference and discards the scene (column 1); Qwen-Image-Edit (C) fails to apply the reference and leaves the slate untouched (column 2); OmniGen2 (C) pastes a flat, unshaded slab (column 3); Paste anchor tiles the reference without relighting (column 4). Each failure is caught by the dimension whose contract it violates (Sec. 4). Scoring against stored reference images is doubly problematic for this task. First, the task is one-to-many. Many navy-suede headboards are equally correct, and penalizing distance to one stored example punishes valid diversity. Second, when all stored reference images come from a single generator, any such metric rewards matching that generator’s rendering style and biases the leaderboard toward that data source. MatReplace therefore treats the stored output as one reference solution that supports qualitative comparison and calibration, never the scoring target. Evaluation instead decomposes the task definition into four independently verifiable properties, each checked only against the provided inputs: whether the masked region became the target material, whether the outside stayed unchanged, whether the inside geometry survived, and whether the new surface matches the scene illumination. The second design choice concerns conditioning, the form of guidance a system receives. Some deployed editors receive only a text instruction, some additionally receive a region mask, and others receive a mask together with a reference image of the target material. Benchmarks mixing these on one leaderboard conflate model quality with input signals. MatReplace defines Track A (instruction), Track B (instruction++mask), and Track C (mask++reference), aligning the same model family across tracks wherever the architecture permits. Holding the model fixed and varying only conditioning measures what a mask or reference is worth; the answer is neither uniform nor always positive. Our contributions are: • MatReplace, a material-replacement benchmark: a frozen split over several material categories and surface types, with taxonomy labels and released inputs. • A reference-free protocol of four independently verifiable dimensions, validated by constructed submissions that incur penalties only on their target. • A three-track leaderboard defined by conditioning, with families aligned across tracks. Masks help only families weak at preservation and able to consume them; reference images degrade all models, in three failure modes the dimensions distinguish. 2 Related Work Instruction-based editing and its benchmarks. End-to-end instruction editing has produced many manual and large-scale benchmarks (Brooks et al. 2023; Zhang et al. 2023; Sheynin et al. 2024; Wang et al. 2023; Kawar et al. 2023; Ku et al. 2024), all scoring primarily by GT-referenced distance plus CLIP text alignment (Hessel et al. 2021) and inheriting both biases above. Recent flow-matching and unified any-to-any editors (Black Forest Labs et al. 2025; Wu et al. 2025; Wu et al. 2026; Deng et al. 2025) motivate our aligned design: one checkpoint enters all three tracks. Reference-guided appearance transfer. Exemplar-based inpainting and object customization insert a reference’s content into a scene (Yang et al. 2023; Chen et al. 2024; Ruiz et al. 2023). Track C is the material-level analogue: the reference specifies appearance (color, texture, micro-structure) to be re-rendered under the scene’s geometry and illumination, not pasted; harmony and structure separate the two. Material editing and design-domain data. A parallel line edits materials without benchmarking the task: parametric control (Sharma et al. 2024), zero-shot single-exemplar transfer (Cheng et al. 2024), light-aware transfer (Lopes et al. 2025), and material-map extraction (Lopes et al. 2024), each on its own small set. No shared suite compares them with general editors, the gap MatReplace fills; despite interiors synthesized at scale from 3D-scene repositories (Fu et al. 2021; Roberts et al. 2021), no leaderboard isolates an interior editing operation. Material understanding and intrinsics. Material recognition in situ is a classic vision problem (Bell et al. 2013; Bell et al. 2015), and the albedo–shading decomposition underlying our harmony dimension goes back to intrinsic-image analysis (Grosse et al. 2009). MatReplace connects this line to generative editing: material correctness via zero-shot SigLIP2 (Tschannen et al. 2025) recognition and DINOv2 (Oquab et al. 2024) appearance similarity, and harmony via shading invariance. Reference-free evaluation. CLIPScore (Hessel et al. 2021) showed reference-free scoring for captioning, a single-obligation task. For editing the right unit is a decomposition into the task’s contract: each dimension ties to a checkable clause, built from established components (LPIPS (Zhang et al. 2018), SSIM (Wang et al. 2004), Depth Anything V2 (Yang et al. 2024)) and validated by an anchor experiment. Unlike a learned judge, every dimension is auditable and deterministic with fixed, released constants. 3 The MatReplace Benchmark Task definition. A task is a tuple (x,m,r,T,c)(x,m,r,T,c): scene x, binary mask m on one surface, reference r of the target material, instruction T, and material-category label c (10-way). A submission is an edited image y y at x’s resolution. The contract: inside m the material becomes c (matching r where provided); outside m the scene is unchanged; the geometry inside is unchanged and the new surface shaded by the scene’s illumination. Why interior scenes. Interior visualization is where material replacement is deployed: design tools iterate over finishes on fixed geometry, so the edit must be surgical. It is also a stress test, with planar surfaces beside upholstery and drapery, strong mixed lighting, and material–object bindings dense enough that text struggles to pin them down. Sources and curation. Figure 2 summarizes construction. Each of 2,017 candidate tasks is one templated session: a GPT-5.6-Sol agent drives one disclosed generator, GPT-Image-2 (itself evaluated), producing the scene, the mask as an edit on that scene (binarized, coverage-checked, regenerated on failure), the reference, an exemplar output, and a canonical instruction. An algorithmic pass checks decodability, resolution agreement, mask binarity and in-bounds coverage, and flags exemplars that redraw the scene (low outside-mask PSNR); 2,015 survive. A semantic pass then audits five properties with a GPT-5.6-Sol agent: mask–object alignment (70.8%70.8\% pass), edit correctness (96.4%96.4\%), outside preservation (83.5%83.5\%), reference–edit consistency (93.3%93.3\%), and watermark absence (99.7%99.7\%), rewriting each instruction to match the realized images. Admission requires an aligned mask and clean imagery; defects in the stored exemplar do not disqualify a task, since scoring never references it. Mask alignment is the dominant yield limiter; the 596 rejects ship as a disjoint rejected pool, released for auditing rather than as supervision. Figure 2: MatReplace Benchmark Construction. One templated generation session per candidate task, an algorithmic validity pass, and a semantic audit by a vision–language agent reduce 2,017 candidate sessions to the frozen 1,421 tasks. The middle bar is drawn to scale, so the admitted and rejected shares are directly comparable and the accounting closes. The 596 rejects ship as a disjoint rejected pool, released for auditing and never used for evaluation. What the rejects look like. The rejects catalogue how a strong generator fails at its own task: masks spill or miss (588, dominant); edits leak outside the mask (332); inside geometry is re-sculpted (73); exemplars mismatch their reference (135) or redraw the scene (153), exactly the behaviors the four dimensions target (Fig. 3). Construction cost. At public GPT-Image-2 list prices (July 2026), a four-call session costs $0.37–$1.48, or $0.7k–$3.0k across 2,017 sessions; absorbing curation yield and the audit, an admitted task costs $0.57–$2.14, and the fully audited benchmark prices below a single small user study. Statistics. MatReplace has 1,421 task specifications, spanning 10 material categories (fabric 187, wood 181, marble 172, stone 170, metal 160, …) and 6 surface types (furniture 825, wall 156, countertop 144, fireplace 125, …), with mask-coverage buckets retained for stratified analysis. Three conditioning tracks. Tracks differ only in inputs: Track A x+Tx+T, Track B x+m+Tx+m+T, and Track C x+m+rx+m+r, the material named only by the image. A family supporting several modes enters several tracks from one checkpoint. Scope and non-goals. MatReplace isolates one edit type; insertion, removal, geometry edits, multi-surface restyling, and global style transfer are out of scope. A single-surface edit is the smallest unit for which preservation, structure, and harmony are simultaneously well defined, whereas composite edits reintroduce ambiguities needing a stored answer. 4 Reference-Free Evaluation Protocol Given (x,m,r,c)(x,m,r,c) and a submission y y, we compute four scores in [0,1][0,1]; overall is their unweighted mean. Reference-free means free of any reference output: every score is a function of the task inputs alone, never of a stored solution. The reference r is itself an input; its Track-A consequence is discussed under the design rationale. Let m~ m denote m dilated by a 77-px boundary band (excluded from outside comparisons), m¯=1−m~ m=1- m its complement, and crop(⋅,m)crop(·,m) the masked crop. Figure 3 shows outputs violating exactly that clause. Figure 3: One failure per dimension: each pair shows the input with the target region outlined, then the model output. SDXL-inpaint paints bare wall instead of navy suede (SmatS_mat). InstructPix2Pix floods the whole room with the target emerald (SpresS_pres). SD3.5-inpaint re-sculpts the rectangular fireplace into an arched portal (SstructS_struct). FLUX-Kontext-ref pastes the reference as flat, unlit patches (SharmS_harm). Material correctness (SmatS_mat). SigLIP2 classification c c of crop(y^,m)crop( y,m) over the material vocabulary, scored at the family level (pairs that vision encoders legitimately confuse, such as marble and granite, are grouped), averaged with a DINOv2 (fDf_D) cosine similarity between the crop and r: Smat=12[fam(c^)=fam(c)+cos(fD(crop(y^,m)),fD(r))]. splitS_mat= 12 [&1\fam( c)=fam(c)\\\ &+ \! (f_D(crop( y,m)),\,f_D(r) ) ]. split (1) Outside preservation (SpresS_pres). LPIPS between x and y y restricted to the outside region: Spres=max(0, 1−LPIPS(x⊙m¯,y^⊙m¯)),S_pres= (0,\,1-LPIPS(x m,\, y m) ), (2) with PSNR/SSIM reported as diagnostics. SpresS_pres is invariant to the resize-back-to-input-resolution step applied to all submissions (round-trip verified). Inside structure (SstructS_struct). Monocular depth d(⋅)d(·) for x and y y, aligned by scale-and-shift on the outside region, then the inside absolute-relative error over m: Sstruct=exp(−AbsRelm(d(x),ad(y^)+b)/τ),S_struct= \! (-AbsRel_m (d(x),\,a\,d( y)+b )/τ ), (3) with temperature τ fixed across all models (all constants in the released harness). Lighting harmony (SharmS_harm). A material swap rescales albedo but must preserve the spatial pattern of illumination. We correlate the low-pass luminance fields inside the mask, NCC=corr(ℓσ(x)∣m,ℓσ(y^)∣m),NCC=corr ( _σ(x)\! _m,\; _σ( y)\! _m ), (4) which is invariant to the albedo scale, and average (after mapping to [0,1][0,1]) with a seam term that penalizes gradient energy appearing just outside the mask boundary (halos, glow) relative to x. Design rationale. Each dimension makes one trade. Material is scored at the family level because fine-grained labels are not reliably separable by current encoders even on clean crops, so strict scoring would grade the scorer, not the model. The DINOv2 term anchors appearance to r even on Track A, where the model never sees r: two valid navy suedes differ in similarity to r, readmitting a bounded one-to-many penalty (at most half of SmatS_mat, one eighth of overall). We keep it because the rewritten instruction describes r’s appearance and removing it costs agreement with the human ranking (Table 5). Preservation excludes the boundary band because mask-edge pixels are ambiguous under resampling. Structure aligns depth on the outside region, isolating inside edits from global depth-scale drift; monocular depth is not perfectly texture-invariant, so a correct swap costs a little structure (the exemplar loses 0.150.15), against which τ is set. Harmony correlates low-pass luminance, since an albedo change rescales brightness but must not restructure illumination, with a seam term watching the ring outside the mask where halos concentrate. Finally, overall is unweighted, so from the released per-task, per-dimension scores we read results dimension-wise. Uncertainty and significance. Every mean carries a 95% percentile-bootstrap interval over tasks (B=10,000B=10,000, fixed seed). Since all submissions share the same tasks, deltas and near-ties are tested with two-sided Wilcoxon signed-rank on per-task overall scores, Holm-corrected across all 14 comparisons. Each entry is a single fixed-seed run, so deltas below 0.020.02 are significant over task sampling but within plausible seed variation, and we flag them (‡ ) in Table 4. Anchor validation. Because the metrics are new, we validate them with three constructed submissions on all 1,421 tasks (Table 1): identity (submit x) saturates preservation/structure/harmony and fails only material (1.000/0.999/1.0001.000/0.999/1.000, material 0.2420.242); paste (tile r, no re-rendering) takes the highest raw material score (0.6730.673) but is caught by structure (0.5480.548) and harmony (0.6970.697), the “sticker” artifact; gt (the exemplar) scores well on all four (0.8290.829) as a soft ceiling; it averages every stored exemplar, including the 328 audit-flagged ones, and moves only to 0.8330.833 on the clean subset. The anchors span the observed range. The exemplar also calibrates conservatism: a correct change costs about 0.150.15 structure and 0.120.12 harmony (depth and low-pass luminance are not fully texture-invariant), compressing raw overall between floor 0.8100.810 and ceiling 0.8290.829. anchor mat pres struct harm overall gt (exemplar) 0.622 0.964 0.853 0.877 0.829 ±.004 identity (input) 0.242 1.000 0.999 1.000 0.810 ±.003 paste (no relight) 0.673 0.951 0.548 0.697 0.717 ±.005 Table 1: Anchor validation on all 1,421 tasks. Bold marks the dimension each anchor is constructed to violate, and no anchor is penalized elsewhere. Metric robustness. Raw overall compresses near identity, so we compare it against two alternatives (Table 2). (i) Scoring SmatS_mat by family accuracy alone (no DINOv2 term) barely moves the ranking (τ=0.91τ=0.91) and lifts the exemplar above paste on material. (i) A gate overallG=S^mat⋅(SpresSstructSharm)1/3overall_G= S_mat·(S_pres\,S_struct\,S_harm)^1/3, with S^mat=clip((Smat−0.242)/0.380, 0, 1) S_mat=clip ((S_mat-0.242)/0.380,\,0,\,1 ) the per-task material score rescaled by the global identity and exemplar anchor means, sends the do-nothing anchor to 0.2360.236 and lifts every entry above it except OmniGen2-ref (0.2040.204) and BAGEL-7B (0.0620.062), both Track C, at a mild re-ranking (τ=0.77τ=0.77). Both agree with the raw ranking on the top of the board and on the Track-C collapse: the compression is a property of the equal-weight mean among strong models, not of the ranking. We therefore report overallGoverall_G as the primary aggregate and raw overall beside it. The gate is also the view closest to human judgment: it agrees best with the Bradley–Terry strengths of Sec. 7 (τH=0.72 _H=0.72 vs. 0.680.68 raw and 0.640.64 family-only; Table 5) and it ranks paste above identity as the raters do (6–3), unlike raw overall. All three aggregates were pre-specified, and the gate’s constants are anchor means rather than parameters fitted to the judgments, so the agreement is a check, not a fit. aggregate identity paste gt >>id τraw _raw raw overall 0.810 0.717 0.829 4/23 1.00 SmatS_mat family-only 0.813 0.749 0.882 9/23 0.91 gate overallGoverall_G 0.236 0.552 0.729 21/23 0.77 Table 2: Robustness of the aggregate. Columns give the three anchors, the number of the 2323 entries scoring above the identity floor, and Kendall τ against the raw ranking. Correlations with the human ranking appear in Table 5. 5 Experiments Evaluated systems. We evaluate 11 systems in 23 track entries, and six families complete all three tracks from one checkpoint or pipeline family. Closed (provider APIs) include GPT-Image-2 (A/B/C, the disclosed data generator) and Nano-Banana-2-Lite (A/B/C). Open, cross-track include FLUX.1-Kontext (Black Forest Labs et al. 2025) (A, inpainting for B, reference mode for C), Qwen-Image-Edit (Wu et al. 2025), OmniGen2 (Wu et al. 2026), and BAGEL-7B (Deng et al. 2025) (all natively multi-image, A/B/C). Open, single-track include InstructPix2Pix (Brooks et al. 2023) (A), FLUX.1-Fill, SDXL-inpainting (Podell et al. 2024), and SD3.5-inpainting (Esser et al. 2024) (B), and Paint-by-Example (Yang et al. 2023) (C). Protocol. All open models run at native or recommended resolution with default guidance and 50 denoising steps, seed fixed. Outputs are resized to the input resolution before scoring. Track prompts are templated identically across models: Track B prepends the mask semantics, and Track C names no material in text. Templating is minimal and untuned, since per-model prompt engineering would reintroduce the input-privilege confound the tracks remove. Track-C reference injection follows each model’s native interface: the multi-image editors (Qwen-Image-Edit, OmniGen2, BAGEL) take an ordered list [x,m,r][x,m,r]; FLUX-Kontext uses its reference port (image_reference=r image\_reference=r) beside image/mask_image image/ mask\_image; Paint-by-Example its exemplar port (example_image=r example\_image=r), no text. The list-based prompts name the roles explicitly (“Image 3 shows a target material”), so role underspecification is not the immediate cause of BAGEL’s failure below. model family acc. material preservation structure harmony raw gate Track A: instruction (input ++ text) Nano-Banana-2-Lite† 0.899 0.640 ±.011 0.953 ±.001 0.820 ±.007 0.857 ±.007 0.817 ±.004 0.760 GPT-Image-2† 0.852 0.616 ±.012 0.962 ±.001 0.853 ±.006 0.878 ±.007 0.827 ±.004 0.739 Qwen-Image-Edit 0.887 0.615 ±.011 0.899 ±.002 0.710 ±.010 0.817 ±.008 0.760 ±.005 0.677 BAGEL-7B 0.797 0.533 ±.012 0.917 ±.002 0.704 ±.010 0.787 ±.009 0.735 ±.004 0.576 FLUX-Kontext 0.780 0.554 ±.013 0.820 ±.002 0.613 ±.011 0.786 ±.008 0.693 ±.005 0.540 OmniGen2 0.832 0.565 ±.012 0.856 ±.005 0.557 ±.015 0.775 ±.009 0.688 ±.006 0.539 InstructPix2Pix 0.521 0.387 ±.014 0.770 ±.008 0.703 ±.015 0.881 ±.008 0.685 ±.006 0.344 Track B: mask ++ text Nano-Banana-2-Lite† 0.894 0.638 ±.011 0.953 ±.001 0.817 ±.007 0.861 ±.007 0.817 ±.004 0.757 GPT-Image-2† 0.833 0.605 ±.012 0.962 ±.001 0.856 ±.006 0.875 ±.007 0.824 ±.004 0.724 Qwen-Image-Edit-inpaint 0.876 0.612 ±.011 0.901 ±.002 0.735 ±.011 0.830 ±.008 0.769 ±.005 0.680 FLUX-Kontext-inpaint 0.805 0.569 ±.012 0.964 ±.001 0.811 ±.006 0.876 ±.007 0.805 ±.004 0.678 BAGEL-7B 0.785 0.538 ±.013 0.856 ±.006 0.731 ±.010 0.741 ±.010 0.716 ±.006 0.570 SD3.5-inpaint 0.841 0.559 ±.011 0.941 ±.001 0.480 ±.013 0.733 ±.009 0.678 ±.005 0.528 OmniGen2-inpaint 0.733 0.506 ±.013 0.819 ±.006 0.507 ±.014 0.786 ±.009 0.654 ±.006 0.449 FLUX.1-Fill 0.551 0.407 ±.014 0.974 ±.001 0.764 ±.009 0.881 ±.007 0.756 ±.005 0.446 SDXL-inpaint 0.477 0.350 ±.014 0.941 ±.001 0.568 ±.013 0.904 ±.006 0.691 ±.005 0.330 Track C: mask ++ reference Nano-Banana-2-Lite† 0.871 0.642 ±.012 0.945 ±.002 0.808 ±.008 0.834 ±.008 0.807 ±.004 0.726 GPT-Image-2† 0.795 0.609 ±.013 0.948 ±.002 0.817 ±.008 0.857 ±.008 0.808 ±.005 0.675 FLUX-Kontext-ref 0.637 0.538 ±.017 0.959 ±.001 0.440 ±.013 0.718 ±.010 0.664 ±.006 0.386 Paint-by-Example 0.453 0.349 ±.014 0.921 ±.001 0.600 ±.013 0.849 ±.008 0.680 ±.006 0.322 Qwen-Image-Edit-ref 0.510 0.423 ±.017 0.702 ±.012 0.527 ±.018 0.827 ±.010 0.620 ±.009 0.307 OmniGen2-ref 0.353 0.318 ±.016 0.739 ±.006 0.518 ±.015 0.843 ±.009 0.605 ±.006 0.204∗ BAGEL-7B 0.612 0.504 ±.016 0.202 ±.004 0.044 ±.004 0.525 ±.010 0.319 ±.005 0.062∗ Table 3: Per-track leaderboard on bench-v1.0 (1,4211,421 tasks, higher is better), ordered by our primary aggregate gate (Sec. 4); raw is the unweighted mean, “family acc.” strict SigLIP2 family accuracy, and ± a 95% bootstrap interval. Bold/underline: best/second per column within a track; † closed; ∗ below the gated do-nothing floor (0.2360.236). Closed editors lead all three tracks; on Track C every family loses ground and two entries fall below that floor. Leaderboard. Table 3 reports per-track results. On Track A the two closed editors lead on both aggregates and clear the exemplar anchor under the gate (0.7600.760/0.7390.739 vs. 0.7290.729; 0.8170.817/0.8270.827 vs. 0.8290.829 raw). GPT-Image-2 generated the data, so its match to its own exemplars is self-consistency; the informative reading is that Nano-Banana-2-Lite, from a different provider, is the one that ranks first. Closed instruction editors thus reach the data-generating process without any spatial guidance, with any home-field advantage bounded by the cross-provider gap. The open field is cleanly separated by architecture: the 20B multi-image editor (Qwen, 0.6770.677) leads, the unified models BAGEL (0.5760.576) and OmniGen2 (0.5390.539) bracket the flow-matching editor (FLUX-Kontext, 0.5400.540), and the SD1.5-era baseline (InstructPix2Pix, 0.3440.344) trails. Track B preserves the closed-model order and promotes mask-specialized pipelines (Qwen and FLUX-Kontext inpainting, 0.6800.680/0.6780.678). Track C inverts the picture: every family drops, and two entries, OmniGen2-ref (0.2040.204) and BAGEL-7B (0.0620.062), fall below the do-nothing floor (0.2360.236). 6 Analysis What is a mask worth? (A→ , same model). Because tracks share checkpoints, Table 4 reads off the causal value of conditioning (p Holm-corrected Wilcoxon throughout). The mask’s value spans 0.150.15 across the six aligned families: +0.111+0.111 for FLUX-Kontext, whose Track-A weakness is precisely preservation (0.820.82) and whose inpainting pipeline enforces it (0.960.96, structure riding along 0.61→0.810.61→0.81); +0.009+0.009 for Qwen (‡ ); statistically zero for both closed editors (pHolm=0.45p_Holm=0.45), which already preserve from text alone; and negative for both unified models (−0.019-0.019 BAGEL ‡ , −0.034-0.034 OmniGen2), which fail to honor it. The gate replicates and sharpens this pattern: FLUX grows to +0.137+0.137 and OmniGen2 falls to −0.090-0.090, while BAGEL’s deficit loses significance. Masks help only models both weak at preservation and able to consume them. What does a reference cost? (B→ , same model). Substituting the reference for the instruction degrades every family under both aggregates (Table 4). (1) Appearance-grounding loss (closed editors, −0.010-0.010 – −0.017-0.017, ‡ -flagged): they still solve the task, just less precisely when the material is shown rather than named. (2) Material-identification collapse (Qwen −0.150-0.150, OmniGen2 −0.050-0.050): with no material named in text, family accuracy falls 0.876→0.5100.876→0.510 (Qwen) and 0.733→0.3530.733→0.353 (OmniGen2). These models cannot translate reference pixels into the semantic target their editing pathway needs, though the pathway executes correctly when the target arrives as text. Nor is this a classifier failure: the same classifier moves the closed editors only 0.0240.024–0.0380.038 from B to C, and the identity anchor’s 0.2540.254 family score is a task prior (originals often already match the target family), not an error rate. (3) Role-binding failure (BAGEL, 0.3190.319): the model misassigns the roles of the three input images and repaints the reference instead of the scene (Fig. 4), collapsing preservation (0.2020.202) and structure (0.0440.044). FLUX-Kontext-ref sits between (2) and (3): it keeps the scene (pres=0.959pres=0.959) but pastes unshaded texture, driving structure to 0.4400.440, near the paste anchor. The prompts name each image’s role explicitly and the other five families never swap roles under the identical template, so the collapse belongs to the model. The open problem is thus not rendering a material, which Track A shows models do from text, but grounding a specification given as pixels. Practical guidance. Where instructions can name the material, closed editors already operate at the exemplar level and masks add friction without accuracy. Where only open models are deployable, the best open entry is a mask-specialized pipeline (FLUX-Kontext-inpaint, 0.8050.805). Reference-driven specification should not ship today: recognize the swatch to a name and route it through text. Figure 4: Role-binding failure: the model repaints the reference (the oak bench), composites the mask silhouette into it, and discards the scene. Overall ties hide opposite failure modes. OmniGen2 and InstructPix2Pix tie on Track A (0.6880.688 vs. 0.6850.685, pHolm=0.86p_Holm=0.86) for opposite reasons: OmniGen2 gets the material (family 0.830.83) but breaks geometry (Sstruct 0.56S_struct\,0.56), InstructPix2Pix preserves geometry (0.700.70) but rarely produces the material (family 0.520.52). SDXL/SD3.5 on Track B repeat the pattern (0.6910.691 vs. 0.6780.678; |Δ|<0.02| |<0.02, single-seed): SDXL blends beautifully (harmony 0.9040.904) but paints the wrong material (0.4770.477), SD3.5 names it (0.8410.841) but deforms the surface (structure 0.4800.480). A holistic score erases these distinctions; the decomposition makes the board actionable. Closed vs. Open. The closed–open gap is concentrated in two dimensions: preservation (0.950.95–0.960.96 vs. 0.770.77–0.900.90 on Track A) and structure (0.820.82–0.850.85 vs. 0.560.56–0.710.71). Material is not the differentiator (0.620.62–0.640.64 vs. up to 0.620.62): open models know what to paint but damage the scene while painting it. The gap is engineering, not material knowledge. Where edits are hard (stratified, Track A). Taxonomy labels let us read the board by stratum. Difficulty tracks mask coverage: on the 188188 large masks every model drops 0.050.05–0.100.10 (GPT-Image-2 0.7550.755 [0.742,0.768][0.742,0.768] vs. 0.8400.840 [0.835,0.845][0.835,0.845] on the 784784 medium; the intervals are disjoint), a bigger region giving geometry and lighting more room to drift. Cluttered “other” surfaces are hardest (GPT 0.6980.698), flat countertops easiest (0.8680.868); leather and rattan are the hardest material classes (both ≈0.77≈ 0.77 for GPT), wood the easiest (0.8810.881). Order is stable across strata, except BAGEL-7B is nearly flat across materials (0.720.72–0.750.75, overlapping intervals) where others swing 0.100.10–0.150.15. The full per-stratum grid ships with the release. Δmask _mask (A→ ) Δref _ref (B→ ) family raw gate raw gate GPT-Image-2† −0.003ns-0.003^ns −0.016ns-0.016^ns −0.017‡-0.017 −0.048-0.048 Nano-Banana-2-Lite† 0.000ns0.000^ns −0.003ns-0.003^ns −0.010‡-0.010 −0.031-0.031 Qwen-Image-Edit +0.009‡+0.009 +0.003‡+0.003 −0.150-0.150 −0.373-0.373 BAGEL-7B −0.019‡-0.019 −0.006ns-0.006^ns −0.397-0.397 −0.508-0.508 FLUX-Kontext +0.111+0.111 +0.137+0.137 −0.141-0.141 −0.291-0.291 OmniGen2 −0.034-0.034 −0.090-0.090 −0.050-0.050 −0.245-0.245 Table 4: Conditioning-aligned cross-track deltas: adding a mask (Δmask _mask) and replacing text with the reference (Δref _ref), as task-paired means under both aggregates (per-track scores in Table 3). Unmarked entries are Holm-corrected Wilcoxon p<10−3p<10^-3; ns not significant; ‡ marks |Δ|<0.02| |<0.02 from a single seed, significant over tasks but not seeds (Sec. 4). Beyond ‡ , the mask helps one family and hurts one; the reference hurts all six, more under the gate. 7 Human Expert Calibration As a calibration pilot, two expert raters produced 459 two-alternative forced-choice judgments over 422 unique tasks, with hidden identities, randomized sides, optional ties, and per-dimension failure tags. The 34 anchor–anchor pairings are held out of scoring; the 25 with winners known by construction serve as attention checks (24 passed, one tied). We fit Bradley–Terry strengths to the remaining 425 scoring judgments (ties as half-wins, bootstrap B=2,000B=2,000). Ranking agreement. Across the 20 entries with at least ten judgments, human strengths correlate with the leaderboard at Kendall τ=0.68τ=0.68 and Spearman ρ=0.87ρ=0.87. On Track A, τ rises from 0.620.62 to 0.870.87 once the single outlier discussed below is excluded (1.001.00 under the gate). Against the exemplar, GPT-Image-2 ties (5W–4L–12T for the model) and Nano-Banana-2-Lite is preferred outright (8W–4L–7T); in the direct closed-vs-closed pairing the raters prefer the non-coupled editor (5–2–10), so generator identity buys no human-visible advantage. The Track-C degradation replicates family by family with the metric’s severity ordering: near-unanimous for the two largest drops (Qwen 18–1, FLUX-Kontext 16–1–2), mild for the closed pair (3–1–4, 3–0–7), with one reversal (OmniGen2, 2–4–3) resting on nine judgments. Metric near-ties are human near-ties: GPT-Image-2 Track A vs. B (Δmask _mask n.s.) draws seven ties in nine pairings, and OmniGen2 vs. InstructPix2Pix (gap 0.0030.003) splits 12–7 (n.s.). The 28% tie rate falls on the pairings the leaderboard also refuses to separate. Ours Established gate raw fam. −-LPIPS CLIP-I CLIPScore CLIPdir τH _H 0.72 0.68 0.64 0.55 0.54 0.05 −0.44-0.44 ρH _H 0.89 0.87 0.86 0.75 0.76 0.15 −0.52-0.52 Table 5: Kendall τH _H and Spearman ρH _H against human Bradley–Terry strengths (425 judgments); GT-referenced baselines use the 1,093 clean-exemplar tasks. Ours exceeds every established metric: significantly for LPIPS and CLIPScore (bootstrap P≥0.985P≥ 0.985), marginally for CLIP-I (Δτ=+0.14 τ=+0.14, CI [−0.01,0.28][-0.01,0.28]). Against existing metrics. The same judgments adjudicate between our protocol and the metrics it critiques (Table 5): every aggregate of ours correlates with the raters more strongly than every baseline, significantly so over LPIPS-to-exemplar and CLIPScore, marginally over CLIP-I. CLIPScore spans only 0.0190.019 across the 20 systems: text alignment cannot separate a correct swap from repainting the room. Directional CLIP is anti-correlated: it rewards large moves toward the target material, and the systems humans rank last make exactly such moves while destroying the scene. The GT-referenced pair scores the exemplar perfectly by construction, where the raters place it fifth of 20. Divergences. The disagreements are as informative as the agreements and share one theme: raters weight the edited region (material identity, shading realism) and discount the outside-region fidelity two dimensions reward. The metric places BAGEL-7B fourth on Track A; raters rank it last (1–14 vs. Qwen and FLUX-Kontext), its losses drawing the most lighting tags, the sole driver of the Track-A τ gap. FLUX-Kontext’s +0.111+0.111 mask gain is invisible (8–1–13): the mask repairs measured fidelity, not perceived quality. Raters prefer paste to doing nothing (6–3) and Qwen-ref above FLUX-ref (7–4–1), punishing sticker pastes harder than misidentification. Per-pair n is 9–20; the OmniGen2 reversal and conservatism probe rest on nine. 8 Limitations The benchmark is fully synthetic and single-source, disclosed: all four task images come from GPT-Image-2, also evaluated. The reference-free protocol removes source bias from scoring and the cross-provider gap (≤0.010≤ 0.010) bounds home-field advantage, but admission conditions on the generator drawing an aligned mask (a selection bias), and transfer to real photographs is unverified. Human agreement is a first answer (Sec. 7) from two raters with non-overlapping judgments; anchor checks substitute for an inter-rater statistic. Mask quality caps yield at 70.8%70.8\%; the curation VLM shares the generator’s provider, under-representing tasks illegible to it; and the taxonomy is furniture-heavy (825 of 1,421), so wall and countertop conclusions rest on smaller strata. Finally, raw overall rewards conservatism (identity 0.8100.810), which the gate view mitigates in Table 2. 9 Reproducibility We release everything11 1 https://anonymous.4open.science/r/MatReplace, including task specifications, the benchmark harness, model generations with per-task scores (∼ 33k images) and the full human study. Closed models are pinned by provider ID and date, open models by checkpoint hash and sampler, libraries by a lockfile. 10 Conclusion MatReplace isolates material replacement, scores it without a ground-truth reference, and aligns model families across three conditioning tracks. Three findings emerge: closed editors match the data-generating process without spatial guidance; the benefit of masks disappears with model strength and can reverse; and reference-driven specification remains open, degrading every family. A two-rater expert study reproduces the ranking and prefers our aggregates to the GT-referenced and CLIP baselines, and the aligned tracks resist gaming by imitation. References Bell et al. (2013) S. Bell, P. Upchurch, N. Snavely, and K. Bala OpenSurfaces: a richly annotated catalog of surface appearance. ACM Transactions on Graphics 32 (4), p. 111:1–111:17. Cited by: §2. Bell et al. (2015) S. Bell, P. Upchurch, N. Snavely, and K. Bala Material recognition in the wild with the materials in context database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 3479–3487. Cited by: §2. Black Forest Labs et al. (2025) Black Forest Labs, S. Batifol, A. Blattmann, F. Boesel, S. Consul, C. Diagne, T. Dockhorn, J. English, Z. English, P. Esser, et al. FLUX.1 kontext: flow matching for in-context image generation and editing in latent space. arXiv preprint arXiv:2506.15742. Cited by: §2, §5. Brooks et al. (2023) T. Brooks, A. Holynski, and A. A. Efros InstructPix2Pix: learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 18392–18402. Cited by: §2, §5. Chen et al. (2024) X. Chen, L. Huang, Y. Liu, Y. Shen, D. Zhao, and H. Zhao AnyDoor: zero-shot object-level image customization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6593–6602. Cited by: §2. Cheng et al. (2024) T. Cheng, P. Sharma, A. Markham, N. Trigoni, and V. Jampani ZeST: zero-shot material transfer from a single image. In European Conference on Computer Vision, p. 370–386. Cited by: §2. Deng et al. (2025) C. Deng, D. Zhu, K. Li, C. Gou, F. Li, Z. Wang, S. Zhong, W. Yu, X. Nie, Z. Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683. Cited by: §2, §5. Esser et al. (2024) P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, Vol. 235, p. 12606–12633. Cited by: §5. Fu et al. (2021) H. Fu, B. Cai, L. Gao, L. Zhang, J. Wang, C. Li, Q. Zeng, C. Sun, R. Jia, B. Zhao, et al. 3D-FRONT: 3D furnished rooms with layouts and semantics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 10933–10942. Cited by: §2. Grosse et al. (2009) R. Grosse, M. K. Johnson, E. H. Adelson, and W. T. Freeman Ground truth dataset and baseline evaluations for intrinsic image algorithms. In 2009 IEEE 12th International Conference on Computer Vision, p. 2335–2342. Cited by: §2. Hessel et al. (2021) J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y. Choi CLIPScore: a reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 7514–7528. Cited by: §2, §2. Kawar et al. (2023) B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani Imagic: text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6007–6017. Cited by: §2. Ku et al. (2024) M. Ku, T. Li, K. Zhang, Y. Lu, X. Fu, W. Zhuang, and W. Chen ImagenHub: standardizing the evaluation of conditional image generation models. In International Conference on Learning Representations, Vol. 2024, p. 46689–46722. Cited by: §2. Lopes et al. (2025) I. Lopes, V. Deschaintre, Y. Hold-Geoffroy, and R. de Charette MatSwap: light-aware material transfers in images. Computer Graphics Forum 44 (4), p. e70168. Cited by: §2. Lopes et al. (2024) I. Lopes, F. Pizzati, and R. de Charette Material palette: extraction of materials from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4379–4388. Cited by: §2. Oquab et al. (2024) M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. DINOv2: learning robust visual features without supervision. Transactions on Machine Learning Research 2024. Cited by: §2. Podell et al. (2024) D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach SDXL: improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, Vol. 2024, p. 1862–1874. Cited by: §5. Roberts et al. (2021) M. Roberts, J. Ramapuram, A. Ranjan, A. Kumar, M. A. Bautista, N. Paczan, R. Webb, and J. M. Susskind Hypersim: a photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 10912–10922. Cited by: §2. Ruiz et al. (2023) N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman DreamBooth: fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 22500–22510. Cited by: §2. Sharma et al. (2024) P. Sharma, V. Jampani, Y. Li, X. Jia, D. Lagun, F. Durand, B. Freeman, and M. Matthews Alchemist: parametric control of material properties with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 24130–24141. Cited by: §2. Sheynin et al. (2024) S. Sheynin, A. Polyak, U. Singer, Y. Kirstain, A. Zohar, O. Ashual, D. Parikh, and Y. Taigman Emu Edit: precise image editing via recognition and generation tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8871–8879. Cited by: §1, §2. Tschannen et al. (2025) M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, et al. SigLIP 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786. Cited by: §2. Wang et al. (2023) S. Wang, C. Saharia, C. Montgomery, J. Pont-Tuset, S. Noy, S. Pellegrini, Y. Onoe, S. Laszlo, D. J. Fleet, R. Soricut, et al. Imagen Editor and EditBench: advancing and evaluating text-guided image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 18359–18369. Cited by: §1, §2. Wang et al. (2004) Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13 (4), p. 600–612. Cited by: §2. Wu et al. (2025) C. Wu, J. Li, J. Zhou, J. Lin, K. Gao, K. Yan, S. Yin, S. Bai, X. Xu, Y. Chen, et al. Qwen-Image technical report. arXiv preprint arXiv:2508.02324. Cited by: §2, §5. Wu et al. (2026) C. Wu, J. Wang, P. Zheng, R. Yan, S. Xiao, X. Luo, Y. Wang, W. Li, X. Jiang, Y. Liu, et al. OmniGen2: towards instruction-aligned multimodal generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 21964–21975. Cited by: §2, §5. Yang et al. (2023) B. Yang, S. Gu, B. Zhang, T. Zhang, X. Chen, X. Sun, D. Chen, and F. Wen Paint by example: exemplar-based image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 18381–18391. Cited by: §2, §5. Yang et al. (2024) L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao Depth anything V2. Advances in Neural Information Processing Systems 37, p. 21875–21911. Cited by: §2. Zhang et al. (2023) K. Zhang, L. Mo, W. Chen, H. Sun, and Y. Su MagicBrush: a manually annotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems 36, p. 31428–31449. Cited by: §1, §2. Zhang et al. (2018) R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 586–595. Cited by: §2.