Paper deep dive

Beyond Single Object: Learning 3D Relations with Large Language Models

Kohsuke Ide, Ryousuke Yamada, Yue Qiu, Xianzheng Ma, Yoshihiro Fukuhara, Hirokatsu Kataoka, Yutaka Satoh

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 121

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 8/22/2026, 2:52:17 AM

Summary

This paper addresses the limitation of existing 3D Large Language Models (3D-LLMs) in performing fine-grained, inter-object comparisons. The authors propose Multi-3DLLM, a framework that augments a PointLLM backbone with a Patch-Interaction Transformer (PIT) to model cross-object relationships while preserving local geometry. They introduce the MO3D dataset for multi-object instruction tuning and two application-driven benchmarks: Shape Mating (geometric compatibility) and Change Captioning (geometric edit understanding). Experiments show that Multi-3DLLM significantly outperforms state-of-the-art 3D-LLMs and 2D-VLMs on these tasks and demonstrates positive transfer to single-object classification.

Entities (9)

Multi-3DLLM · model · 98%MO3D · dataset · 97%Patch-Interaction Transformer · model-component · 96%Change Captioning · benchmark · 95%Shape Mating · benchmark · 95%PointLLM · model · 92%Objaverse-Cap3D · dataset · 90%LLaVA-7B · model · 88%ShapeNet · dataset · 85%

Relation Signals (8)

Multi-3DLLM → usescomponent → Patch-Interaction Transformer

confidence 96% · Multi-3DLLM, using a minimal Patch-Interaction Transformer (PIT) that models inter-/intra-object relationships

Multi-3DLLM → trainedon → MO3D

confidence 95% · Multi-3DLLM trained on our mixture data learns geometric reasoning, surpasses all baselines on MO3D

Change Captioning → testscapability → geometric_edit_understanding

confidence 93% · CC, serves as a benchmark for grounding geometric edits in language... describes the transformation from the Anchor to the Positive

Shape Mating → testscapability → geometric_compatibility

confidence 93% · SM benchmark as a downstream task centered on geometric compatibility... determining whether components can physically fit together

Multi-3DLLM → builton → PointLLM

confidence 92% · To validate our data-driven paradigm, we extend PointLLM Xu et al. 2024.

MO3D → derivedfrom → Objaverse-Cap3D

confidence 90% · The dataset is built on Objaverse-Cap3D... by grouping Objaverse-Cap3D... instances

Multi-3DLLM → outperforms → PointLLM

confidence 90% · Multi-3DLLM... surpasses all baselines on MO3D... zero‑shot single‑object classification improves from 50.7% to 54.2%

Multi-3DLLM → →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We address a fundamental gap in 3D-LLMs: existing models focus on single-object/scene description, struggling with detailed, inter-object comparison. We propose a framework for detailed object-level reasoning across multiple objects with three components: (1) MO3D (Multi-Object in 3D), an instruction dataset requiring fine-grained multi-object comparison; (2) Multi-3DLLM, using a minimal Patch-Interaction Transformer (PIT) that models inter-/intra-object relationships while preserving local geometry; (3) Mini-apps, two application-driven benchmarks (Shape Mating, Change Captioning) that probe geometric understanding for practical use. Recent 3D-LLMs and 2D-VLMs perform poorly on these tasks, lacking both comparison-centric design and geometric awareness. In contrast, Multi-3DLLM trained on our mixture data learns geometric reasoning, surpasses all baselines on MO3D, and provides positive transfer to single-object classification.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

118,219 characters extracted from source content.

Expand or collapse full text

Beyond Single Object: Learning 3D Relations with Large Language Models Kohsuke Ide Affiliation: [2m] AIST Affiliation: University of Tsukuba Ryousuke Yamada Affiliation: [2m] AIST Affiliation: University of Technology Nuremberg Yue Qiu Affiliation: [2m] AIST Xianzheng Ma Affiliation: University of Oxford [1m] Yoshihiro Fukuhara Affiliation: [2m] AIST Hirokatsu Kataoka Affiliation: [2m] AIST Affiliation: University of Oxford Yutaka Satoh Affiliation: [2m] AIST Affiliation: University of Tsukuba Abstract We address a fundamental gap in 3D-LLMs: existing models focus on single-object/scene description, struggling with detailed, inter-object comparison. We propose a framework for detailed object-level reasoning across multiple objects with three components: (1) MO3D (Multi-Object in 3D), an instruction dataset requiring fine-grained multi-object comparison; (2) Multi-3DLLM, using a minimal Patch-Interaction Transformer (PIT) that models inter-/intra-object relationships while preserving local geometry; (3) Mini-apps, two application-driven benchmarks (Shape Mating, Change Captioning) that probe geometric understanding for practical use. Recent 3D-LLMs and 2D-VLMs perform poorly on these tasks, lacking both comparison-centric design and geometric awareness. In contrast, Multi-3DLLM trained on our mixture data learns geometric reasoning, surpasses all baselines on MO3D, and provides positive transfer to single-object classification. Figure 1: Overview of our proposed multi-object 3D understanding framework. Left: existing models fail at complex, multi-object 3D tasks. Standard 3D-LLMs cannot perform fine-grained comparisons, and 2D-VLMs are geometrically-unaware, failing tasks like complementarity. Right: our Multi-3DLLM, trained on our MO3D and Mini-apps datasets, addresses these challenges, enabling robust comparison (top) and geometric understanding (bottom). 1 Introduction The emergence of 3D large language models (3D-LLMs) has opened the door to direct interaction with physical space Xu et al. 2024; Qi et al. 2024a; Qi et al. 2024b. For the coming era of physical intelligence, however, these models must support not only description but also comparison and relational reasoning over 3D objects Malisiewicz and Efros 2009; Rosch 2024. Existing approaches largely fall into two paradigms: object-centric models that provide high-fidelity descriptions of a single object, and scene-level models Wang et al. 2023; Huang et al. 2024; Huang et al. 2023a; Fu et al. 2024; Chen et al. 2024; Abdelreheem et al. 2025 that process multiple objects for global context. Neither is designed for detailed, object-to-object comparison, which is crucial for applications such as a robot distinguishing similar tools or an augmented reality system comparing furniture. We argue that high-fidelity geometric comparison across distinct point clouds is a critical missing capability for 3D-LLMs. This capability has been stalled by two key gaps. First, object‑centric 3D‑LLMs such as PointLLM Xu et al. 2024 and ShapeLLM Qi et al. 2024a are trained on corpora like Objaverse that consist mainly of isolated objects with no collective or comparative descriptions. Scene‑level models Wang et al. 2023, in turn, often rely on object‑level coarse semantics, discarding local geometry needed for precise comparison. As a result, there is no architecture explicitly designed to capture fine‑grained cross‑object relationships, nor a large‑scale instruction dataset that teaches geometric comparison rather than generic scene understanding. Without appropriate data and models, existing methods cannot learn or be evaluated on detailed relationships, differences, and collective properties among multiple 3D objects. To address this critical data gap, we introduce MO3D dataset, a large‑scale instruction dataset for multi‑object 3D reasoning, together with two targeted mini‑applications, Shape Mating and Change Captioning (Fig. 2). MO3D dataset forms semantically related sets of point clouds with multiple objects (basically 2 or 3 objects) by grouping Objaverse‑Cap3D Deitke et al. 2023; Luo et al. 2023 instances with CLIP Radford et al. 2021 embeddings, then prompts an LLM, conditioned on multi‑view renderings and a hierarchy of constraints, to synthesize grounded question–answer pairs. The pipeline explicitly suppresses language priors by balancing question categories, mixing hard positive/negative groupings, and enforcing order invariance, yielding up to 70k positional, comparative, and holistic queries that demand fine‑grained geometric discrimination. On the modeling side, we propose Multi‑3DLLM, which augments a PointLLM Xu et al. 2024 backbone with a lightweight patch‑interaction transformer. By concatenating patch tokens from all objects and applying a shallow, scalar‑gated self‑attention block, the model captures cross‑object, patch‑level dependencies without destabilizing pretrained alignments or incurring prohibitive cost. This minimal, architecture-agnostic adaptation highlights a key finding: interaction granularity matters. While object-level aggregation smooths details, patch-level processing retains the local geometry required for fine-grained comparative tasks. This minimal modification is architecture‑agnostic and illustrates how the broader 3D‑LLM framework can be extended when combined with MO3D dataset. Extensive experiments reveal that state‑of‑the‑art 3D‑LLMs Tang et al. 2024; Qi et al. 2024a; Tang et al. 2025 and competitive 2D-VLMs Deitke et al. 2025; Liu et al. 2023a perform poorly on MO3D’s open‑ended relational tasks, providing empirical evidence showing clear limitations in multi‑object settings. Here, the proposed Multi‑3DLLM trained on our holistic data mixture substantially outperforms the strongest baselines on MO3D dataset, while surpassing models overspecialized to each Mini-App, evidence of generalization rather than memorization. These gains arise from data design: by forcing the model to resolve subtle geometric contrasts instead of relying on textual co‑occurrence, MO3D dataset teaches new relational skills and strengthens foundational ones, notably, zero‑shot single‑object classification improves from 50.7% to 54.2%, indicating positive transfer without forgetting. We summarize our contributions as follows. Conceptual contribution (Fig. Beyond Single Object: Learning 3D Relations with Large Language Models). We extend 3D‑LLMs from single‑object or purely scene‑context regimes toward a multi‑object relational paradigm. This enables fine‑grained and multi-object comparison. Dataset contribution (Fig. 2). In line with this concept, we construct a suite of benchmarks to greatly expand the capabilities of 3D-LLMs. This includes the MO3D (Multi-Object in 3D) dataset, our instruction dataset for multi-object comparison, along with two critical application-driven benchmarks, Shape Mating and Change Captioning. Model contribution (Fig. 3). To further unlock the 3D-LLMs potential, we propose Multi-3DLLM. By training with our MO3D dataset, the model can acquire functionalities with minimal modifications to an existing 3D-LLM (e.g., PointLLM). This minimal yet effective contribution demonstrates the extensibility of the 3D-LLM paradigm itself. Experimental contribution (Sec. 5). We provide a comprehensive benchmark showing that existing methods are fundamentally limited on complex, geometry‑aware tasks such as Shape Mating and Change Captioning. Training Multi‑3DLLM on our mixed data yields strong performance on these tasks and produces positive transfer to standard single‑object and point‑wise benchmarks, underscoring the broader applicability of our approach. Figure 2: Overview of our three proposed benchmarks. MO3D tasks models with multi-object comparison, including Positional, Comparative, and Holistic understanding. Shape Mating is a geometrically-intensive task requiring the model to identify the single correct complementary pair from a set of three components. Change Captioning tasks the model with generating a natural language description of the geometric edit that transforms an anchor object (left chair) into a target object (right chair). 2 Related Work Object-Centric 3D-LLMs. Research in 3D-LLMs has been largely advanced by an object-centric paradigm Tang et al. 2024; Yu et al. 2022; Xue et al. 2023; Xue et al. 2024; Qi et al. 2024b; Tang et al. 2025; Qi et al. 2023; Pang et al. 2023; Huang et al. 2023b. This line was established by PointLLM Xu et al. 2024, which created a direct pipeline from 3D point clouds to LLMs, trained on large-scale caption datasets Deitke et al. 2023; Luo et al. 2023. ShapeLLM Qi et al. 2024a pursued universal object understanding and embodied interaction by introducing a new 3D encoder, and MiniGPT-3D Tang et al. 2024 demonstrated training efficiency using 2D knowledge. A different paradigm focuses on pre-alignment with unified multi-modal embeddings Guo et al. 2023; Liu et al. 2023b. While these models excel at high-fidelity description and recognition Qi et al. 2024b of an object, their models and training data confine them to this single-object paradigm. They are not designed to process or compare multiple distinct objects simultaneously. Scene-Level 3D-LLMs. A separate line of work has focused on scene-level understanding, processing multiple objects within a full scene context Chen et al. 2020; Azuma et al. 2022; Hong et al. 2023; Li et al. 2023; Yin et al. 2023. Models in this category Wang et al. 2023; Huang et al. 2024; Huang et al. 2023a; Fu et al. 2024; Chen et al. 2024; Abdelreheem et al. 2025 handle multiple objects, typically by aggregating points into coarse object-level tokens to capture global context. While effective for spatial and contextual relationships Ma et al. 2022; Wald et al. 2020; Zhu et al. 2023; Qian et al. 2024, this object-level pooling smooths out the local, patch-level details required for intrinsic geometric comparison Chen et al. 2022; Geng et al. 2023; Zakka et al. 2020. Consequently, they struggle with the fine-grained geometric analysis our tasks require. Our work addresses the gap between the two paradigms: the patch-level fidelity of object-centric models and the multi-object capability of scene-level models. Multi-image and Multi-entity VLMs. The trajectory of 2D-VLMs provides a compelling blueprint for the advancement of 3D understanding. The field experienced a paradigm shift with instruction-tuning, exemplified by InstructBLIP Dai et al. 2023, which unlocked a remarkable ability to follow diverse natural language instructions. Flamingo Alayrac et al. 2022, Qwen-VL Bai et al. 2023 additionally demonstrated the ability to compare multiple visual inputs. The concept of “comparison” also has historical precedent in 2D tasks like change captioning Park et al. 2019; Jhamtani and Berg-Kirkpatrick 2018 and compositional reasoning Suhr and Artzi 2019; Suhr et al. 2019. This evolution clearly indicates that enabling multi-entity input is key to unlocking a higher order of understanding. However, this 2D blueprint is often insufficient for tasks demanding strict 3D geometric fidelity. While 2D-VLMs excel at semantic or contextual comparison from projections Dai et al. 2023; Alayrac et al. 2022; Bai et al. 2023; Yang et al. 2024; Deitke et al. 2025; Liu et al. 2023a, they lack direct access to the underlying 3D structure Liu et al. 2025; Zheng et al. 2025; Huang et al. 2025. A core part of our experimental contribution (Sec. 5) is to empirically demonstrate that 2D-VLMs, despite their strong language and semantic priors, struggle with the fine-grained, geometry-centric tasks that our 3D-native framework is designed to solve. 3 MO3D: Multi-Object in 3D Dataset To extend the capabilities of 3D-LLMs from single-object description to multi-object comparison, we introduce MO3D (Multi-Object in 3D) dataset, an instruction-tuning dataset designed to support both training and evaluation of relational understanding by multi-object comparison, shape mating, and change captioning. The dataset is built on Objaverse-Cap3D Deitke et al. 2023; Luo et al. 2023, which pairs 3D objects from the Objaverse with detailed captions generated by Cap3D. Our pipeline generates approximately 70k high-quality QA pairs, each involving 2-3 point clouds. MO3D dataset construction follows a two-stage pipeline. First, we use a Qwen2-72B-Instruct Yang et al. 2024 to extract naturally occurring attributes from the captions and manually organize them into six core comparison categories. Next, we sample semantically related object groups and generate QA pairs using GPT-4. Critically, to ensure questions are grounded in visual facts beyond just the text, we provide GPT-4 with both the Cap3D captions and multi-view renderings of a point cloud as input. This process targets three types of relational tasks: positional, comparative, and holistic. Grouping. We sample groups of 2-3 objects based on the semantic similarity of their captions. We employ a dual sampling inspired by the complementary pair approach Goyal et al. 2017 to create a diverse mix of scenarios, including both fine-grained comparisons (e.g., three chairs) and broader comparisons involving semantically dissimilar items. To ensure topical diversity, we pre-define six comparison categories (geometry, taxonomy, function, material, style, color) and use weighted sampling to balance their distribution. QA Generation. We use an LLM to generate QA pairs for each group. Critically, to ensure questions are grounded in visual facts beyond just the text, we provide the language model with both the Cap3D Luo et al. 2023 captions and multi-view renderings of the point clouds as input. This process is guided by a strict prompt hierarchy that enforces visual grounding and ensures clear, unambiguous question formulation. This pipeline targets three core task types. • Positional understanding requires the model to address objects based on their relative order. It requires context-dependent referencing according to each object’s role. • Comparative understanding focuses on comparing the properties of multiple objects. It involves analyzing 3D properties like shape, structure, and spatial features. • Holistic understanding demands a binary Yes/No judgment that synthesizes information across the entire object set. As the highest level in the hierarchy, it assesses the ability to grasp the whole set and perform reasoning. These three tasks (positional, comparative, and holistic) are not a random assortment. They form a coherent and structured learning curriculum designed to progressively develop sophisticated comparison capabilities. Dataset Statistics and Quality Control. The final 70k examples are split into training and test sets. Thanks to our balanced sampling, the ‘holistic’ subset features a near 50:50 Yes/No ratio, mitigating linguistic bias for binary questions. To ensure data reliability, we conducted a human audit of 500 randomly sampled QA pairs from the test set. Three independent annotators evaluated each pair, achieving an 81.0% unanimous agreement rate. Disagreements largely stemmed from intrinsic ambiguities, such as subjective perceptions of abstract shapes or occlusions (detailed in Supp.). 3.1 Mini-Apps: Application-Driven Benchmarks Beyond MO3D, we curate two application-driven benchmarks, namely shape mating and change captioning. Mini-App A: Shape Mating (SM). While MO3D focuses on semantic comparison, we introduce the SM benchmark as a downstream task centered on geometric compatibility. This task is motivated by applications in robotics and industrial design, where determining whether components can physically fit together is crucial. Given three component point clouds, the model must (1) identify which pair, if any, can successfully mate and (2) provide a geometric rationale explaining its decision. The benchmark is built from meshes sourced from Thingi10K Zhou and Jacobson 2016, using the procedural mesh-cutting process from Neural Shape Mating Chen et al. 2022 to split objects into complementary “part A” and “part B” components. This provides a clear ground truth for mating pairs. We create challenging 4-choice scenarios by sampling a ground-truth and other three incompatible parts. Mini-App B: Change Captioning (C). Our second mini-application, C, serves as a benchmark for grounding geometric edits in language. This capability is critical for 3D software applications. The dataset is derived from the ShapeNet Chang et al. 2015 variation of ShapeTalk dataset Achlioptas et al. 2023, which pairs 3D models with instructions describing the edit. This benchmark is built upon a contrastive quadruplet, a triplet of point clouds paired with a human-written instruction. The point clouds consist of an Anchor, which is the original, pre-edit point cloud, a Positive, post-edit point cloud that matches the instruction, and a Negative, a distractor that does not follow the Instruction. The Instruction itself describes the transformation from the Anchor to the Positive (e.g., “thicken the legs”). To create a challenging scenario, the Negative model is deliberately sampled to be confusing, originating from the same Anchor but corresponding to a different edit instruction (e.g., “shorten the legs”). Based on these components, we generate two distinct task variations of Verification (Binary Classification) and Delta Captioning (Generative). For Verification, the model is given the Anchor, the Instruction, and either the Positive or the Negative model. It must then answer Yes/No to whether the candidate correctly satisfies the instruction. For Delta Captioning, the model is given the Anchor and the Positive models and is asked to generate a natural language description of the geometric edits that occurred. 4 Multi-3DLLM Architecture To validate our data-driven paradigm, we extend PointLLM Xu et al. 2024. Originally built for single objects, we modify it to process a set of N objects O=o1,…,oNO=\o_1,…,o_N\ (Fig. 3). Each object oio_i is independently encoded by g⁡(⋅)g(·) and mapped by projector f⁡(⋅)f(·) into a token sequence Zi=f⁡(g⁡(oi))∈ℝT×dZ_i=f\! (g(o_i) ) ^T× d, where T is the number of patch tokens and d is the LLM dimension. We concatenate these to form a set-wide representation: Zcat=Concat(Z1,Z2,…,ZN)∈ℝ(NT)×d.Z_cat=Concat(Z_1,Z_2,…,Z_N) ^(NT)× d. (1) Patch-Interaction Transformer (PIT) block. We insert a minimal Transformer encoder FθF_θ performing self-attention jointly over all patch tokens. We use a scalar-gated residual: Z^cat=Zcat+γΔ=(1−γ)Zcat+γFθ(Zcat). Z_cat\;=\;Z_cat+γ\, \;=\;(1-γ)\,Z_cat+γ\,F_θ(Z_cat). (2) This lets γ control the PIT contribution. Crucially, initializing γ≈0γ≈ 0 preserves pre-trained alignments, allowing the model to safely learn cross-object relations without catastrophic forgetting of foundational single-object knowledge. Figure 3: Multi-3DLLM architecture. N point clouds are encoded independently by the Point Encoder and Projector. A PIT Block then performs self-attention over the concatenated point tokens, injecting cross-object relations before the tokens are passed to the LLM. This representation Z^cat Z_cat reflects our core design principle: interaction granularity matters. Unlike prior scene-level work Wang et al. 2023 that compresses each object into a single vector, our patch-level interaction preserves the fine-grained local geometry required for complex geometric tasks. This formulation endows the model with multi-object reasoning capabilities, motivating our instruction-tuning (Sec. 4.1). 4.1 Training Strategy Training procedure for our Multi-3DLLM is structured into two distinct phases, building upon the two-stage strategy of PointLLM Xu et al. 2024. The objective is to first align the core feature representations, then adapt the model to a multi-object context, and finally, teach relational reasoning skills. Phase 1: Feature Alignment. We begin by inheriting the methodology from PointLLM’s first training stage. The parameters of the point cloud encoder and the LLM are frozen, and only the MLP projector is trained. This phase uses the large-scale, brief-description instruction dataset to effectively align the features of point clouds with the LLM’s text embedding space. Phase 2: Holistic Task-Mixture Fine-tuning. We unfreeze the MLP projector, PIT block, and LLM, and instruction-tune on a unified mixture dataset of ≈ 150K pairs, sampled from the MO3D train split (≈ 63K), SM (≈ 44K), and C (≈ 40K). Note that MO3D contains ≈ 70K pairs in total across all splits. This end-to-end mixed-task training lets the projection layer and PIT block learn representations directly optimized for our target tasks, in contrast to methods that pre-train separate components Wang et al. 2023 on potentially out-of-domain data, which require more data and time and can trap the projector in a sub-optimal, poorly aligned solution. Across both phases, we minimize the negative log-likelihood of the response tokens, enabling the model to effectively integrate geometric, linguistic, and relational information. Table 1: Comprehensive comparison of Multi-3DLLM against SOTA 3D-LLM and 2D-VLM baselines across all proposed tasks: MO3D, Shape Mating, and Change Captioning. Metrics are reported as: M = Semantic Accuracy, B = Binary Accuracy, R = Reasoning Accuracy, and S = Selection Accuracy (all in %). For 3D-LLM baselines, “Concat.” denotes the point cloud concatenation strategy. For 2D-VLM baselines, “(n images)” and “(2*n images)” refer to providing 1 view and 2 opposing views per object, respectively. Model MO3D Shape Mating Change Captioning Positional (%) Comparative (%) Holistic (%) Selection (%) Verify (%) Delta Caption (%) M M B R S R B R M LLaVA-7B (n images) 25.8 11.7 61.1 41.9 20.4 2.0 48.2 29.6 7.6 LLaVA-7B (2*n images) 21.3 4.8 59.9 31.0 21.9 2.5 49.1 34.6 9.3 Molmo-7B (n images) 16.5 10.5 61.8 41.6 20.7 0.2 27.8 29.6 0.0 Molmo-7B (2*n images) 16.8 6.9 63.3 36.5 22.7 0.3 34.3 28.1 0.0 PointLLM-7B (Concat.) 15.6 4.2 66.0 33.4 21.4 1.7 42.7 33.8 0.4 ShapeLLM-7B (Concat.) 22.6 11.2 49.8 34.6 17.5 0.1 47.2 43.2 1.0 MiniGPT-3D (Concat.) 20.1 7.8 62.4 35.1 22.2 0.1 4.8 1.2 0.4 Multi-3DLLM (No-Interaction) 45.5 21.8 81.3 53.0 34.4 36.7 49.1 37.1 48.0 Multi-3DLLM (Object-Level) 52.9 32.3 81.0 49.7 25.0 23.7 51.7 34.7 49.6 Multi-3DLLM (Ours) 56.3 33.8 81.7 57.2 37.1 36.8 51.2 37.3 51.0 Table 2: Zero-shot classification (ModelNet40) results for single-object (1-Obj) versus multi-object (Multi-Obj) inputs. ‘M’ denotes the ordinal position of the queried object in 2-input and 3-input scenes, ‘MI’ indicates multiple 3D point cloud inputs, and ‘Ours’ employs Multi-3DLLM architecture. 1-Obj Multi-Obj Task (%) Task 2 Inputs 3 Inputs Model (%) M=1 M=2 M=1 M=2 M=3 PointLLM Xu et al. 2024 50.7 - - - - - PointLLM Xu et al. 2024 w/ MI - 7.4 31.2 2.1 8.7 27.6 Ours w/o PIT block 53.4 53.9 52.1 52.9 51.7 50.2 Ours 54.2 54.7 51.1 52.5 51.2 51.5 5 Experiments We first describe the experimental setup, including datasets, evaluation metrics, baseline models, and shared implementation details. We then present results on our main multi-object comparison task, MO3D, where we compare our Multi-3DLLM against baselines. We evaluate performance on two specialized mini-applications, SM and C, and then assess foundational capabilities on a standard zero-shot classification benchmark to measure knowledge transfer and potential catastrophic forgetting. Finally, we present ablation studies analyzing key design choices. Datasets. Our experiments use MO3D dataset, our primary multi-object comparison dataset, and two application-driven benchmarks, SM (geometric compatibility) and C (edit-grounded understanding). For the zero-shot classification task, we also use the ModelNet40 benchmark Wu et al. 2015. Evaluation Metrics. To comprehensively assess performance, we use task-specific metrics. For binary questions, including MO3D (holistic) and C (verify), we adopt a two-stage protocol. First, Binary Accuracy (B) checks whether the answer starts with the correct “Yes” or “No” prefix. For SM, Selection Accuracy (S) measures accuracy in a 4-way choice among “(1,2)”, “(1,3)”, “(2,3)”, and “None”. Second, for all choice-based tasks (measured by B or S), Reasoning Accuracy (R) is computed on the subset of correctly answered samples: an LLM evaluator (gpt-4o-mini) assigns a binary score. To validate this metric, a human audit of 300 responses showed 91.0% unanimous agreement with the LLM’s judgments. For open-ended questions, we report Semantic Accuracy (M), where the evaluator assigns a correctness score using task-specific criteria. For MO3D, it checks semantic alignment with the ground-truth text and, if misaligned, then checks whether the answer is visually grounded in the multi-view images; a response is marked incorrect only if both tests fail. For C (delta caption), we decompose the ground truth into edit components (e.g., “thicken chair legs”, “soften edges”) and score answers on a 10-point scale based on how many components are correctly identified, missed, or contradicted, with any contradiction yielding a score of 0. For zero-shot classification on ModelNet40 Wu et al. 2015, we use CLIP-based Radford et al. 2021 retrieval accuracy, comparing the cosine similarity between the embedding of the generated text and those of the 40 class names. Baseline Models. We compare our approach against a representative set of state-of-the-art object-centric 3D large language models (3D-LLMs), including the original PointLLM Xu et al. 2024, ShapeLLM Qi et al. 2024a, and MiniGPT-3D Tang et al. 2024. All selected 3D baselines are conversational and accept point clouds as their primary 3D input. In addition, we compare against leading 2D Vision-Language Models (VLMs), specifically LLaVA Liu et al. 2023a and Molmo Deitke et al. 2025. Input Format for Baselines. For fairness, we match each baseline with an input format suited to its modality. 3D-LLMs, originally single-input, are extended to our multi-object setting via a point cloud concatenation strategy that merges multiple point clouds into one 8,192-point cloud while preserving geometric separation (see Supplementary Material). For 2D-VLMs, we replace point cloud inputs with rendered views of the point clouds, using either one view (1-view) or two opposite views (2-view) per object. Additional baseline comparisons (including scene-level and text-only models) and zero-shot evaluations on real-world datasets for Shape Mating are detailed in the Supplementary Material. Implementation Details. Our Multi-3DLLM extends PointLLM, reusing the Point-BERT encoder and Vicuna-7B backbone while adding key architectural changes for multi-object comparison (see Section 4). All experiments were conducted on 8 NVIDIA H200 GPUs; remaining training details are given in the supplementary. Table 3: Ablation study on the training data mixture. We compare our final model (trained on a uniform mixture of all three datasets) against specialized models trained only on a single downstream task. This analysis measures the effect of co-training with our main MO3D dataset on the performance of the mini-applications. Training Strategy MO3D Shape Mating Change Captioning Positional (%) Comparative (%) Holistic (%) Selection (%) Verify (%) Delta Caption (%) M M B R S R B R M Multi-3DLLM (MO3D-only) 45.5 21.9 84.6 53.0 - - - - - Multi-3DLLM (ShapeMating-only) - - - - 34.4 36.7 - - - Multi-3DLLM (ChangeCaptioning-only) - - - - - - 49.7 34.0 48.2 Multi-3DLLM (All Tasks) 56.3 33.8 81.7 57.2 37.1 36.8 51.2 37.3 51.0 5.1 Multi-Object Comparison (MO3D) Objective. We evaluate Multi-3DLLM on the MO3D dataset to assess its multi-object comparison capabilities against SOTA 3D-LLM and 2D-VLM baselines. Experimental Design. We test all models on the MO3D test set. Our model is compared with 3D-LLMs (via point cloud concatenation) and 2D-VLMs (via rendered images). We report Semantic Accuracy (M) for positional and comparative tasks, and Binary (B) / Reasoning (R) Accuracy for the holistic task. Results. Table 1 shows that our Multi-3DLLM consistently outperforms all baselines across the four MO3D metrics. It scores 56.3 on Positional (M) and 81.7 on Holistic (B), substantially higher than the best 3D-LLM baseline, ShapeLLM, which scored 22.6 and 49.8, respectively. Our model also exceeds the strongest 2D-VLM baselines, LLaVA and Molmo. The gap is most pronounced on open-ended tasks: in Comparative (M), our model achieves 33.8, nearly three times the next best 3D baseline, ShapeLLM, which scored 11.2, and the best 2D baseline, LLaVA, which scored 11.7, demonstrating a significantly stronger ability to articulate fine-grained differences. A key observation from Table 1 is that both 2D-VLM and 3D-LLM baselines fail on the open-ended Positional and Comparative tasks, even though 3D-LLMs see full 3D point clouds and 2D-VLMs see 2D projections. This suggests that while captions offer a strong prior, 2D-VLMs struggle with geometric awareness, and existing 3D-LLMs face challenges in multi-object comparison. 5.2 Performance on Mini-Applications Objective. Beyond the main comparison dataset, we evaluate two specialized mini-applications, SM and C, to test whether our model, which is trained on a holistic mixture of tasks, supports fine-grained geometric compatibility and edit-grounded instruction following, and to assess the value of geometry-aware 3D modeling. Both tasks heavily depend on detailed 3D geometry. We therefore hypothesize that 2D-VLM baselines, which only process 2D projections and lack geometric awareness, will exhibit clear limitations, underscoring the value of 3D-native models for complex spatial reasoning. Experimental Design. We evaluate our model, Multi-3DLLM, on the held-out test sets for both SM and C. We compare its performance against the full suite of 3D-LLM and 2D-VLM baselines to test our hypothesis on the necessity of 3D-native models. Performance is measured using the task-specific metrics defined in our Evaluation Metrics section: Selection (S) and Reasoning (R) Accuracy for SM, Binary (B) and Reasoning (R) Accuracy for C (Verify), and Semantic Accuracy (M) for C (Delta Caption). Results. Table 1 summarizes results on the two mini-applications and provides evidence for the necessity of geometrically-aware 3D-LLMs. On SM, our hypothesis from the Objective is strongly validated: all 2D-VLM and 3D-LLM baselines score well below the 25% four-way chance level, indicating a failure to capture the geometric compatibility signal. In contrast, Multi-3DLLM achieves a Selection (S) score of 37.1%, outperforming all baselines. Moreover, all baseline models score near-zero on the Reasoning (R) metric, showing that their already-poor selections lack valid geometric justification and are effectively random. C yields a more nuanced picture. For the generative Delta Caption (M) task, we again see near-zero performance from all 2D and 3D baselines, while our model reaches 51.0% and is the only architecture that demonstrates comprehension. The Verification (B) task is exceptionally challenging, with all models performing near the 50.0% chance baseline (further analysis on real-world domain transfer and enhanced reasoning strategies is provided in Supp). 5.3 Evaluation on Zero-Shot Classification Objective. We evaluate the model’s foundational capabilities using a zero-shot classification task. This evaluation serves two main purposes. We first test whether the model can ground positional language (e.g., “the first object”) to the correct point cloud in multi-object scenes, a critical skill for all our tasks. We then examine how our holistic training strategy affects this fundamental skill. By comparing our final model with the original PointLLM, we assess whether this training strategy leads to positive knowledge transfer or catastrophic forgetting, while comparison with the variant w/o PIT block isolates the impact of our architectural changes on high-level semantic recognition. Experimental Design. We perform zero-shot classification on ModelNet40 benchmark Wu et al. 2015 under three settings (see Table 2): (1) Single-Object, (2) 2-Input, and (3) 3-Input. In the multi-object settings, the model receives a set of point clouds from different ModelNet40 classes, with one designated as the target and the rest as distractors, and is queried with an ordinal prompt such as “What is the class of the first object?” or “Identify the category of the n-th object” For the 2-Input task we query both positions (M=1M=1, M=2M=2); for the 3-Input task we query all three (M=1M=1, M=2M=2, M=3M=3). Classification accuracy is measured using the CLIP-based Radford et al. 2021 retrieval protocol from the original PointLLM. Results. Table 2 shows that PointLLM w/ MI, which only adds multi-input capacity without multi-object tuning, fails critically: it exhibits strong positional bias Zheng et al. 2023, with accuracy collapsing on early objects (e.g., a score of 2.1 for M=1M=1 in the 3-Input task) and recovering only for the final object. This indicates that positional grounding must be learned. In contrast, fine-tuned models achieve stable accuracy across all positional queries, avoid catastrophic forgetting, and even surpass PointLLM on the single-object setting. This shows that task-mixture training yields positive knowledge transfer and strengthens semantic understanding. Finally, our full model performs similarly to the Multi-3DLLM (w/o PIT) variant, suggesting that the PIT block, while crucial for fine-grained comparative and geometric tasks (Sec. 5.1), is not required for high-level semantic classification and does not degrade its basic capability. Table 4: Quantitative comparison of attention patterns between our Multi-3DLLM model and a model trained only on MO3D baseline. Metrics are computed on 266 object instances from the MO3D test set. Sparsity (Gini) measures attention inequality (1 = max concentration). Entropy measures attention uniformity (0 = max concentration). Top-K% Conc indicates proportion of attention mass on the top K% of points. Effective Rank approximates the number of points receiving substantial attention, computed as exp⁡(H) (H) where H is the entropy. Model Sparsity (Gini) Entropy Top-10% Conc. Top-20% Conc. Effective Rank Multi-3DLLM (MO3D-Only) 0.605 0.822 0.538 0.652 2209 Multi-3DLLM 0.726 0.716 0.675 0.763 1088 5.4 Ablation Studies Impact of the PIT block. We ablate the PIT block by comparing three architectural variants, shown in Table 1. Multi-3DLLM (w/o PIT block), hereafter No-Interaction, feeds the concatenated tokens to the LLM without cross-object mixing. Multi-3DLLM, hereafter Object-Level, uses an object-level encoder that mean-pools each object’s patch tokens, runs a self-attention over the resulting N object tokens, and broadcasts one residual update per object back to its T patch tokens, as in prior scene-level designs Wang et al. 2023. Our full model, Multi-3DLLM (Ours), hereafter Ours (PIT), instead applies a patch-level Transformer that computes per-token residuals, as detailed in Sec. 4. The results in Table 1 validate our patch-level interaction design. On MO3D, Ours (PIT) improves fine-grained positional and comparative reasoning over No-Interaction by 10.8 points on Positional (M) and 12.0 points on Comparative (M), showing that patch-level cross-object mixing is crucial for relational understanding. Object-Level performs particularly poorly on SM: its Selection accuracy drops to 25.0 versus 34.4 for No-Interaction, and its Rationale accuracy to 23.7 versus 36.7. This indicates that mean-pooling and broadcasting a single object-wise residual washes out token-level cues such as mating interfaces. By computing token-specific updates, Ours (PIT) preserves these local correspondences and achieves the best SM scores, 37.1 on Selection and 36.8 on Rationale. On C, Ours (PIT) also performs strongly suggesting that patch-level interactions support more specific and faithful descriptions. Overall, Ours (PIT) yields the most robust gains on tasks requiring both inter-object and intra-object (patch-level) reasoning. Attention analysis We evaluate our training strategy by comparing the attention patterns of a model trained on MO3D + Mini-apps with a MO3D-only baseline. We analyze 266 object-level attention distributions from 100 MO3D test sets and compute four metrics (Table 4): Gini Coefficient, Normalized Entropy, Top‑K Concentration, and Effective Rank. Effective Rank, defined as exp⁡(H) (H) where H is the Shannon entropy Shannon 1948, approximates the number of points receiving substantial attention, which we call the “used points”. Table 4 shows that multitask learning yields markedly more efficient and sparser attention: the Effective Rank drops by 50.7%, indicating that our model processes less than half as many points while achieving higher overall performance (Table 3). We hypothesize that this acts as an attention regularizer—by solving fine-grained tasks such as SM, the model learns to aggressively discard irrelevant information and focus on salient geometric features (Fig. 4). The resulting sparsity not only improves performance but also offers promising opportunities for future computational optimizations and enhanced interpretability. Figure 4: Attention visualization comparison. Attention weights overlaid on a 3D point cloud from the MO3D test set. Left (MO3D Only): The task-specific model shows distributed, diffuse attention across the entire object. Right (MO3D + Mini-apps): Our multitask model exhibits concentrated and sparse attention. Attention weights are mapped to colors, with yellow indicating high attention and darker colors indicating low attention. The multitask model achieves competitive performance while attending to significantly fewer points (see Table 4). 6 Conclusion We expanded conventional 3D-LLMs, previously limited to single object, to handle multi-object point clouds for comparison, shape mating, and change captioning. The MO3D dataset provides a robust benchmark, while our Multi-3DLLM, trained on a holistic data mixture with a minimal patch-level adaptation, significantly outperforms prior methods (e.g., PointLLM). We further demonstrate its practical utility on complex application-driven tasks. Limitations and future work. Our pipeline relies on existing captions, inheriting potential biases in description granularity. Extending to scenes with more objects introduces token budget trade-offs, and transferring to real-world scans highlights point density challenges. Future work will explore hierarchical part-based annotations, efficient token scaling strategies, and robust real-world domain adaptation. References Abdelreheem et al. (2025) Ahmed Abdelreheem, Filippo Aleotti, Jamie Watson, Zawar Qureshi, Abdelrahman Eldesokey, Peter Wonka, Gabriel Brostow, Sara Vicente, and Guillermo Garcia-Hernando. Placeit3d: Language-guided object placement in real 3d scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025. Achlioptas et al. (2023) Panos Achlioptas, Ian Huang, Minhyuk Sung, Sergey Tulyakov, and Leonidas Guibas. ShapeTalk: A language dataset and framework for 3d shape edits and deformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12685–12694, 2023. Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 23716–23736, 2022. Azuma et al. (2022) Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page 19129–19139, 2022. Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005. Chang et al. (2015) Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015. Chen et al. (2020) Dave Zhenyu Chen, Angel X. Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In Proceedings of the European Conference on Computer Vision (ECCV), page 202–221, 2020. Chen et al. (2024) Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26428–26438, 2024. Chen et al. (2022) Yun-Chun Chen, Haoda Li, Dylan Turpin, Alec Jacobson, and Animesh Garg. Neural shape mating: Self-supervised object assembly with adversarial shape priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page 12724–12733, 2022. Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, Ion Stoica, and Eric P Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. lmsys.org, 2023. Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), pages 49250–49267, 2023. Deitke et al. (2023) Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13142–13153, 2023. Deitke et al. (2025) Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 91–104, 2025. Fu et al. (2024) Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401, 2024. Ganesan (2018) Kavita Ganesan. Rouge 2.0: Updated and improved measures for evaluation of summarization tasks. arXiv preprint arXiv:1803.01937, 2018. Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821, 2021. Geng et al. (2023) Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and He Wang. Gapartnet: Cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7081–7091, 2023. Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6904–6913, 2017. Guo et al. (2023) Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.00615, 2023. Hong et al. (2023) Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482–20494, 2023. Huang et al. (2024) Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. Advances in Neural Information Processing Systems, 37:113991–114017, 2024. Huang et al. (2023a) Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871, 2023a. Huang et al. (2023b) Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22157–22167, 2023b. Huang et al. (2025) Xiaohu Huang, Jingjing Wu, Qunyi Xie, and Kai Han. Mllms need 3d-aware representation supervision for scene understanding. arXiv preprint arXiv:2506.01946, 2025. Jhamtani and Berg-Kirkpatrick (2018) Harsh Jhamtani and Taylor Berg-Kirkpatrick. Learning to describe differences between pairs of similar images. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4024–4034, 2018. Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2019. Li et al. (2023) Mingsheng Li, Xin Chen, Chi Zhang, Sijin Chen, Hongyuan Zhu, Fukun Yin, Gang Yu, and Tao Chen. M3dbench: Let’s instruct large models with multi-modal 3d prompts. arXiv preprint arXiv:2312.10763, 2023. Lin (2004) Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004. Liu et al. (2025) Dingning Liu, Cheng Wang, Peng Gao, Renrui Zhang, Xinzhu Ma, Yuan Meng, and Zhihui Wang. 3daxisprompt: Promoting the 3d grounding and reasoning in gpt-4o. Neurocomputing, 637:130072, 2025. Liu et al. (2023a) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023a. Liu et al. (2023b) Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding. Advances in neural information processing systems, 36:44860–44879, 2023b. Luo et al. (2023) Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. In Advances in Neural Information Processing Systems (NeurIPS), pages 75307–75337, 2023. Ma et al. (2022) Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022. Malisiewicz and Efros (2009) Tomasz Malisiewicz and Alyosha Efros. Beyond categories: The visual memex model for reasoning about object relationships. In Advances in Neural Information Processing Systems (NeurIPS), 2009. Pang et al. (2023) Yatian Pang, Eng Hock Francis Tay, Li Yuan, and Zhenghua Chen. Masked autoencoders for 3d point cloud self-supervised learning. World Scientific Annual Review of Artificial Intelligence, 1:2440001, 2023. Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002. Park et al. (2019) Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4624–4633, 2019. Qi et al. (2023) Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In Proceedings of the International Conference on Machine Learning (ICML), pages 28223–28243. PMLR, 2023. Qi et al. (2024a) Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. ShapeLLM: Universal 3d object understanding for embodied interaction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 214–238, 2024a. Qi et al. (2024b) Zhangyang Qi, Ye Fang, Zeyi Sun, Xiaoyang Wu, Tong Wu, Jiaqi Wang, Dahua Lin, and Hengshuang Zhao. Gpt4point: A unified framework for point-language understanding and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26417–26427, 2024b. Qian et al. (2024) Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 4542–4550, 2024. Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), pages 8748–8763. PmLR, 2021. Rosch (2024) Eleanor Rosch. Principles of categorization. In Cognition and categorization, pages 27–48. Routledge, 2024. Shannon (1948) Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948. Suhr and Artzi (2019) Alane Suhr and Yoav Artzi. Nlvr2 visual bias analysis. arXiv preprint arXiv:1909.10411, 2019. Suhr et al. (2019) Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 6418–6428, 2019. Tang et al. (2024) Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Yixue Hao, Long Hu, and Min Chen. Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors. In Proceedings of the ACM International Conference on Multimedia (ACM M), page 6617–6626, 2024. Tang et al. (2025) Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Jinfeng Xu, Yixue Hao, Long Hu, and Min Chen. More text, less point: Towards 3d data-efficient point-language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7284–7292, 2025. Wald et al. (2020) Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3961–3970, 2020. Wang et al. (2023) Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.08769, 2023. Wu et al. (2015) Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1912–1920, 2015. Xu et al. (2024) Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. In Proceedings of the European Conference on Computer Vision (ECCV), pages 131–147, 2024. Xue et al. (2023) Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1179–1189, 2023. Xue et al. (2024) Le Xue, Ning Yu, Junnan Li, Roberto Martín-Martín, Jiajun Wu, Ran Xu, Juan Carlos Niebles, Caiming Xiong, and Silvio Savarese. ULIP-2: Towards scalable multimodal pre-training for 3D understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27091–27101, 2024. Yang et al. (2024) An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. Yin et al. (2023) Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, et al. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. Advances in Neural Information Processing Systems, 36:26650–26685, 2023. Yu et al. (2022) Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-BERT: Pre-training 3D point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19313–19322, 2022. Zakka et al. (2020) Kevin Zakka, Andy Zeng, Johnny Lee, and Shuran Song. Form2fit: Learning shape priors for generalizable assembly from disassembly. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9404–9410. IEEE, 2020. Zheng et al. (2025) Duo Zheng, Shijia Huang, Yanyang Li, and Liwei Wang. Learning from videos for 3d world: Enhancing mllms with 3d vision geometry priors. arXiv preprint arXiv:2505.24625, 2025. Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023. Zhou and Jacobson (2016) Qingnan Zhou and Alec Jacobson. Thingi10k: A dataset of 10,000 3d-printing models. arXiv preprint arXiv:1605.04797, 2016. Zhu et al. (2023) Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2911–2921, 2023. Supplementary Material Appendix A Implementation Details Model Checkpoints We initialized our system using publicly available pre-trained models from the PointLLM Xu et al. 2024 framework: Language Model: Vicuna-7B-v1.5 (lmsys/vicuna-7b-v1.5) Chiang et al. 2023 Point Cloud Encoder: Point-BERT Yu et al. 2022, pre-trained via ULIP2 Xue et al. 2024 CLIP used for classification, grouping: OpenCLIP ViT-L/14 backbone Radford et al. 2021 A.1 Training Hyperparameters We detail the hyperparameters used for our two-stage training strategy in Table 5. All models were trained using the AdamW optimizer with a cosine learning rate scheduler and a warmup ratio of 0.03. Table 5: Training Hyperparameters. Parameter Phase 1 Phase 2 Feature Alignment Holistic Task-Mixture Batch Size 16 14 Learning Rate 2e-3 2e-5 Weight Decay 0.0 0.0 Number of Epochs 3 3 LR Scheduler Cosine w/ Warmup Cosine w/ Warmup Warmup Ratio 0.03 0.03 Trained Params Projector Projector, PIT block, LLM Frozen Point Encoder Point Encoder Params PIT block, LLM Hardware Training was conducted on 8 NVIDIA H200 GPUs (140GB VRAM). Training Time. The training process was completed in two phases with the following durations: 70 minutes for feature alignment (Phase 1), 12 hours for Training on holistic mixture of datasets (MO3D, Shape Mating, Change Captioning). Appendix B Dataset Generation Details B.1 MO3D Dataset Data-Driven Category Definition. To ensure our benchmark covers diverse aspects of 3D objects, we established six core categories through a two-stage process. First, we prompted Qwen2-72B-Instruct Bai et al. 2023 to identify common attribute types from 70k samples of Objaverse-Cap3D Deitke et al. 2023; Luo et al. 2023 captions. Then, we manually grouped these outputs into six semantically distinct and comprehensive categories, followed by manual curation. These categories, detailed in Table 8, form the foundation for our balanced generation pipeline. Figure 5: Category Distribution and Weighted Balancing. We transform the naturally occurring, unbalanced attribute distribution (left) into a curated target distribution (middle). This process explicitly prioritizes geometric and structural categories (e.g., Geometry, Material) while suppressing superficial visual cues (e.g., Color) to enforce 3D understanding. The stacked bars (right) verify that this balanced distribution is consistently applied across all three task types: Positional, Comparative, and Holistic QA. Table 6: Overall Statistics of the MO3D Dataset. The dataset maintains a perfectly balanced distribution across the three core tasks and a near-even split between positive and negative object groupings. Total Samples Task Types Group Types Positional Comparative Holistic Positive Negative 69,996 23,332 23,332 23,332 34,926 35,070 (100%) (33.3%) (33.3%) (33.3%) (49.9%) (50.1%) Table 7: Distribution of Question Categories. We compare our design target weights with the actual distribution in the final dataset. The resulting distribution closely matches our goal of prioritizing geometric and structural comparison. Category Target Weight Actual Count (%) Geometry / Structure 30% 21,264 (30.4%) Material 20% 14,316 (20.5%) Function 15% 10,578 (15.1%) Taxonomy 15% 10,200 (14.6%) Style / Aesthetics 15% 10,206 (14.6%) Color 5% 03,432 0(4.9%) Table 8: The six core reasoning categories curated from our data-driven analysis. We define each category and provide examples of the raw, LLM-discovered themes that were merged into it. Core Category Description and Example Question Focus Examples of Merged Raw Categories geometry/structure The shape, size, number of parts, and structural complexity of an object. e.g., “Does it have more than four legs?” shape, size, components, part, feature, features, detail, base, form taxonomy The general class, type, or identity of an object. e.g., “Is this object a piece of furniture?” objecttype, type, object_type, object, theme, category function The intended purpose, use, or action associated with an object. e.g., “What is this object used for?” function, action, purpose, usage, activity material The physical substance an object is made of. This is distinct from color. e.g., “Is the frame made of metal or wood?” material, substance, composition style/aesthetics The visual style, era, pattern, decoration, or contextual setting of an object. e.g., “Does this object have a writing on the surface?” style, decoration, pattern, design, era, location, context color The surface color or hue of an object or its parts. e.g., “Is the main color of the object brown?” color, hue, finish Group Formation Strategy. The foundation of each query is a meaningful group of objects. We formulated these groups based on the semantic similarity of captions from the Objaverse-Cap3D corpus. To quantify similarity efficiently at scale, we computed embeddings using a pre-trained CLIP text encoder (ViT-L/14) Radford et al. 2021 and leveraged Faiss Johnson et al. 2019 to pre-compute and cache the top-50 nearest neighbors for every object. At generation time, we select an anchor object and n−1n-1 companions. We specifically focus on group sizes of n∈2,3n∈\2,3\. The inclusion of 3-object groups is a deliberate design choice: a group of three is the minimal configuration required to introduce complex relational concepts, such as identifying a semantic “outlier” or finding a “majority” property, which are impossible with only pairwise comparisons. To ensure a balanced distribution of comparison scenarios, we employ a Dual Sampling strategy with specific thresholds: • Positive Sampling: Selects companion objects from the anchor’s top-50 semantically similar neighbors. This results in groups with high conceptual overlap, suitable for fine-grained comparison (e.g., distinguishing between two different chairs). • Negative Sampling: Selects companions from the pool of items that do not appear in the anchor’s top-50 neighbors. This produces groups with low conceptual overlap for broader distinctions. Finally, the sequence of selected objects is randomly shuffled to ensure the task is order-invariant and to prevent the model from learning positional biases. Guided Question Generation. We use GPT-4 to generate QA pairs, inputting both the Cap3D captions and multi-view renderings. To prevent hallucinations and ensure quality, we enforce a strict Prompt Hierarchy: • Grounding Constraint (Highest Priority): All information must be strictly visually grounded in the provided inputs. Invented details are prohibited. • Task-Specific Constraints : Each task type (e.g., geometry) enforces specific keywords (e.g., use “shape” instead of ambiguous “feature”). • Category Coverage: We use weighted balancing to target specific categories, prioritizing geometric understanding over simpler visual cues as shown in Figure 5. If a category is not applicable (e.g., no color information), the model falls back to a valid alternative. The full prompt used for generation is provided in Figure 9. Balance Correction. To mitigate linguistic priors (e.g., the tendency to answer ”Yes”), we implement a post-hoc balance correction. We analyze the holistic subset and identify any imbalance. We then regenerate a subset of samples using a constrained prompt that forces the generation of a ”No” question (e.g., asking about a property not shared by the group), ensuring a near 50:50 distribution in the final dataset. Dataset Statistics. The final MO3D dataset consists of approximately 70k multi-object instruction-following examples. As detailed in Table 6, the dataset is perfectly balanced across the three core tasks (Positional, Comparative, Holistic), with each constituting exactly one-third of the data. The object grouping strategies (Positive vs. Negative sampling) are also balanced near 50:50 to ensure diverse comparison scenarios. Furthermore, Table 7 demonstrates that our weighted sampling strategy successfully aligned the generated question categories with our target distribution, prioritizing geometric and structural understanding (30.4%) over simpler attributes like color (4.9%). We further analyze the linguistic complexity in Table 9 and Figure 12. Data Reliability and Intrinsic Ambiguity. To rigorously validate the reliability of the MO3D dataset, we conducted a human audit on 500 randomly sampled QA pairs from the test set. Three independent annotators evaluated each pair, achieving a strong unanimous agreement rate of 81.0%. Through this audit, we found that the remaining disagreements did not stem from incorrect ground truths, but rather from the intrinsic ambiguity of 3D object interpretation. As illustrated in Figure 6, subjective perceptions of highly abstract shapes or partial occlusions naturally lead to divergent valid interpretations among humans. This highlights the inherent complexity of 3D relational tasks compared to standard 2D QA. Crucially, when evaluated on this high-quality unanimous subset, our model achieves an average accuracy of 59.2% across the MO3D tasks, closely aligning with our 57.3% average on the full test set. This verifies that intrinsic ambiguities do not artificially distort the evaluation metrics. Figure 6: Intrinsic Ambiguity. Disagreements in human verification often arise from subjective perception (e.g. abstract shapes or occlusions) Table 9: Linguistic Complexity Statistics. We report the average and maximum length in words for instructions and responses. Shape Mating features long responses due to the requirement of geometric rationales. Dataset Task Type Instruction Response Mean Max Mean Max MO3D Positional 10.8 21 10.0 25 Comparative 13.3 30 18.1 33 Holistic 9.9 18 12.8 31 Shape Mating Selection 31.8 34 73.5 130 Change Cap. Verify 64.3 191 25.4 83 Delta 16.8 18 25.6 96 B.2 Mini-App A: Shape Mating Details Data Construction and Sampling. We source base 3D meshes from the Thingi10K dataset Zhou and Jacobson 2016 and generate mating pairs using the cut shell operation from Neural Shape Mating Chen et al. 2022. We utilize five distinct cut geometries: Planar, Sine, Square, Pulse, and Parabolic. Crucially, we selected the cut shell operation to ensure domain consistency with our point cloud encoder, Point-BERT Yu et al. 2022. Since Point-BERT is pre-trained solely on object surface points rather than interior cross-sections, the introduction of artificial flat cut-faces inherent to solid cutting methods would result in a domain gap. Consequently, we utilize the cut shell method to preserve the surface-shell characteristic, thereby aligning the input distribution with the pre-training regime of the encoder. Finally, we perform uniform random sampling on the mesh surface to generate dense point clouds of 8,192 points per part (Figure 7), capturing the fine-grained geometric details required for the mating task. To strictly enforce the 4-choice classification task, we employ a targeted sampling logic for scene composition. For 1-Mate (Positive) scenarios, we randomly select a valid ground-truth pair (Part A and Part B from the same instance) and sample a third ”decoy” part. This decoy is carefully selected to be non-mating due to specific reasons, such as originating from a different object, a different cut type, or a different cut position (Phase Mismatch), and its position is randomized. For 0-Mate (Negative) scenarios, we sample three parts such that no combination forms a valid pair, acting as a hard negative that forces the model to verify all possible connections. Figure 7: Point cloud generation pipeline for Shape Mating. Starting from a raw mesh (Left), we apply the cut shell operation to split the object into two complementary halves (Middle). Unlike solid cuts, this operation preserves the hollow, surface-only structure of the object. Finally, each part is uniformly sampled into 8,192 points (Right) to serve as the input for our model. Figure 8: Hard Negative Sampling for Change Captioning. We construct contrastive triplets where the Negative candidate (red captions) is not random, but a “hard negative” sharing the same source Anchor as the Positive target (green captions). This Negative sample is crucial for the Verification task to generate challenging “No” instances Question and Rationale Formulation. We construct the QA pairs using a two-step process to ensure both linguistic diversity and geometric grounding. First, for the question component, we employ a set of 15 distinct templates (detailed in Table 16) to ensure consistent task formulation while providing linguistic variety. These templates explicitly list the four options and mandate a reasoning-based response. Second, for the rationale component, we train the model to explain why a pair does not mate by automatically assigning structured error tags to non-mating pairs. We then use GPT-4o-mini to paraphrase these tags into natural language justifications. The full prompt used for this paraphrasing is provided in Figure 10. The error types are defined as follows: • cut mismatch indicates that the two parts possess disparate cut interfaces, such as a planar surface versus a sinusoidal one. • object mismatch signifies that although the parts share the same cut type, they originate from distinct source objects and thus do not align globally. • phase mismatch occurs when parts share the same object and cut geometry but are derived from different cut instances or positions, preventing an exact fit. • same side denotes topological incompatibility, where the selected parts represent the same side of the object, such as two “Part A” components. Dataset Statistics. We provide a detailed analysis of linguistic complexity in Table 9 and visualize the length distributions in Figure 13. As shown in the table, Shape Mating involves particularly long responses due to the requirement for detailed geometric reasoning. B.3 Mini-App B: Change Captioning Details Data Construction and Sampling. We construct this benchmark using the ShapeNet subset of the ShapeTalk dataset Achlioptas et al. 2023. The core unit is a contrastive triplet consisting of an Anchor, a Positive, and a Negative point cloud, paired with an Instruction. We employ a strict Hard Negative Sampling strategy to ensure difficulty, as illustrated in Figure 8. For a given Anchor-Positive pair (e.g., ”thinner back”), we prioritize sampling a Negative shape that shares the same Anchor but corresponds to a different edit instruction (e.g., ”thicker seat”). This forces the model to ground the specific semantic details of the instruction, rather than relying on coarse object recognition. If no such hard negative exists, we fall back to a random object from the same semantic class. Question and Rationale Formulation. We transform the raw triplet data into model inputs using standardized templates. As detailed in Table 17, we employ distinct template sets for each task to ensure consistent definition while introducing phrasing variety. To ensure high-quality linguistic output, we employ GPT-4o-mini for both tasks. For the Verification task, we explicitly randomize the input order of the Anchor and Candidate point clouds to prevent the model from memorizing positional cues (e.g., assuming the answer is always the second object). We used GPT-4o-mini to construct a natural language rationale based on the original instruction associated with the Positive/Negative object. For the Delta Captioning task, we similarly use GPT-4o-mini to paraphrase and consolidate multiple raw utterances into a single, fluent description. Dataset Statistics. We report the linguistic complexity in Table 9. Notably, the Verification task has the longest average instruction length of 64.3 words because the input explicitly includes the full list of geometric requirements (from the instruction) that the model must check. B.4 Data Splitting Across all our benchmarks (MO3D dataset, Shape Mating, and Change Captioning), we employ a strict leakage-free splitting strategy to ensure rigorous evaluation, as simple random splitting is insufficient when 3D assets share underlying geometries or appear in multiple grouping scenarios. To prevent data leakage, we utilize a graph-based approach where every unique 3D asset is represented as a node, and edges are drawn between nodes if they appear together in the same sample (e.g., within a triplet) or share the same source object. We then compute the connected components of this graph and atomically assign entire components to a single split (Train or Test). This methodology guarantees that no object instance, nor any of its co-occurring or geometrically related variants, ever leaks across splits, ensuring that our evaluation measures true generalization rather than memorization. Appendix C Additional Ablation Studies C.1 Baselines & Input Format Fairness To ensure a fair comparison across fundamentally different architectures, we carefully designed modality-specific input formatting strategies: Object-Centric 3D-LLMs: Models like PointLLM and ShapeLLM natively accept a single point cloud. For our multi-object tasks, we implement a concatenation with separation strategy. Each object’s point cloud is individually normalized into a unit sphere, and then translated along a single axis (e.g., the x-axis) with a fixed margin. This preserves the intrinsic local geometry of each object while keeping them spatially distinct within a single 8,192-point input limit. Scene-Level Models: We evaluated scene-level models, including Chat-Scene Huang et al. 2024 and L3DA Chen et al. 2024. Since these models are designed to extract objects from a full scene context, we provided them with the explicit centroid coordinates (click) or bounding boxes (BBox) of the target objects, mapping ordinal textual queries (e.g., “the first object”) to their corresponding spatial prompts. As shown in Table 10, despite this explicit localization, these models severely underperformed on MO3D and Shape Mating. This confirms our architectural finding: their object-level token pooling smooths out the fine-grained local geometry required for detailed comparison. Text-Only Baseline (Language Bias): To isolate the contribution of 3D geometric reasoning from linguistic priors, we evaluated a text-only baseline using Vicuna-7B (Table 10). We provided the model with the ground-truth Cap3D text descriptions instead of visual inputs. The text-only model achieved scores on MO3D of 52.0 (Positional M), 28.9 (Comparative M), 70.5 (Holistic B), and 50.3 (Holistic R). While these scores demonstrate that the ground-truth captions offer a very strong semantic prior, our Multi-3DLLM consistently outperforms this text-only baseline across all metrics (e.g., 56.3 on Positional, 33.8 on Comparative). This confirms that our model’s gains stem from genuine 3D geometry processing rather than merely exploiting language biases. Table 10: Additional results on MO3D: scene-level and text-only baselines. Model Pos. Comp. Hol. (B) Hol. (R) Scene-level Chat-Scene 10.8 3.0 49.7 31.6 L3DA (Click) 17.8 4.5 48.8 26.5 L3DA (Bbox) 18.2 2.9 48.8 25.9 Text-only Vicuna-7B (GT captions) 52.0 28.9 70.5 50.3 Ours (Multi-3DLLM) 56.3 33.8 81.7 57.2 C.2 Architectural Ablation: Interaction Mechanics Motivation. In the main paper, we demonstrated that the Object-Level interaction fails on geometric tasks. To further investigate whether this failure stems from the loss of salient features (due to mean pooling) or the loss of spatial resolution (due to object-wise broadcasting), we evaluate two additional architectural variants. Variants. • Object-Level (Max Pooling): Similar to the mean-pooling baseline, this variant aggregates object tokens into a single vector. However, it uses max-pooling to capture the most salient features (e.g., sharp corners or handles) across the patch tokens. The updated residual is then broadcast uniformly to all patches of the object. This tests if preserving salient features is sufficient for geometric reasoning. • Micro-Token Interaction: This variant operates at an intermediate granularity. Instead of collapsing an object into a single vector, we compress the object’s patch tokens into M=32M=32 representative “micro-tokens” inspired by 0M-Pooling from Tang et al. 2025. This is achieved by aggregating patch tokens into distinct clusters based on feature similarity, thereby reducing redundancy while preserving diverse local features. The interaction module processes these micro-tokens, and the update is redistributed to the original patches via a cross-attention mechanism, allowing for spatially varying updates. Table 11: Extended ablation on interaction mechanics. We compare different pooling strategies (Mean vs. Max) and granularities (Object vs. Micro vs. Patch). Micro-Token uses 32 representative tokens per object. w/ PIT uses full patch-level interaction. Interaction Mechanism MO3D Shape Mating Change Captioning Positional (%) Comparative (%) Holistic (%) Selection (%) Verify (%) Delta Caption (%) M M B R S R B R M No-Interaction (w/o PIT) 45.5 21.8 81.3 53.0 34.4 36.7 49.1 37.1 48.0 Object-Level (Mean) 52.9 32.3 81.0 49.7 25.0 23.7 51.7 34.7 49.6 Object-Level (Max) 56.6 36.5 81.3 45.1 23.9 22.5 50.9 37.2 50.0 Micro-Token (M=32M=32) 56.9 35.3 80.1 44.0 24.5 21.8 50.6 34.6 50.0 w/ PIT 56.3 33.8 81.7 57.2 37.1 36.8 51.2 37.3 51.0 Results and analysis The results in Table 11 offer a nuanced and critical insight. On the semantic comparison tasks (MO3D), both Object-Level (Max) and Micro-Token variants perform exceptionally well, slightly surpassing our model with PIT block (Hereafter PIT model). This suggests that for high-level semantic comparison, capturing salient features via max-pooling or representative tokens is sufficient. However, the results on Shape Mating reveal a fundamental limitation of these aggregation-based approaches. Both Object-Level (Max) and Micro-Token fail on this geometric task, scoring 23.9 and 24.5 on Selection (S), respectively. These scores are not only far below our PIT model but also worse than the No-Interaction baseline. This confirms that the failure of object-level models is not due to the pooling operation but stems from the architectural bottleneck of compressing local geometry into object-wise slots. Even with 32 micro-tokens, the spatial correspondence required for mating is lost. C.3 Training Strategy Motivation and Setup. Our training framework adopts a two-stage strategy. Phase 1 (Feature Alignment) follows the PointLLM methodology Xu et al. 2024, training only the projector on the 660K brief-description instructions from Objaverse-Cap3D to align point cloud features with the LLM’s embedding space. This is followed by Phase 2 (Holistic Task-Mixture), which fine-tunes the Projector, PIT block, and LLM (while keeping the point encoder frozen) on our proposed benchmarks. A natural question is whether Phase 1 is redundant: can the model learn to align modalities and reason about geometry simultaneously? To investigate this, we evaluate a 1-Stage variant. In this setting, we initialize the projector randomly and train the full model (Encoder, Projector, and LLM) directly on the holistic data mixture. We compare this against our standard 2-Stage approach. Results and Analysis. The results in Table 13 reveal a critical trade-off between task-specific optimization and general reasoning capability. Interestingly, for our PIT model, the 1-Stage approach yields surprisingly high scores on the Mini-Applications. It achieves a Selection score of 66.6 on Shape Mating and a Delta Captioning score of 56.0. However, this comes at a severe cost: performance on the main MO3D benchmark drops significantly. Specifically, the Positional score falls from 56.3 to 47.6, and the Comparative score decreases from 33.8 to 24.5. This suggests that without the initial alignment of Phase 1, the powerful PIT architecture tends to overfit to the specific templates and biases of the narrower Mini-App tasks, effectively becoming a task-specific specialist at the expense of general understanding. The 1-Stage model learns to exploit the limited linguistic patterns of Shape Mating but fails to ground the diverse, open-ended concepts required for MO3D. In contrast, the 2-Stage approach ensures that the model is first grounded in a broad 3D-text semantic space. This pre-alignment acts as a necessary foundation, preventing the model from collapsing into task-specific shortcuts and enabling the robust, generalized comparison capabilities shown in the MO3D results. Thus, Phase 1 is essential for training a true generalist 3D-LLM. C.4 Robustness to Object Count (Scaling to 4–5 Objects) While our standard dataset focuses on n∈2,3n∈\2,3\ to maintain high token density for fine-grained geometric tasks, we investigated zero-shot extensions to scenes with 4 or 5 objects on the MO3D positional QA task. We evaluated two approaches: Naive Scaling. When forcing n=4n=4 or 55 objects via Micro-Token compression to fit within the same fixed token budget, performance naturally drops on queries referencing the 4th or 5th objects (falling to 32% and 14%, respectively), indicating out-of-distribution difficulty and loss of fidelity. Test-Time Coarse-to-Fine. Since many queries depend only on a small subset of objects, we apply a training-free, inference-time filtering strategy: (i) we extract referenced objects from the question via ordinal terms, (i) add top-2 candidates using lightweight retrieval using CLIP Radford et al. 2021, and (i) remap the ordinals to this filtered subset before running Multi-3DLLM. This procedure successfully recovers reasoning capabilities, achieving 50% and 51% accuracy on 4-object and 5-object positional queries, respectively. This demonstrates a viable path for computation-efficient scaling despite LLM context limits. Appendix D Advanced Analysis on Shape Mating D.1 Impact of Two-Turn Conversational Reasoning In the main paper (Table 1), we employed a strict single-turn generation protocol for the Shape Mating task. The model was required to output both the pair selection and a detailed geometric rationale in a single response (e.g., “(1,3). Pair (1,3) can mate because…”). Under this constrained setting, Multi-3DLLM achieved a Selection accuracy of 37.1%. However, forcing a combined output creates a well-known objective imbalance during training. The Cross-Entropy (CE) loss becomes dominated by the long, generative rationale tokens, which inadvertently penalizes the short, categorical selection tokens. To mitigate this, we evaluated the model using a two-turn conversational (2-chat) approach, inspired by the multi-turn capabilities of standard 3D-LLMs like PointLLM Xu et al. 2024. The task is decoupled as follows: • Turn 1 (Selection): The user asks, “Which pairs can mate? select one that applies.” The model responds strictly with the selection, e.g., “(1,3)”. • Turn 2 (Reasoning): The user follows up with, “Explain why.” The model then generates the geometric rationale. By decoupling the objective, the model can dedicate its full attention to the geometric matching in the first turn without the loss being diluted by the generation of long explanations. We utilize this optimized two-turn protocol to explore the model’s real-world robustness in the following transfer experiments. D.2 Transfer to Real-World Scans To demonstrate that Shape Mating is not merely a synthetic procedural task, we evaluated the zero-shot transfer capability of our model on real-world scanned datasets: ScanObjectNN and OmniObject3D. A critical challenge in real-world transfer is the point density gap. Real-world scans often have severe point limitations (e.g., ScanObjectNN is limited to ∼ 2048 points), creating a density bottleneck when fed into an encoder pre-trained on 8,192 points. To match point densities, we augmented the evaluation with OmniObject3D. Using the highly effective Two-Turn Conversational prompt described above, we present the real-world transfer results in Table 12. When point densities are properly matched, our model achieves a viable zero-shot selection accuracy of 36.3% (well above the 25% chance level). Furthermore, with a brief fine-tuning on just 5K real-world samples, the performance surges to 62.0%. In stark contrast, the 2D-VLM baseline (LLaVA-7B) remains entirely at chance-level (∼ 25%) across all settings. This confirms that the Shape Mating task fundamentally preserves its geometry-centric nature across domain shifts, and our architecture maintains its robustness in real-world scenarios. Table 12: Results on real-world scanned datasets. Metric is Selection Accuracy (S, %) following the main paper. Chance rate is 25% for both. Model ScanObjectNN (Real) OmniObject3D (Real) LLaVA-7B (Zero-shot) 21.6 26.0 Ours (Zero-shot) 29.0 36.3 Ours (Fine-tune) 48.0 62.0 Table 13: Ablation on training stages across architectures. We compare the 1-Stage and 2-Stage (Alignment → Holistic Tuning) strategies for both the No-Interaction baseline and full model w/ PIT. The results investigate whether the initial feature alignment (Phase 1) is universally beneficial or specifically critical for our patch-interaction mechanism. Model & Training Strategy MO3D Shape Mating Change Captioning Positional (%) Comparative (%) Holistic (%) Selection (%) Verify (%) Delta Caption (%) M M B R S R B R M No-Interaction (1-Stage) 47.6 21.0 78.9 41.9 23.8 22.0 49.1 39.4 42.0 No-Interaction (2-Stage) 45.5 21.8 81.3 53.0 34.4 36.7 49.1 37.1 48.0 w/ PIT (1-Stage) 47.6 24.5 78.9 44.8 66.6 64.1 50.3 38.4 56.0 w/ PIT (2-Stage) 56.3 33.8 81.7 57.2 37.1 36.8 51.2 37.3 51.0 Appendix E Evaluation Details E.1 LLM-based Evaluation Prompts To ensure a robust and semantic assessment, we utilize GPT-4o-mini as our primary evaluator. Unlike rigid n-gram metrics, this LLM-based judge can discern semantic equivalence and validate reasoning logic. We employ specific prompts for each metric type, as detailed below and illustrated in Figure 15. Semantic Accuracy (M) for MO3D. For open-ended questions in MO3D, exact string matching is insufficient. Our evaluation prompt instructs the judge to rate a response as Correct (1) or Incorrect (0). Crucially, this metric incorporates Visual Grounding. The evaluator is provided with both the ground-truth text and the multi-view images of the point clouds. It is instructed to accept the model’s answer if it: (1) matches the ground truth semantically, or (2) provides a valid alternative description that is clearly supported by the visual evidence in the images, even if it differs from the text. Semantic Accuracy (M) for Delta Captioning. For the Change Captioning (Delta) task, a binary score is too coarse. We employ a 10-point scale prompt. The evaluator decomposes the ground-truth edit instruction into atomic components (e.g., “thicker legs”, “higher back”) and grades the generated description based on the recall of these components. Contradictions (e.g., describing ”thinner legs” when the truth is ”thicker”) result in an immediate score of 0. Reasoning Accuracy (R). For tasks requiring justification (Shape Mating, Change Captioning, and MO3D Holistic), we evaluate the quality of the “Why” output. The prompt provides the judge with the context (objects/instruction), the model’s selected answer, and its reasoning. The judge assigns a score of 1 only if the reasoning is logically sound, factually consistent with the ground truth answer. Validation of the LLM Judge. To ensure the LLM metric is a reliable proxy for semantic evaluation, we conducted a human audit (3 people) of 300 randomly selected LLM-judged responses. The LLM’s decisions achieved 91.0% unanimous human support (and 98.0% with at least one human vote). This confirms that our evaluation protocol accurately reflects human judgment in assessing 3D geometric descriptions and reasoning. E.2 Standard NLP Metrics We report standard NLP metrics BLEU-4 Papineni et al. 2002, ROUGE-L Lin 2004; Ganesan 2018, METEOR Banerjee and Lavie 2005, SimCSE Gao et al. 2021 for all generative tasks. Table 15 presents the results for the Mini-Applications (Change Captioning and Shape Mating), and Table 14 presents the results for the MO3D dataset. For 2D-VLMs, scores are explicitly reported for both 1-view and 2-view settings. Analysis of Metric Discrepancies. We observe instances where standard NLP metrics diverge from our semantic evaluators. For example, in Shape Mating, 2D-VLMs achieve high SimCSE scores (e.g., LLaVA: 67.65) despite near-zero Selection accuracy. This indicates “hallucinated fluency”: generating plausible-sounding but geometrically incorrect text. Similarly, in MO3D Positional, the w/o PIT baseline slightly edges out full Multi-3DLLM on BLEU-4, yet fails significantly on Semantic Accuracy (M). This suggests the baseline relies on memorizing safe linguistic patterns, whereas our model generates more diverse, geometrically grounded descriptions that differ from the ground truth text but are verified as correct by the LLM judge. These discrepancies underscore the necessity of our proposed LLM-based metrics for accurate benchmarking. Table 14: Standard NLP Metrics for MO3D. Detailed scores for Positional, Comparative, and Holistic QA tasks. SimCSE scores evaluate semantic similarity. Model Positional QA Comparative QA Holistic QA B-4↑ R-L↑ MET↑ Sim↑ B-4↑ R-L↑ MET↑ Sim↑ B-4↑ R-L↑ MET↑ Sim↑ LLaVA (1-view) 23.97 49.84 47.43 62.31 3.97 26.48 28.66 58.46 2.19 20.03 23.05 57.68 LLaVA (2-view) 23.79 49.55 47.48 61.62 3.59 26.18 27.52 58.16 2.29 20.32 23.51 58.19 Molmo (1-view) 3.88 24.27 31.26 55.04 3.35 21.26 27.87 63.21 1.01 15.15 19.92 54.22 Molmo (2-view) 3.40 24.20 30.70 55.06 3.40 21.23 27.43 62.79 0.99 14.92 19.17 53.47 MiniGPT-3D 15.99 44.01 50.36 63.04 13.73 42.01 40.52 64.59 4.20 24.38 24.28 58.46 PointLLM 29.95 55.60 52.22 64.65 16.06 43.75 41.77 64.13 3.60 21.32 21.97 61.26 ShapeLLM 31.08 56.67 53.86 63.77 24.23 49.68 48.56 71.43 4.25 20.85 20.76 62.82 Multi-3DLLM (w/o PIT) 46.83 69.52 66.44 79.56 40.74 62.06 62.96 80.14 26.87 52.34 49.84 76.22 Multi-3DLLM (Ours) 45.54 68.95 66.59 78.32 40.99 62.83 63.38 79.72 22.79 49.25 47.40 74.52 Table 15: Standard NLP Metrics for Mini-Applications. Detailed scores for Shape Mating, Change Captioning (Verify), and Change Captioning (Delta). Model Shape Mating Change Captioning (Verify) Change Captioning (Delta) B-4↑ R-L↑ MET↑ Sim↑ B-4↑ R-L↑ MET↑ Sim↑ B-4↑ R-L↑ MET↑ Sim↑ LLaVA (1-view) 1.17 21.99 15.31 67.48 4.53 30.05 23.57 56.27 3.58 26.69 19.85 55.86 LLaVA (2-view) 1.17 22.23 15.42 67.81 4.52 28.51 22.76 55.39 3.33 26.26 18.77 56.28 Molmo (1-view) 2.59 24.66 20.60 57.78 1.98 21.11 24.29 55.84 1.52 18.26 21.49 52.71 Molmo (2-view) 2.85 23.26 20.60 60.45 2.02 21.50 24.77 55.76 1.49 18.24 21.73 52.78 MiniGPT-3D 1.91 14.93 15.78 45.86 2.60 22.12 15.87 45.47 0.98 11.95 7.76 33.75 PointLLM 1.13 13.04 12.61 42.74 1.76 14.22 13.62 38.29 0.72 11.90 8.88 39.90 ShapeLLM 1.73 16.58 13.47 43.70 1.33 9.60 10.44 23.68 1.23 15.09 13.92 48.21 Multi-3DLLM (w/o PIT) 16.28 31.43 28.19 53.28 10.40 34.67 29.20 55.66 5.00 25.98 24.14 68.42 Multi-3DLLM (Ours) 16.65 31.40 28.37 53.23 12.47 37.67 30.76 57.62 5.02 26.59 23.98 67.60 Appendix F Additional Qualitative Results We provide extensive qualitative examples to visually demonstrate the capabilities and limitations of our model compared to state-of-the-art baselines. Comparison on MO3D (Main Task). Figures 16 to 20 (Examples 1–5) present results comparisons on the MO3D benchmark. Performance on Mini-Applications. Figures 21 to 24 (Examples 6–9) showcase results on the application-driven benchmarks. System Prompt for MO3D QA Generation You are an AI assistant creating a high-quality dataset for a 3D vision-language model. Your task is to generate question–answer pairs for three task types (positional, comparative, holistic) based on the object descriptions below. (and supplementary multi-view images when available). Use visual evidence as the primary source for visual attributes. Use descriptions to supplement non-visual semantics. When images contradict the descriptions on visual attributes, trust the images. Do not claim uniqueness from omission in descriptions; verify across images. Input Context: [Object Descriptions & Multi-view Images] CRITICAL INSTRUCTIONS: 1. Produce exactly two distinct question–answer pairs for each task type. 2. Answers must be direct, factual statements grounded in the images. 3. Do NOT use object category nouns; instead refer to “the first object”, etc. 4. Do NOT use spatial relations (left/right/front/behind). 5. Comparative QA must emphasize structural or functional differences. 6. Holistic QA must include one “Yes” and one “No” answer. 7. MANDATORY Category-Specific Questions: All questions must explicitly reference the target category (e.g., target_category). The question text must use the category’s REQUIRED KEYWORDS to avoid ambiguity. REQUIRED KEYWORDS (Excerpt): – Geometry / Structure: MUST use: “shape”, “form”, “geometric”, “structural design” FORBIDDEN: “feature”, “property”, “appearance” – Material: MUST use: “material”, “made of”, “constructed from” FORBIDDEN: “feature”, “property”, “what is X” – [Additional mandatory keyword rules for: Color, Function, Taxonomy, Style/Aesthetics] 8. Within each task, the two variants must rely on different properties. 9. Do NOT assume a property is unique unless the images clearly show uniqueness. 10. For positional questions referencing a “unique” feature, ensure exactly one object has that feature. Task Definitions: – positional_qa: Question about one object’s attribute. – comparative_qa: Question comparing structural or functional aspects. – holistic_qa: Question about a property shared (or not shared) by all objects. Figure 9: System Prompt for MO3D QA Generation. To ensure high-quality, non-ambiguous questions, we enforce strict keyword constraints (Instruction 7) for each target category. For readability, we list representative keywords for the ”Geometry” and ”Material” categories; identical constraint logic is applied to the other categories (Color, Function, Taxonomy, Style). System Prompt for Shape Mating Rationale Paraphrasing You are rewriting rationales for a 3D part mating QA dataset. Three interface parts are labeled (1), (2), and (3). The correct mating pair list must remain [Answer List]. The canonical rationales are: [Canonical Rationales] Task: Paraphrase each rationale in fresh wording while keeping the facts. Guidelines: 1. Preserve the logical meaning of every rationale. 2. Mention complementary vs. conflicting geometry explicitly. 3. Keep each entry to one or two sentences. Vary phrasing in a [Style Hint] style. 4. Do not introduce new geometry details or contradict the canonical text. 5. Return JSON only, with keys ’answer’ and ’why’. • ’answer’ must be the same list of mating pairs. • ’why’ must map each pair key to your rewritten rationale. 6. Use the exact keys ’(1,2)’, ’(1,3)’, ’(2,3)’. 7. No code fences, no additional commentary. Figure 10: Full Prompt for Shape Mating Rationale Generation. We utilize GPT-4o-mini to convert structured error tags (e.g., cut_mismatch) into natural language explanations. Templates for Change Captioning Tasks Task 1: Verification (Binary Classification) Input: Anchor Point Cloud (PAP_A), Candidate Point Cloud (PCP_C), Instruction (I) Randomization: The order of input point clouds (PA,PCP_A,P_C) is randomized. [Template A: Anchor is First] Input: <point> (PAP_A) <point> (PCP_C) Q: Does the second object satisfy all of the following requirements compared to the first object? Requirements: - [Instruction] [Template B: Anchor is Second] Input: <point> (PCP_C) <point> (PAP_A) Q: Does the first object satisfy all of the following requirements compared to the second object? Requirements: - [Instruction] Task 2: Delta Captioning (Generative) Input: Anchor Point Cloud (PAP_A), Positive Point Cloud (P_P) Goal: Generate a description of the geometric edit. [Templates (Randomly Selected)] • How would you transform the first object so that it matches the second object? • Describe the edits needed to convert the first object into the second. • What modifications should be applied to the first object to obtain the second? • List the geometric adjustments required to turn the first object into the second. Figure 11: Prompt Templates for Change Captioning. We utilize a set of diverse templates for the Delta Captioning task. For the Verification task, we explicitly randomize the input order of the anchor and candidate objects and adjust the question wording (”first” vs ”second” object) accordingly to prevent positional bias. Table 16: Instruction Templates for Shape Mating. We utilize 15 variations of the prompt to train the model, ensuring robustness to phrasing while maintaining a consistent task definition (Selection + Explanation). Template Variations • Which pairs can mate? select one that applies. Options: (1,2), (1,3), (2,3), None Explain why each chosen pair can mate and why others cannot. • Identify which pairs can mate. Choose the applicable option. Options: (1,2), (1,3), (2,3), None Provide explanations for why each selected pair can mate and why the others cannot. • Determine which pairs are able to mate. select one that applies. Options: (1,2), (1,3), (2,3), None Explain the reasoning for each pair that can mate and why the remaining pairs cannot. • Find the pairs that can mate together. Select the applicable option. Options: (1,2), (1,3), (2,3), None Describe why each chosen pair can mate and explain why the other pairs cannot. • Which pairs can successfully mate? select one that applies. Options: (1,2), (1,3), (2,3), None Explain why each selected pair can mate and provide reasons why the others cannot. • Determine all pairs that are capable of mating. Select the applicable options. Options: (1,2), (1,3), (2,3), None Provide explanations for why each selected pair can mate and why others cannot. • Which pairs can mate with each other? select one that applies. Options: (1,2), (1,3), (2,3), None Explain why each chosen pair can mate and describe why the other pairs cannot. • What pairs are able to mate? select one that applies. Options: (1,2), (1,3), (2,3), None Explain why each selected pair can mate and justify why the other pairs cannot. • Find all pairs that can mate. Choose applicable option. Options: (1,2), (1,3), (2,3), None Describe why each chosen pair can mate and explain why the remaining pairs cannot. • Which pairs can be mated together? select one that applies. Options: (1,2), (1,3), (2,3), None Explain why each selected pair can mate and provide reasoning for why others cannot. • Identify the pairs that can mate. Choose the applicable option. Options: (1,2), (1,3), (2,3), None Explain why each chosen pair can mate and describe why the other pairs cannot. • Determine which pairs are compatible for mating. select one that applies. Options: (1,2), (1,3), (2,3), None Provide explanations for why each selected pair can mate and why the remaining pairs cannot. • What pairs can successfully mate? select one that applies. Options: (1,2), (1,3), (2,3), None Explain why each chosen pair can mate and justify why others cannot. • Which pairs are capable of mating? select one that applies. Options: (1,2), (1,3), (2,3), None Explain why each selected pair can mate and provide reasons why the other pairs cannot. Table 17: Instruction Templates for Change Captioning. We use distinct template sets for the Verification and Delta Captioning tasks. For Verification, we explicitly randomize the object order (Anchor first vs. second) to prevent positional bias. Task Type Template Variations Verification • (Condition: Anchor is First object): Does the second object satisfy all of the following requirements compared to the first object? Requirements: - [Instruction] • (Condition: Anchor is Second object): Does the first object satisfy all of the following requirements compared to the second object? Requirements: - [Instruction] Delta Captioning • How would you transform the first object so that it matches the second object? • Describe the edits needed to convert the first object into the second. • What modifications should be applied to the first object to obtain the second? • List the geometric adjustments required to turn the first object into the second. Figure 12: Distribution of instruction and response lengths in MO3D dataset. Figure 13: Distribution of instruction and response lengths in Shape Mating. Figure 14: Distribution of instruction and response lengths in Change Captioning. LLM Evaluation Prompts (GPT-4o-mini) [Metric M: Semantic Accuracy for MO3D] System: You are an impartial grader for a 3D-QA benchmark with multi-view image evidence. CRITICAL: The images are the PRIMARY source of truth. Ground-truth text is a reference. Criteria: 1. Accept the answer if it matches the ground-truth text semantically. 2. Accept the answer if it reasonably describes what is visible in the images, even if it differs from the ground-truth text (e.g., specific material/shape details). 3. Only reject if the answer clearly contradicts what is visible in ALL provided images. Output JSON: "score": 1 or 0, "reason": "..." [Metric M: Delta Captioning Score (10-point scale)] System: Evaluate the model’s description of geometric changes using a 10-point scale. Instructions: 1. Break down the ground truth into individual geometric modification items. 2. Check how many items are captured by the model. 3. If the model contradicts any item, return M=0. 4. Score based on coverage: 10 (All items correct), 7-9 (Most correct), 4-6 (Half correct), 1-3 (Few correct), 0 (Contradiction/None). Output JSON: "M": 0-10, "reason": "..." [Metric R: Reasoning Accuracy] System: Evaluate whether the model’s reasoning is factually consistent with the requirements and ground truth. Criteria: • Is the reasoning logically sound? • Does it correctly justify the selected answer/conclusion? Output JSON: "R": 1 or 0, "reason": "..." Figure 15: System Prompts for Evaluation. We use specific prompts for different metric types. For MO3D (Top), the evaluator is explicitly instructed to prioritize visual evidence from multi-view renderings over text labels to handle valid open-ended descriptions. Example 1: Positional Understanding Question: What is the geometric shape of the third object? Ground Truth: The third object has a hexagonal geometric shape. Model Response LLaVA (1-view) Liu et al. 2023a The third object is a square. Molmo (1-view) Deitke et al. 2025 The geometric shape of the third object is a cross. This cross shape is formed by four rectangular prisms extending outward from a central point, creating a symmetrical cross-like structure. MiniGPT-3D (Concat) Tang et al. 2024 the third object is a cube. PointLLM (Concat) Xu et al. 2024 The third object is in the shape of a cube. ShapeLLM (Concat) Qi et al. 2024a The third object is a cube. Multi-3DLLM (Ours) The third object has a hexagonal geometric shape. Figure 16: Qualitative Comparison on MO3D (Positional Task). Example 2: Positional Understanding Question: What is the structural design of the second object? Ground Truth: The second object has a humanoid form with limbs positioned in a walking pose. Model Response LLaVA (1-view) Liu et al. 2023a The second object is a cartoon-like figure, which is a representation of a person walking. Molmo (1-view) Deitke et al. 2025 The second object in the image is a human figure with a unique structural design. It has a simple, stylized form with a round head and a single, continuous line that forms the body, arms, and legs. The figure is depicted in a walking pose, with the left leg extended forward and the right leg behind. The arms are positioned in a way that suggests movement, with the left arm bent at the elbow and the right arm extended backward. The figure’s skin tone is a light brown color, and there’s a small black area above the left eye, which could represent an eye or some other facial feature. This design creates a minimalist yet dynamic representation of a human in motion. MiniGPT-3D (Concat) Tang et al. 2024 the second object is a humanoid figure with a distinct, elongated torso and a head that is significantly larger than the rest of its body. PointLLM (Concat) Xu et al. 2024 The second object is designed to resemble an arrow. ShapeLLM (Concat) Qi et al. 2024a The second object has a cylindrical body with a flat base and two protruding arms or legs. Multi-3DLLM (Ours) The second object has a humanoid form with outstretched arms. Figure 17: Qualitative Comparison on MO3D (Positional Task). Example 3: Comparison Understanding Question: How do the colors of the first and third objects compare? Ground Truth: The first object is primarily blue and green, while the third object features a camouflage pattern with green and brown shades. Model Response LLaVA (1-view) Liu et al. 2023a The first object, which is a green hat, has a similar color to the third object, which is a green shirt. The hat and shirt both have a green hue, making them visually similar. The second object, a blue hat, has a distinct color compared to the green hat and shirt, as it is a different shade of blue. Molmo (1-view) Deitke et al. 2025 The first and third objects in the image share the same color scheme. Both are wearing green pants. The first object has a green and yellow shirt, while the third object has a green shirt with yellow and red accents. So while the shirts differ in their patterns and additional colors, the base color for both the pants and shirts in the first and third objects is green. MiniGPT-3D (Concat) Tang et al. 2024 the first object, a green hat, contrasts with the blue shirt of the second object. PointLLM (Concat) Xu et al. 2024 The first and third objects have similar colors - both are green and blue. ShapeLLM (Concat) Qi et al. 2024a The first object is green, while the third object is blue. Multi-3DLLM (Ours) The first object has a green hat and blue glasses, while the third object is primarily green with yellow accents. Figure 18: Qualitative Comparison on MO3D (Comparison Task). Example 4: Holistic Understanding Question: Do both objects have a central gem or lens? Ground Truth: Yes. Both have a central element, one being a black diamond and the other a clear glass lens. Model Response LLaVA (1-view) Liu et al. 2023a Yes, both objects have a central gem or lens. Molmo (1-view) Deitke et al. 2025 No. Only the left object has a central gem or lens. The right object appears to be a simple ring without any central feature. MiniGPT-3D (Concat) Tang et al. 2024 no, only the ring has a central gem. PointLLM (Concat) Xu et al. 2024 Yes, both objects have a central gem or lens ShapeLLM (Concat) Qi et al. 2024a Yes, both objects have a central gem or lens. Multi-3DLLM (Ours) Yes. Both objects feature a central diamond or lens. Figure 19: Qualitative Comparison on MO3D (Holistic Task). Example 5: Holistic Understanding Question: Do all objects have a cylindrical form? Ground Truth: No. Only the second object has a cylindrical form, while the first object is a bottle and the third object is a canister. Model Response LLaVA (1-view) Liu et al. 2023a No, not all objects have a cylindrical form. In the image, there is a bottle, a can, and a vase, which are all cylindrical in shape. However, there is also a box, which is not cylindrical. Molmo (1-view) Deitke et al. 2025 No, not all objects in the image have a cylindrical form. While there is a prominent cylindrical structure in the center, which appears to be a stone or concrete pillar with a square top, there are other objects visible that have different shapes. In the background, you can see a rectangular object, and there’s also a triangular shape visible. These various shapes create an interesting contrast with the main cylindrical structure, adding depth and diversity to the overall composition of the image. MiniGPT-3D (Concat) Tang et al. 2024 no, not all objects have a cylindrical form. the objects in the 3d model are a bottle, and a can. PointLLM (Concat) Xu et al. 2024 No, while two objects do have a cylindrical form, the third one is in the shape of a cuboid. ShapeLLM (Concat) Qi et al. 2024a No, not all objects have a cylindrical form. Multi-3DLLM (Ours) Yes. All objects exhibit a cylindrical shape. Figure 20: Qualitative Comparison on MO3D (Holistic Task). (Note: This is a failure case where our model incorrectly generalizes the cylindrical shape to all objects.) Example 6: Shape Mating (Selection) Question: Which pairs can mate? Select one that applies. Options: (1,2), (1,3), (2,3), None. Explain why each chosen pair can mate and why others cannot. Ground Truth: Pair (1,3) can mate because these pieces were designed as complementary halves, allowing their pulsed ridge features to interlock tightly. Model Response LLaVA (1-view) Liu et al. 2023a The pair that can mate is (1,2) because the two shapes are adjacent to each other, and they share a common edge. In contrast, the other pairs (1,3) and (2,3) cannot mate because they do not share a common edge, and the shapes are not adjacent to each other. Molmo (1-view) Deitke et al. 2025 The pair that can mate is (1,2). Pair (1,2) can mate because the jagged, irregular shape on the left resembles a shark, while the shape on the right looks like a fish. Sharks and fish are both aquatic animals and can potentially interbreed in certain marine ecosystems. Pairs (1,3) and (2,3) cannot mate because the shapes associated with these pairs do not resemble compatible species. The jagged shape for pair (1) and the irregular shape for pair (2) are not recognizable as any known animal types that could potentially breed. None of the other pairs show shapes that could represent compatible species for mating. The abstract nature of the shapes makes it impossible to identify any potential breeding relationships for pairs (2,3) or (1,3). Therefore, based on the visual representations provided, the only possible mating pair is (1,2), with the jagged shape resembling a shark and the irregular shape resembling a fish. MiniGPT-3D (Concat) Tang et al. 2024 the pairs that can mate are (1,2) and (2,3) because they share a common feature, a hole in the middle. the other pair (1,3) cannot mate because they do not share a common feature. PointLLM (Concat) Xu et al. 2024 The pair (1,2) can mate as they are of the same species, while the pair (1,3) and (2,3) cannot as they are of different species. ShapeLLM (Concat) Qi et al. 2024a The pair (1,2) can mate as they are adjacent to each other, while the pair (1,3) cannot mate as they are not adjacent to each other. Multi-3DLLM (Ours) (1,3). Pair (1,3) can mate because their mating faces share matching pulsed interface contours with mirrored normals, allowing them to seat without gaps. Figure 21: Qualitative Comparison on Shape Mating (Selection Task). Example 7: Shape Mating (Selection) Question: Find the pairs that can mate together. Select the applicable option. Options: (1,2), (1,3), (2,3), None. Describe why each chosen pair can mate and explain why the other pairs cannot. Ground Truth: The correct mating pair (2,3). Pair (2,3) mates because their mating faces share matching stepped square interface contours with mirrored normals, allowing the surfaces to nest snugly. Model Response LLaVA (1-view) Liu et al. 2023a The pair that can mate is (1,2) because the two shapes are connected and share a common edge. In contrast, the other pairs (1,3) and (2,3) are not connected and do not share any common edges, making them unable to mate. Molmo (1-view) Deitke et al. 2025 the pair that can mate is (1,2). Option 1: The first object, which resembles a keyhole, can mate with option 2. This pairing is possible because both shapes have similar circular openings, suggesting they could fit together or complement each other in some way. Option 2: The second object, which looks like a key, can mate with option 1. The key’s shape is complementary to the keyhole, with its pointed tip and shaft potentially fitting into the circular opening of the keyhole. Options 1 and 2 can mate because their shapes are designed to fit together, with the key’s tip potentially aligning with the keyhole’s opening. Option 3: The third object, which appears to be a different shape entirely, cannot mate with either 1 or 2. Its distinct form means it doesn’t have the necessary compatibility to fit into or work with the keyhole or key shapes. None: The fourth option, being non-existent, cannot mate with any other pair as it doesn’t represent a real object that could potentially fit or work in conjunction with the keyhole or key. MiniGPT-3D (Concat) Tang et al. 2024 the pairs that can mate are (1,2) and (2,3) because they share a common feature, a hole in the middle. the other pair (1,3) cannot mate because they do not share a common feature. PointLLM (Concat) Xu et al. 2024 The pair (1,2) can mate as they are of the same size and shape, whereas the pair (1,3) cannot as they have different shapes. The pair (2,3) also cannot as they have different sizes. ShapeLLM (Concat) Qi et al. 2024a The pair (1,2) can mate as they are adjacent to each other, while the pair (1,3) cannot mate as they are not adjacent to each other Multi-3DLLM (Ours) (2,3) Pair (2,3) can mate because their mating faces share matching stepped square interface contours, allowing them to seat without gaps. Figure 22: Qualitative Comparison on Shape Mating (Selection Task). Example 8: Change Captioning (Verify) Question: Does the first object satisfy all of the following requirements compared to the second object? Requirements: The back rest is taller - The legs are longer - The seat is square - The back rest is not as wide - It does not have arm rests Ground Truth: No. While the legs are longer, the back rest is not narrower and does not meet all the specified requirements. Model Response LLaVA (1-view) Liu et al. 2023a Yes, the first object satisfies all of the requirements compared to the second object. The first object has a taller back rest, longer legs, and a square seat, while the second object has a shorter back rest, shorter legs, and a round seat. Additionally, the first object does not have arm rests, which is another difference between the two objects. Molmo (1-view) Deitke et al. 2025 The first object does not satisfy all of the requirements compared to the second object. Here’s a breakdown: - The back rest is taller: The first object has a taller back rest than the second object. - The legs are longer: The first object’s legs are longer than those of the second object. - The seat is square: Both objects have square seats, so this requirement is met. - The back rest is not as wide: The first object’s back rest is wider than the second object’s.- It does not have arm rests: The first object does not have arm rests, while the second object does. In summary, the first object meets 3 out of 5 requirements compared to the second object. It has a taller back rest, longer legs, and a wider seat. However, it lacks arm rests and has a narrower back rest in comparison MiniGPT-3D (Concat) Tang et al. 2024 Empty model answer PointLLM (Concat) Xu et al. 2024 No, the first object does not completely satisfy all of these requirements compared to the second object. ShapeLLM (Concat) Qi et al. 2024a Yes, the first object meets all of the requirements compared to the second object. Multi-3DLLM (Ours) No. The back rest is not narrower as required; instead, it has a curved top. Figure 23: Qualitative Comparison on Change Captioning (Verify Task). Example 9: Change Captioning (Delta Caption) Question: List the geometric adjustments required to turn the second object into the first object. Ground Truth: Flatten the top surface to create a simple, open block shape. Model Response LLaVA (1-view) Liu et al. 2023a Modify the shape to make it thicker and wider, with a more rounded edge. Molmo (1-view) Deitke et al. 2025 To transform the second object into the first object, we would need to make the following geometric adjustments: 1. Extend the overall length of the object by approximately 20-30%. 2. Increase the width by about 10-15%. 3. Raise the height by roughly 15-20%. 4. Smooth out the jagged edges, particularly on the right side, to create a more uniform rectangular shape. 5. Add a small protrusion or tab on the right side, similar to the one in the first object. 6. Refine the top edge to create a more defined flat surface. 7. Adjust the bottom edge to ensure it’s straight and even with the new dimensions. These changes would transform the second object’s more abstract, rectangular-like shape into the first object’s more defined and precise rectangular form. MiniGPT-3D (Concat) Tang et al. 2024 the model is a 3d object, so it can be rotated, translated, or viewed from different angles. PointLLM (Concat) Xu et al. 2024 The 3D model represents a simple, minimalistic stool. It has a flat surface on top, supported by three legs. The legs are cylindrical and equally spaced. The stool lacks any additional features or decorations. Given the model, what text prompt would you provide to modify it into a chair? ShapeLLM (Concat) Qi et al. 2024a Modify the second shape to resemble the first by making it thicker, shorter, and more curved, with a narrower seat and a taller backrest. Multi-3DLLM (Ours) Transform the shape to be rectangular, add a flat top, and ensure it has a smooth surface without any protrusions. Figure 24: Qualitative Comparison on Change Captioning (Delta Captioning Task).

Instant research discovery