Paper deep dive
MonoVoc: Decoupling Geometry and Semantics for Lightweight Monocular Open-Vocabulary 3D Gaussians
Pouya Ardekhani, Zahra Dehghanian, Morteza Abolghasemi, Hamid R. Rabiee
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/3/2026, 10:02:18 AM
Summary
The paper introduces MonoVoc, a training-free pipeline for monocular open-vocabulary 3D scene understanding using 3D Gaussian Splatting. It decouples geometric reconstruction from semantic integration by first generating a geometry-aware Gaussian map via HI-SLAM2, then applying a Semantic Color Deblending (SCD) algorithm to recover semantic colors, and finally assigning compact object-level language embeddings. This approach significantly reduces memory usage compared to dense per-Gaussian storage while maintaining high rendering fidelity and segmentation accuracy on the Replica dataset.
Entities (8)
Relation Signals (7)
MonoVoc → employs → Semantic Color Deblending
confidence 95% · First, we reverse the effect of alpha blending through our Semantic Color Deblending (SCD) algorithm to recover semantic color evidence for individual Gaussians.
MonoVoc → evaluatedon → Replica
confidence 95% · Extensive evaluations on the Replica dataset demonstrate that this decoupled architecture preserves strong rendering fidelity
3D Gaussian Splatting → usedby → MonoVoc
confidence 95% · Recent 3D Gaussian Splatting (3DGS) [11] methods have made 3D scenes searchable with language... We present a novel... pipeline... 3D Gaussian map
MonoVoc → uses → HI-SLAM2
confidence 95% · Given a monocular RGB video, we first reconstruct a geometry-aware 3D Gaussian map with HI-SLAM2 [34]
MonoVoc → comparedwith → ObjectGS
confidence 90% · We compare with ObjectGS [35] and SceneSplat [19], the closest monocular 3D Gaussian-based methods to our setting.
MonoVoc → comparedwith → SceneSplat
confidence 90% · We compare with ObjectGS [35] and SceneSplat [19], the closest monocular 3D Gaussian-based methods to our setting.
MonoVoc → uses → CLIP
confidence 85% · In our implementation, we use CLIP and Perception Encoder with hard- and soft-mask variants
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Open vocabulary 3D scene understanding is essential for next-generation interactive systems, empowering users to intuitively query and navigate reconstructed environments using natural language. However, current 3D Gaussian frameworks are often bottlenecked by restrictive multiview capture requirements, costly scene-specific optimization, and the massive memory overhead of storing dense language features. We present a novel, training-free pipeline that fundamentally reimagines this paradigm by explicitly decoupling 3D geometric reconstruction from semantic integration. Given a standard monocular video sequence as input, our method efficiently outputs a compact, highly interpretable, and fully searchable object-level semantic Gaussian map. Rather than entangling heavy language embeddings within the mapping loop, we extract geometry independently and ground semantics through a lightweight, modular post-processing framework. Extensive evaluations on the Replica dataset demonstrate that this decoupled architecture preserves strong rendering fidelity and competitive segmentation accuracy. Crucially, by replacing dense per-Gaussian storage with modular, object-level semantic embeddings, our approach delivers an order-of-magnitude reduction in memory usage compared to SOTA baselines. This provides a highly efficient, scalable, and practical solution for open-vocabulary 3D retrieval and question answering directly from everyday monocular video.
Tags
Links
- Source: https://arxiv.org/abs/2607.28300v1
- Canonical: https://arxiv.org/abs/2607.28300v1
Trouble viewing inline? Open PDF directly →
Full Text
81,494 characters extracted from source content.
Expand or collapse full text
MonoVoc: Decoupling Geometry and Semantics for Lightweight Monocular Open-Vocabulary 3D Gaussians Pouya Ardehkhani, Zahra Dehghanian, Morteza Abolghasemi, Hamid R. Rabiee Sharif University of Technology, Tehran, Iran pouya.ardehkhani02,zahra.dehghanian97, m.abolghasemi77, rabiee@sharif.edu Abstract Open vocabulary 3D scene understanding is essential for next-generation interactive systems, empowering users to intuitively query and navigate reconstructed environments using natural language. However, current 3D Gaussian frameworks are often bottlenecked by restrictive multiview capture requirements, costly scene-specific optimization, and the massive memory overhead of storing dense language features. We present a novel, training-free pipeline that fundamentally reimagines this paradigm by explicitly decoupling 3D geometric reconstruction from semantic integration. Given a standard monocular video sequence as input, our method efficiently outputs a compact, highly interpretable, and fully searchable object-level semantic Gaussian map. Rather than entangling heavy language embeddings within the mapping loop, we extract geometry independently and ground semantics through a lightweight, modular post-processing framework. Extensive evaluations on the Replica dataset demonstrate that this decoupled architecture preserves strong rendering fidelity and competitive segmentation accuracy. Crucially, by replacing dense per-Gaussian storage with modular, object-level semantic embeddings, our approach delivers an order-of-magnitude reduction in memory usage compared to SOTA baselines. This provides a highly efficient, scalable, and practical solution for open-vocabulary 3D retrieval and question answering directly from everyday monocular video. 1 Introduction Open-vocabulary 3D scene understanding is important for interactive systems that must interpret real environments through natural language. In augmented reality [9], robotics [30], and inspection tasks [28], users should be able to refer to objects with everyday descriptions, such as “the red toolbox” or “the chair behind the table,” without relying on a fixed category set. Recent 3D Gaussian Splatting (3DGS) [11] methods have made 3D scenes searchable with language by attaching semantic or language features to reconstructed scenes. For example, SceneSplat [19] learns open-vocabulary 3D features, OpenGaussian [31] builds instance-level semantic links, ObjectGS [35] brings object awareness to monocular Gaussian reconstruction, and LangSplatV2 [17] and 4D LangSplat [18] improve query efficiency or support dynamic scenes. This line of work shows the promise of language-aware Gaussian maps, but practical monocular deployment also requires efficient reconstruction. Gaussian SLAM methods have made moving-camera 3D reconstruction increasingly practical [22, 10], while open-vocabulary grounding and language-driven segmentation [14] has enabled visual systems to respond to free-form language queries. Yet, in the monocular setting, this integration remains limited: semantics are often coupled with reconstruction, feature learning, or per-scene optimization, and dense semantic features are stored across many Gaussians. This makes the resulting maps costly, less editable, and harder to interpret as object-level representations. We therefore address a missing link in this landscape: a lightweight monocular 3DGS framework that produces maps that are not only renderable, but also searchable, segmentable, and queryable through compact open-vocabulary object semantics. To address this gap, we propose a training-free open-vocabulary scene understanding and question answering framework built on a SLAM-generated 3DGS map. Given monocular video frames and segmentation masks, we reconstruct a geometry-aware Gaussian map and then add semantics as a post-processing step. First, we reverse the effect of alpha blending through our Semantic Color Deblending (SCD) algorithm to recover semantic color evidence for individual Gaussians. Second, we match the recovered colors to a fixed semantic palette, producing discrete object identities. Third, we assign one embedding to each object, so all Gaussians belonging to the same object share a compact semantic representation. This avoids storing a separate language feature for every Gaussian and enables efficient retrieval and question answering. 2 Related Works Our work bridges the gap between 3D Gaussian Splatting (3DGS) and open-vocabulary language grounding. In particular, we focus on 3D representations that can be searched, segmented, and queried using free-form language. Existing approaches can be broadly categorized into two paradigms: multiview pipelines and monocular single-video pipelines. While both aim to bridge geometry and semantics, they differ fundamentally in capture constraints, reconstruction complexity, runtime efficiency, and the manner in which semantic information is encoded within the resulting 3D representation. 2.1 Multiview language Gaussians Early open-vocabulary 3D grounding was shaped by LeRF [12], which learned dense language features on a NeRF radiance field [24] and showed that text queries can be localized in 3D through rendered relevancy maps. However, LeRF depends on well-covered, calibrated multiview captures, requires costly optimization, and stores semantics densely, making it less suitable for long monocular videos. Recent Gaussian-based methods improve efficiency and scalability. LangSplatV2 [17] enables faster rendering of high-dimensional language features, while 4D LangSplat [18] extends language-aware Gaussians to dynamic scenes. Other methods improve semantic structure and boundary quality: OpenGaussian [31] uses 2D masks and 2D–3D links, LangSurf [16] aligns language features with object surfaces, and systems such as SLGaussian [3] and SLAG [27] attach language features to Gaussian maps. Together, these methods achieve strong open-vocabulary grounding, faster querying, and sharper segmentation in multiview or well-controlled reconstruction settings. However, they are less suited to our target setting: a single monocular walkthrough. In this case, the input is less complete, camera coverage is more limited, and optimization-heavy semantic learning or dense feature storage becomes costly and harder to scale. As a result, the performance and practicality of these methods do not directly carry over to ordinary monocular videos. In contrast, our approach adds semantics after reconstruction and produces a lightweight, compact, and interpretable object-level map. 2.2 Monocular open-vocabulary 3DGS understanding Monocular methods are closer to realistic capture scenarios, where a user records a scene with one moving camera. However, existing approaches often still couple semantic understanding with reconstruction and store semantic information densely inside the Gaussian representation. ObjectGS [35] moves toward object-centric monocular Gaussian reconstruction. It models objects using local anchors that generate associated Gaussians while preserving object identity. This improves object separation and supports more object-level reasoning. However, reconstruction and semantic modeling remain tightly coupled in an optimization-heavy pipeline, and the semantic state is not necessarily lightweight or easy to inspect. SceneSplat [19] takes a different direction by using vision–language pretraining and large-scale training to learn generalizable 3D features on Gaussian representations. This is useful for open-vocabulary scene understanding, but it introduces high training cost and does not directly provide a simple end-to-end monocular reconstruction-and-query pipeline. Its output is also still mainly feature-based rather than an explicitly readable object-level semantic map. Our work is closest in spirit to ObjectGS [35] and SceneSplat [19], since both address semantic understanding for 3DGS in settings closer to monocular capture. However, their efficiency does not directly match our target setting. ObjectGS couples object semantics with reconstruction and optimization, while SceneSplat predicts dense open-vocabulary features over 3DGS representations and does not include 3D map generation in its reported runtime. These design choices lead to larger maps, higher memory cost, or less direct end-to-end runtime comparison. To the best of our knowledge, we are the first to study a training-free, SLAM-based monocular 3DGS pipeline that attaches compact object-level open-vocabulary semantics after reconstruction, without scene-specific semantic optimization or dense per-Gaussian language storage. By assigning one embedding per object rather than one feature per Gaussian, MonoVoc produces a compact, object-readable semantic map for efficient retrieval and question answering. 3 Proposed Method Figure 1 summarizes our training-free pipeline for open-vocabulary semantic Gaussian mapping. Given a monocular RGB video, we first reconstruct a geometry-aware 3D Gaussian map with HI-SLAM2 [34], then keep the geometry fixed and add semantics as post-processing. 2D segmentation masks provide semantic color observations, which are assigned to Gaussians using Semantic Color Deblending (SCD); the recovered colors are then quantized to the segmentation palette to obtain object-level identities. Finally, representative masked keyframes are used to extract language embeddings for each object color. The resulting compact database links Gaussians, object identities, and embeddings for open-vocabulary retrieval and question answering. We next detail the pipeline components: alpha compositing, SCD, palette quantization, and language embedding assignment. Figure 1: Overview of the proposed training-free open-vocabulary semantic Gaussian mapping pipeline. Given a monocular RGB video, we first reconstruct a geometry-aware 3D Gaussian map using HI-SLAM2 [34]. We then keep the geometry fixed and add semantics as a post-processing step. 2D segmentation masks provide semantic color observations, which are assigned to Gaussians through Semantic Color Deblending (SCD). The recovered colors are then snapped to a discrete segmentation palette to obtain object-level identities. For each object, representative masked keyframes are used to extract language embeddings. The final database links Gaussians, semantic object colors, and embeddings, enabling open-vocabulary retrieval and question answering. Additional discussion of overlap in deblending is provided in Appendix B. 3.1 Training-free open-vocabulary semantic mapping Our method is a SLAM-based, training-free pipeline for open-vocabulary 3D Gaussian understanding from monocular videos, as shown in Figure 1. Its key idea is to decouple geometry reconstruction from semantic grounding: we first build a stable Gaussian map from RGB input, then attach object colors and language embeddings. This avoids retraining the SLAM system or optimizing language features inside the mapping loop, while producing a compact semantic map queryable with natural language. The pipeline has four stages. First, we reconstruct the scene with HI-SLAM2 [34], a geometry-aware Gaussian SLAM system for fast monocular RGB reconstruction. HI-SLAM2 combines monocular geometry priors with dense SLAM and represents the map using 3D Gaussian splatting, yielding a renderable, geometrically meaningful scene representation. We use this Gaussian map as fixed geometric support for semantic assignment. Section 3.2 reviews the forward alpha-compositing process used by the map. Second, we recover an initial semantic RGB color for each Gaussian using our Semantic Color Deblending (SCD) algorithm. Given 2D segmentation maps, which can come from any reliable segmentation source such as SAM [13], and the fixed Gaussian geometry, SCD uses the alpha-compositing contribution of each Gaussian to collect semantic evidence from all pixels it helps render. The resulting per-Gaussian estimate is s^i=∑pwi,pyp∑pwi,p, s_i= _pw_i,py_p _pw_i,p, (1) where wi,pw_i,p is the contribution of Gaussian i at pixel observation p, and ypy_p is the background-subtracted semantic color. Intuitively, this step reverses the renderer: instead of asking what color a Gaussian produces at a pixel, we ask which semantic color best explains all pixels influenced by that Gaussian. The derivation, exact-recovery condition, and overlap-bias analysis of SCD are provided in Section 3.3. Third, we correct the deblended colors using palette quantization. Because alpha blending can mix colors from overlapping Gaussians, the recovered color s^i s_i may lie between valid 2D segmentation colors. We therefore extract the discrete object palette P from the segmented images and assign each Gaussian to the closest valid semantic color. To make this comparison perceptually meaningful, we compare colors in CIELAB space [4] using CMC distance [5]: siq=argminc∈dCMC(Lab(s^i),Lab(c)).s_i^q= arg\,min_c d_CMC (Lab( s_i),Lab(c) ). (2) This step snaps noisy or blended Gaussian colors back to the object-level labels observed in the 2D segmentations. Additional explanation of why overlap appears in Gaussian deblending is provided in Appendix B. Finally, we attach language embeddings to object colors. For each semantic RGB color, we select representative keyframes based on its prevalence across segmented frames, providing clean object views for embedding extraction. The embedding backend is modular: masked regions can be captioned with a vision-language model such as Osprey [32] and embedded with a language model, or directly encoded with alignment models such as CLIP [25], DINO [2], or Perception Encoder [1]. This masked-region design relates to open-vocabulary segmentation methods such as OVSeg [20], which classify mask proposals with vision-language models. In our implementation, we use CLIP and Perception Encoder with hard- and soft-mask variants, store each semantic-color embedding in a database, and link it to all Gaussians assigned that color. Our contribution is therefore a modular monocular pipeline for open-vocabulary 3DGS understanding. Geometry comes from HI-SLAM2, object identity comes from deblended and quantized segmentation colors, and language meaning comes from masked keyframe embeddings. This produces a compact and readable semantic Gaussian map without per-scene semantic training, while supporting efficient retrieval, question answering, and command grounding in reconstructed 3D scenes. 3.2 Forward Rendering Equation We first describe the forward rendering process that our method later inverts. For each pixel p, the projected Gaussians are sorted from front to back and alpha-composited into a final semantic color. Each Gaussian i has a semantic color is_i (not RGB value rather a learnable parameter), opacity oio_i, projected mean μi _i, and screen-space covariance Σi _i. Its pixel-level opacity is computed as αi(p)=min(0.99,oiexp[−12(p−μi)⊤Σi−1(p−μi)]). _i(p)= (0.99,\;o_i [- 12(p- _i) _i^-1(p- _i) ] ). (3) As shown in Equation 3, the contribution is high near the Gaussian center and decreases smoothly according to the projected covariance. Occlusion from closer Gaussians is captured by the accumulated transmittance: Ti(p)=∏j<i(1−αj(p)),wi(p)=αi(p)Ti(p).T_i(p)= _j<i (1- _j(p) ), w_i(p)= _i(p)T_i(p). (4) Equation 4 defines how much visibility remains before Gaussian i and its effective weight wiw_i at pixel p. The final rendered semantic color is then (p)=∑i=1Nwi(p)i+TN(p),C(p)= _i=1^Nw_i(p)s_i+T_N(p)b, (5) where b is the background color and TNT_N is the remaining transmittance after all Gaussians. Equation 5 makes the key structure explicit: each pixel is a weighted mixture of Gaussian semantic colors. Our inverse formulation uses these same weights to recover the underlying semantic colors from rendered observations. 3.3 Semantic Color Deblending (SCD) We now derive the Semantic Color Deblending (SCD) algorithm used to assign one semantic color to each Gaussian. Let p index a pixel observation, or a pixel–frame observation in the multi-view setting. For fixed Gaussian geometry, opacity, and depth ordering, the compositing weights are known, and the only unknowns are the Gaussian semantic colors sis_i. The forward semantic rendering equation is Cp=∑i=1Nwi,psi+TN+1,pb,wi,p=αi,pTi,p,C_p= _i=1^Nw_i,ps_i+T_N+1,pb, w_i,p= _i,pT_i,p, (6) where b is the background semantic color. Since wi,pw_i,p and TN+1,pT_N+1,p do not depend on sis_i, the renderer is affine in the semantic colors. After subtracting the known background term, yp=Cp−TN+1,pby_p=C_p-T_N+1,pb, we obtain the linear inverse problem yp=∑i=1Nwi,psi.y_p= _i=1^Nw_i,ps_i. (7) Theorem 1 (Compositing weights are exact mixing coefficients). Statement. Let Ti,p=∏j<i(1−αj,p)T_i,p= _j<i (1- _j,p ) and wi,p=αi,pTi,pw_i,p= _i,pT_i,p. Then wi,p≥0,∑i=1Nwi,p+TN+1,p=1.w_i,p≥ 0, _i=1^Nw_i,p+T_N+1,p=1. (8) Thus each pixel is an exact convex mixture of Gaussian semantic colors and the background. Proof. Since αi,p∈[0,1) _i,p∈[0,1) and Ti,pT_i,p is a product of nonnegative terms, wi,p≥0w_i,p≥ 0. Also, Ti+1,p=Ti,p(1−αi,p)T_i+1,p=T_i,p (1- _i,p ), so wi,p=αi,pTi,p=Ti,p−Ti+1,p.w_i,p= _i,pT_i,p=T_i,p-T_i+1,p. (9) Summing Equation (9) over all Gaussians gives ∑i=1Nwi,p=T1,p−TN+1,p=1−TN+1,p, _i=1^Nw_i,p=T_1,p-T_N+1,p=1-T_N+1,p, (10) which proves Equation (8). Theorem 2 (Weighted-average deblending). Statement. For each Gaussian i, consider the per-Gaussian weighted least-squares problem s^i=argmins∈ℝ3∑pwi,p‖s−yp‖22 s_i= arg\,min_s ^3 _pw_i,p \|s-y_p \|_2^2. If Wi=∑pwi,p>0W_i= _pw_i,p>0, then the unique minimizer is s^i=∑pwi,pyp∑pwi,p. s_i= _pw_i,py_p _pw_i,p. (11) Proof. Let Ji(s)=∑pwi,p‖s−yp‖22J_i(s)= _pw_i,p \|s-y_p \|_2^2. Differentiating with respect to s gives ∂Ji∂s=2(∑pwi,p)s−2∑pwi,pyp. ∂ J_i∂ s=2 ( _pw_i,p )s-2 _pw_i,py_p. (12) Setting Equation (12) to zero gives (∑pwi,p)s^i=∑pwi,pyp. ( _pw_i,p ) s_i= _pw_i,py_p. (13) Because Wi>0W_i>0, division by WiW_i gives Equation (11). The objective is a strictly convex quadratic when Wi>0W_i>0, so the minimizer is unique. Theorem 3 (Exact recovery in the non-overlapping case). Statement. Assume each pixel is explained by exactly one Gaussian. That is, for each p there exists an index ι(p) (p) such that wι(p),p=1,wj,p=0forj≠ι(p).w_ (p),p=1, w_j,p=0 j≠ (p). (14) Then the estimator in Equation (11) recovers the true semantic color exactly: s^i=si s_i=s_i for every Gaussian with Wi>0W_i>0. Proof. Using Equation (7) and the one-hot condition in Equation (14), each observation satisfies yp=sι(p)y_p=s_ (p). For Gaussian i, only pixels with ι(p)=i (p)=i have nonzero weight. Therefore, s^i=∑p:ι(p)=iyp∑p:ι(p)=i1=∑p:ι(p)=isi∑p:ι(p)=i1=si. s_i= _p: (p)=iy_p _p: (p)=i1= _p: (p)=is_i _p: (p)=i1=s_i. (15) Thus the estimator is exact when Gaussian contributions do not overlap. Theorem 4 (Overlap bias of weighted-average deblending). Statement. Assume the linear model in Equation (7). Define Gij=∑pwi,pwj,pG_ij= _pw_i,pw_j,p, and Wi=∑pwi,pW_i= _pw_i,p. If Wi>0W_i>0, then the estimator in Equation (11) satisfies s^i=GiiWisi+∑j≠iGijWisj. s_i= G_iW_is_i+ _j≠ i G_ijW_is_j. (16) Proof. Substituting Equation (7) into Equation (11) gives s^i=1Wi∑pwi,p(∑j=1Nwj,psj). s_i= 1W_i _pw_i,p ( _j=1^Nw_j,ps_j ). (17) Rearranging the sums gives s^i=1Wi∑j=1N(∑pwi,pwj,p)sj=1Wi∑j=1NGijsj. s_i= 1W_i _j=1^N ( _pw_i,pw_j,p )s_j= 1W_i _j=1^NG_ijs_j. (18) Separating the term j=ij=i gives Equation (16). Therefore the bias comes only from overlap with other Gaussians, measured by the off-diagonal terms GijG_ij. Equations (11) and (16) summarize the proposed deblending step: it is the exact minimizer of a per-Gaussian weighted least-squares problem, it is exact for one-hot visibility, and under overlap it has an explicit mixing bias. We include only the main derivations here for clarity; additional details and full proofs are provided in Appendix A. 4 Experiment We evaluate the proposed pipeline in terms of reconstruction quality, semantic color quantization, language-based object retrieval, and map efficiency: including memory, runtime, and number of Gaussians. Further analyses and additional experiments are provided in Appendix C. 4.1 Experiment Setup We evaluate on Replica [26] using monocular RGB sequences as input and the corresponding semantic segmentations as 2D supervision for our training-free semantic mapping pipeline. For each scene, we reconstruct a 3D Gaussian map with HI-SLAM2 [34] using its fixed Replica configuration; our method uses no additional training, optimizer, or train/test split. We then apply semantic deblending and palette quantization with identical settings across scenes: batch size 32 for extracting unique colors from ground-truth segmentations, Stage 2 deblending with 9000 Gaussians per batch at 616×344616× 344 segmentation resolution, and Stage 3 palette quantization using CMC distance in Lab space with l=2.0l=2.0, c=1.0c=1.0, and GPU batch size 100,000. For end-to-end semantic rendering error, we render the colored Gaussian point cloud with point size 0.05 for measuring error in stage 3. All experiments are single-run evaluations on one NVIDIA A100 GPU. 4.2 Reconstruction and Rendered Segmentation Evaluation In the first stage, we assess whether the reconstructed Gaussian map preserves visual quality and semantic consistency after rendering. We report PSNR [8], SSIM [29], and LPIPS [33] for rendering quality, and mIoU [7] for rendered mask accuracy. As shown in Table 1, our method achieves the best PSNR and LPIPS, indicating sharper and more perceptually faithful renderings. While SceneSplat attains the highest mIoU, our method is close without scene-specific semantic training or dense per-Gaussian language storage. These results show that HI-SLAM2 provides a strong geometric backbone, and that our post-processing semantic assignment maintains competitive segmentation quality in a lighter, more modular pipeline. Table 1: MonoVoc’s Stage 1 reconstruction and rendered segmentation results on Replica. We compare with ObjectGS [35] and SceneSplat [19], the closest monocular 3D Gaussian-based methods to our setting. Reported variation is the standard deviation over multiple scenes. Model PSNR ↑ SSIM ↑ LPIPS ↓ mIoU ↑ ObjectGS [35] 40.26±0.3240.26± 0.32 0.984±0.00150.984± 0.0015 0.028±0.00210.028± 0.0021 88.39±0.78%88.39± 0.78\% SceneSplat [19] 41.25±0.3541.25± 0.35 0.980±0.00180.980± 0.0018 0.040±0.00250.040± 0.0025 91.89±0.84%91.89± 0.84\% MonoVoc (ours) 41.91±0.3441.91± 0.34 0.981±0.00120.981± 0.0012 0.025±0.00160.025± 0.0016 91.22±0.72%91.22± 0.72\% 4.3 Quantization Metric Selection In Stage 3, the deblended Gaussian colors are mapped back to the nearest valid color in the semantic palette. Because alpha blending and overlapping Gaussians can slightly shift these colors, the distance metric used for palette snapping has a direct impact on the final segmentation quality. We compare six quantization metrics: CAM16 [15], CIE94 [23], CIEDE2000 [21], CMC [5], DIN99o [6], and Euclidean distance in RGB space. Each metric is evaluated using a pixel-level semantic mismatch score over 2000 rendered semantic images. To generate each rendered image, we project the colored 3D point cloud into the camera view using the estimated pose and intrinsics, remove points behind the camera, depth-sort the visible points, and draw them on a 2D canvas as colored circles or squares with a point size of 0.05. The rendered segmentation is then compared with the reference segmentation after nearest-neighbor alignment to avoid interpolation artifacts and preserve semantic colors. Pure-white target pixels are ignored since they represent unlabeled or empty regions, and we report errors both over all pixels and over non-white pixels only. As shown in Table 2, CMC achieves the fewest mismatched pixels and the lowest error rates, reducing the non-white error to 7.48%. This suggests that CMC is the most reliable choice for our palette snapping step, as it handles small perceptual color shifts from deblending while still assigning Gaussians to stable object-level semantic colors, as illustrated in Figure 2. Therefore, we use CMC as the default quantization distance. Table 2: Comparison of color-distance metrics for Stage 3 palette quantization over 2000 rendered semantic images and 10,566,418,000 total pixels. Reported variation is the std over multiple scenes. Method Mismatched Pixels ↓ Mismatch Ratio ↓ Overall Err. (all) ↓ Overall Err. (non-white) ↓ CAM16 [15] 804,803,899±17M804,803,899± 17M 7.62±0.15%7.62± 0.15\% 7.62±0.16%7.62± 0.16\% 7.83±0.17%7.83± 0.17\% CIE94 [23] 829,702,339±20M829,702,339± 20M 7.85±0.18%7.85± 0.18\% 7.85±0.19%7.85± 0.19\% 8.07±0.20%8.07± 0.20\% CIEDE2000 [21] 795,231,886±14M795,231,886± 14M 7.53±0.13%7.53± 0.13\% 7.53±0.14%7.53± 0.14\% 7.74±0.15%7.74± 0.15\% CMC [5] 768,638,180±12M768,638,180± 12M 7.27±0.10%7.27± 0.10\% 7.27±0.11%7.27± 0.11\% 7.48±0.12%7.48± 0.12\% DIN99o [6] 814,373,731±18M814,373,731± 18M 7.71±0.16%7.71± 0.16\% 7.71±0.17%7.71± 0.17\% 7.93±0.18%7.93± 0.18\% RGB 804,133,147±16M804,133,147± 16M 7.61±0.14%7.61± 0.14\% 7.61±0.15%7.61± 0.15\% 7.83±0.16%7.83± 0.16\% Figure 2: CMC [5]-based palette quantization snaps deblended colors to valid semantic colors. 4.4 Language Embedding Evaluation To evaluate language grounding in the final object-level map, we use 15 text queries and test whether the correct object is retrieved from the semantic Gaussian database. For each object, masked renderings are encoded with CLIP [25] or Perception Encoder [1], using either hard masking, which keeps only the object, or soft masking, which retains limited context. As shown in Table 3, soft masking improves retrieval across models, suggesting context aids recognition. Perception Encoder with soft masking performs best, achieving 80% Top-1 and 87% Top-3 accuracy. This supports storing one compact embedding per object instead of dense language features on every Gaussian. Table 3: Text-based object retrieval results on MonoVoc using 15 text queries. Embedding Masking Type Top-1 Accuracy (%) ↑ Top-3 Accuracy (%) ↑ CLIP [25] Hard mask 66.7±5.066.7± 5.0 73.3±3.673.3± 3.6 CLIP [25] Soft mask 73.3±5.073.3± 5.0 80.0±6.280.0± 6.2 Perception Encoder [1] Hard mask 73.3±7.173.3± 7.1 80.0±5.080.0± 5.0 Perception Encoder [1] Soft mask 80.0±5.080.0± 5.0 86.7±3.686.7± 3.6 4.5 Efficiency Analysis We also evaluate the efficiency of the generated 3D semantic maps in terms of map size, memory usage, and end-to-end runtime. As shown in Table 4, MonoVoc produces a much smaller representation than the compared methods, using only 140K Gaussians per scene, versus 1M in ObjectGS [35] and 1.5M in SceneSplat [19]. This compact map reduces memory usage to 14 MB per scene, nearly an order of magnitude below both baselines. Runtime is also considered: although SceneSplat reports a shorter time, it excludes 3D map generation, making the comparison less direct. In contrast, MonoVoc includes both SLAM-based reconstruction and semantic post-processing, better reflecting the cost of a full monocular pipeline. Overall, these results show that decoupling semantics from reconstruction and storing compact object-level representations, rather than dense per-Gaussian language features, yields a much lighter semantic map while remaining practical in memory and runtime. Table 4: Efficiency comparison in terms of average map size, average memory usage, and average runtime. MonoVoc uses fewer Gaussians and substantially less memory while maintaining a practical end-to-end runtime. Model # Gaussians per Scene ↓ Memory Usage per Scene ↓ Runtime ↓ ObjectGS [35] 1M 90 MB 186 min ± 34 min SceneSplat [19] 1.5M 149 MB 40 min∗ ± 22 min MonoVoc (ours) 140K 14 MB 81 min ± 17 min ∗Runtime is reported without 3D map generation; SceneSplat does not generate the 3D map. 5 Discussion and Limitations Discussion. The results support our main design choice: geometry and semantics need not be learned jointly for monocular open-vocabulary 3D understanding. Using HI-SLAM2 [34] as a fixed geometric backbone, we add semantics after reconstruction without retraining the mapping pipeline, achieving strong rendering quality, competitive rendered segmentation accuracy, and a far more compact semantic map than ObjectGS and SceneSplat. Ablations show that deblending and semantic color correction are important, with CMC in Lab space [4, 5] yielding the lowest semantic mismatch after alpha blending. For language grounding, Perception Encoder with soft masking performs best, though the pipeline is model- and segmenter-agnostic. Overall, compact object-level semantics suffice for efficient open-vocabulary retrieval and question answering without storing dense language features on every Gaussian. Limitations. The method has two main limitations, both in semantic assignment. First, palette quantization can be ambiguous when object colors are close in the semantic palette or when deblending yields intermediate colors. We partly address this with CMC distance in Lab space, which is more perceptually meaningful than RGB distance, but performance could improve with better-separated or adaptive palettes. Second, SCD uses an efficient per-Gaussian approximation rather than solving the full global inverse problem over all overlapping Gaussians. This keeps the method lightweight, but may cause errors near object boundaries or in highly overlapping regions. Quantization reduces many mixed-color errors, while future work could add confidence weighting or sparse global refinement. Broader impact, privacy considerations, and deployment risks are discussed in Appendix D. 6 Conclusion We presented a training-free pipeline for open-vocabulary 3D scene understanding from monocular videos. Rather than learning dense language features during reconstruction, our method first builds a geometry-aware Gaussian map with HI-SLAM2, then assigns semantics in post-processing. Semantic colors are recovered for individual Gaussians via deblending, refined through perceptual palette quantization, and linked to compact object-level language embeddings. This keeps the map readable and lightweight while supporting text-based object retrieval and question answering. Experiments on Replica show that this modular design is effective, achieving strong reconstruction quality, competitive rendered segmentation accuracy, and substantially lower memory usage than baselines. Overall, our work suggests a simple direction for semantic 3D Gaussian maps: keep geometry, object identity, and language meaning modular. Although the method still depends on segmentation quality, color quantization, and approximate deblending, it provides a compact foundation for monocular open-vocabulary scene understanding. Future work can extend it to real-world videos, diverse environments, stronger object-level reasoning, and more robust handling of overlap, occlusion, and noisy semantic observations. References [1] D. Bolya, P. Huang, P. Sun, J. H. Cho, A. Madotto, C. Wei, T. Ma, J. Zhi, J. Rajasegaran, H. Rasheed, J. Wang, M. Monteiro, H. Xu, S. Dong, N. Ravi, D. Li, P. Dollár, and C. Feichtenhofer (2025) Perception encoder: the best visual embeddings are not at the output of the network. arXiv:2504.13181. Cited by: §3.1, §4.4, Table 3, Table 3. [2] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin (2021-10) Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 9650–9660. Cited by: §3.1. [3] K. Chen, B. Dai, M. Qin, D. Zhang, P. Li, Y. Zou, and H. Wang (2025) Slgaussian: fast language gaussian splatting in sparse views. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 3047–3056. Cited by: §2.1. [4] CIE (2007) Colorimetry – part 4: CIE 1976 L*a*b* colour space. Note: CIE Standard S 014-4/E:2007 Cited by: §3.1, §5. [5] F. J. J. Clarke, R. McDonald, and B. Rigg (1984) Modification to the JPC79 colour–difference formula. Journal of the Society of Dyers and Colourists 100 (4), p. 128–132. External Links: Document Cited by: §3.1, Figure 2, §4.3, Table 2, §5. [6] G. Cui, M. R. Luo, B. Rigg, G. Roesler, and K. Witt (2002) Uniform colour spaces based on the din99 colour-difference formula. Color Research & Application 27 (4), p. 282–290. External Links: Document Cited by: §4.3, Table 2. [7] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman (2010) The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision 88 (2), p. 303–338. External Links: Document Cited by: §4.2. [8] A. Hore and D. Ziou (2010) Image quality metrics: PSNR vs. SSIM. Proceedings of the 20th International Conference on Pattern Recognition (ICPR), p. 2366–2369. External Links: Document Cited by: §4.2. [9] S. Kato, S. Kurita, C. Chu, and S. Kurohashi (2023) ARKitSceneRefer: text-based localization of small objects in diverse real-world 3d indoor scenes. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 784–799. External Links: Document Cited by: §1. [10] N. Keetha, J. Karhade, K. M. Jatavallabhula, G. Yang, S. Scherer, D. Ramanan, and J. Luiten (2024) SplaTAM: splat, track & map 3d gaussians for dense rgb-d slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §1. [11] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis (2023) 3D gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42 (4). External Links: Document Cited by: §1. [12] J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik (2023) Lerf: language embedded radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, p. 19729–19739. Cited by: §2.1. [13] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, P. Dollár, and R. Girshick (2023) Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 4015–4026. Cited by: §3.1. [14] B. Li, K. Q. Weinberger, S. Belongie, V. Koltun, and R. Ranftl (2022) Language-driven semantic segmentation. In International Conference on Learning Representations, Cited by: §1. [15] C. Li, Z. Li, Z. Wang, Y. Xu, M. R. Luo, G. Cui, M. Melgosa, M. H. Brill, and M. R. Pointer (2017) Comprehensive color solutions: cam16, cat16, and cam16-ucs. Color Research & Application 42 (6), p. 703–718. External Links: Document Cited by: §4.3, Table 2. [16] H. Li, R. Qin, Z. Zou, D. He, B. Li, B. Dai, D. Zhang, and J. Han (2024) LangSurf: language-embedded surface gaussians for 3d scene understanding. arXiv preprint arXiv:2412.17635. External Links: Link Cited by: §2.1. [17] W. Li, Y. Zhao, M. Qin, Y. Liu, Y. Cai, C. Gan, and H. Pfister (2025) LangSplatV2: high-dimensional 3D language Gaussian splatting with 450+ FPS. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2507.07136, Link Cited by: §1, §2.1. [18] W. Li, R. Zhou, J. Zhou, Y. Song, J. Herter, M. Qin, G. Huang, and H. Pfister (2025-06) 4D langsplat: 4D language Gaussian splatting via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 22001–22011. Cited by: §1, §2.1. [19] Y. Li, Q. Ma, R. Yang, H. Li, M. Ma, B. Ren, N. Popovic, N. Sebe, E. Konukoglu, T. Gevers, L. Van Gool, M. R. Oswald, and D. P. Paudel (2025-10) SceneSplat: Gaussian splatting-based scene understanding with vision-language pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 4961–4972. Cited by: §1, §2.2, §2.2, §4.5, Table 1, Table 1, Table 4. [20] F. Liang, B. Wu, X. Dai, K. Li, Y. Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu (2023) Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 7061–7070. Cited by: §3.1. [21] M. R. Luo, G. Cui, and B. Rigg (2001) The development of the cie 2000 colour-difference formula: ciede2000. Color Research & Application 26 (5), p. 340–350. External Links: Document Cited by: §4.3, Table 2. [22] H. Matsuki, R. Murai, P. H. J. Kelly, and A. J. Davison (2024) Gaussian splatting slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §1. [23] R. McDonald and K. J. Smith (1995) CIE94—a new colour-difference formula. Journal of the Society of Dyers and Colourists 111 (12), p. 376–379. External Links: Document Cited by: §4.3, Table 2. [24] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng (2020) NeRF: representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, p. 405–421. Cited by: §2.1. [25] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021-18–24 Jul) Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 8748–8763. Cited by: §3.1, §4.4, Table 3, Table 3. [26] J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, A. Clarkson, M. Yan, B. Budge, Y. Yan, X. Pan, J. Yon, Y. Zou, K. Leon, N. Carter, J. Briales, T. Gillingham, E. Mueggler, L. Pesqueira, M. Savva, D. Batra, H. M. Strasdat, R. De Nardi, M. Goesele, S. Lovegrove, and R. Newcombe (2019) The replica dataset: a digital replica of indoor spaces. arXiv preprint arXiv:1906.05797. External Links: Link Cited by: §4.1. [27] L. Szilagyi, F. Engelmann, and J. Bohg (2025) SLAG: scalable language-augmented gaussian splatting. IEEE Robotics and Automation Letters. Cited by: §2.1. [28] O. Tasneem and R. Pieters (2026) Human–robot collaborative visual inspection with large language models. Robotics and Computer-Integrated Manufacturing 98, p. 103154. External Links: Document Cited by: §1. [29] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli (2004) Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13 (4), p. 600–612. External Links: Document Cited by: §4.2. [30] A. Werby, C. Huang, M. Büchner, A. Valada, and W. Burgard (2024) Hierarchical open-vocabulary 3d scene graphs for language-grounded robot navigation. In Robotics: Science and Systems (RSS), Cited by: §1. [31] Y. Wu, J. Meng, H. Li, C. Wu, Y. Shi, X. Cheng, C. Zhao, H. Feng, E. Ding, J. Wang, and J. Zhang (2024) OpenGaussian: towards point-level 3D Gaussian-based open vocabulary understanding. In Advances in Neural Information Processing Systems, Vol. 37, p. 19114–19138. External Links: Document Cited by: §1, §2.1. [32] Y. Yuan, W. Li, J. Liu, D. Tang, X. Luo, C. Qin, L. Zhang, and J. Zhu (2024) Osprey: pixel understanding with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §3.1. [33] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang (2018) The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 586–595. External Links: Document Cited by: §4.2. [34] W. Zhang, Q. Cheng, D. Skuddis, N. Zeller, D. Cremers, and N. Haala (2025-10) HI-SLAM2: geometry-aware gaussian SLAM for fast monocular scene reconstruction. IEEE Transactions on Robotics 41, p. 6478–6493. Note: Originally available as arXiv:2411.17982 External Links: Document, Link Cited by: Figure 1, §3.1, §3, §4.1, §5. [35] R. Zhu, M. Yu, L. Xu, L. Jiang, Y. Li, T. Zhang, J. Pang, and B. Dai (2025-10) ObjectGS: object-aware scene reconstruction and scene understanding via Gaussian splatting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 8350–8360. Cited by: §1, §2.2, §2.2, §4.5, Table 1, Table 1, Table 4. Appendix A Additional deblending proofs This appendix gives the full derivations for the deblending formulation. We use v for a pixel coordinate and m for a general observation, e.g., a pixel or a pixel–frame pair. All semantic colors lie in ℝ3R^3, and all vector equations hold channel-wise. Lemma A.1 (Semantic-independence of alpha). Statement. Fix a pixel v. Let i∈1,…,Ni∈\1,…,N\ index Gaussians ordered along the ray through v. Let the unknown semantic color of Gaussian i be si∈ℝ3s_i ^3. Let θi _i denote all non-semantic parameters of Gaussian i, such as mean, covariance, and opacity. Define αi(v)=gi(v;θi)∈[0,1), _i(v)=g_i(v; _i)∈[0,1), (19) and assume gig_i does not depend on skk=1N\s_k\_k=1^N. Then, for all i,ki,k, ∂αi(v)∂sk=0. ∂ _i(v)∂ s_k=0. (20) Proof. By assumption, αi(v)=gi(v;θi) _i(v)=g_i(v; _i) and θi _i is independent of the semantic colors. Hence, for any k, ∂αi(v)∂sk=∂skgi(v;θi)=0. ∂ _i(v)∂ s_k= ∂ s_kg_i(v; _i)=0. (21) Lemma A.2 (Semantic-independence of transmittance and weights). Statement. Assume Lemma A.1. Define T1(v)=1,T_1(v)=1, (22) Ti(v)=∏j=1i−1(1−αj(v))fori≥2,T_i(v)= _j=1^i-1 (1- _j(v) ) i≥ 2, (23) wi(v)=αi(v)Ti(v),w_i(v)= _i(v)T_i(v), (24) and TN+1(v)=∏j=1N(1−αj(v)).T_N+1(v)= _j=1^N (1- _j(v) ). (25) Then, for all i,ki,k, ∂Ti(v)∂sk=0,∂wi(v)∂sk=0,∂TN+1(v)∂sk=0. ∂ T_i(v)∂ s_k=0, ∂ w_i(v)∂ s_k=0, ∂ T_N+1(v)∂ s_k=0. (26) Proof. For i≥2i≥ 2, Ti(v)=∏j=1i−1(1−αj(v)).T_i(v)= _j=1^i-1 (1- _j(v) ). (27) Differentiating with respect to sks_k gives ∂Ti(v)∂sk=∑ℓ=1i−1(∏j=1,j≠ℓi−1(1−αj(v)))(−∂αℓ(v)∂sk). ∂ T_i(v)∂ s_k= _ =1^i-1 ( _ j=1,\ j≠ ^i-1 (1- _j(v) ) ) (- ∂ _ (v)∂ s_k ). (28) By Lemma A.1, each derivative ∂αℓ(v)/∂sk∂ _ (v)/∂ s_k is zero. Therefore ∂Ti(v)∂sk=0. ∂ T_i(v)∂ s_k=0. (29) Also T1(v)=1T_1(v)=1, so ∂T1(v)/∂sk=0∂ T_1(v)/∂ s_k=0. Since wi(v)=αi(v)Ti(v)w_i(v)= _i(v)T_i(v), ∂wi(v)∂sk=∂αi(v)∂skTi(v)+αi(v)∂Ti(v)∂sk=0. ∂ w_i(v)∂ s_k= ∂ _i(v)∂ s_kT_i(v)+ _i(v) ∂ T_i(v)∂ s_k=0. (30) Finally, TN+1(v)T_N+1(v) is also a product of terms that are independent of the semantic colors, so the same product-rule argument gives ∂TN+1(v)∂sk=0. ∂ T_N+1(v)∂ s_k=0. (31) Theorem A.3 (Forward rendering is affine in semantic colors). Statement. Fix a pixel v. Let si∈ℝ3s_i ^3 be unknown semantic colors and let bg∈ℝ3bg ^3 be a known background color. The forward alpha-compositing equation is C(v)=∑i=1Nwi(v)si+TN+1(v)bg.C(v)= _i=1^Nw_i(v)s_i+T_N+1(v)bg. (32) Then the map (s1,…,sN)↦C(v)(s_1,…,s_N) C(v) is affine. Proof. Let S=[s1⋮sN]∈ℝ3N.S= [ array[]cs_1\\ \\ s_N array ] ^3N. (33) Define A(v)=[w1(v)I3w2(v)I3⋯wN(v)I3],A(v)= [ array[]cw_1(v)I_3&w_2(v)I_3&·s&w_N(v)I_3 array ], (34) and b(v)=TN+1(v)bg.b(v)=T_N+1(v)bg. (35) Then ∑i=1Nwi(v)si=A(v)S, _i=1^Nw_i(v)s_i=A(v)S, (36) and therefore C(v)=A(v)S+b(v).C(v)=A(v)S+b(v). (37) By Lemma A.2, A(v)A(v) and b(v)b(v) are independent of the unknown semantic colors. Thus the map from the semantic colors to C(v)C(v) is affine. Corollary A.4 (Linearity after background subtraction). Statement. Define the background-subtracted color C~(v)=C(v)−TN+1(v)bg. C(v)=C(v)-T_N+1(v)bg. (38) Then C~(v)=∑i=1Nwi(v)si. C(v)= _i=1^Nw_i(v)s_i. (39) Proof. Using Equation (32), C~(v)=(∑i=1Nwi(v)si+TN+1(v)bg)−TN+1(v)bg. C(v)= ( _i=1^Nw_i(v)s_i+T_N+1(v)bg )-T_N+1(v)bg. (40) The background terms cancel, giving C~(v)=∑i=1Nwi(v)si. C(v)= _i=1^Nw_i(v)s_i. (41) Thus, after background subtraction, the inverse problem is linear in the unknown semantic colors. Corollary A.5 (Stacked linear system over many observations). Statement. Let ℐI index observations, such as pixels or pixel–frame pairs. For each m∈ℐm , define wm,iw_m,i as the weight of Gaussian i at observation m and tm=TN+1,mt_m=T_N+1,m. Stack the observations as Y=[Cm1⋮Cm|ℐ|],B=[tm1bg⋮tm|ℐ|bg].Y= [ array[]cC_m_1\\ \\ C_m_|I| array ], B= [ array[]ct_m_1bg\\ \\ t_m_|I|bg array ]. (42) Then there exists a matrix W with block rows Wm=[wm,1I3wm,2I3⋯wm,NI3]W_m= [ array[]cw_m,1I_3&w_m,2I_3&·s&w_m,NI_3 array ] (43) such that Y=WS+B,Y−B=WS.Y=WS+B, Y-B=WS. (44) Proof. For each observation m∈ℐm , Theorem A.3 gives Cm=∑i=1Nwm,isi+tmbg.C_m= _i=1^Nw_m,is_i+t_mbg. (45) Stacking these equations over all observations gives Y=WS+B.Y=WS+B. (46) Subtracting B from both sides gives Y−B=WS.Y-B=WS. (47) Lemma A.6 (Recursive transmittance update). Statement. Fix a pixel v and define T1(v)=1,Ti(v)=∏j=1i−1(1−αj(v))fori≥2.T_1(v)=1, T_i(v)= _j=1^i-1 (1- _j(v) ) i≥ 2. (48) Then, for all i∈1,…,Ni∈\1,…,N\, Ti+1(v)=Ti(v)(1−αi(v)).T_i+1(v)=T_i(v) (1- _i(v) ). (49) Proof. By definition, Ti+1(v)=∏j=1i(1−αj(v))=(∏j=1i−1(1−αj(v)))(1−αi(v)).T_i+1(v)= _j=1^i (1- _j(v) )= ( _j=1^i-1 (1- _j(v) ) ) (1- _i(v) ). (50) The product in parentheses is Ti(v)T_i(v), so Ti+1(v)=Ti(v)(1−αi(v)).T_i+1(v)=T_i(v) (1- _i(v) ). (51) Lemma A.7 (Weight equals transmittance drop). Statement. Let wi(v)=αi(v)Ti(v).w_i(v)= _i(v)T_i(v). (52) Then, for all i∈1,…,Ni∈\1,…,N\, wi(v)=Ti(v)−Ti+1(v).w_i(v)=T_i(v)-T_i+1(v). (53) Proof. Using Lemma A.6, Ti(v)−Ti+1(v)=Ti(v)−Ti(v)(1−αi(v)).T_i(v)-T_i+1(v)=T_i(v)-T_i(v) (1- _i(v) ). (54) Therefore, Ti(v)−Ti+1(v)=Ti(v)αi(v)=wi(v).T_i(v)-T_i+1(v)=T_i(v) _i(v)=w_i(v). (55) Theorem A.8 (Partition of unity). Statement. Let TN+1(v)=∏j=1N(1−αj(v)).T_N+1(v)= _j=1^N (1- _j(v) ). (56) Then ∑i=1Nwi(v)=1−TN+1(v),∑i=1Nwi(v)+TN+1(v)=1. _i=1^Nw_i(v)=1-T_N+1(v), _i=1^Nw_i(v)+T_N+1(v)=1. (57) Proof. By Lemma A.7, ∑i=1Nwi(v)=∑i=1N(Ti(v)−Ti+1(v)). _i=1^Nw_i(v)= _i=1^N (T_i(v)-T_i+1(v) ). (58) This sum telescopes: ∑i=1Nwi(v)=(T1(v)−T2(v))+⋯+(TN(v)−TN+1(v))=T1(v)−TN+1(v). _i=1^Nw_i(v)= (T_1(v)-T_2(v) )+·s+ (T_N(v)-T_N+1(v) )=T_1(v)-T_N+1(v). (59) Since T1(v)=1T_1(v)=1, ∑i=1Nwi(v)=1−TN+1(v). _i=1^Nw_i(v)=1-T_N+1(v). (60) Adding TN+1(v)T_N+1(v) to both sides gives ∑i=1Nwi(v)+TN+1(v)=1. _i=1^Nw_i(v)+T_N+1(v)=1. (61) Corollary A.9 (Nonnegativity and boundedness). Statement. If αi(v)∈[0,1) _i(v)∈[0,1), then, for all i, 0≤Ti+1(v)≤Ti(v)≤1,0≤wi(v)≤1,0≤TN+1(v)≤1.0≤ T_i+1(v)≤ T_i(v)≤ 1, 0≤ w_i(v)≤ 1, 0≤ T_N+1(v)≤ 1. (62) Proof. Since αi(v)∈[0,1) _i(v)∈[0,1), each factor 1−αi(v)1- _i(v) lies in (0,1](0,1]. Therefore every transmittance product is nonnegative and at most one: 0≤Ti(v)≤1.0≤ T_i(v)≤ 1. (63) By Lemma A.6, Ti+1(v)=Ti(v)(1−αi(v)),T_i+1(v)=T_i(v) (1- _i(v) ), (64) so 0≤Ti+1(v)≤Ti(v)0≤ T_i+1(v)≤ T_i(v). By Lemma A.7, wi(v)=Ti(v)−Ti+1(v).w_i(v)=T_i(v)-T_i+1(v). (65) Thus wi(v)≥0w_i(v)≥ 0 and wi(v)≤Ti(v)≤1w_i(v)≤ T_i(v)≤ 1. The same product argument gives 0≤TN+1(v)≤10≤ T_N+1(v)≤ 1. Theorem A.10 (Exact mixing form of alpha compositing). Statement. The rendered color C(v)=∑i=1Nwi(v)si+TN+1(v)bgC(v)= _i=1^Nw_i(v)s_i+T_N+1(v)bg (66) is an exact convex mixture of the Gaussian semantic colors and the background: wi(v)≥0,TN+1(v)≥0,∑i=1Nwi(v)+TN+1(v)=1.w_i(v)≥ 0, T_N+1(v)≥ 0, _i=1^Nw_i(v)+T_N+1(v)=1. (67) Proof. The nonnegativity of wi(v)w_i(v) and TN+1(v)T_N+1(v) follows from Corollary A.9. The sum-to-one property follows from Theorem A.8. Therefore the coefficients in Equation (66) form a convex mixture over the semantic colors sii=1N\s_i\_i=1^N and the background bgbg. Lemma A.11 (Convex quadratic form). Statement. Fix a Gaussian index i. Let ℳM index observations. For each m∈ℳm , let ym∈ℝ3y_m ^3 be a known target and wi,m≥0w_i,m≥ 0 be the known contribution weight of Gaussian i. Define Ji(s)=∑m∈ℳwi,m‖s−ym‖22.J_i(s)= _m w_i,m \|s-y_m \|_2^2. (68) Then JiJ_i is a convex quadratic in s. Its Hessian is ∇2Ji(s)=2(∑m∈ℳwi,m)I3.∇^2J_i(s)=2 ( _m w_i,m )I_3. (69) If ∑m∈ℳwi,m>0 _m w_i,m>0, then JiJ_i is strictly convex and has a unique minimizer. Proof. We expand the squared norm: ‖s−ym‖22=(s−ym)⊤(s−ym). \|s-y_m \|_2^2=(s-y_m) (s-y_m). (70) Hence Ji(s)=∑mwi,m(s−ym)⊤(s−ym).J_i(s)= _mw_i,m(s-y_m) (s-y_m). (71) Differentiating gives ∇Ji(s)=∑m2wi,m(s−ym)=2(∑mwi,m)s−2∑mwi,mym.∇ J_i(s)= _m2w_i,m(s-y_m)=2 ( _mw_i,m )s-2 _mw_i,my_m. (72) Differentiating again gives ∇2Ji(s)=2(∑mwi,m)I3.∇^2J_i(s)=2 ( _mw_i,m )I_3. (73) If ∑mwi,m>0 _mw_i,m>0, the Hessian is positive definite, so JiJ_i is strictly convex and the minimizer is unique. Theorem A.12 (Weighted-average estimator). Statement. Assume Wi=∑m∈ℳwi,m>0.W_i= _m w_i,m>0. (74) Then the unique minimizer of s^i=argmins∈ℝ3∑m∈ℳwi,m‖s−ym‖22 s_i= arg\,min_s ^3 _m w_i,m \|s-y_m \|_2^2 (75) is s^i=∑m∈ℳwi,mym∑m∈ℳwi,m. s_i= _m w_i,my_m _m w_i,m. (76) Proof. By Lemma A.11, the objective is strictly convex when Wi>0W_i>0, so the unique minimizer satisfies ∇Ji(s)=0∇ J_i(s)=0. From Equation (72), 0=2Wis−2∑mwi,mym.0=2W_is-2 _mw_i,my_m. (77) Thus Wis=∑mwi,mym.W_is= _mw_i,my_m. (78) Since Wi>0W_i>0, s=∑mwi,mymWi.s= _mw_i,my_mW_i. (79) Therefore the minimizer is Equation (76). Corollary A.13 (Accumulation form). Statement. Define color_accum[i]=∑m∈ℳwi,mym,weight_accum[i]=∑m∈ℳwi,m.color\_accum[i]= _m w_i,my_m, \_accum[i]= _m w_i,m. (80) If weight_accum[i]>0weight\_accum[i]>0, then s^i=color_accum[i]weight_accum[i]. s_i= color\_accum[i]weight\_accum[i]. (81) Proof. This follows directly from Theorem A.12 by substituting the two accumulator definitions from Equation (80). Theorem A.14 (Exact recovery under one-hot mixing). Statement. Let ℳM index observations. Assume the forward model Cm=∑i=1Nwi,msi+TN+1,mbgC_m= _i=1^Nw_i,ms_i+T_N+1,mbg (82) and define ym=Cm−TN+1,mbg.y_m=C_m-T_N+1,mbg. (83) Assume one-hot mixing: for every m∈ℳm , there exists ι(m)∈1,…,N (m)∈\1,…,N\ such that wι(m),m=1,wj,m=0forj≠ι(m).w_ (m),m=1, w_j,m=0 j≠ (m). (84) Let ℳi=m∈ℳ:ι(m)=i.M_i=\m : (m)=i\. (85) If |ℳi|>0|M_i|>0, then the estimator s^i=∑m∈ℳwi,mym∑m∈ℳwi,m s_i= _m w_i,my_m _m w_i,m (86) recovers the true semantic color: s^i=si. s_i=s_i. (87) Proof. From Equations (82) and (83), ym=∑i=1Nwi,msi.y_m= _i=1^Nw_i,ms_i. (88) Under one-hot mixing, ym=wι(m),msι(m)+∑j≠ι(m)wj,msj=sι(m).y_m=w_ (m),ms_ (m)+ _j≠ (m)w_j,ms_j=s_ (m). (89) Fix a Gaussian i with |ℳi|>0|M_i|>0. Then ∑m∈ℳwi,mym=∑m∈ℳiym=∑m∈ℳisi=|ℳi|si. _m w_i,my_m= _m _iy_m= _m _is_i=|M_i|s_i. (90) Also, ∑m∈ℳwi,m=∑m∈ℳi1=|ℳi|. _m w_i,m= _m _i1=|M_i|. (91) Therefore s^i=|ℳi|si|ℳi|=si. s_i= |M_i|s_i|M_i|=s_i. (92) Theorem A.15 (Overlap decomposition of the estimator). Statement. Assume the background-subtracted forward model ym=∑j=1Nwj,msj.y_m= _j=1^Nw_j,ms_j. (93) Define Wi=∑m∈ℳwi,m,Gij=∑m∈ℳwi,mwj,m.W_i= _m w_i,m, G_ij= _m w_i,mw_j,m. (94) If Wi>0W_i>0, then the estimator s^i=∑m∈ℳwi,mymWi s_i= _m w_i,my_mW_i (95) satisfies s^i=GiiWisi+∑j=1,j≠iNGijWisj. s_i= G_iW_is_i+ _j=1,\,j≠ i^N G_ijW_is_j. (96) Proof. Substitute Equation (93) into Equation (95): s^i=1Wi∑m∈ℳwi,m(∑j=1Nwj,msj). s_i= 1W_i _m w_i,m ( _j=1^Nw_j,ms_j ). (97) Rearranging the sums gives s^i=1Wi∑j=1N(∑m∈ℳwi,mwj,m)sj. s_i= 1W_i _j=1^N ( _m w_i,mw_j,m )s_j. (98) Using the definition of GijG_ij, s^i=1Wi∑j=1NGijsj. s_i= 1W_i _j=1^NG_ijs_j. (99) Separating the j=ij=i term gives Equation (96). Corollary A.16 (Explicit bias under overlap). Statement. Define ei=s^i−si.e_i= s_i-s_i. (100) Then ei=(GiiWi−1)si+∑j=1,j≠iNGijWisj.e_i= ( G_iW_i-1 )s_i+ _j=1,\,j≠ i^N G_ijW_is_j. (101) Equivalently, ei=−Wi−GiiWisi+∑j=1,j≠iNGijWisj,e_i=- W_i-G_iW_is_i+ _j=1,\,j≠ i^N G_ijW_is_j, (102) where Wi−Gii=∑m∈ℳwi,m(1−wi,m).W_i-G_i= _m w_i,m(1-w_i,m). (103) Proof. Subtract sis_i from Equation (96): ei=s^i−si=(GiiWi−1)si+∑j=1,j≠iNGijWisj.e_i= s_i-s_i= ( G_iW_i-1 )s_i+ _j=1,\,j≠ i^N G_ijW_is_j. (104) Also, Wi−Gii=∑mwi,m−∑mwi,m2=∑mwi,m(1−wi,m).W_i-G_i= _mw_i,m- _mw_i,m^2= _mw_i,m(1-w_i,m). (105) Therefore GiiWi−1=−Wi−GiiWi, G_iW_i-1=- W_i-G_iW_i, (106) which gives Equation (102). Corollary A.17 (Norm bound in terms of overlap). Statement. Under Corollary A.16, ‖ei‖2≤|GiiWi−1|‖si‖2+∑j=1,j≠iNGijWi‖sj‖2. \|e_i \|_2≤ | G_iW_i-1 | \|s_i \|_2+ _j=1,\,j≠ i^N G_ijW_i \|s_j \|_2. (107) Proof. Apply the triangle inequality to Equation (101): ‖ei‖2=‖(GiiWi−1)si+∑j=1,j≠iNGijWisj‖2. \|e_i \|_2= \| ( G_iW_i-1 )s_i+ _j=1,\,j≠ i^N G_ijW_is_j \|_2. (108) Thus, ‖ei‖2≤|GiiWi−1|‖si‖2+∑j=1,j≠iNGijWi‖sj‖2. \|e_i \|_2≤ | G_iW_i-1 | \|s_i \|_2+ _j=1,\,j≠ i^N G_ijW_i \|s_j \|_2. (109) Lemma A.18 (Normal equations of the global least-squares inverse). Statement. Let ℳM index M observations. Stack the unknown semantic colors as S=[s1⊤⋮sN⊤]∈ℝN×3,S= [ array[]cs_1 \\ \\ s_N array ] ^N× 3, (110) and stack the background-subtracted targets as Y=[ym1⊤⋮ymM⊤]∈ℝM×3.Y= [ array[]cy_m_1 \\ \\ y_m_M array ] ^M× 3. (111) Let W∈ℝM×NW ^M× N be the matrix with entries Wm,i=wm,i.W_m,i=w_m,i. (112) Consider S⋆=argminS∈ℝN×3‖WS−Y‖F2.S = arg\,min_S ^N× 3 \|WS-Y \|_F^2. (113) Then S⋆S satisfies (W⊤W)S=W⊤Y.(W W)S=W Y. (114) Equivalently, defining G=W⊤W,B=W⊤Y,G=W W, B=W Y, (115) the system is GS=B,GS=B, (116) with entries Gij=∑m∈ℳwm,iwm,j,Bi=∑m∈ℳwm,iym.G_ij= _m w_m,iw_m,j, B_i= _m w_m,iy_m. (117) Proof. Expand the objective: ‖WS−Y‖F2=tr((WS−Y)⊤(WS−Y)). \|WS-Y \|_F^2=tr ((WS-Y) (WS-Y) ). (118) Differentiating with respect to S gives ∇S‖WS−Y‖F2=2W⊤(WS−Y). _S \|WS-Y \|_F^2=2W (WS-Y). (119) At a minimizer, the gradient is zero: W⊤(WS−Y)=0.W (WS-Y)=0. (120) Therefore (W⊤W)S=W⊤Y.(W W)S=W Y. (121) Using G=W⊤WG=W W and B=W⊤YB=W Y gives GS=BGS=B. Entrywise, Gij=(W⊤W)ij=∑mWm,iWm,j=∑mwm,iwm,j,G_ij=(W W)_ij= _mW_m,iW_m,j= _mw_m,iw_m,j, (122) and Bi=(W⊤Y)i=∑mWm,iYm=∑mwm,iym.B_i=(W Y)_i= _mW_m,iY_m= _mw_m,iy_m. (123) Theorem A.19 (Approach 1 as a diagonal approximation). Statement. The exact normal equations can be written row-wise as ∑j=1NGijsj=Bi. _j=1^NG_ijs_j=B_i. (124) Using the diagonal-only approximation G≈D=diag(G11,…,GNN)G≈ D=diag(G_11,…,G_N) (125) gives the decoupled estimator s~i=BiGii=∑m∈ℳwm,iym∑m∈ℳwm,i2,Gii>0. s_i= B_iG_i= _m w_m,iy_m _m w_m,i^2, G_i>0. (126) If wm,i2=wm,iw_m,i^2=w_m,i for all m, then Gii=WiG_i=W_i and s~i=∑m∈ℳwm,iym∑m∈ℳwm,i=s^i. s_i= _m w_m,iy_m _m w_m,i= s_i. (127) More generally, if wm,i2≈wm,iw_m,i^2≈ w_m,i, then s~i≈s^i s_i≈ s_i. Proof. From Equation (124), Giisi+∑j=1,j≠iNGijsj=Bi.G_is_i+ _j=1,\,j≠ i^NG_ijs_j=B_i. (128) The diagonal approximation sets the off-diagonal terms GijG_ij, j≠ij≠ i, to zero, giving Giisi≈Bi.G_is_i≈ B_i. (129) Thus si≈BiGii=s~i.s_i≈ B_iG_i= s_i. (130) If wm,i2=wm,iw_m,i^2=w_m,i for all m, then Gii=∑mwm,i2=∑mwm,i=Wi.G_i= _mw_m,i^2= _mw_m,i=W_i. (131) Substituting Gii=WiG_i=W_i into Equation (126) gives s~i=∑mwm,iym∑mwm,i=s^i. s_i= _mw_m,iy_m _mw_m,i= s_i. (132) If wm,i2≈wm,iw_m,i^2≈ w_m,i, then Gii=∑mwm,i2≈∑mwm,i=Wi,G_i= _mw_m,i^2≈ _mw_m,i=W_i, (133) so the diagonal estimator is close to the weighted-average estimator. Theorem A.20 (Multi-frame weighted average as a single-index weighted LS). Statement. Let ℱF index frames and let fP_f index pixels in frame f. Define ℳ=(p,f):f∈ℱ,p∈f.M=\(p,f):f ,\ p _f\. (134) For each observation (p,f)(p,f), let yp,f∈ℝ3y_p,f ^3 be a known target and let wi,p,f≥0w_i,p,f≥ 0 be the known weight of Gaussian i. If Wi=∑f∈ℱ∑p∈fwi,p,f>0,W_i= _f _p _fw_i,p,f>0, (135) then the minimizer of s^i=argmins∈ℝ3∑f∈ℱ∑p∈fwi,p,f‖s−yp,f‖22 s_i= arg\,min_s ^3 _f _p _fw_i,p,f \|s-y_p,f \|_2^2 (136) is s^i=∑f∈ℱ∑p∈fwi,p,fyp,f∑f∈ℱ∑p∈fwi,p,f. s_i= _f _p _fw_i,p,fy_p,f _f _p _fw_i,p,f. (137) Proof. Using the observation set ℳM, the objective can be written as s^i=argmins∈ℝ3∑(p,f)∈ℳwi,p,f‖s−yp,f‖22. s_i= arg\,min_s ^3 _(p,f) w_i,p,f \|s-y_p,f \|_2^2. (138) Differentiating and setting the derivative to zero gives 0=∑(p,f)∈ℳ2wi,p,f(s−yp,f).0= _(p,f) 2w_i,p,f(s-y_p,f). (139) Therefore 0=2Wis−2∑(p,f)∈ℳwi,p,fyp,f.0=2W_is-2 _(p,f) w_i,p,fy_p,f. (140) Thus Wis=∑(p,f)∈ℳwi,p,fyp,f.W_is= _(p,f) w_i,p,fy_p,f. (141) Since Wi>0W_i>0, s^i=∑(p,f)∈ℳwi,p,fyp,fWi. s_i= _(p,f) w_i,p,fy_p,fW_i. (142) Expanding ℳM back into frames and pixels gives Equation (137). Corollary A.21 (Accumulation across frames). Statement. Define color_accum[i]=∑f∈ℱ∑p∈fwi,p,fyp,f,weight_accum[i]=∑f∈ℱ∑p∈fwi,p,f.color\_accum[i]= _f _p _fw_i,p,fy_p,f, \_accum[i]= _f _p _fw_i,p,f. (143) If weight_accum[i]>0weight\_accum[i]>0, then s^i=color_accum[i]weight_accum[i]. s_i= color\_accum[i]weight\_accum[i]. (144) Proof. Substituting the accumulator definitions in Equation (143) into Equation (137) gives Equation (144). Corollary A.22 (Stacked multi-frame linear-system interpretation). Statement. Let S∈ℝN×3S ^N× 3 stack the Gaussian semantic colors, and let Y∈ℝ|ℳ|×3Y ^|M|× 3 stack the targets yp,f\y_p,f\. Let W∈ℝ|ℳ|×NW ^|M|× N have entries W(p,f),i=wi,p,f.W_(p,f),i=w_i,p,f. (145) Then the multi-frame forward model is Y=WS,Y=WS, (146) and the estimator in Theorem A.20 is the weighted least-squares minimizer over the expanded index set ℳM. Proof. By definition, each observation (p,f)(p,f) satisfies yp,f=∑i=1Nwi,p,fsi.y_p,f= _i=1^Nw_i,p,fs_i. (147) Stacking all observations gives Y=WS.Y=WS. (148) The multi-frame objective in Equation (136) is therefore the same per-Gaussian weighted least-squares problem as before, but with the observation index set expanded to include all pixels across all frames. Appendix B Why Overlap Happens? This appendix explains why semantic overlap appears in our pipeline and why it is a natural consequence of alpha-composited Gaussian rendering. The main point is that a rendered pixel is rarely produced by a single Gaussian. Instead, it is usually formed by several projected Gaussians whose contributions are blended in image space. Our deblending step tries to reverse this process from the rendered 2D observations, but when the same Gaussian contributes to pixels belonging to different semantic regions, its recovered color can become a mixture of those regions. Figure 3 illustrates common situations where overlap appears. Figure 3: Typical overlap cases in semantic Gaussian deblending. Overlap is caused by image-space alpha blending. In Gaussian splatting, each 3D Gaussian is projected to a 2D elliptical footprint on the image plane. This footprint is not a hard region. Its opacity is highest near the projected center and decreases smoothly toward the boundary according to the projected covariance. Therefore, even if a Gaussian mainly belongs to one object, its projected footprint may still cover nearby pixels from another object, especially near object boundaries. For a pixel observation p, the background-subtracted semantic color can be written as yp=∑i=1Nwi,psi,y_p= _i=1^Nw_i,ps_i, (149) where sis_i is the semantic color of Gaussian i and wi,pw_i,p is its alpha-compositing weight at pixel p. Equation 149 shows that the observed color at a pixel is a weighted mixture of all Gaussians that influence that pixel. If only one Gaussian contributes to the pixel, then the observation directly represents that Gaussian. However, when several Gaussians overlap in the same pixel, the observation contains semantic evidence from multiple objects at once (Figure 4). Figure 4: Visualization of the overlap effect in semantic deblending. Object boundaries are the most affected regions. Overlap is especially visible around object boundaries. At the boundary between two objects, the projected Gaussian footprints from both sides can cover the same pixels. For example, a Gaussian located on the edge of a chair may still splat partly onto pixels labeled as the wall or the floor. Similarly, a Gaussian on the edge of a table may influence both table pixels and background pixels. Since the renderer blends these contributions smoothly, the resulting semantic observation near the boundary may lie between the valid object colors. This does not mean that the geometry is incorrect. It is a normal result of using soft Gaussian primitives rather than hard object surfaces. A Gaussian is a smooth basis function with spatial extent, so its influence naturally spreads beyond a single ideal pixel or object mask. The inverse problem is coupled. Our deblending step assigns a semantic color to each Gaussian by collecting all pixel observations in which that Gaussian participates: s^i=∑pwi,pyp∑pwi,p. s_i= _pw_i,py_p _pw_i,p. (150) This estimator is exact when the Gaussian only contributes to pixels of one semantic color. However, if Gaussian i also contributes to pixels where other Gaussians are visible, then the observations ypy_p already contain the colors of those other Gaussians. Substituting Equation 149 into Equation 150 gives s^i=1Wi∑j=1N(∑pwi,pwj,p)sj,Wi=∑pwi,p. s_i= 1W_i _j=1^N ( _pw_i,pw_j,p )s_j, W_i= _pw_i,p. (151) The term Gij=∑pwi,pwj,pG_ij= _pw_i,pw_j,p (152) measures how often Gaussian i and Gaussian j influence the same observations. When GijG_ij is large for j≠ij≠ i, the recovered color of Gaussian i is pulled toward the color of Gaussian j. This is the overlap bias described in Theorem 4. In other words, the deblending formula is solving the best per-Gaussian average, but the rendered observations themselves are not purely per-Gaussian; they are already mixtures produced by the forward renderer. Multi-view observations can increase ambiguity. The same Gaussian is seen from many frames and viewing directions. In one frame, it may project mostly inside its own object mask. In another frame, due to a different viewpoint, camera pose, occlusion ordering, or segmentation boundary, the same Gaussian may partially overlap a neighboring object. Therefore, the semantic evidence collected for one Gaussian may not be perfectly consistent across all frames. This is important because our estimator uses all observations where the Gaussian has nonzero weight. A Gaussian that is mostly assigned to one object can still receive weaker evidence from another object if it repeatedly overlaps that object in some views. As a result, its averaged semantic color may drift away from the true palette color and move toward an intermediate value. Small opacity does not always mean small accumulated influence. Although a Gaussian may have only a small contribution to a single wrong pixel, this effect can accumulate across many frames. If the Gaussian appears in many views and repeatedly touches a neighboring semantic region, the total weighted evidence from that region can become noticeable. Thus, overlap is not only a single-frame boundary artifact; it is also a multi-view accumulation effect. Why quantization helps. The 2D segmentation maps provide a discrete set of valid semantic colors. However, after deblending, a Gaussian color s^i s_i may lie between two or more valid colors because of alpha blending. Palette quantization corrects this by snapping the recovered color back to the closest valid segmentation color (Figure 5): siq=argminc∈dCMC(Lab(s^i),Lab(c)).s_i^q= arg\,min_c d_CMC (Lab( s_i),Lab(c) ). (153) This step does not remove the fact that overlap happened. Instead, it uses the discrete structure of the 2D segmentation labels to make the final Gaussian map semantically cleaner. The deblending step estimates where the Gaussian color lies from all rendered observations, and the quantization step maps this estimate back to a valid object-level label. Figure 5: Visualization of CMC-based semantic palette quantization. The overall error metric shows 7.48% error across rendered images. Appendix C Further Experiments In this appendix, we provide additional experiments and observations about the sensitivity of our method to 2D segmentation quality. Since our pipeline assigns semantics to 3D Gaussians from 2D segmentation evidence, the quality of the input segmentations directly affects the quality of the final semantic Gaussian map. These experiments are not meant to evaluate a specific segmentation model. Instead, they show what types of segmentation errors are harmful, why they affect the deblending and quantization stages, and what practical conditions make the method work reliably. Failure case from segmentation noise. Figure 6 shows a representative failure case caused by noisy 2D segmentations. The right image shows the semantic rendering obtained from clean segmentation inputs, while the left image shows the result when the input segmentations are corrupted by artificial noise and segmentation artifacts. The noisy input produces a less consistent semantic rendering from the 3D Gaussian map. This happens because our method relies on the 2D segmentation maps as semantic observations. During deblending, each Gaussian collects color evidence from all pixels that it helps render. If the segmentation color at a pixel is wrong, then that incorrect color is also accumulated as evidence for the Gaussians that contributed to that pixel. When this happens repeatedly across frames, the estimated Gaussian semantic color can be pulled away from the correct object color. The problem becomes more visible near object boundaries, where a Gaussian may already contribute to multiple neighboring regions due to alpha blending. Therefore, noisy segmentation does not only create local 2D errors. It can also propagate into the 3D map because the same Gaussian is observed in multiple frames. If a wrong semantic label appears consistently, or if boundary noise is present in many views, the final assigned color of a Gaussian may become incorrect. This explains why segmentation quality is an important practical requirement for our pipeline. Figure 6: Failure case caused by noisy 2D segmentation inputs. Left: semantic rendering when the input segmentation contains artificial noise and segmentation artifacts. Right: semantic rendering obtained from clean segmentation inputs. Since our method uses 2D segmentation colors as semantic evidence for the Gaussians, noisy labels can be accumulated during deblending and can lead to incorrect semantic assignments in the final 3D map. It is important to note that our method is not tied to a particular segmentation model. The user can choose any segmentation method as long as the resulting segmentation maps are reliable enough. In practice, better segmentation masks lead to cleaner semantic Gaussian assignments, while noisy masks naturally lead to noisier 3D semantics. Color-confusable object experiment. We also evaluate a more challenging setting where different object instances are assigned visually similar segmentation colors. This experiment is designed to test the palette quantization stage. Since palette quantization maps each deblended Gaussian color to the closest valid color in the segmentation palette, it can become difficult when two valid colors are very close in the chosen color space. Figure 7: Example of color-confusable segmentation inputs. Different objects or object parts are assigned similar colors, making the palette quantization step more challenging. The experiment tests whether the recovered Gaussian colors can still be mapped back to the correct discrete semantic colors. Figure 8: Semantic rendering result for the color-confusable experiment. The method remains reliable when similar colors are still sufficiently separated in the selected color space. When colors become too close, however, palette quantization can become ambiguous. Figure 7 shows examples of similar colors used for different parts of the scene, and Figure 8 shows the corresponding semantic Gaussian rendering. In our experiments, the method remains stable even when the colors are close, as long as the color distance is still large enough for the quantization metric to separate them. This is because the deblending stage aggregates evidence over many pixels and frames, and the quantization stage then snaps the estimated color to one of the valid palette entries. However, this experiment also highlights an important limitation. If two object colors are almost identical in the quantization space, then palette quantization can fail because the nearest-color decision becomes ambiguous. In such cases, a Gaussian whose recovered color lies between two very similar palette entries may be assigned to the wrong object. This is not a limitation of a specific implementation, but a natural ambiguity caused by using colors as object identifiers but still our method can handle them at an acceptable margin. In our implementation, we use CMC distance in CIELAB space because it gives a perceptually meaningful comparison between colors. Nevertheless, the choice of quantization metric is modular. For special cases, the user may choose a different color space or distance function if it better separates the specific palette used by the segmentation method. In general, assigning more separated colors to different objects makes the quantization stage more robust. Ablation on segmentation quality. To better understand how segmentation quality affects the final semantic map, we perform an ablation study by changing the quality of the 2D segmentation inputs. The purpose of this ablation is to isolate the dependency of our method on the semantic observations. The 3D Gaussian geometry, camera poses, rendering procedure, and quantization method are kept fixed, while only the segmentation inputs are changed. We consider several segmentation quality settings. In the clean setting, the original segmentation maps are used without additional corruption. In the boundary-noise setting, errors are added around object boundaries, where Gaussian overlap is already more likely. In the random-label-noise setting, a percentage of pixels is randomly assigned to incorrect palette colors. In the mixed-boundary setting, boundary pixels are replaced by intermediate RGB values between neighboring object colors. Finally, in the region-artifact setting, small holes, thin noisy structures, or isolated mislabeled regions are added to simulate typical segmentation artifacts. The results show that clean segmentations produce the most consistent semantic Gaussian maps. Boundary noise and mixed boundary colors are especially harmful because they occur in the same regions where Gaussian footprints naturally overlap. Random label noise also degrades the result, but its effect depends on how often the corrupted labels are observed and accumulated across frames. Region artifacts can also create incorrect local assignments, especially when small noisy structures are repeatedly visible from multiple viewpoints. This ablation suggests three practical rules of thumb for preparing segmentation inputs for our method. First, segmentation masks should avoid mixed RGB values at object boundaries. Some segmentation pipelines produce smooth transitions between two object colors near boundaries, but this is harmful for our method because those intermediate colors do not correspond to valid object labels. Second, object colors should be well separated in the color space used by the quantization algorithm. Larger distances between palette colors reduce the chance that a blended or noisy Gaussian color is snapped to the wrong object. Third, segmentation masks should avoid sharp isolated artifacts, such as thin noisy structures or small mislabeled regions, because these can introduce incorrect semantic evidence during the deblending stage. Artificial corruption of 2D segmentations. Finally, we artificially corrupt the 2D segmentation maps and measure the drop in semantic quality. Starting from the clean segmentation maps, we add increasing levels of noise and then run the same semantic assignment pipeline. This allows us to measure how robust the method is when the segmentation input becomes less reliable. Let mIoUcleanmIoU_clean be the result obtained using the original segmentation maps, and let mIoUnoisemIoU_noise be the result after corrupting the segmentations. We report the performance drop as ΔmIoU=mIoUclean−mIoUnoise. =mIoU_clean-mIoU_noise. (154) We corrupt the segmentation maps using random pixel-level label noise. For a noise level ρ, a fraction ρ of pixels is randomly selected and replaced with another color from the valid segmentation palette. This keeps the corrupted image inside the same palette, but breaks the spatial consistency of the labels. We also test boundary-focused corruption, where noise is applied mostly around object boundaries. This setting is more challenging because boundary pixels are the most likely to influence multiple Gaussians through alpha blending. After corruption, the final semantic rendering quality decreases compared with the clean segmentation setting. The drop in mIoU becomes larger as the corruption level increases. Boundary-focused corruption causes a stronger drop than uniform random noise, because boundary regions already contain larger alpha-compositing ambiguity. In other words, errors near object edges are more likely to affect the semantic color assigned to nearby Gaussians. These results confirm that corrupted 2D segmentations lead to weaker final semantic maps. This is expected because the 2D segmentation maps are the semantic source used by the pipeline. When the input labels are corrupted, the deblending stage receives incorrect observations, and the quantization stage may snap some Gaussians to wrong palette colors. Taken together, these experiments show where the method works well and where it needs cleaner input. The pipeline can handle moderate noise and visually close palette colors when the masks are consistent and the object colors remain separable. However, errors that repeat across frames, especially near object boundaries, can be accumulated by the Gaussians and appear again in the final 3D semantic map. For this reason, the segmentation stage should be treated as an important input preparation step rather than a separate detail of the pipeline. Appendix D Broader Impact This work is primarily methodological and operates on monocular videos provided by the user. Therefore, its societal impact depends strongly on the input data and deployment context. Potential positive uses include lightweight 3D scene understanding for robotics, augmented reality, inspection, and assistive systems. However, if applied to sensitive indoor spaces without consent, the same capability could raise privacy concerns or support unauthorized spatial mapping. Practical deployments should therefore use consented data, avoid privacy-sensitive scenes when possible, and follow local privacy and data-protection requirements. We do not release any new scraped dataset or high-risk generative model; any released code is intended to reproduce the proposed semantic mapping pipeline on consented or public benchmark data.