Paper deep dive
From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentation
Rit Gangopadhyay, Alex Wong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/31/2026, 3:32:35 AM
Summary
The paper introduces Distortion Extenders (DEX), a lightweight, architecture- and task-agnostic method to generalize vision foundation models from perspective to fisheye cameras. DEX uses learnable parameters to model fisheye distortion coefficients and align latent embeddings of fisheye images with those of perspective images via self-supervised alignment loss. It improves monocular depth estimation and open-vocabulary segmentation on indoor and outdoor datasets without requiring camera calibration or input rectification at test time.
Entities (13)
Relation Signals (10)
Distortion Extenders → targets → Fisheye Cameras
confidence 95% · We propose a method to generalize vision foundation models to fisheye cameras... DEX... model the fisheye distortion coefficients
Distortion Extenders → usedfor → Monocular Depth Estimation
confidence 95% · We demonstrate DEX on monocular depth estimation and open-vocabulary segmentation...
Distortion Extenders → usedfor → Open-Vocabulary Segmentation
confidence 95% · We demonstrate DEX on monocular depth estimation and open-vocabulary segmentation...
Rit Gangopadhyay → affiliatedwith → Yale Vision Laboratory
confidence 90% · Rit Gangopadhyay Alex Wong Affiliation: Yale Vision Laboratory, Yale University...
Alex Wong → affiliatedwith → Yale Vision Laboratory
confidence 90% · Rit Gangopadhyay Alex Wong Affiliation: Yale Vision Laboratory, Yale University...
Distortion Extenders → improvesperformanceon → WoodScape
confidence 90% · strong zero-shot performance on real fisheye datasets for both monocular depth estimation (ScanNet++, KITTI-360) and open-vocabulary segmentation (WoodScape).
Distortion Extenders → improvesperformanceon → ScanNet++
confidence 90% · strong zero-shot performance on real fisheye datasets for both monocular depth estimation (ScanNet++, KITTI-360)...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision foundation models are capable of generalizing across 3-dimensional (3D) scenes with high-fidelity estimates; their empirical success can be attributed to training on large-scale datasets of perspective images. However, when transferred to wide field-of-view (FoV) images, such as those captured by fisheye cameras, they return erroneous outputs due to a covariate shift stemming from the radial distortion on the image pixels. We propose a method to generalize vision foundation models to fisheye cameras. The crux of our method lies in a set of learnable parameters, termed Distortion Extenders (DEX), that model the fisheye distortion coefficients and the distributional shift between fisheye and perspective images encoded in the latent space. By minimizing a self-supervised alignment loss, DEX transforms the latent embeddings of fisheye images to resemble those of perspective images to recover high-fidelity estimates. DEX is architecture- and task-agnostic: We demonstrate DEX on monocular depth estimation and open-vocabulary segmentation for convolution- and Transformer-based architectures, where we consistently improve over baselines across indoor and outdoor fisheye datasets. As a byproduct, the activations of DEX can also be decoded to distortion coefficients to support camera calibration. Code available at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.27860v1
- Canonical: https://arxiv.org/abs/2608.27860v1
Trouble viewing inline? Open PDF directly →
Full Text
79,376 characters extracted from source content.
Expand or collapse full text
From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentation Rit Gangopadhyay Alex Wong Affiliation: Yale Vision Laboratory, Yale University, New Haven, CT 06520, USA E-mail rit.gangopadhyay,alex.wong@yale.edu Abstract Vision foundation models are capable of generalizing across 3-dimensional (3D) scenes with high-fidelity estimates; their empirical success can be attributed to training on large-scale datasets of perspective images. However, when transferred to wide field-of-view (FoV) images, such as those captured by fisheye cameras, they return erroneous outputs due to a covariate shift stemming from the radial distortion on the image pixels. We propose a method to generalize vision foundation models to fisheye cameras. The crux of our method lies in a set of learnable parameters, termed Distortion Extenders (DEX), that model the fisheye distortion coefficients and the distributional shift between fisheye and perspective images encoded in the latent space. By minimizing a self-supervised alignment loss, DEX transforms the latent embeddings of fisheye images to resemble those of perspective images to recover high-fidelity estimates. DEX is architecture- and task-agnostic: We demonstrate DEX on monocular depth estimation and open-vocabulary segmentation for convolution- and Transformer-based architectures, where we consistently improve over baselines across indoor and outdoor fisheye datasets. As a byproduct, the activations of DEX can also be decoded to distortion coefficients to support camera calibration. Code available at: https://github.com/Suchisrit/DEX. Keywords: Adaptation Monocular Depth Estimation Open-Vocabulary Segmentation Camera Calibration Self-Supervised Learning 1 Introduction Figure 1: 3D reconstructions and open-vocabulary segmentation of fisheye imagery produced by foundational models, UniDepthV2 [42] and LSeg [27]. The distortion in the input leads to erroneous and distorted reconstructions and segmentations. Vision foundation models are typically trained on tens to hundreds of millions of images, enabling them to generalize across a wide range of 3-dimensional (3D) scenes and across tasks, spanning from recognition to 3D reconstruction. These models leverage the large volumes of internet images available, which are predominantly captured by standard perspective cameras of 3D scenes. The relationship of 3D points and their projection onto a 2D image plane can be described by the projective geometry of the camera, where the perspective cameras capturing the majority of the large-scale training dataset are well-approximated by a simple pinhole projection model. When training on such datasets, biases of the camera projective geometry are “baked into” these foundation models, which are often deployed onto mixed camera systems to enable spatial applications, from extended realities to robotic manipulation and autonomous navigation. Yet, within these mixed camera systems are wide field-of-view (FoV) cameras necessary for environmental coverage (e.g., fisheye and other wide angle cameras). Their projective geometry differs from that of perspective cameras, in that 3D points follow a highly non-linear projection model that induces strong radial distortions. As the FoV increases, objects that are projected near the image borders become increasingly warped or distorted, deviating from the assumptions of projective geometry under which foundational models are trained. When these models are applied to fisheye images, the mismatch in projection induces a systematic distribution shift in local pixel arrangements, and in turn, the encoded features – leading to distorted 3D reconstructions and degraded recognition performance, as seen in Fig. 1. A natural response is to “undo” this image distortion; hence, one line of work rectifies input images using known camera intrinsics, or re-projects them into a canonical representation such as equirectangular, tangent, spherical, or cube-map projections before processing [28, 8, 34, 32, 18, 73]. While this retains compatibility with models (pre)trained on perspective images, it introduces multiple resampling steps, calibration dependencies, and characteristic artifacts (e.g., stretching near poles, seams between faces, aliasing, and gaps between patches). Another approach is to train dedicated models (e.g., depth estimators or segmentation networks) for fisheye cameras [26, 84, 2], which often suffer in generalization across 3D scenes. This is partly due to the difficulty in assembling large-scale fisheye training datasets, as there exist fewer available fisheye images than perspective images. While one may finetune publicly available (foundational) models for specific cameras to improve generalization across scenes, this risks parameter drift and may degrade performance on previously learned cameras [11]. An alternative is to supplement the shortage via mixed-camera training [41], yet this trades generalizability for fidelity across cameras. More importantly, the main school of thought aims to introduce inductive biases tailored to each camera model, which entails specialized designs and thus limits the end user’s flexibility in choosing architectures. Separately, the design of these methods is also task-specific, whether reconstruction or recognition, leading to disparate approaches across vision tasks. Given that a pretrained model can already faithfully infer properties of the 3D scene from perspective images, erroneous estimates on fisheye images stem from a covariate shift in the values of latent embeddings due to local operations on image pixels displaced and rearranged by distortion. As such, if one can modulate their values to align with those that produced high-fidelity estimates in perspective images, then we hypothesize that one can extend the capabilities of models pretrained on images captured by perspective cameras to those of fisheye cameras. To ensure high utility, we consider an architecture- and task-agnostic approach, such that we can apply our method to both convolution-based and Transformer-based architectures for reconstruction and recognition tasks. To do so, we begin by forgoing explicit modeling of camera projective geometry in the input and latent space of the model and build on the insight that the values of embeddings, rather than coordinates, can be adjusted to recover high-fidelity estimates. To this end, we freeze the backbone to preserve its performance on perspective images and propose a set of lightweight, learnable modules that can be inserted into each block of a frozen backbone to modulate feature embeddings of fisheye images. Rather than re-projecting pixels to a canonical image plane, these parameters implicitly model the distortion in fisheye images and operate directly in latent space to map fisheye embeddings toward the distribution of perspective image embeddings. Hence, our method bypasses the need for rectification or camera-specific inductive biases and the cost to retrain the backbone to extend perspective vision models to fisheye cameras. We term our method Distortion Extenders, or DEX for short. We train DEX in a self-supervised manner using only calibrated perspective images. Given a perspective image, we synthetically distort it to obtain a fisheye counterpart. We pass the perspective image through a frozen vision model and treat its output (or intermediate features) as a reference. The fisheye image is then processed by the same backbone but augmented with DEX, and an inverse geometric transformation aligns the fisheye output back to the perspective reference frame. We minimize an alignment loss that drives the Extenders to reconcile the two estimates. This procedure requires neither ground truth nor real fisheye training data, and can be applied to both convolutional and Transformer architectures and across reconstruction (e.g., monocular depth estimation) and recognition tasks (e.g., open-vocabulary segmentation). At test time, fisheye images are fed directly into the augmented model – no calibration, re-projection, or knowledge of additional inference-time geometry is required. We demonstrate our method on two vision tasks: monocular depth estimation and open-vocabulary segmentation. For monocular depth estimation, we further change the output parameterization from Cartesian depth z to spherical range R, leveraging the structure of fisheye projection models to more naturally express distances in high-FoV settings and to counteract the radial attenuation observed in models trained on perspective images. For open-vocabulary segmentation, we formulate our objective in the latent space to better take advantage of the affinities between an image and its associated text. Our Contributions are as follows: (1) We introduce Distortion Extenders (DEX), a set of lightweight, feature-space modulators that generalizes pretrained models from perspective to fisheye imagery by transforming fisheye embeddings to align with those of perspective images. (2) We provide analysis showing that DEX indeed does model fisheye distortion coefficients and serves as latent distribution alignment. (3) DEX is architecture-agnostic and applicable to both convolution- and Transformer-based architectures. (4) DEX is also task-agnostic and enables strong zero-shot performance on real fisheye datasets for both monocular depth estimation (ScanNet++, KITTI-360) and open-vocabulary segmentation (WoodScape). As a byproduct, activations of DEX can be decoded to distortion coefficients for calibration purposes. 2 Related Works Vision Foundation Models. Recently, foundation models (pretrained models designed to provide strong initializations for a variety of tasks) have seen significant growth in the vision domain. These models leverage large datasets through both self-supervised and supervised learning. Their flexibility enables them to adapt to and perform a wide range of tasks. CLIP [44] demonstrates broad task applicability by using contrastive learning on a large vision-language dataset and has been a core component in open-vocabulary segmentation [27, 71]. DINO [3, 37] uses self-supervision to extract effective features and has been used as an encoding module in depth estimators [75, 76, 43]. However, despite being trained on extensive datasets, they experience performance degradation when used with data captured by fisheye cameras, where errors propagate to downstream models. Transformation-based Solutions. To enable depth estimation on unrectified images, methods typically rely on alternate representations such as equi-rectangular projection (ERP) [28, 61], tangent projection [8, 29], spherical projection [34, 73] or cube mapping [32, 60]. However, each of these representations has drawbacks: aside from resampling artifacts, ERP suffers from vertical stretching distortion and angular compression towards the poles, tangent projection exhibits radial stretching and nonuniform area scaling towards the edges, and cube maps have discontinuous transitions between each face of the cube. These artifacts can introduce spurious errors when processed by standard neural network operations, such as convolutions, tokenization, and patch embedding. Spherical projections further require specialized network operations, which increases implementation complexity. Additionally, these projections introduce latency during inference and require accurate camera calibration at test time, which can be sensitive to small physical changes in the system. Depth Estimation for Specific Cameras. Hence, to avoid projecting input images to a canonical representation, another line of work seeks to train depth estimators tailored to specific camera models [10, 9, 17, 19, 26, 35, 38, 40, 49, 47, 52, 54, 57, 77, 67, 66, 65, 64, 68, 84], e.g., perspective, wide-angle, fisheye, omnidirectional. Yet, it is challenging to acquire training datasets for wide FoV cameras compared to publicly available perspective camera datasets. As such, it is difficult to train foundation models for wide FoV cameras. Nonetheless, one can finetune existing large-scale pretrained models, e.g., foundational depth estimators, for specific cameras [1, 6, 69]. While finetuning can improve performance for a target camera, it also introduces the risk of parameter drift [5, 39], where the resulting models may lose their generalizability across 3D scenes and previously learned cameras. It also produces camera-specific models, so deployment platforms (e.g., autonomous vehicles, XR, robotic systems) with mixed camera systems must maintain multiple depth estimators, which increases operational complexity. Generalizable Depth Estimators. To obtain models that generalize across 3D scenes and cameras, recent work has focused on developing foundational monocular depth estimators (FMDEs) [75, 43, 45]. Aside from training on large-scale datasets of mixed cameras, methods in this vein also incorporate the previously mentioned projection to some canonical space, either in the input or in the internal latent representation: DepthAnyCamera [18] applies equirectangular projection (ERP) to the input image before processing it through the network and backprojects the output to the original image space; FoVA-Depth [32] instead utilizes cube mapping of the input image; Unik3D [41] adopts a spherical internal representation and trains on 23 different datasets of mixed cameras to achieve state-of-the-art performance. However, the fidelity across all cameras is reduced. Additionally, the architectural complexity introduced by these models amounts to hundreds of millions of parameters, which imposes less flexibility on deployment platforms with specific computational requirements. Image Segmentation has also benefited from large-scale pretrained networks. Classic convolution-based architectures like U-Net [51] and DeepLab [4] remain popular, while Transformer-based methods [72] have also shown success in capturing global context. Recent developments such as the Segment Anything Model [23] highlight the power of large-scale self-supervision and prompt-based adaptation. Semantic segmentation [4, 25, 24, 51, 63] involves recognizing the identity of segmented regions in the image, extensions of which include Open-vocabulary Semantic Segmentation, where the possible classes for objects in the scene are unknown until inference time. Many of such methods [30, 33, 20, 81, 83] rely on the CLIP vision-language model because of its strong alignment between image and language embeddings, which can be used to assign labels to regions of an image. Recent methods such as LSeg [27] and SED [71] encode text and language separately at input and compare their embeddings to assign regions of the image to the given set of classes. Similar to depth estimation, segmentation pipelines can suffer when encountering heavy distortions. However, most existing work to handle fisheye images can only be applied to depth estimation. We instead show how our method can be easily adapted to recognition tasks. Differentiation from Model Extension and Adaptation Methods. The closest (concurrent) work to ours is Calibration Tokens [7, 11], which extend FMDEs pretrained on perspective images to fisheye images by appending trainable tokens to Transformer blocks. Unlike them, the proposed Distortion Extenders are distinct in: (1) Our method is architecture-agnostic and can be applied to feature map-based architectures (e.g., convolutional neural networks, CNNs) as well as token-based architectures (e.g. Transformers); Calibration Tokens are limited to Transformers due to their use of tokens. (2) Calibration Tokens require manual removal of its appended tokens after each layer, which adds complexity; unlike them, our method does not require appending or discarding of any element, but automatically determines the weighting of the Extenders as a convex combination for modulation based on their affinity towards fisheye features. (3) Calibration Tokens utilize a loss specific to depth, which limits their tasks, whereas our loss is applicable to both depth estimation and segmentation. (4) A minor difference is that Calibration Tokens output depth in z-depth (Cartesian); whereas, we output in range (Euclidean) for depth estimation. Also related are cross-attention [58] and LoRA [21]. Cross-attention determines an added bias by learning projections of different inputs to queries, keys, and values; whereas, LoRA projects the input directly to an added bias by learning a weight matrix. Unlike them, we learn Extender modulators, and a projection matrix to select amongst them, that allow the input features to directly determine weights for convex combinations of the Extenders. The result of which will modulate each encoder block within a backbone. Intuitively, one may analogize Extenders to “centers” or entries of a “codebook” that model distortion coefficients and the distributional shift between perspective and fisheye images. 3 Method Figure 2: Distortion Extenders. The figure illustrates the DEX training pipeline and architecture. During training, we pass perspective inputs and fisheye inputs augmented by DEX into a frozen backbone model. The fisheye features select convex combinations of Extenders for translation in the latent space. We optimize the Distortion Extenders to minimize the difference between the fisheye and perspective output. Formalization. We assume that we are given a model pretrained on perspective images, where its architecture comprises an image encoder f(⋅)f(·) and a task decoder g(⋅)g(·). We note that f and g are parameterized functions, but for ease of notation, we omit their parameters as both sets are fixed or frozen in our method. Let I:Ω⊂ℝ2→ℝ3I: ^2 ^3 denote an input RGB image, where Ω denotes the image space. Inference for a task is given by y^=g(f(I)) y=g(f(I)) for the estimate y^∈ℝH×W×C y ^H× W× C, where H and W are its spatial dimensions. In the case of regression tasks, e.g., depth estimation, C=1C=1 and y y denotes a 2.5D range or depth map. In the case of classification tasks, e.g., open-vocabulary segmentation, C varies depending on the number of input classes or text labels and y y are the logits corresponding to affinities between the image and text features. To extend the pretrained model to handle fisheye images with similar fidelity as perspective images, we propose Distortion Extenders (DEX) to model distortions observed in the image and to correct for the covariate shift in values of the latent embeddings stemming from fisheye distortion. To this end, the crux of our method lies in a set of modulators that translate the latent embeddings of fisheye images such that they resemble those of perspective images; the result of which is a latent alignment of fisheye image embeddings to the distribution of perspective image embeddings. This naturally necessitates two steps: (1) determine the degree of distortion and their effect on the latent embeddings and (2) determine the modulation necessary to correct for their effect. To make the modeling of the broad range of distortions tractable, each Extender serves as a discrete “center” in this continuous space, where convex combinations of multiple Extenders determine the translation necessary to correct for the effect of image distortion on latent embeddings of a fisheye image. As such, DEX is comprised of two primary components: (1) a “soft” selection mechanism that is parameterized by a weight matrix ∈ℝD×DW ^D× D and (2) a set of M Extender modulation parameters ∈ℝM×DE ^M× D, where D denotes the dimensions of latent embeddings encoded by f. Thus, the full architecture of DEX is θ=(l),(l)l=1Lθ=\E^(l),W^(l)\_l=1^L for L layers. An overview of our method is shown in Fig. 2. Extender Mechanism. Let (l)=f(l)((l−1))X^(l)=f^(l)(X^(l-1)) be the output of the llth layer or block of an image encoder (where (0)=IX^(0)=I). We learn Distortion Extenders θ, to modulate the layer’s output, (l)∈ℝ(h×w)×DX^(l) ^(h× w)× D, for all L layers. To reduce over-parametrization, we use lower rank matrices to represent =1⊤2W=W_1 W_2, where 1,2∈ℝd×DW_1,W_2 ^d× D with d<<Dd<<D. As the covariate shift stems from changes in calibration, we hypothesize a low dimensional matrix can encode the transformation. Our modulation after each transformer block reads: =,=softmax(⊤D),←+,S=EW, =softmax( XS D),\ \ \ X +AE, (1) which allows us to add a convex combination of Extenders to every latent embedding (whether token or feature vector) in X, weighted by the affinity A between each latent embedding and each Extender’s selection in S. The modulated embeddings are then used in the next layer and any skip connections: (l+1)=f(l+1)((l)+()(l))=f(l+1)(f(l)((l−1)+()(l−1))+()(l))X^(l+1)=f^(l+1)(X^(l)+(AE)^(l))=f^(l+1)\! (f^(l)(X^(l-1)+(AE)^(l-1))+(AE)^(l) ) (2) Distortion Extenders can be thought of as "centers" that model the latent calibration differences between perspective and fisheye cameras. As we take convex combinations of Extenders, they form a boundary around a range of possible modulations to correct for an image’s distortion in the latent space. The intuition is that an input image’s features are able to select the weights of the combination of Extenders that represents its own distortion. We study this further in Sec. 5, where we use this principle to decode Distortion Extenders into KB distortion coefficients using an image’s selected weights. Training. We aim to optimize our learnable Distortion Extenders, θ, at each of the L encoder blocks to align fisheye image latent embeddings with those of perspective images. We use a self-supervised training scheme, based on AugUndo [70]. We start with calibrated, perspective images I, and distort them with a fisheye transformation T to yield a synthetic fisheye image counterpart If=∘I_f=T I. We freeze a backbone model f trained on perspective images. Then, we retrieve the output of the perspective image y^=g(f(I)) y=g(f(I)), and the output of the fisheye image augmented with DEX, e.g., y^f=g(fθ(If)) y_f=g(f_θ(I_f)). We then apply the inverse fisheye transformation −1T^-1 on y^f y_f in the output space to facilitate comparison between the perspective output and the spatially aligned fisheye output in the perspective reference frame. We formulate the optimization as: argminθ1N∑n=1Nℒ(y^(n),−1∘y^f(n)). _θ 1N _n=1^NL( y^(n),T^-1 y_f^(n)). (3) Distortion Extenders can be applied to both reconstruction and recognition tasks. Thus, we derive two different forms of the learning objective from Eq. 3. Monocular Depth Estimation. In the case of depth estimation, our Distortion Extenders also learn to re-project the output from a Cartesian coordinate system to a spherical one. We treat our perspective output as the supervision signal z^=g(f(I)) z=g(f(I)), and retrieve the output of the fisheye image with Distortion Extenders R^f=g(fθ(If)) R_f=g(f_θ(I_f)). However, the baseline model’s output is in depth. Thus, we convert the perspective output from z- to R-coordinates using the following process for each point on the depth map: ′=[x′y′1]⊤,=−1′=[X/zY/z1]⊤,p = bmatrixx &y &1 bmatrix , _proj=K^-1p = bmatrixX/z&Y/z&1 bmatrix , (4) R=|z⋅|=||=X2+Y2+z2.R=|z·P_proj|=|P|= X^2+Y^2+z^2. (5) Finally, we use the following loss to optimize the Distortion Extenders θ: ℒ(R^(n),∘R^f(n))=log(|R^(n)−∘R^f(n)|+1).L( R^(n),T R_f^(n))= (| R^(n)-T R_f^(n)|+1). (6) Open-Vocabulary Segmentation. Most architectures for open-vocabulary segmentation involve separately encoding the image fi(I)f_i(I) and a set of text labels fl(t)f_l(t), representing the desired classes for classification. Then, some form of similarity is computed between the image embeddings and the language embeddings in order to assign a class to each pixel of the image. Thus, instead of computing our loss on the entire network’s output, which includes a multiplication with the language modality, we compare the perspective and fisheye image embeddings: ℒ(fi(L)(I(n)),−1∘fi,θ(L)(If(n)))=log(|fi(L)(I(n))−1∘fi,θ(L)(If(n)))|+1).L(f_i^(L)(I^(n)),T^-1 f^(L)_i,θ(I_f^(n)))= (|f_i^(L)(I^(n))-T^-1 f^(L)_i,θ(I_f^(n)))|+1). (7) Inference. At inference time, no rectification or knowledge of calibration is necessary to handle fisheye images. For depth estimation, we simply attach our Distortion Extenders to the backbone model and produce our output reconstruction R^=fθ(If) R=f_θ(I_f). The same follows for segmentation to produce our image embedding fi,θ(If)f_i,θ(I_f), which is later compared with language embeddings fl(t)f_l(t) in the baseline model’s architecture using the inner product. The additional overhead in storage and time for inference is minimal, as shown in Tab. 5. 4 Experiments Datasets. During the training process, we synthesize fisheye images using the KB distortion model, and optimize our Distortion Extenders by comparing with the perspective image output. As such, only calibrated, perspective images are necessary for training. Ground truth or real fisheye images are not necessary. We evaluate our method on real indoor and outdoor fisheye images, where we train just one set of Distortion Extenders for both domains, showing the ability of our method to generalize across 3D scenes. Training datasets: NYUv2 [53] provides a wide range of indoor perspective imagery; VOID [66] includes perspective office, classroom, and stairwell scenes; IRS [62] contains synthetic, rendered perspective views of homes, restaurants, and retail environments; Hypersim [50] offers high-quality photorealistic indoor scenes, including both residential and commercial, captured in a perspective projection. The Waymo dataset [55] supplements these with a broad collection of perspective urban driving scenes. Validation datasets: ScanNet++ [78] provides real fisheye images and ground-truth depth, collected via laser scanning and DSLR fisheye capture, enabling evaluation on authentic indoor fisheye data. KITTI-360 [31] features suburban driving scenes recorded with a multi-sensor rig that includes fisheye cameras, along with 3D ground-truth points for assessing reconstruction accuracy. WoodScape [80] offers similar driving scenes with 2D semantic instance labels. We use this dataset to evaluate performance in the open-vocabulary segmentation task. See Supp. Mat. for more details on datasets. Models. Distortion Extenders are architecture- and task-agnostic. As such, we evaluate them with state-of-the-art monocular depth estimation models using the Vision Transformer architecture: MiDaS [46], DepthAnything [75], and UniDepthV2 [42]. We also evaluate our Extenders with LSeg [27], a state-of-the-art model for open-vocabulary semantic segmentation. To show that DEX is architecture agnostic, we additionally evaluate with CNN-based models, VNL [79] for monocular depth estimation and SED [71] for open-vocabulary segmentation. Figure 3: MDE Qualitative Results. We show 3D reconstructions of indoor and outdoor samples from the evaluation in Tab. 1. Previous methods struggle with the outermost regions in the fisheye images, which have the most severe distortion. DEX is more robust to these distortions. In the outdoor scenes with the highest distortions, we are still able to accurately reconstruct many of the points at the edges of the images. 4.1 Evaluation on Fisheye Datasets We evaluate our method’s performance when used with various MDEs in Tab. 1, testing on real fisheye images in both indoor and outdoor settings with the same set of Extenders. We also compare our method to UniK3D [41], a state-of-the-art method for fisheye monocular depth estimation. Additionally, we compare our method to LoRA adaptation and Calibration Tokens adapted for R. As there are currently no other models to perform segmentation specifically for fisheye images, we evaluate our method in conjunction with LSeg [27] and SED [71], which perform open-vocabulary semantic segmentation. Note: We further adapted Calibration Tokens [11] for LSeg for an additional comparison, but could not for VNL and SED as Calibration Tokens require Transformers. We additionally compare DEX against a parameter-efficient LoRA adaptation of UniK3D to our training set. UniK3D already includes large-scale fisheye training data from ASE, aiMotive, HOI4D, and DL3DV, totaling approximately 3.263M examples, as well as ScanNet++ pinhole data. DEX achieves lower RMSE and higher δ1 _1 on both ScanNet++ and KITTI-360. We hypothesize that UniK3D performance has largely saturated and that further adaptation with our mix of 200K examples makes little difference. Figure 4: Segmentation Qualitative Results. We evaluate LSeg and SED with and without DEX on the WoodScape outdoor dataset. We see improvements in the open-vocabulary segmentation, highlighting DEX’s generalizability to this recognition-based task. We evaluate on the valid regions of the images specified by the ground truth. Indoor MDE. We evaluate the performance of MDE models with and without DEX on various scenes from ScanNet++ [78], an indoor fisheye dataset. We compare the output 3D reconstructions against the ground truth in the ground truth’s Cartesian coordinate system. Across the baseline monocular depth estimators, DEX improves by 19% in RMSE and 15% in δ1 _1 on average over LoRA and Calibration Tokens. Additionally, when applied to UniDepthV2, our method improves over UniK3D by 8% averaged over the metrics. Table 1: MDE Performance on Indoor and Outdoor. After training a set of Distortion Extenders, we test their zero-shot performance on ScanNet++ and KITTI-360, which contain real fisheye images with varying scenes and distortion parameters. Testset Experiment Model Train Dataset RMSE ↓ δ1 _1 ↑ ScanNet++ [78] Base Model MiDaS [46] Mix 1.4M 0.672 0.479 w/ LoRA [21] MiDaS [46] Mix 200K 0.561 0.599 w/ Calibration Tokens [11] MiDaS [46] Mix 200K 0.413 0.688 w/ Distortion Extenders MiDaS [46] Mix 200K 0.397 0.702 Base Model DepthAnything [75] Mix 63.5M 0.705 0.454 w/ LoRA [21] DepthAnything [75] Mix 200K 0.706 0.445 w/ Calibration Tokens [11] DepthAnything [75] Mix 200K 0.479 0.667 w/ Distortion Extenders DepthAnything [75] Mix 200K 0.414 0.708 Base Model UniDepthV2 [42] Mix 16M 0.329 0.671 w/ LoRA [21] UniDepthV2 [42] Mix 200K 0.235 0.854 w/ Calibration Tokens [11] UniDepthV2 [42] Mix 200K 0.223 0.841 w/ Distortion Extenders UniDepthV2 [42] Mix 200K 0.200 0.872 Base Model VNL [79] NYUD-V2 29K 0.680 0.521 w/ Distortion Extenders VNL [79] Mix 80K 0.372 0.704 Comparison DepthAnyCamera [18] Indoor 670K 0.390 0.852 Comparison UniK3D [41] Mix 12M 0.223 0.825 w/ LoRA UniK3D [41] Mix 200K 0.218 0.835 KITTI-360 [31] Base Model MiDaS [46] Mix 1.4M 6.111 0.312 w/ LoRA [21] MiDaS[46] Mix 200K 2.479 0.601 w/ Calibration Tokens [11] MiDaS[46] Mix 200K 2.588 0.699 w/ Distortion Extenders MiDaS[46] Mix 200K 2.451 0.657 Base Model DepthAnything [75] Mix 63.5M 6.484 0.318 w/ LoRA [21] DepthAnything [75] Mix 200K 2.897 0.550 w/ Calibration Tokens [11] DepthAnything [75] Mix 200K 2.224 0.676 w/ Distortion Extenders DepthAnything [75] Mix 200K 2.022 0.745 Base Model UniDepthV2 [42] Mix 16M 7.093 0.262 w/ LoRA [21] UniDepthV2 [42] Mix 200K 1.916 0.771 w/ Calibration Tokens [11] UniDepthV2 [42] Mix 200K 1.788 0.763 w/ Distortion Extenders UniDepthV2 [42] Mix 200K 1.663 0.842 Base Model VNL [79] KITTI 24K 7.719 0.245 w/ Distortion Extenders VNL [79] Mix 80K 2.222 0.605 Comparison DepthAnyCamera [18] Outdoor 130K 3.641 0.789 Comparison UniK3D [41] Mix 12M 2.969 0.812 w/ LoRA UniK3D [41] Mix 200K 2.802 0.818 Table 2: Open Vocabulary Segmentation. We evaluate the zero-shot performance of our method on the WoodScape dataset, which contains real fisheye images with varying scenes and distortion parameters. Testset Experiment Model Train Dataset mIoU ↑ weighted IoU ↑ WoodScape [80] Baseline LSeg [27] Mix 500K 0.305 0.819 w/ Calibration Tokens LSeg [27] Mix 50K 0.321 0.823 w/ Distortion Extenders LSeg [27] Mix 50K 0.362 0.838 Baseline SED [71] Mix 120K 0.439 0.829 w/ Distortion Extenders SED [71] Mix 50K 0.449 0.832 Outdoor MDE. We also evaluate on the KITTI-360 outdoor fisheye image dataset. These images feature much higher FoV and as such, much more severe fisheye distortion. Predicting in spherical coordinates is especially helpful in this case. We compare all output 3D reconstructions against the ground truth in the ground truth’s spherical coordinate system. On average, we see an 11% improvement in RMSE and 11% improvement in δ1 _1 over LoRA and Calibration Tokens across the baseline models. Additionally, the same set of Extenders is used for the indoor and outdoor setting, which demonstrates how DEX takes advantage of the existing backbone model’s inherent generalization. Open-Vocabulary Segmentation. We evaluate the LSeg and SED open-vocabulary semantic segmentation models on the WoodScape segmentation dataset with and without Distortion Extenders. We query the model with the following classes during evaluation: road, lane markings, curb, human, cyclist, cars, bicycle, motorcycle, street sign, other. We find that DEX improves over Calibration Tokens by around 13% in the mIoU metric. The experiments are shown in Tab. 2. 4.2 DEX as Task-Agnostic To further evaluate whether DEX is task-agnostic, we apply it to PanopticDepth [12], a multitask framework for depth estimation and depth-aware panoptic segmentation. We jointly optimize DEX by summing both depth and segmentation loss terms. The depth term similarly compares DEX’s fisheye depth output to PanopticDepth’s perspective depth output. The segmentation term is computed in this case by comparing DEX’s fisheye semantic logits to PanopticDepth’s perspective semantic logits. We are able to reuse the same training set for the experiment because we use the PanopticDepth baseline outputs as supervision. As shown in Tab. 3, DEX improves performance in both depth estimation on KITTI-360 and segmentation on WoodScape. Table 3: DEX with PanopticDepth. We apply DEX to PanopticDepth and evaluate both depth estimation and segmentation on WoodScape. Method Depth Segmentation RMSE ↓ δ1↑ _1 mIoU ↑ weighted IoU ↑ PanopticDepth [12] 7.273 0.267 0.160 0.300 w/ Distortion Extenders 3.583 0.354 0.282 0.833 5 Analysis Ablation Study. In order to characterize the contributions of various architectural decisions, we ablate them and evaluate the performance of depth estimation with the UniDepthV2 model. As expected, learning to estimate depth in the Cartesian coordinate system has an especially negative effect in the outdoor setting, which has a much wider FoV with stronger fisheye distortion. Additionally, without the low-rank decomposition, there is a slight performance degradation due to over-parametrization. See Tab. 4 for the full results. Dataset Method RMSE δ1 _1 ScanNet++ Distortion Extenders 0.200 0.872 w/o R R Extension 0.249 0.792 w/o Low-Rank Decomposition 0.216 0.849 KITTI-360 Distortion Extenders 1.663 0.842 w/o R R Extension 7.024 0.271 w/o Low-Rank Decomposition 1.777 0.800 Table 4: Ablation Experiments. We evaluate DEX’s performance without low rank decomposition, and without extension to spherical coordinates. Model UniDepthV2 LSeg Base memory 0.7 GB 2.9 GB DEX memory 2.8 MB 6.1 MB Inference +0.3 ms +2.21 ms Table 5: Computational Overhead. Additional memory and inference time of Distortion Extenders over the backbone models. DEX adds minimal complexity. Figure 5: t-SNE Plot of Fisheye and Perspective Image Features. When using DEX, fisheye image features align much closer with corresponding perspective features. Feature Alignment. We claim that DEX is able to make fisheye images more conducive to models trained on perspective imagery by modulating fisheye image features towards the distribution of perspective features. We verify this claim in Fig. 5, where we use a t-SNE dimensionality reduction to visualize the UniDepthV2 image features of a perspective image, and its corresponding fisheye image with and without DEX. When the model is augmented with DEX, the fisheye image features align very closely with the perspective image features. DEX Decoding. To better understand the internal representations learned by DEX, we attempt to decode the Extenders into KB distortion coefficients for an image input. We posit that because DEX learns to align fisheye features to perspective features, Distortion Extenders implicitly encode KB distortion coefficients. Thus, we should be able to find the Extenders that encode a particular image’s distortion coefficients using the weightings of its activated Extenders. As such, we pass an image input through UniDepthV2 with our trained DEX, and extract the affinities A along with the Extenders E for all L layers. We then use a simple linear probe D∈ℝD×4W_D ^D× 4 to decode each Extender into four values [α1(i)α2(i)α3(i)α4(i)]i=1M=D bmatrix _1^(i)& _2^(i)& _3^(i)& _4^(i) bmatrix_i=1^M=EW_D. For each layer, we take a linear combination across the α values for each Extender, using the affinities A averaged across image features. We then predict the final KB distortion coefficients [k1k2k3k4] bmatrixk_1&k_2&k_3&k_4 bmatrix with one more linear combination across layers, with a learnable weighting L∈ℝLW_L ^L. As we also optimize DEX to extend the baseline model to spherical coordinates, this learnable weighting across layers helps us select the layers with Extenders most relevant to the fisheye distortion correction. We optimize DW_D and LW_L using the same training data in the DEX training pipeline, and evaluate on the same ScanNet++ evaluation set. As seen in Tab. 6, with a simple linear regressor, we are able to predict k1k_1 with the highest accuracy, and k4k_4 with lower accuracy. The degradation pattern from k1k_1 to k4k_4 aligns with the fisheye distortion effect from each parameter. Changes in k1k_1 will result in a much greater visual effect than changes in k4k_4, e.g., moderate changes in k4k_4 do not yield visual differences, so the earlier coefficients dominate the discriminativeness in determining distortion. As such, Distortion Extenders encode later distortion coefficients with wider uncertainty. We show the visual difference for perturbing the distortion coefficients in an image by the error percentages produced by our method in Fig. 6. There is very little difference caused by this shift in distortion parameters, indicating difficulty in recovering them with higher fidelity. Table 6: KB Distortion Parameter Decoding. Distortion Extenders are able to recover KB distortion parameters for an image input. ScanNet++ k1k_1 k2k_2 k3k_3 k4k_4 Average MSE ↓ 2.04×10−62.04×10^-6 1.53×10−61.53×10^-6 3.53×10−73.53×10^-7 8.87×10−98.87×10^-9 9.81×10−79.81×10^-7 MAE ↓ 1.17×10−31.17×10^-3 9.88×10−49.88×10^-4 5.27×10−45.27×10^-4 7.05×10−57.05×10^-5 6.88×10−46.88×10^-4 MAPE (%) ↓ 3.91 14.30 41.86 78.61 34.67 Figure 6: Error in KB Distortion Prediction Visualized. We provide an example for the visual difference in distortion when we generate a fisheye image with a set of distortion parameters representative of ScanNet++, and when we change all of the distortion coefficients by our average error in prediction. Absolute difference between the images generated by the two different sets of distortion coefficients has little visual difference. Absolute difference is best viewed in 5×. 6 Discussion DEX provides a simple but effective mechanism for adapting perspective-trained vision foundation models to fisheye imagery without modifying the backbone or performing explicit geometric rectification at test time. By operating directly in feature space, our approach mitigates the projection mismatch that arises from high-FoV lenses. The improvements observed across multiple backbones and datasets (Tab. 1, 2) suggest that many of the errors in distorted views stem from a systematic feature-space shift. More broadly, this work illustrates the flexibility of our training pipeline. The same self-supervised alignment framework of using a frozen backbone, synthetic domain shift, inverse geometric alignment, and lightweight extenders can be reused across architectures and tasks. Limitations. During training, we do not need ground truth or fisheye images. However, this means that our performance is upper-bounded by the quality of the backbone models we use as our supervision signal, which implies we must carefully select training data that the backbone model can produce high-fidelity outputs for. Luckily, FMDEs and open-vocabulary segmentation models have been trained on tens to hundreds of millions of images. This allows us to casually select training datasets. We hypothesize that one must take care in dataset selection for smaller-scale models. Acknowledgements This work is supported by a Google Gift Award, NSF-2112562 Athena AI Institute and the Global Industrial Technology Cooperation Center (GITCC) through a grant agreement with the Korea Institute for Advancement of Technology (KIAT), project number P0028922. References [1] B. Berenguel-Baeta, M. Santos-Villafranca, J. Bermudez-Cameo, A. Pérez-Yus, and J. J. Guerrero (2023) Convolution kernel adaptation to calibrated fisheye. In 34th British Machine Vision Conference 2023, BMVC 2023, Aberdeen, UK, November 20-24, 2023, Cited by: §2. [2] G. Blott, M. Takami, and C. Heipke (2018) Semantic segmentation of fisheye images. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, p. 0–0. Cited by: §1. [3] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin (2021) Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, p. 9650–9660. Cited by: §2. [4] L. Chen, G. Papandreou, I. Kokkinos, K. P. Murphy, and A. L. Yuille (2016) DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, p. 834–848. Cited by: §2. [5] Y. Chung, H. Park, P. Rim, X. Zhang, J. He, Z. Zeng, S. Cicek, B. Hong, J. S. Duncan, and A. Wong (2025) ETA: energy-based test-time adaptation for depth completion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Cited by: Appendix 0.I, §2. [6] G. P. de La Garanderie, A. A. Abarghouei, and T. P. Breckon (2018) Eliminating the blind spot: adapting 3d object detection and monocular depth estimation to 360 panoramic imagery. In Proceedings of the European Conference on Computer Vision (ECCV), p. 789–807. Cited by: §2. [7] R. Duan, E. Hong, E. Turner, A. Wong, and Y. Zhou (2026) Fisheye3r: adapting unified 3d feed-forward foundation models to fisheye lenses. In European Conference on Computer Vision, Cited by: Appendix 0.I, §2. [8] M. Eder, M. Shvets, J. Lim, and J. Frahm (2020) Tangent images for mitigating spherical distortion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 12426–12434. Cited by: §1, §2. [9] V. Ezhov, H. Park, Z. Zhang, R. Upadhyay, H. Zhang, C. C. Chandrappa, A. Kadambi, Y. Ba, J. Dorsey, and A. Wong (2024) All-day depth completion. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Cited by: §2. [10] X. Fei, A. Wong, and S. Soatto (2019) Geo-supervised visual depth prediction. IEEE Robotics and Automation Letters 4 (2), p. 1661–1668. Cited by: Appendix 0.I, §2. [11] S. Gangopadhyay, J. Kim, X. Chen, P. Rim, H. Park, and A. Wong (2025) Extending foundational monocular depth estimators to fisheye cameras with calibration tokens. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Cited by: Appendix 0.E, Appendix 0.I, §1, §2, §4.1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1. [12] N. Gao, F. He, J. Jia, Y. Shan, H. Zhang, X. Zhao, and K. Huang (2022) Panopticdepth: a unified framework for depth-aware panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 1632–1642. Cited by: §4.2, Table 3. [13] R. Garg, V. K. Bg, G. Carneiro, and I. Reid (2016) Unsupervised cnn for single view depth estimation: geometry to the rescue. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, p. 740–756. Cited by: Appendix 0.I. [14] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun (2013) Vision meets robotics: the kitti dataset. The international journal of robotics research 32 (11), p. 1231–1237. Cited by: Appendix 0.I. [15] C. Godard, O. Mac Aodha, and G. J. Brostow (2017) Unsupervised monocular depth estimation with left-right consistency. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 270–279. Cited by: Appendix 0.I. [16] C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow (2019) Digging into self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF international conference on computer vision, p. 3828–3838. Cited by: Appendix 0.I. [17] X. Gui and X. Zhang (2022) An efficient dense depth map estimation algorithm using direct stereo matching for ultra-wide-angle images. In Computer Graphics International Conference, p. 117–128. Cited by: §2. [18] Y. Guo, S. Garg, S. M. H. Miangoleh, X. Huang, and L. Ren (2025) Depth any camera: zero-shot metric depth estimation from any camera. arXiv preprint arXiv:2501.02464. Cited by: §1, §2, Table 1, Table 1. [19] Z. Guo, Y. Seo, A. Fu, M. Hao, I. Xia, J. J. Lee, X. M. Xie, H. Park, A. Dollar, and A. Wong (2026) Vista3d: a dataset and benchmark for thin object reconstruction from vision, tactile, and 3d point clouds. In European Conference on Computer Vision, Cited by: §2. [20] W. He, S. Jamonnak, L. Gou, and L. Ren (2023) Clip-s4: language-guided self-supervised semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 11207–11216. Cited by: §2. [21] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. ICLR 1 (2), p. 3. Cited by: §2, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1. [22] J. Kannala and S. Brandt (2004) A generic camera calibration method for fish-eye lenses. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., Vol. 1, p. 10–13. Cited by: §0.D.1. [23] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, P. Dollár, and R. Girshick (2023) Segment anything. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Vol. , p. 3992–4003. Cited by: §2. [24] D. Lao, Y. Wu, T. Y. Liu, A. Wong, and S. Soatto (2024) Sub-token vit embedding via stochastic resonance transformers. In International Conference on Machine Learning, Cited by: §2. [25] D. Lao, F. Yang, D. Wang, H. Park, S. Lu, A. Wong, and S. Soatto (2024) On the viability of monocular depth pre-training for semantic segmentation. In European Conference on Computer Vision, Cited by: §2. [26] J. Lee, G. Cho, J. Park, K. Kim, S. Lee, J. Kim, S. Jeong, and K. Joo (2023) Slabins: fisheye depth estimation using slanted bins on road environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 8765–8774. Cited by: §1, §2. [27] B. Li, K. Q. Weinberger, S. Belongie, V. Koltun, and R. Ranftl (2022) Language-driven semantic segmentation. arXiv preprint arXiv:2201.03546. Cited by: Appendix 0.C, Appendix 0.I, Figure 1, Figure 1, §2, §2, §4.1, Table 2, Table 2, Table 2, §4. [28] M. Li, X. Jin, X. Hu, J. Dai, S. Du, and Y. Li (2022) MODE: multi-view omnidirectional depth estimation with 360 cameras. In European Conference on Computer Vision, p. 197–213. Cited by: §1, §2. [29] Y. Li, Y. Guo, Z. Yan, X. Huang, Y. Duan, and L. Ren (2022) Omnifusion: 360 monocular depth estimation via geometry-aware fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 2801–2810. Cited by: §2. [30] F. Liang, B. Wu, X. Dai, K. Li, Y. Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu (2023) Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 7061–7070. Cited by: §2. [31] Y. Liao, J. Xie, and A. Geiger (2022) Kitti-360: a novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (3), p. 3292–3310. Cited by: Appendix 0.E, Table 1, §4. [32] D. Lichy, H. Su, A. Badki, J. Kautz, and O. Gallo (2024) FoVA-depth: field-of-view agnostic depth estimation for cross-dataset generalization. In 2024 International Conference on 3D Vision (3DV), p. 1–10. Cited by: §1, §2, §2. [33] Y. Lin, M. Chen, W. Wang, B. Wu, K. Li, B. Lin, H. Liu, and X. He (2023) Clip is also an efficient segmenter: a text-driven approach for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15305–15314. Cited by: §2. [34] J. Liu, Y. Xu, S. Li, and J. Li (2024) Estimating depth of monocular panoramic image with teacher-student model fusing equirectangular and spherical representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 1262–1271. Cited by: §1, §2. [35] T. Y. Liu, P. Agrawal, A. Chen, B. Hong, and A. Wong (2022) Monitored distillation for positive congruent depth completion. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part I, p. 35–53. Cited by: Appendix 0.I, §2. [36] C. Mei and P. Rives (2007) Single view point omnidirectional camera calibration from planar grids. In Proceedings 2007 IEEE International Conference on Robotics and Automation, p. 3945–3950. Cited by: §0.D.2. [37] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2023) Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: §2. [38] H. Park, R. Chen, P. Rim, D. Lao, and A. Wong (2026) Orcas: unsupervised depth completion via occluded region completion as supervision. The Fourteenth International Conference on Learning Representations. Cited by: Appendix 0.I, §2. [39] H. Park, A. Gupta, and A. Wong (2024) Test-time adaptation for depth completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 20519–20529. Cited by: Appendix 0.I, §2. [40] P. Peer and F. Solina (2002) Panoramic depth imaging: single standard camera approach. International Journal of Computer Vision 47 (1), p. 149–160. Cited by: §2. [41] L. Piccinelli, C. Sakaridis, M. Segu, Y. Yang, S. Li, W. Abbeloos, and L. Van Gool (2025) UniK3D: universal camera monocular 3d estimation. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 1028–1039. Cited by: §1, §2, §4.1, Table 1, Table 1, Table 1, Table 1. [42] L. Piccinelli, C. Sakaridis, Y. Yang, M. Segu, S. Li, W. Abbeloos, and L. Van Gool (2025) UniDepthV2: universal monocular metric depth estimation made simpler. arXiv preprint arXiv:2502.20110. Cited by: Appendix 0.I, Figure 1, Figure 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, §4. [43] L. Piccinelli, Y. Yang, C. Sakaridis, M. Segu, S. Li, L. Van Gool, and F. Yu (2024) UniDepth: universal monocular metric depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 10106–10116. Cited by: Appendix 0.I, §2, §2. [44] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §2. [45] R. Ranftl, A. Bochkovskiy, and V. Koltun (2021) Vision transformers for dense prediction. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Cited by: §2. [46] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun (2022) Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer . IEEE Transactions on Pattern Analysis & Machine Intelligence 44 (03), p. 1623–1637. Cited by: Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, §4. [47] P. Rim, H. Park, V. Ezhov, J. Moon, and A. Wong (2025) Radar-guided polynomial fitting for metric depth estimation. arXiv preprint arXiv:2503.17182. Cited by: §2. [48] P. Rim, H. Park, V. Ezhov, J. Moon, and A. Wong (2026) Radar-guided polynomial fitting for metric depth estimation. In Proceedings of the Computer Vision and Pattern Recognition Conference, Cited by: Appendix 0.I. [49] P. Rim, H. Park, Z. Zeng, Y. Chung, and A. Wong (2025) Protodepth: unsupervised continual depth completion with prototypes. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 6304–6316. Cited by: Appendix 0.I, §2. [50] M. Roberts, J. Ramapuram, A. Ranjan, A. Kumar, M. A. Bautista, N. Paczan, R. Webb, and J. M. Susskind (2021) Hypersim: a photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference on computer vision, p. 10912–10922. Cited by: Appendix 0.E, §4. [51] O. Ronneberger, P. Fischer, and T. Brox (2015) U-net: convolutional networks for biomedical image segmentation. ArXiv abs/1505.04597. Cited by: §2. [52] Z. Shen, C. Lin, K. Liao, L. Nie, Z. Zheng, and Y. Zhao (2022) PanoFormer: panorama transformer for indoor 360 depth estimation. In European Conference on Computer Vision, p. 195–211. Cited by: §2. [53] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus (2012) Indoor segmentation and support inference from rgbd images. In European Conference on Computer Vision, External Links: Link Cited by: Appendix 0.E, Appendix 0.I, §4. [54] A. D. Singh, Y. Ba, A. Sarker, H. Zhang, A. Kadambi, S. Soatto, M. Srivastava, and A. Wong (2023) Depth estimation from camera image and mmwave radar point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9275–9285. Cited by: §2. [55] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. (2020) Scalability in perception for autonomous driving: waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 2446–2454. Cited by: Appendix 0.E, §4. [56] J. Tu, F. Yang, C. Ma, X. Yu, Z. Zeng, S. Wu, H. Zhao, Z. Tao, C. Zhang, H. Qian, and A. Wong (2026) Unitac: a unified multimodal model for cross-sensor tactile understanding and generation. In European Conference on Computer Vision, Cited by: Appendix 0.I. [57] R. Upadhyay, H. Zhang, Y. Ba, E. Yang, B. Gella, S. Jiang, A. Wong, and A. Kadambi (2023) Enhancing diffusion models with 3d perspective geometry constraints. ACM Transactions on Graphics (TOG) 42 (6), p. 1–15. Cited by: §2. [58] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in neural information processing systems 30. Cited by: §2. [59] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell (2021) Tent: fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, Cited by: Appendix 0.I. [60] F. Wang, Y. Yeh, M. Sun, W. Chiu, and Y. Tsai (2020) Bifuse: monocular 360 depth estimation via bi-projection fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 462–471. Cited by: §2. [61] N. Wang, B. Solarte, Y. Tsai, W. Chiu, and M. Sun (2020) 360sd-net: 360 stereo depth estimation with learnable cost volume. In 2020 IEEE International Conference on Robotics and Automation (ICRA), p. 582–588. Cited by: §2. [62] Q. Wang, S. Zheng, Q. Yan, F. Deng, K. Zhao, and X. Chu (2021) Irs: a large naturalistic indoor robotics stereo dataset to train deep models for disparity and surface normal estimation. In 2021 IEEE International Conference on Multimedia and Expo (ICME), p. 1–6. Cited by: Appendix 0.E, §4. [63] A. Wong, A. Chen, Y. Wu, S. Cicek, A. Tiard, B. Hong, and S. Soatto (2021) Small lesion segmentation in brain mris with subpixel embedding. In International MICCAI Brainlesion Workshop, p. 75–87. Cited by: §2. [64] A. Wong, S. Cicek, and S. Soatto (2021) Learning topology from synthetic data for unsupervised depth completion. IEEE Robotics and Automation Letters 6 (2), p. 1495–1502. Cited by: Appendix 0.I, §2. [65] A. Wong, X. Fei, B. Hong, and S. Soatto (2021) An adaptive framework for learning unsupervised depth completion. IEEE Robotics and Automation Letters 6 (2), p. 3120–3127. Cited by: Appendix 0.I, §2. [66] A. Wong, X. Fei, S. Tsuei, and S. Soatto (2020) Unsupervised depth completion from visual inertial odometry. IEEE Robotics and Automation Letters 5 (2), p. 1899–1906. Cited by: Appendix 0.E, Appendix 0.I, §2, §4. [67] A. Wong and S. Soatto (2019) Bilateral cyclic constraint and adaptive regularization for unsupervised monocular depth prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 5644–5653. Cited by: Appendix 0.I, §2. [68] A. Wong and S. Soatto (2021) Unsupervised depth completion with calibrated backprojection layers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 12747–12756. Cited by: Appendix 0.I, §2. [69] W. Wu, Z. Zhang, X. Tan, Z. Zhang, and L. Ma (2025) DepthFisheye: efficient fine-tuning of depth estimation models for fisheye cameras. In International Conference on Computational Visual Media, p. 3–18. Cited by: §2. [70] Y. Wu, T. Y. Liu, H. Park, S. Soatto, D. Lao, and A. Wong (2024) Augundo: scaling up augmentations for monocular depth completion and estimation. In European Conference on Computer Vision, p. 274–293. Cited by: Appendix 0.I, §3. [71] B. Xie, J. Cao, J. Xie, F. S. Khan, and Y. Pang (2024) Sed: a simple encoder-decoder for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 3426–3436. Cited by: Appendix 0.C, Appendix 0.I, §2, §2, §4.1, Table 2, Table 2, §4. [72] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo (2021) SegFormer: simple and efficient design for semantic segmentation with transformers. In Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), Cited by: §2. [73] Q. Yan, Q. Wang, K. Zhao, B. Li, X. Chu, and F. Deng (2022) Spheredepth: panorama depth estimation from spherical domain. In 2022 International Conference on 3D Vision (3DV), p. 1–10. Cited by: §1, §2. [74] F. Yang, C. Feng, Z. Chen, H. Park, D. Wang, Y. Dou, Z. Zeng, X. Chen, R. Gangopadhyay, A. Owens, and A. Wong (2024) Binding touch to everything: learning unified multimodal tactile representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 26340–26353. Cited by: Appendix 0.I. [75] L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao (2024) Depth anything: unleashing the power of large-scale unlabeled data. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Appendix 0.I, §2, §2, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, §4. [76] L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao (2024) Depth anything v2. Advances in Neural Information Processing Systems 37, p. 21875–21911. Cited by: Appendix 0.I, §2. [77] Y. Yang, A. Wong, and S. Soatto (2019) Dense depth posterior (ddp) from single image and sparse range. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3353–3362. Cited by: §2. [78] C. Yeshwanth, Y. Liu, M. Nießner, and A. Dai (2023) Scannet++: a high-fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 12–22. Cited by: Appendix 0.E, §4.1, Table 1, §4. [79] W. Yin, Y. Liu, C. Shen, and Y. Yan (2019) Enforcing geometric constraints of virtual normal for depth prediction. In Proceedings of the IEEE/CVF international conference on computer vision, p. 5684–5693. Cited by: Table 1, Table 1, Table 1, Table 1, §4. [80] S. Yogamani, C. Hughes, J. Horgan, G. Sistu, P. Varley, D. O’Dea, M. Uricár, S. Milz, M. Simon, K. Amende, et al. (2019) Woodscape: a multi-task, multi-camera fisheye dataset for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 9308–9318. Cited by: Appendix 0.E, Table 2, §4. [81] D. Zhang, F. Liu, and Q. Tang (2025) CorrCLIP: reconstructing patch correlations in clip for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 24677–24687. Cited by: §2. [82] X. Zhang, B. Hong, H. Park, D. H. Pak, A. Rickmann, L. H. Staib, J. S. Duncan, and A. Wong (2025) Progressive test time energy adaptation for medical image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Cited by: Appendix 0.I. [83] Y. Zhang, M. Guo, M. Wang, and S. Hu (2024) Exploring regional clues in clip for zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3270–3280. Cited by: §2. [84] G. Zhao, Y. Liu, W. Qi, F. Ma, M. Liu, and J. Ma (2025) Fisheyedepth: a real scale self-supervised depth estimation model for fisheye camera. In 2025 IEEE International Conference on Robotics and Automation (ICRA), p. 3780–3787. Cited by: §1, §2. From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentation SUPPLEMENTARY MATERIAL Appendix 0.A Affinity Maps of Distortion Extenders Fig. 7 visualizes the affinity of various Distortion Extenders toward patch embeddings of the input image. Different Extenders learn to modulate different parts of the image. Figure 7: Affinity Maps of DEX. Each Extender exhibits different affinity toward different regions of the fisheye image features, enabling a selective, yet cooperative modulation of the fisheye features to align them back to those of perspective images. Appendix 0.B Sensitivity Studies In Tab. 7, we study sensitivity to number of extenders M, to decomposition dimension d, and to using DEX at a subset of layers for UniDepthV2. For the layer placement strategies, we tested DEX at every other layer ("Every Other"), just the first layer ("First"), and a single trained set of Extenders shared and reused across all layers ("Shared"). We also study sensitivity to different sets of labels used in open-vocabulary segmentation evaluation for LSeg in Tab. 8. We note the study on different sets of labels (Tab. 8) demonstrates that regardless of labels, DEX always improves performance. As stated in the Limitations section in Sec. 6 of the main paper and the extended discussion in Sec. 0.I, we note that the improvement of DEX is related to that of the base model (LSeg). As such, differences in performance across label sets are artifacts of the base model rather than evidence of inherent sensitivity in DEX. Table 7: Sensitivity study on hyperparameters and layer placement. We analyze the effect of the number of extenders M, decomposition dimension d, and different layer placement strategies on monocular depth estimation performance for ScanNet++ and KITTI-360. Within the "Layers" section of the table, "Every Other" refers to inserting DEX at every other encoder block; "First" refers to only using DEX at the first encoder block; "Shared" refers to training one set of Extenders and reusing it for every encoder block. M d ScanNet++ KITTI-360 RMSE ↓ δ1↑ _1 RMSE ↓ δ1↑ _1 Sensitivity 12 64 0.209 0.845 1.704 0.826 12 20 0.200 0.860 1.663 0.842 12 10 0.207 0.858 1.695 0.844 12 8 0.207 0.852 1.709 0.847 12 4 0.212 0.847 1.746 0.840 32 20 0.201 0.860 1.685 0.815 16 20 0.203 0.859 1.723 0.814 8 20 0.204 0.858 1.740 0.796 4 20 0.218 0.832 1.708 0.818 Layers Every Other 0.207 0.847 1.653 0.833 First 0.250 0.792 1.979 0.681 Shared 0.233 0.812 1.665 0.779 Table 8: Sensitivity to label wording on WoodScape open-vocabulary segmentation. We evaluate how different textual label sets affect the performance of LSeg with DEX on WoodScape. We note that the sensitivity of outcomes for different label sets is more of a property of the base model (LSeg) than it is of DEX. Label Set Method mIoU ↑ weighted IoU ↑ [miscellaneous, street, lanemarks, curbstone, person, rider, vehicles, bike, motorbike, traffic sign] LSeg 0.289 0.812 w/ DEX 0.305 0.822 [background, roadway, road markings, sidewalk edge, pedestrian, riding person, automobiles, cycle, moto, road sign] LSeg 0.227 0.792 w/ DEX 0.293 0.816 [other, road, lane markings, curb, human, cyclist, cars, bicycle, motorcycle, street sign] LSeg 0.305 0.819 w/ DEX 0.362 0.838 Appendix 0.C Implementation Details For the depth estimation experiments, we train 12 Distortion Extenders (M=12M=12) for each transformer block, and for VNL, we used 72, 18, 9, 4, and 2 Extenders for the 5 residual blocks in the encoder. For segmentation, we train 24 Distortion Extenders for each transformer block in LSeg [27], and 24, 12, 6, and 3 Extenders for each resolution of CNN encoder blocks in SED [71]. For all experiments, we learn projection matrices ,W_1,W_2 with a smaller inner dimension d=20d=20. We train using the Adam optimizer with a learning rate of 10−310^-3, and a batch size of 32 for depth estimation and 16 for segmentation. We create our synthetic fisheye training images by randomly sampling distortion coefficients from the following ranges: k1 from -1.3 to -0.005, k2 from -1.3 to -0.005, k3 from -1.3 to -0.005, and k4 from -0.5 to 0.0, inclusive. We trained DEX for 8 epochs using 4 RTX3090 GPUs. Code and checkpoints for DEX will be released for reproducibility. Appendix 0.D Evaluation Protocols As mentioned in the main paper, all evaluations on depth estimation are conducted in the coordinate frame of the ground truth: ScanNet++ in z-coordinates, and KITTI-360 in R-coordinates (resolutions of 462 × 616 and 700 × 700, respectively). Hence, we convert our depth estimate from R to z for ScanNet++, but no conversion is needed for KITTI-360. Similarly, outputs of FMDEs (natively in z) are converted for KITTI-360, but no change is needed for ScanNet++. The z- to R-coordinate transformation for evaluation follows Eqs. 4 and 5 in the main paper, but we must undo fisheye distortion after inverting the intrinsics. We formulate this conversion and the conversion from R to z below. 0.D.1 R to z Transformation For each pixel ′p in the R fisheye distance map, we start by recovering the originally projected point by inverting the intrinsics K and the fisheye distortion model transformation T (KB [22] for the case of ScanNet++): ′=[x′y′1],=−1′=[uv1],p = bmatrixx \\ y \\ 1 bmatrix, _dist=K^-1p = bmatrixu\\ v\\ 1 bmatrix, (8) =−1=[X/zY/z1]P_proj=T^-1P_dist= bmatrixX/z\\ Y/z\\ 1 bmatrix (9) Now, P_proj is in the direction of the actual 3D point P, so we back-project by scaling with R and extract the z component to get the depth of the pixel: =R||⋅=[XYz].P= R|P_proj|·P_proj= bmatrixX\\ Y\\ z bmatrix. (10) 0.D.2 z to R Transformation For this transformation, we start with the above process to recover P_proj (with the MEI [36] model for KITTI-360). We then scale by z and take the magnitude, similarly to the training process: R=|z⋅|=||=X2+Y2+z2.R=|z·P_proj|=|P|= X^2+Y^2+z^2. (11) Appendix 0.E Dataset Details Training Datasets: NYUv2 [53] contains 464 indoor scenes spanning homes, offices, and shared spaces, producing roughly 400k RGB–depth pairs at 640×480 resolution. NYUv2 is a standard indoor benchmark and forms one of our primary supervised training sources. IRS [62] offers a large suite of synthetic indoor environments, from compact apartments to larger commercial layouts. Up to 103,316 rendered frames with accurate depth provide complementary coverage that broadens scene diversity beyond real indoor scans. VOID [66] (Visual Odometry and Indoor Depth) supplies around 58k pairs of RGB image and depth maps captured in corridors, classrooms, and open common areas. Hypersim [50] is a physically based synthetic dataset comprising approximately 77k photorealistic RGB–depth pairs. The scenes include carefully modeled geometry and lighting across diverse architectural settings, enabling the model to observe highly controlled yet visually rich interiors before evaluation on real data. Waymo [55] contributes about 230k frames with synchronized camera and LiDAR across varied outdoor driving conditions. Although originally targeted at autonomous driving perception tasks, we include Waymo to broaden the distribution of our training data to wide-range outdoor imagery featuring complex lighting and large depth ranges. Testing Datasets: Our evaluation focuses on real-world datasets featuring fisheye or wide-FOV imaging. ScanNet++ [78] extends the ScanNet family with additional indoor scenes and fisheye acquisitions. Its fisheye depth ground truth allows us to directly stress-test our method under strong lens distortion in cluttered interiors. KITTI-360 [31] is an outdoor, large-scale autonomous driving dataset equipped with panoramic fisheye cameras and high-resolution LiDAR. Its suburban and semi-rural sequences help measure how well our approach generalizes to wide-FOV outdoor imagery in challenging real driving scenarios. WoodScape [80] is a multi-camera fisheye dataset specifically built for automotive perception, providing four surround-view fisheye cameras with full 360° coverage as well as LiDAR, IMU, and GNSS measurements. It includes annotations for nine tasks—such as semantic segmentation, depth estimation, 2D/3D object detection, motion segmentation, soiling detection, and visual odometry/SLAM—with over 10,000 images annotated at the instance level and more than 100,000 frames annotated for other tasks. Training sets: Mix 200k for Monocular Depth Estimation incorporates images from all of the training datasets. We take subsamples from each dataset in the training set (subsample 25k from NYUv2, 60k from IRS, 30k from VOID, 60k from Hypersim, and 25k from Waymo). Note: we follow the subsampling strategy of Calibration Tokens [11] for fair comparison. Mix 80k for Monocular Depth Estimation removes IRS and Hypersim because VNL was not trained on synthetic data. Including synthetic data introduces a sim-to-real gap, which degrades accuracy of estimates during training. This reflects the stated limitation of our self-supervised training scheme as discussed in "Limitations" in the Sec. 6 ("Discussion") of the main paper. Mix 50k for Open-Vocabulary Segmentation removes IRS, Hypersim, and VOID to speed up training time due to computational resource limitations. Figure 8: Additional Qualitative Examples. We show additional 3D reconstructions of diverse indoor and outdoor samples using DEX. Appendix 0.F Additional Qualitative Examples We provide additional qualitative figures for both indoor (ScanNet++) and outdoor (KITTI-360) scenarios in Fig. 8. All samples are created using the UniDepthV2 model with DEX. Our method is able to recover the L-shaped road (top left) as well as the straight road (top right) for outdoors. We are also able to get the farther portion of the room (bottom left) and straight edges in the room (bottom right) for indoors. Table 9: Error metrics for depth estimation and open-vocabulary segmentation. These evaluation metrics compute the error between predicted output values and the ground truth. For depth estimation metrics (e.g. RMSE and δ1 _1), d d denotes the prediction and d the ground truth. For open-vocabulary segmentation metrics (e.g., mIoU, weighted IoU), TP denotes true positive, FP denotes false positive, FN denotes false negative, and nkn_k denotes the sum of pixels in the ground truth for the class k. Metric Definition RMSE ↓ 1|Ω|∑p∈Ω(d^(p)−d(p))2 1| | _p∈ ( d(p)-d(p) )^2 δ1 _1 ↑ 1|Ω|∑p∈Ω(max(d^(p)d(p),d(p)d^(p))<1.25) 1| | _p∈ 1\! ( \! ( d(p)d(p),\, d(p) d(p) )<1.25 ) mIoU↑mIoU 1K∑k=1KTPkTPk+FPk+FNk 1K _k=1^K TP_kTP_k+FP_k+FN_k weighted IoU↑weighted IoU ∑k=1KwkTPkTPk+FPk+FNk, _k=1^Kw_k\, TP_kTP_k+FP_k+FN_k, wk=nk∑j=1Knjw_k= n_k _j=1^Kn_j Appendix 0.G Evaluation Metrics Table 9 provides the mathematical definitions for evaluation metrics on monocular depth estimation and open-vocabulary segmentation. For monocular depth estimation, we adopt standard benchmarking metrics to evaluate our predictions. Specifically, we use Root Mean Squared Error (RMSE), which aggregates the magnitude of errors in linear space. Lower values in this metric indicate better accuracy. Additionally, we measure a threshold-based accuracy δ1 _1, which quantifies the percentage of pixels whose predicted output values lie within a bound of the ground-truth values. For open-vocabulary segmentation, we adopt mean intersection-over-union (mIoU) and weighted intersection-over-union (weighted IoU). mIoU computes the average intersection-over-union across all classes, treating each class equally regardless of its frequency. Weighted IoU accounts for class imbalance by weighting each class according to its ground-truth proportion. Appendix 0.H Comparison to Standard Undistortion Pipeline We compare DEX to a standard undistortion preprocessing pipeline, followed by inference with UniDepthV2 in Tab. 10. Since undistortion changes the effective field of view and can reduce scene coverage, we alter our evaluation protocol to use Chamfer distance and F1 score to measure both accuracy and completeness of the reconstructed point cloud. DEX substantially improves over using undistortion as preprocessing on both ScanNet++ and KITTI-360. Table 10: Comparison to standard undistortion preprocessing pipelines. We compare undistortion preprocessing followed by inference with UniDepthV2 against UniDepthV2 with DEX using Chamfer distance and F1 score. Base Model Method Dataset Chamfer ↓ F1 ↑ UniDepthV2 Undistortion ScanNet++ 0.360 0.317 UniDepthV2 DEX ScanNet++ 0.117 0.841 UniDepthV2 Undistortion KITTI-360 2.428 0.196 UniDepthV2 DEX KITTI-360 0.343 0.887 Appendix 0.I Extended Discussion on Limitations and Future Work In the main paper, we mentioned that a limitation of our method is that it is bounded by the performance of the baseline model. We illustrate this point with the VNL model, which has separate checkpoints trained exclusively for indoors and outdoors. We evaluate the VNL model (NYUv2 and KITTI) with DEX on both the indoor and outdoor testing sets in Tab. 11. As expected, we found that the performance of indoors-trained VNL with DEX degrades on outdoor data, and vice versa. This is because the models are trained on specific domains; hence, using it to generate self-supervision for a domain it was not trained on creates an out-of-distribution setting with respect to the training dataset (either NYUv2 [53] or KITTI [14]). This degrades the supervision signal, leading to less improvement for the domain that the model was not originally trained on. Table 11: Limitations. We evaluate the performance of the indoor and outdoor VNL models with DEX on both the indoor and outdoor test sets. VNL ScanNet++ KITTI-360 RMSE ↓ δ1↑ _1 RMSE ↓ δ1↑ _1 NYUv2 0.372 0.704 6.283 0.301 KITTI 1.153 0.377 2.222 0.605 The key insight behind the development of Distortion Extenders lies in the fact that existing pretrained foundation models [27, 43, 42, 75, 76, 71] can already infer properties of the 3D scene. This begs the question of whether such models can be extended to a different projection model with the same (high) fidelity. As the forward image formation model can be seen as a composition of extrinsic (camera within the 3D scene) and intrinsic (camera internal parameters) factors, we see this as motivation to leverage existing knowledge of the model about (extrinsic) 3D scenes to extend them to (intrinsic) new cameras. Related are continual learning methods in cameras [11, 7], scenes [49], and sensors [48, 74, 56]. Because our method is self-supervised and task-agnostic, we also foresee it being applicable to test-time adaptation, e.g., for reconstruction [39, 5], [59, 82], where the parameters of extenders are learned online. Finally, our self-supervised loss can also be made unsupervised if given a sequence of images [10, 13, 15, 16, 67] and additional sensors [35, 38, 66, 65, 64, 68, 70], making this approach suitable for deployment to platforms with heterogeneous sensors.