Paper deep dive
Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration
Zhixin Cheng, Jiacheng Deng, Xiaotian Yin, Baoqun Yin, Richang Hong, Tianzhu Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/17/2026, 4:30:13 AM
Summary
The paper introduces M23D, an adaptive dual-masked autoencoder network for image-to-point cloud registration. It addresses domain discrepancies and limited sensitivity in detection-free methods by proposing the Intermodal Dual-MAE Framework (ID-MAE) and a Similarity-based RL Masking Strategy (SRLM). ID-MAE uses bidirectional cross-modal reconstruction to align features, while SRLM adaptively selects informative regions for masking using reinforcement learning. The method achieves state-of-the-art performance on RGB-D Scenes v2 and 7-Scenes benchmarks.
Entities (10)
Relation Signals (8)
M23D → contains → ID-MAE
confidence 98% · M23D... featuring two core components: the Intermodal Dual-MAE Framework (ID-MAE) and the Similarity-based RL Masking Strategy (SRLM).
M23D → contains → SRLM
confidence 98% · M23D... featuring two core components: the Intermodal Dual-MAE Framework (ID-MAE) and the Similarity-based RL Masking Strategy (SRLM).
M23D → evaluatedon → 7-Scenes
confidence 95% · Experiments on RGB-D Scenes v2 and 7-Scenes benchmarks show that our method achieves state-of-the-art performance
M23D → evaluatedon → RGB-D Scenes v2
confidence 95% · Experiments on RGB-D Scenes v2 and 7-Scenes benchmarks show that our method achieves state-of-the-art performance
SRLM → uses → Reinforcement Learning
confidence 95% · SRLM... leveraging cross-modal similarity and reinforcement learning
Zhixin Cheng → affiliatedwith → Hefei University of Technology
confidence 90% · Zhixin Cheng and Richang Hong is with the School of Computer Science and Technology, Hefei University of Technology
Richang Hong → affiliatedwith → Hefei University of Technology
confidence 90% · Zhixin Cheng and Richang Hong is with the School of Computer Science and Technology, Hefei University of Technology
Tianzhu Zhang → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Detection-free methods for image-to-point cloud registration are prone to erroneous correspondences caused by domain and modality discrepancies, limited sensitivity of feature extractors, and the presence of non-overlapping regions. The Masked Autoencoder (MAE) has shown strong performance in visual representation for images and point clouds. It may be helpful to apply this approach to image-to-point cloud registration, a task that requires unified feature extraction and accurate cross-modal correspondences. Standard MAE's random masking may overlook key regions due to limited camera views, reducing registration effectiveness. To address this, we propose the Intermodal Dual-MAE Framework (ID-MAE) with a Similarity-based RL Masking Strategy (SRLM), which adaptively masks informative positions by leveraging cross-modal similarity and reinforcement learning, thus narrowing the modality gap. Our method enhances cross-modal representation learning by enforcing representation consistency during feature extraction, thereby enabling more reliable 2D-3D correspondence estimation. Experiments on RGB-D Scenes v2 and 7-Scenes benchmarks show that our method achieves state-of-the-art performance in image-to-point cloud registration.
Tags
Links
- Source: https://arxiv.org/abs/2609.18088v1
- Canonical: https://arxiv.org/abs/2609.18088v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
88,927 characters extracted from source content.
Expand or collapse full text
Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration Zhixin Cheng Jiacheng Deng Xiaotian Yin Baoqun Yin Richang Hong Tianzhu Zhang †thanks: Zhixin Cheng and Richang Hong is with the School of Computer Science and Technology, Hefei University of Technology, Hefei 230601, China. (e-mail: zxcheng@hfut.edu.cn; hongrc.hfut@gmail.com.)†thanks: Jiacheng Deng, Baoqun Yin and Tianzhu Zhang are with the School of Information Science and Technology, University of Science and Technology of China, Hefei 230027, China (e-mail: chengzhixin@mail.ustc.edu.cn; dengjc@mail.ustc.edu.cn; bqyin@ustc.edu.cn; tzzhang@ustc.edu.cn).†thanks: Xiaotian Yin is with the Institute of Advanced Technology, University of Science and Technology of China, Hefei 230027, China (e-mail: xiaotianyin@mail.ustc.edu.cn).†thanks: Corresponding author: Tianzhu Zhang and Richang Hong.†thanks: Manuscript received April 19, 2021; revised August 16, 2021. Abstract Detection-free methods for image-to-point cloud registration are prone to erroneous correspondences caused by domain and modality discrepancies, limited sensitivity of feature extractors, and the presence of non-overlapping regions. The Masked Autoencoder (MAE) has shown strong performance in visual representation for images and point clouds. It may be helpful to apply this approach to image-to-point cloud registration—a task that requires unified feature extraction and accurate cross-modal correspondences. Standard MAE’s random masking may overlook key regions due to limited camera views, reducing registration effectiveness. To address this, we propose the Intermodal Dual-MAE Framework (ID-MAE) with a Similarity-based RL Masking Strategy (SRLM), which adaptively masks informative positions by leveraging cross-modal similarity and reinforcement learning, thus narrowing the modality gap. Our method enhances cross-modal representation learning by enforcing representation consistency during feature extraction, thereby enabling more reliable 2D–3D correspondence estimation. Experiments on RGB-D Scenes v2 and 7-Scenes benchmarks show that our method achieves state-of-the-art performance in image-to-point cloud registration. Index Terms: Image-to-Point cloud Registration, Masked Autoencoder, Masking Strategy. I Introduction Image-to-point cloud registration (I2P) aims to determine the rigid transformation from the point cloud to the camera coordinate system, which involves the cross-modal matching of image and point cloud, followed by a pose estimator to compute rotation and translation matrices. Such registration is essential for tasks like 3D reconstruction[39, 40, 80, 25, 26], SLAM [32, 33, 22, 34, 74], and visual localization[35, 36, 29, 77, 78]. However, images are represented as regular, dense 2D grids, whereas point clouds are unordered, sparse, and irregular 3D points. The significant modality gap makes designing a model that effectively interacts between RGB and geometric modalities a challenging task. Fig. 1: (a) Schematic of point cloud features aiding image feature reconstruction, where point cloud values denote attention-based correspondences. (b) SRLM Strategy Diagram: Due to the presence of overlapping regions (within the yellow box) and high-value registration areas (star-shaped regions) in the scene, the orange blocks in the random masking strategy indicate ineffective masks. Therefore, we use similarity-based initialization to select patches (gray blocks) in the overlapping regions. Subsequently, we employ RL-driven refinement to seek more informative masking regions (green blocks). SRLM reduces redundant masking and focuses on critical matching regions. Image-to-point cloud registration methods are typically classified into detect-then-match [19, 3, 42] and detection-free approaches [43, 41]. Detect-then-match methods rely on independently detecting 2D and 3D keypoints and matching them via semantic features, but suffer from modality gaps and limited descriptor precision. Detection-free methods, such as 2D3D-MATR [1], use a coarse-to-fine pipeline to establish patch-level matches and refine them to dense correspondences, improving inlier ratios via contextual cues and multi-scale receptive fields. However, image feature extraction relies on texture information, whereas point cloud feature extraction depends on structural information. As a result, the gaps between modalities and the limitations of feature extractors continue to impede further accuracy improvements. Masked Autoencoders [51] have shown strong representation capability in both image and point cloud domains. However, despite some cross-modal attempts [52, 61], their potential for image-to-point cloud registration remains worth exploring. As a result, effectively leveraging MAE for cross-modal tasks remains challenging, since these tasks demand reconstructing features across different modalities while preserving precise correspondence alignment. Moreover, the commonly used random masking strategy may overlook critical regions or include irrelevant areas, limiting its ability to improve registration accuracy. Therefore, adapting both the MAE architecture and masking mechanism is essential to unlock its full potential for cross-modal alignment. Fig. 2: T-SNE visualization and MMD metrics (detailed in IV-A) showing improved alignment between image and point cloud features after training. Based on the above discussions, we identify two key issues that need to be addressed to achieve accurate and robust image-to-point cloud registration. (1) How to design a unified framework that leverages MAE to enhance feature extraction across different modalities. Although MAE has demonstrated remarkable improvements in feature extraction for single-modal tasks, designing a unified cross-modal MAE paradigm remains challenging. If we can enhance the feature extraction capability and further leverage cross-modal guidance for local structure reconstruction, it will greatly improve the ability to extract high-quality features and alleviate domain discrepancies (see in Fig. 2), ultimately increasing the matching success rate. As shown in Fig. 1(a), the attention maps highlight the correspondences between point cloud and image features, with the point cloud features aiding in the reconstruction of the image features. (2) How to select masked regions to maximize the effectiveness of our design. Random masking may be suboptimal, as it can mask irrelevant areas due to the camera’s limited field of view, while missing key regions for registration. This not only increases computational cost but also degrades point cloud reconstruction, limiting the effectiveness of the MAE module. To address this, we propose a masking strategy focused on critical patches for registration in overlapping regions between the image and point cloud. As shown in Fig. 1(b), our method reduces redundant masked areas (orange) and selects more informative regions (green). To address the above challenges, we propose the Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration (M23D), featuring two core components: the Intermodal Dual-MAE Framework (ID-MAE) and the Similarity-based RL Masking Strategy (SRLM). ID-MAE introduces a unified cross-modal MAE with a bidirectional design, where point cloud features assist image reconstruction. Partial masking is applied to both modalities, and a low-resolution point cloud constrains the MAE-reconstructed output, enhancing feature quality. Image reconstruction is guided by both point cloud features and original image supervision, aligning modalities and establishing correspondences. SRLM improves MAE’s effectiveness through a similarity-based initialization mechanism and RL-driven refinement. The masking strategy first targets overlapping regions by computing cross-modal similarity maps and selecting the top-k most relevant areas for masking. Then, we apply reinforcement learning [69, 70, 71] to make the selection of high-value regions differentiable, effectively focusing reconstruction on discriminative regions and improving registration accuracy. Unlike transformer-based matching methods such as 2D3D-MATR, which enhance correspondence estimation by refining interactions among pre-extracted features, our approach addresses cross-modality discrepancies earlier. Specifically, the proposed Intermodal Dual-MAE explicitly constrains feature learning via cross-modal reconstruction, encouraging image and point cloud representations to encode consistent geometric and semantic cues from the outset. Recent interaction-based methods have made notable progress in addressing the cross-modality challenge of image-to-point-cloud registration. In particular, Flow-I2P introduces an innovative information-geometric perspective, reformulating I2P registration as a manifold alignment problem and leveraging Beltrami flow to progressively refine cross-modal feature manifolds. This design effectively enhances feature interaction and improves generalization under limited training data. Our work is motivated by a complementary observation. Beyond refining feature interactions or manifold structures after feature extraction, the cross-modality gap can also be alleviated early in the representation learning process. Instead of explicitly modeling manifold evolution, we employ an intermodal dual-masked autoencoder. This method implicitly encodes cross-modal consistency through reconstruction, encouraging image and point cloud features to share aligned geometric semantics from the outset. In this sense, our approach focuses on learning more compatible representations before correspondence reasoning. It therefore complements existing interaction-based or manifold-alignment methods. In summary, our work can be summarized as follows: • We propose the Similarity Driven Dual-Masked Autoencoder Network (M23D), a unified architecture of the Intermodal Dual-MAE Framework (ID-MAE) with a novel Similarity-based RL Masking Strategy (SRLM) achieves strong accuracy and robustness. To our knowledge, this is the first MAE-based cross-modal design for image-to-point cloud registration. • ID-MAE aligns modalities and establishes correspondences via a dual-MAE framework, while SDMS reduces modality gaps by selecting informative mask regions, improving efficiency and reconstruction quality. • Extensive experiments and ablations on RGB-D Scenes v2 and 7-Scenes demonstrate the effectiveness of our approach, setting a new state-of-the-art in image-to-point cloud registration. I Related Work In this section, we briefly overview related works on image-to-point cloud registration, including stereo image registration, point cloud registration, and inter-modality registration. Stereo Image Registration. Detector-based methods have long dominated stereo image registration. Prior to deep learning, key points were detected using handcrafted techniques like SIFT [8] and ORB [9], which built 2D matches from local features. The advent of deep learning introduced neural network-based detection, transforming the field. SuperGlue [11] was pioneering in using Transformers [23] for image registration, greatly enhancing local feature matching. However, the challenge of detecting repeatable interest points in non-salient areas has led to the rise of detector-free methods. Approaches like LoFTR [6] and Efficient LoFTR [7] use a coarse-to-fine pipeline with Transformers to efficiently estimate dense image matches through global receptive fields. Fig. 3: Overall pipeline of M23D. We extract image and point cloud features through the intermodal dual-MAE framework, which are then processed via attention mechanism. The point cloud and image features generate a score map using cosine similarity and maximum operations, enabling coarse-level matching and refining fine-level correspondences. Finally, PnP + RANSAC is applied to estimate the rigid transformation. Point Cloud Registration. Point cloud registration has advanced from handcrafted descriptors like PPF [12, 13] and FPFH [14] to deep learning-based approaches. CoFiNet [16] introduced detector-free registration with a coarse-to-fine strategy, and recent methods have replaced RANSAC [5] with deep robust estimators for better speed and accuracy. GeoTransformer [17] improves inlier ratios by integrating global information with the transformer and introduces a local-to-global method for RANSAC-free registration. Cross-modal Masked Autoencoder Methods. Existing 2D–3D studies have explored cross-modal representation learning and MAE frameworks from different perspectives. PiMAE [50] employs an interactive dual-branch MAE architecture, where image and point cloud tokens are reconstructed through complementary cross-modal masking based on geometric projection. Such a design enhances object-level semantic representations but relies on projection-based mask generation rather than adaptive selection of informative matching regions. I2P-MAE [49] transfers knowledge from large-scale 2D pre-trained models through semantic-guided point masking and hierarchical point reconstruction, where visible point tokens are selected according to 2D semantic priors. Although effective for generic 3D representation learning, its masking strategy is determined by fixed semantic guidance rather than pair-specific cross-modal correspondence. CrossNet [75] aligns image and point cloud feature spaces via cross-modal contrastive learning, emphasizing global feature consistency and discriminative capability without adopting MAE-based masked reconstruction. Inter-MAE [76] introduces image features to supervise point-cloud MAE pre-training through cross-modal contrastive learning while retaining conventional random masking for point reconstruction. Consequently, cross-modal interaction is mainly achieved through feature supervision instead of adaptive mask optimization. Joint-MAE [79] constructs a unified 2D–3D MAE by jointly encoding image and point cloud tokens and reconstructing both modalities with modality-specific decoders. It adopts random masking for both modalities and emphasizes generic multimodal pre-training, without explicitly considering overlap-aware masking or registration-oriented correspondence learning. Overall, these methods demonstrate the effectiveness of MAE and cross-modal learning for 2D–3D representation modeling and provide valuable insights into cross-modal feature interaction. However, their architectures are primarily designed for generic representation learning or perception tasks, while their masking strategies are generally projection-based, random, or guided by fixed semantic priors. None of them explicitly optimize mask generation according to cross-modal overlap, correspondence reliability, or the registration objective. In contrast, our method introduces a dual-MAE framework tailored for image-to-point cloud registration, where image and point cloud features are jointly optimized through cross-modal reconstruction. Furthermore, the proposed Similarity-based RL Masking Strategy (SRLM) first initializes masks using bidirectional cross-modal similarity and then refines them via reinforcement learning, enabling adaptive selection of informative regions that directly benefit cross-modal correspondence learning and registration accuracy. Inter-modality Registration. Inter-modal registration presents greater challenges compared to intra-modal registration because of the significant domain discrepancies involved. Traditional approaches typically employ a detect-then-match strategy. For instance, 2D3D-Matchnet [19] uses SIFT [8] and ISS [18] to extract key points from images and point clouds, constructing patches around these key points. It then uses CNNs and PointNet [38] to extract features and build descriptors for matching. P2-Net [3] introduces a joint learning framework with a comprehensive reception mechanism, using a single forward pass to detect key locations and extract descriptors, enabling efficient matching through contrastive constraints. Unfortunately, the inefficiency of keypoint extraction in cross-modal scenarios has led to significant accuracy loss, prompting the emergence of detection-free methods. 2D3D-MATR [1] adopts a coarse-to-fine matching process, using a transformer-based approach to establish patch-level matches and then seeking fine-grained matches within them, followed by regressing rigid transformations using PNP+RANSAC [4, 5]. This detection-free method overcomes the challenge of obtaining repeatable key points and makes 2D-3D descriptors more consistent. Using transformers’ global receptive fields and multi-level methods significantly increases the inlier ratio of matches. B2-3Dnet [2] further enhance cross-modal correspondence learning by leveraging covariance-guided feature alignment to improve the robustness and consistency of descriptors. Based on this, CA-I2P [62] introduces channel adaptation and global optimal selection to better align cross-modal features and reduce redundant matches, achieving improved registration accuracy. Flow-I2P [10] improves image-to-point-cloud registration by using Beltrami flow for better manifold alignment and enhanced registration accuracy. Our method advances detection-free approaches by introducing a dual MAE framework (ID-MAE) combined with a Similarity-based RL Masking Strategy (SRLM), where the dual MAE framework aligns modalities and establishes correspondences, while SRLM selectively masks high-value regions through similarity-based initialization and RL-driven refinement, making it a state-of-the-art solution for image-to-point cloud registration. I Method I-A Overview Fig. 4: Overall pipeline of ID-MAE. After selecting the mask regions through SRLM, the point cloud MAE module constructs a loss constraint by reconstructing the point cloud from its low-resolution version. The image MAE module incorporates point cloud features to assist in reconstruction and is supervised by the original features. After post-processing, fif_i and fpf_p are output for subsequent matching. Given an image I∈ℝH×W×3I ^H× W× 3 and a point cloud P∈ℝN×3P ^N× 3 from the same scene, the objective of the registration between images and point clouds is to determine the rigid transformation [R,t][R,t] from the point cloud coordinate system to the camera coordinate system. Here, W and H are the image’s width and height, while N is the number of points. The transformation consists of a 3D rotation R∈(3)R (3) and a 3D translation vector t∈ℝ3t ^3. Our method, M2-3D, depicted in Fig. 3, adopt a dual MAE-enhanced detection-free registration paradigm, constructing the modality-aligned and corresponded image and point cloud features through the intermodal dual-MAE framework. After passing through a series of attention modules, similarity is calculated to generate a score map, enabling patch-level matches. High-resolution images and point cloud features are then used to refine dense correspondences from the patch matches. Finally, the PnP+RANSAC [4] algorithm effectively regresses the rigid transformation. I-B Intermodal Dual-MAE Framework We utilize ResNet [24] with FPN [27] and KPFCNN [28] to extract features from images and point clouds, respectively. The 2D features and 3D features are downsampled at the lowest resolution to obtain fi1∈ℝ(h×w)×cf_i1 ^(h× w)× c and fp1∈ℝn×cf_p1 ^n× c. For the extracted features at the lowest resolution fif_i and fpf_p, positional encoding is applied to enhance the features. To prepare for the subsequent MAE, we do not adopt the conventional pretraining mode of MAE, as pretraining significantly increases computational overhead and is often constrained in certain scenarios. Instead, we aim to enhance the feature extraction capability of the network encoder through unsupervised MAE during normal training. This approach facilitates modality aggregation and establishes correspondences between images and point clouds. We can observe the overall pipeline of Intermodal Dual-MAE Framework (ID-MAE) from Fig. 4. I-B1 Dual-modal MAE Enhancement Point MAE module adopts a self-supervised process, where the point cloud feature fp1f_p1 is masked based on the Similarity-based RL Masking Strategy (SRLM) (we will elaborate on this later), resulting in fpm∈ℝ(n−mp)×cf_pm ^(n-m_p)× c, where mpm_p denotes the number of masked points. For the encoder, we employ local self-attention instead of global self-attention, to construct a Transformer-based encoder. We fill the masked positions in fpmf_pm with learnable mask tokens of dimension ℝ1×cR^1× c, completing the feature to obtain fpc∈ℝn×cf_pc ^n× c with the exact dimensions as fp1f_p1. After passing through the decoder, the mask tokens learn the corresponding point cloud features, completing the point cloud reconstruction process. We utilize standard point cloud cross-attention blocks in the point cloud decoder, along with self-attention blocks. The reconstructed point cloud features fp1∈ℝn×cf_p1 ^n× c are finally mapped to the point cloud P2∈ℝn×3P_2 ^n× 3, while the features fp2f_p2 obtained from the encoder are decoded to P1∈ℝn×3P_1 ^n× 3. The two point clouds P1P_1 and P2P_2 are supervised by an L2L_2 loss. Here, n denotes the number of points in the reconstructed point cloud, and m indexes the m-th point. ℒpmae=1M∑m=1M‖P1,m−P2,m‖22,L_pmae= 1M _m=1^M\|P_1,m-P_2,m\|_2^2, (1) after filtering out some excessively small point cloud patches, we output the point cloud feature fpf_p for subsequent matching. As illustrated in Fig. 4, we adopt a KPFCNN backbone that naturally produces a hierarchical multi-scale point pyramid from the input point cloud P. The point MAE operates on the coarsest-scale level of this pyramid as its reconstruction target, while finer-scale point features are preserved for subsequent matching and registration. Therefore, the effective downsampling ratio of the low-resolution point set is implicitly determined by the backbone’s subsampling configuration, and can be reproduced by reporting the actual point counts at different hierarchy levels. Our masking strategy further selects informative regions on the coarsest-level point tokens, producing masked subsets PmP_m for the training-only MAE branch. At the dataset level, a random subsampling is optionally applied only to cap the maximum number of raw input points for efficiency; this preprocessing neither defines the low-resolution point cloud used in ID-MAE nor affects the hierarchical structure. Importantly, the low-resolution point cloud is solely used as a reconstruction target during training and is removed at inference, thus it does not participate in similarity computation, masking selection, or pose estimation. This design naturally aligns with the coarse-to-fine registration paradigm, where global structural consistency is first captured at coarse resolution and progressively refined at finer scales. For point clouds, we reconstruct them for supervision due to their sparsity, while for images, we directly supervise the features. And since the point cloud perspective contains more information, we aim to introduce cross-modal information to assist in the reconstruction of the image. I-B2 Cross-Modal Guided Reconstruction Typically, MAE encoders benefit from learning a generalized encoder capable of capturing high-dimensional data representations of both images and point clouds. Due to the differences between the two modalities, a dedicated decoder is required to decode the high-level latent data across two different modalities. The encoder part of image MAE module is similar to that of the point cloud. The image feature fi1f_i1 undergoes SRLM, resulting in fim∈ℝ(h×w−mi)×cf_im ^(h× w-m_i)× c, where mim_i denotes the number of masked patches. We leverage point cloud features to restore the masked regions in the image, facilitating the identification of corresponding structures between the two modalities and enhancing semantic alignment. A learnable mask token of dimension ℝ1×cR^1× c is introduced and duplicated to fill the missing positions in the masked image features, yielding a complete feature map fic∈ℝ(h×w)×cf_ic ^(h× w)× c. The feature ficf_ic is used as the query (Q), while the point cloud representation fp1f_p1 serves as both the key (K) and value (V) in the cross-attention mechanism to reconstruct image features. These components are jointly fed into the decoder to model inter-modal dependencies and produce the updated and restored feature fir∈ℝ(h×w)×cf_ir ^(h× w)× c. This decoding process enhances alignment and improves reconstruction accuracy. The restored feature firf_ir is obtained by aggregating relevant point cloud features through a weighted summation based on feature similarity, as defined below: fir=∑b∈fp1(a)esab∑l∈fp1(a)esalficb,f_ir= _b _f_p1(a) e^s_ab _l _f_p1(a)e^s_alf_ic^b, (2) where fp1(a)N_f_p1(a) denotes the neighborhood of the a-th feature in fp1f_p1 associated with ficf_ic. The similarity sabs_ab between fp1af_p1^a and ficbf_ic^b is computed as: sab=fp1a⊤Wficb,s_ab=f_p1^a Wf_ic^b, (3) where W is a learnable parameter matrix. This formula calculates the similarity between ficf_ic and fp1f_p1 features, applies a weighted summation over the neighboring image features fp1f_p1, and generates the reconstructed target feature firf_ir. This process effectively fuses the point cloud and image features, providing the necessary support for subsequent decoding. The fi1f_i1 feature is passed through the encoder to obtain fi2∈ℝ(h×w)×cf_i2 ^(h× w)× c, which is constrained during the reconstruction process. N represents the number of patches extracted from the image, corresponding to the total number of feature tokens in the first dimension of fi2f_i2. The n-th patch refers to a specific patch among these N patches. ℒimae=1N∑n=1N(fi2,n−fir,n)2,L_imae= 1N _n=1^N (f_i2,n-f_ir,n )^2, (4) a lightweight three-stage CNN [64] is used to extract image features at three different scales, capturing multi-level spatial information. The resulting features form a feature pyramid denoted as fif_i, which serves as input for subsequent process. Discussion. Why do we choose to reconstruct image features using point cloud features rather than the other way around? The reason lies in the fact that point clouds and images do not correspond exactly. Compared to images, point clouds provide a broader scene view and richer information. Thus, reconstructing point cloud features from images may suffer from information loss, degrading reconstruction quality and constraints. Moreover, restricting point cloud features to the image range requires ground truth R,tR,t, which we aim to predict, making this approach unreasonable. Fig. 5: Overall pipeline of SRLM, where similarity-based masking selection is used for prior initialization, followed by RL-driven refinement. I-C Similarity-based RL Masking Strategy In image-to-point-cloud registration, due to the existence of non-overlapping regions in the image and point cloud views, when image or point cloud patches fall within these areas, it can lead to completely incorrect matches, making these computations meaningless. Additionally, since the approach follows a coarse-to-fine paradigm, this can result in large misalignments during fine-grained matching, which is undesirable. Therefore, we aim to increase the likelihood of selecting regions within the overlapping areas through similarity-based selection. However, due to domain differences in image and point cloud features, areas with high similarity cannot be directly considered as critical matching regions. Since the task of selecting key regions is inherently non-differentiable, we aim to address this challenge by employing a reinforcement learning strategy, allowing us to effectively solve the problem. Thus to improve cross-modal learning, we propose the Similarity-based RL Masking Strategy (SRLM), which masks regions of critical patches. SRLM comprises a Similarity-based Masking Selection (SMS) and RL-driven Refinement (RLR). The overall process of SDMS is presented in Fig. 5. I-C1 Similarity-based Masking Selection The camera’s limited field of view may miss discriminative areas for registration. Expanding the masked regions increases computational costs and degrades point cloud reconstruction. Observing the matching process, we find that correspondences may rely on high similarity between image and point cloud features, which should be masked as initial state to enhance the model’s perception. We compute the similarity maps: the Image-to-Point Map (I2P Map) and the Point-to-Image Map (P2I Map). By setting similarity thresholds and performing top-k selection, we determine the positions to mask for both images and point clouds, denoted as image mask and point mask, respectively. Using these guided positions, we apply Mask-Guided Selection, masking the corresponding areas to obtain ImI_m and PmP_m. Through the above design, we roughly determine the initial positions of the image-point cloud mask blocks, which will be used as the initial state input in the subsequent reinforcement learning process. For features fim∈ℝ(h×w−mi)×cf_im ^(h× w-m_i)× c and fpm∈ℝ(n−mp)×cf_pm ^(n-m_p)× c, mim_i and mpm_p represent the number of masked regions in the image and point cloud, respectively. While similarity evaluation is indeed widely used in registration tasks, it is typically applied at the correspondence or matching stage. In contrast, our similarity-aware masking strategy leverages such cues during representation learning, guiding the MAE to focus on informative cross-modal regions before correspondence estimation. This shift in usage is crucial: rather than directly establishing correspondences, similarity is used to prioritize which regions should be preserved or reconstructed during masked modeling, enabling the MAE to learn more correspondence-aware representations. However, due to repetitive object textures in the scene and domain gaps in the initialization between images and point clouds, a similarity-only selection strategy may not be sufficient to fully unleash the potential of our bidirectional MAE architecture. I-C2 RL-driven Refinement To enhance the accuracy of image-to-point cloud matching, we introduce a reinforcement learning (RL)-based strategy optimization to refine the selection of masked regions (as shown in Algorithm 1). Instead of relying on fixed or heuristic masking schemes, the proposed RL framework adaptively adjusts which regions to mask or preserve during training, enabling the model to focus on more informative and reliable cross-modal areas. The RL component is designed as an auxiliary optimization mechanism that operates alongside the MAE framework, rather than a standalone decision-making module. By observing the current similarity-aware state, the policy network dynamically refines the selection of masked regions, thereby reducing masking randomness and stabilizing MAE optimization. A task-aligned reward function derived from the matching loss guides policy updates, ensuring that the learned masking strategy consistently supports the main registration objective. Through this design, the RL-based masking strategy facilitates more reliable feature learning for image-to-point cloud matching while avoiding additional inference overhead, as it is only applied during training. The initial state in the RL process is determined by the similarity obtained from SMS. The reward function plays a critical role in this process, as it guides the optimization of the policy. In our approach, the reward is computed based on LiL_i (described in Equation 11 in I-D), where the circle loss effectively constrains overall matching accuracy, making it an ideal candidate for the reward. Specifically, since better matching performance results in smaller values for LiL_i, the reward increases as matching accuracy improves. Thus, the reward is defined as: R=1Li+δ,R= 1L_i+δ, (5) where δ is a small constant to prevent division by zero. The probabilities of selecting image and point cloud masks, denoted as ProbiProb_i and ProbpProb_p, are computed based on the output of the policy network for both regions. These probabilities reflect the network’s confidence in selecting each region and are calculated as follows: Probi=log(πθ(aimask|simage)),Prob_i= ( _θ(a_imask|s_image)), (6) Probp=log(πθ(apmask|spoint)),Prob_p= ( _θ(a_pmask|s_point)), (7) where πθ(aimask|simage) _θ(a_imask|s_image) and πθ(apmask|spoint) _θ(a_pmask|s_point) represent the probabilities output by the policy network for the image and point cloud mask selections, respectively. The overall log-probability, logprob, is computed by averaging the log-probabilities of both modalities as follows: logprob=12log[1+(1h⋅w∑i=1h⋅wProbi)⋅(1n∑j=1nProbp)], ^prob= 12 [1+ ( 1h· w _i=1^h· wProb_i )· ( 1n _j=1^nProb_p ) ], (8) where h⋅wh· w and n represent the number of elements in the image and point cloud regions, respectively. This averaging process ensures that both image and point cloud regions contribute equally to the overall decision-making process by combining their confidence levels. In RL, the action corresponds to selecting the masked regions based on the current state. In our approach, actions involve determining which regions should be masked and which should remain visible. The action space consists of all possible combinations of masked regions, and the policy network learns to select the optimal regions by maximizing the reward. The network computes the log-probability of the selected action, reflecting its confidence in the chosen masked regions. Specifically, the gradient of the policy is estimated as: ∇θJ(θ)=t[∇logπθ(at|st)⋅Rt],∇θ J(θ)=E_t [∇ _θ(a_t|s_t)· R_t ], (9) where πθ(at|st) _θ(a_t|s_t) represents the probability of selecting action ata_t given state sts_t, parameterized by θ. The log-probability ∇logπθ(at|st)∇ _θ(a_t|s_t) is used to compute the gradient for policy update, while RtR_t is the reward associated with the action taken at time t, guiding the policy towards optimal actions. The reinforcement loss LtL_t is then computed as: Lt=−logprob×R.L_t=-log^prob× R. (10) This policy loss is added to the main loss, steering the optimization of the network parameters. By maximizing the reward and minimizing the total loss, the policy network gradually learns to select the best masked regions, ultimately improving the matching accuracy between the image and point cloud. This design helps stabilize the MAE training process by reducing randomness in mask selection and alleviating sensitivity to hyperparameters. The resulting adaptive masking strategy facilitates more reliable image-to-point cloud matching and improves robustness in practice. Algorithm 1 RL-driven Refinement for Mask Region Selection Input: Image feature IF_I, point-cloud feature PF_P, SMS, policy network πθ _θ, MAE network ℳM, matching network G, and constant δ. Output: Image mask aimaska_imask and point-cloud mask apmaska_pmask. Similarity-aware initialization: (simage,spoint)←SMS(I,P).(s_image,s_point) (F_I,F_P). Policy-based mask refinement: • Predict mask-selection distributions: πθ(aimask∣simage),πθ(apmask∣spoint). _θ(a_imask s_image), 17.00024pt _θ(a_pmask s_point). • Sample masking actions: aimask∼πθ(⋅∣simage),apmask∼πθ(⋅∣spoint).a_imask _θ(· s_image), 17.00024pta_pmask _θ(· s_point). • Apply the masks: ~I←Mask(I,aimask),~P←Mask(P,apmask). F_I (F_I,a_imask), 17.00024pt F_P (F_P,a_pmask). Reconstruction and matching: • Reconstruct the masked features: (Ir,Pr)←ℳ(~I,~P).(F_I^r,F_P^r) ( F_I, F_P). • Estimate correspondences and matching loss: ←(Ir,Pr),Li←CircleLoss().C (F_I^r,F_P^r), 17.00024ptL_i (C). Reward: R←1Li+δ.R← 1L_i+δ. Policy probability: • Compute modal log-probabilities: Probi Prob_i ←logπθ(aimask∣simage), ← _θ (a_imask s_image ), Probp Prob_p ←logπθ(apmask∣spoint). ← _θ (a_pmask s_point ). • Aggregate the two modalities: logprob←12log[1+(1hw∑i=1hwProbi)(1n∑j=1nProbp)]. ^prob← 12 \! [1+ ( 1hw _i=1^hwProb_i ) ( 1n _j=1^nProb_p ) ]. Policy optimization: • Compute the policy and total losses: Lt←−logprobR,Ltotal←Lmain+λtLt.L_t←- ^probR, 17.00024ptL_total← L_main+ _tL_t. • Update the policy and main-network parameters: θ←θ−ηθ∇θLt,ϕ←ϕ−ηϕ∇ϕLtotal.θ←θ- _θ _θL_t, 17.00024ptφ←φ- _φ _φL_total. Return: aimaska_imask and apmaska_pmask. Training stability discussion. Although the RL module is tightly coupled with the registration task, several design choices are adopted to ensure stable optimization. First, the RL policy is not activated from the beginning of training. As shown in our implementation, the network is first trained with MAE-based reconstruction losses during a warm-up phase (set for the first 20 epochs), allowing cross-modal representations to stabilize before policy learning is introduced. This significantly reduces the variance and sensitivity of subsequent policy updates. Second, the reward signal is dense and continuous, defined as the inverse of the main task loss at each iteration. Unlike sparse or delayed rewards, this design provides immediate and consistent feedback to the policy, effectively mitigating reward sparsity. Third, the policy loss is incorporated as a lightweight regularization term with a small weighting factor, ensuring that the primary optimization objective remains dominated by the registration loss. As a result, the RL component softly biases the masking strategy rather than aggressively steering the feature-learning process, reducing the risk of co-adaptation or training collapse. To further improve stability, the policy employs factorized Bernoulli sampling at the token level, and the log-probabilities of mask selections are averaged across multiple token-wise decisions. This design significantly reduces policy gradient variance compared to combinatorial masking actions. Finally, the policy state is initialized and constrained by similarity-based priors, limiting exploration to informative regions and further mitigating co-adaptation with the main network. As a result, the RL component serves as a lightweight refinement mechanism rather than a dominant optimization driver. I-D Model Training & Inference Our training consists of two stages: first, we train the point MAE module, followed by the image MAE module. As image feature reconstruction depends on point cloud features, we prioritize training the point MAE module to ensure high-quality feature extraction. Our ID-MAE is employed solely during training and omitted at inference. To obtain the total loss, first let us examine the loss functions for the coarse and fine matching networks. Both ℒcoarseL_coarse and ℒfineL_fine use the general circle loss [45, 44]. Given an anchor descriptor did_i, the descriptors of its positive and negative pairs are iPD^P_i and iND^N_i, respectively: ℒi=1γlog[1+(∑dj∈iPeβpi,j(dij−Δp))⋅(∑dk∈iNeβni,k(Δn−dik))], _i= 1γ [1+ ( _d^j ^P_ie^β^i,j_p(d^j_i- _p) )· ( _d^k ^N_ie^β^i,k_n( _n-d^k_i) ) ], (11) where dijd^j_i is the L2L_2 feature distance, βpi,j=γλpi,j(dij−Δp)β^i,j_p=γλ^i,j_p(d^j_i- _p) and βni,k=γλni,k(Δn−dik)β^i,k_n=γλ^i,k_n( _n-d^k_i) are the individual weights for the positive and negative pairs, with λpi,jλ^i,j_p and λni,kλ^i,k_n as scaling factors [65]. Combining the discussions above, the total loss is composed of three key components: the MAE loss for image and point cloud (ℒimaeL_imae, ℒpmaeL_pmae), the aggregation loss ℒtL_t from the SRLM module, and the matching loss ℒdL_d (including ℒcoarseL_coarse and ℒfineL_fine) from the matching process. The total loss is computed as: ℒtotal=γ1ℒimae+γ2ℒpmae+γ3ℒt+γ4ℒd,L_total= _1L_imae+ _2L_pmae+ _3L_t+ _4L_d, (12) where γi _i are hyperparameters balancing the contribution of different loss terms. IV Experiments IV-A Datasets and Implementation Details Based on the 2D3D-MATR benchmark, we conducted extensive experiments and ablation studies on two challenging benchmarks: RGB-D Scenes v2 [20] and 7Scenes [21]. Dataset. RGB-D Scenes Dataset v2 consists of 14 scenes containing furniture (chair, coffee table, sofa, table) and a subset of the objects in the RGB-D Object Dataset (bowls, caps, cereal boxes, coffee mugs, and soda cans). For each scene, we create point cloud fragments from every 25 consecutive depth frames and sample one RGB image per 25 frames. We select image-point-cloud pairs with an overlap ratio of at least 30%. Scenes 1-8 are used for training, 9-10 for validation, and 11-14 for testing, resulting in 1,748 training pairs, 236 validation pairs, and 497 testing pairs. The 7-Scenes dataset is a collection of tracked RGB-D camera frames. All 7 indoor scenes were recorded from a handheld Kinect RGB-D camera at 640×480 resolution. We select image to point-cloud pairs from each scene with at least 50% overlap, adhering to the official sequence split for training, validation, and testing. This results in 4,048 training pairs, 1,011 validation pairs, and 2,304 testing pairs. The KITTI Odometry dataset contains 22 image and point cloud sequences, with 11 sequences providing ground-truth calibration. We use sequences 00–08 for training and 09–10 for testing. To simulate mis-registration, a 2D translation within ±10± 10 m and an unconstrained rotation around the up-axis are applied. Images are downsampled to 160×512160× 512 and point clouds to 40,960 points. T-SNE & MMD. t-SNE [63] is a dimensionality reduction technique that projects high-dimensional data into 2D or 3D while preserving local structure, making it useful for visualizing feature distributions. MMD is a non-parametric metric that measures the difference between two distributions in a kernel-based feature space. Both are commonly used in tasks such as domain adaptation and cross-modal representation analysis. Implementation Details. We use an NVIDIA Geforce RTX 3090 GPU for training. The entire pipeline is implemented using PyTorch. In the image decoder, the cross-attention module is applied once, while the self-attention module is applied three times. The threshold for the I2P map is set to 0.5, with k1=35k_1=35 in the top-k selection. Similarly, the threshold for the P2I map is set to 0.5, with k2=15k_2=15 in the top-k selection. For loss γ1=γ2=γ3=γ4=1 _1= _2= _3= _4=1. TABLE I: Evaluation results on RGB-D Scenes v2. Teal numbers highlight the best, the second best are Boldfaced and the baseline are underlined. Model Scene.11 Scene.12 Scene.13 Scene.14 Mean Mean depth (m) 1.74 1.66 1.18 1.39 1.49 Inlier Ratio ↑ FCGF-2D3D [67] 6.8 8.5 11.8 5.4 8.1 P2-Net [3] 9.7 12.8 17.0 9.3 12.2 Predator-2D3D [68] 17.7 19.4 17.2 8.4 15.7 2D3D-MATR [1] 32.8 34.4 39.2 23.3 32.4 FreeReg [55] 36.6 34.5 34.2 18.2 30.9 B2-3Dnet [2] 36.4 32.7 43.8 27.4 35.1 CA-I2P [62] 38.6 40.6 38.9 24.0 35.5 Flow-I2P [10] 49.6 44.0 36.5 30.4 40.1 M23D (ours) 48.1 48.4 46.3 31.4 43.5 Feature Matching Recall ↑ FCGF-2D3D [67] 11.1 30.4 51.5 15.5 27.1 P2-Net [3] 48.6 65.7 82.5 41.6 59.6 Predator-2D3D [68] 86.1 89.2 63.9 24.3 65.9 2D3D-MATR [1] 98.6 98.0 88.7 77.9 90.8 FreeReg [55] 91.9 93.4 93.1 49.6 82.0 B2-3Dnet [2] 100.0 99.0 92.8 85.8 94.4 CA-I2P [62] 100.0 100.0 91.8 82.7 93.6 Flow-I2P [10] 100.0 100.0 94.5 78.7 93.3 M23D (ours) 100.0 100.0 92.8 83.6 94.1 Registration Recall ↑ FCGF-2D3D [67] 26.5 41.2 37.1 16.8 30.4 P2-Net [3] 40.3 40.2 41.2 31.9 38.4 Predator-2D3D [68] 44.4 41.2 21.6 13.7 30.2 2D3D-MATR [1] 63.9 53.9 58.8 49.1 56.4 FreeReg+Kabsch [55] 38.7 51.6 30.7 15.5 34.1 FreeReg+PnP [55] 74.2 72.5 54.5 27.9 57.3 B2-3Dnet [2] 58.3 60.8 74.2 60.2 63.4 CA-I2P [62] 68.1 73.5 63.9 47.8 63.3 Flow-I2P [10] 90.0 65.9 54.8 63.0 68.4 M23D (ours) 88.9 76.5 83.5 63.3 78.0 TABLE I: Evaluation results on 7Scenes. Teal numbers highlight the best, the second best are Boldfaced and the baseline are underlined. Model Chs Fr Hds Off Pmp Kit Strs Mean Mean depth(m) 1.78 1.55 0.80 2.03 2.25 2.13 1.84 1.77 Inlier Ratio ↑ FCGF-2D3D [67] 34.2 32.8 14.8 26 23.3 22.5 6.0 22.8 P2-Net [3] 55.2 46.7 13.0 36.2 32.0 32.8 5.8 31.7 Predator-2D3D [68] 34.7 33.8 16.6 25.9 23.1 22.2 7.5 23.4 2D3D-MATR [1] 72.1 66.0 31.3 60.7 50.2 52.5 18.1 50.1 B2-3Dnet [2] 73.8 66.7 33.1 61.7 50.8 52.3 18.1 50.9 CA-I2P [62] 73.6 66.4 34.5 62.4 52.1 52.8 19.1 51.6 Flow-I2P [10] 76.6 64.7 37.1 62.0 52.3 52.8 18.5 52.0 M23D(ours) 75.0 68.3 37.7 65.5 53.1 55.2 18.7 53.4 Feature Matching Recall ↑ FCGF-2D3D [67] 99.7 98.2 69.9 97.1 83.0 87.7 16.2 78.8 P2-Net [3] 100.0 99.3 58.9 99.1 87.2 92.2 16.2 79 Predator-2D3D [68] 91.3 95.1 76.6 88.6 79.2 80.6 31.1 77.5 2D3D-MATR [1] 100.0 99.6 98.6 100.0 92.4 95.9 58.2 92.1 B2-3Dnet [2] 100.0 100.0 98.6 100.0 92.7 95.6 64.9 93.1 CA-I2P [62] 100.0 100.0 98.6 100.0 92.0 95.5 60.8 92.4 Flow-I2P [10] 100.0 99.7 95.1 99.9 93.1 96.8 56.7 91.6 M23D(ours) 100.0 100.0 100.0 100.0 93.8 96.0 59.5 92.8 Registration Recall ↑ FCGF-2D3D [67] 89.5 79.7 19.2 85.9 69.4 79.0 6.8 61.4 P2-Net [3] 96.9 86.5 20.5 91.7 75.3 85.2 4.1 65.7 Predator-2D3D [68] 69.6 60.7 17.8 62.9 56.2 62.6 9.5 48.5 2D3D-MATR [1] 96.9 90.7 52.1 95.5 80.9 86.1 28.4 75.8 B2-3Dnet [2] 98.3 90.5 56.2 96.4 84.0 86.1 32.4 77.7 CA-I2P [62] 99.0 90.7 68.5 96.2 83.0 88.1 31.1 79.5 Flow-I2P [10] 98.8 90.0 58.4 93.9 82.1 88.6 37.6 78.4 M23D(ours) 97.6 95.6 67.1 98.7 84.4 89.0 35.1 81.3 Metrics. We evaluate our method using several key metrics: Inlier Ratio (IR), Feature Matching Recall (FMR), and Registration Recall (R). Inlier Ratio (IR) quantifies the proportion of inliers among all putative pixel-point correspondences. A correspondence is deemed an inlier if its 3D distance is less than a threshold τ1=5 cm _1=5 cm under the ground-truth transformation →ℐ∗T^*_P : IR=1|C|∑(xi,yi)∈C[‖→ℐ∗(xi)−1(yi)‖2<τ1].IR= 1|C| _(x_i,y_i)∈ C [ \|T^*_P (x_i)-K^-1(y_i) \|_2< _1 ]. (13) Here, [⋅][·] denotes the Iverson bracket, xi∈x_i , and yi∈⊆ℐy_i are pixel coordinates. The function −1K^-1 projects a pixel to a 3D point based on its depth value. Feature Matching Recall (FMR) represents the fraction of image-point-cloud pairs with an IR above a threshold τ2=0.1 _2=0.1. It measures the likelihood of successful registration: FMR=1M∑i=1M[IRi>τ2],FMR= 1M _i=1^M [IR_i> _2 ], (14) where M is the total number of image-point-cloud pairs. Registration Recall (R) measures the fraction of image-point-cloud pairs that are correctly registered. A pair is correctly registered if the root mean square error (RMSE) between the ground-truth-transformed and predicted point clouds →ℐT_P is less than τ3=0.1 m _3=0.1 m: RMSE=1||∑pi∈‖→ℐ(pi)−→ℐ∗(pi)‖22,RMSE= 1|P| _p_i \|T_P (p_i)-T^*_P (p_i) \|_2^2, (15) R=1M∑i=1M[RMSEi<τ3].R= 1M _i=1^M [RMSE_i< _3 ]. (16) TABLE I: Evaluation Results on KITTI Dataset. Teal numbers highlight the best. Method Type RTE(m) ↓ RRE (∘) ↓ vpc + GeoTransformer[17] Point-to-Point 4.27 ± 7.14 8.67 ± 8.55 vpc + Hunter Point-to-Point 4.59 ± 5.22 6.23 ± 5.17 DeepI2P(2D) [42] Image-to-Point 5.15 ± 7.35 9.14 ± 8.02 CorrI2P [41] Image-to-Point 4.24 ± 7.26 6.47 ± 5.20 VP2P [56] Image-to-Point 2.05 ± 3.23 4.01 ± 6.37 2D3D-MATR [1] Image-to-Point 1.86 ± 3.79 2.59 ± 4.46 RetrI2P [53] Image-to-Point 1.61 ± 2.39 3.16 ± 2.85 FreeReg [55] Image-to-Point 1.78 ± 1.76 2.89 ± 4.47 CFI2P [54] Image-to-Point 1.95 ± 2.97 2.63 ± 3.19 M23D (Ours) Image-to-Point 1.58 ± 2.33 2.34 ± 3.29 TABLE IV: Ablation studies on 7Scenes. Teal numbers highlight the best. Model Image-MAE Point-MAE Cross Modal IR FMR R BL 50.1 92.1 75.8 F1 ✓ 49.8 91.2 78.0 F2 ✓ 52.2 92.4 78.3 F3 ✓ ✓ 50.4 92.0 80.4 Full ✓ ✓ ✓ 53.4 92.8 81.3 These metrics provide a comprehensive evaluation of the model’s capability to accurately match features and register image-point-cloud pairs, ensuring robust alignment and effective correspondence. IV-B Evaluations on Dataset We compare our approach with 2D3D-MATR [1] and other baseline methods [67, 3, 68, 55, 2, 62, 10] on the RGB-D Scenes v2 dataset, as shown in Table I. Our method integrates the ID-MAE framework to enhance feature extraction and better align the image and point cloud modalities. The introduction of the SRLM strategy further refines the model by adaptively focusing on semantically meaningful regions while suppressing noise from irrelevant areas. These improvements lead to a substantial increase in performance across all evaluation metrics. Specifically, our method achieves a 11.1 percentage point gain in inlier ratio (from 32.4 to 43.5), a 3.3 percentage point improvement in feature matching recall (from 90.8 to 94.1), and a 21.6 percentage point increase in registration recall (from 56.4 to 78.0) compared to the baseline method. Notably, our method sets a new state-of-the-art in registration recall, with a remarkable improvement of 7.8 percentage points, demonstrating its robustness in accurately estimating correspondences across different modalities. Fig. 6: The visualization results of M23D. To rigorously analyze the performance, we set the error threshold to a strict 50px. TABLE V: Ablation studies of different masking strategies on RGB-D Scenes v2. Teal numbers highlight the best. Method Random Similarity RL SRLM GFLOPs IR FMR R M1 388.9 32.5 91.0 56.4 M2 ✓ 423.8 42.9 93.2 63.5 M3 ✓ 457.2 37.1 92.9 67.2 M4 ✓ 834.8 42.7 93.4 73.1 M5 ✓ 535.1 43.5 94.1 78.0 Compared to RGB-D Scenes v2, 7-Scenes exhibit greater scale variations, yet our method still outperforms others, as shown in Table I. In the Inlier Ratio metric, we achieve 53.4%, surpassing 2D3D-MATR by 3.3 percentage points (50.1%). For Feature Matching Recall, our method reaches 92.8%, demonstrating robustness in correspondence establishment, outperforming 2D3D-MATR by 0.7 percentage points (92.1%). In Registration Recall, we achieve 81.3%, maintaining an advantage of 5.5 percentage points over 2D3D-MATR’s 75.8%. Notably, 2D3D-MATR shows improvements in the challenging Heads and Pumpkin scenes, with the Heads scene involving small errors due to texture-less areas, and the Pumpkin scene containing repetitive patterns. Our ID-MAE effectively addresses these challenges, ensuring superior performance across all scenes. Although B2-3Dnet achieves slightly better performance in FMR due to its targeted modeling of image features, our method offers a greater overall advantage when considering the comprehensive results. Fig. 7: Ablation studies on the hyperparameters. To better evaluate our performance, we conducted experiments on the outdoor dataset KITTI to verify the robustness of our method. As shown in Table I, we evaluate our method, M23D, on the KITTI dataset and compare it with several state-of-the-art techniques. Our evaluation is based on two key metrics: Reprojection Error (RTE) and Rotation Error (RRE), both of which are essential for assessing the accuracy of 3D point cloud registration. Our method outperforms all other approaches in both RTE and RRE. Specifically, M23D achieves a significantly lower RTE of 1.58 ± 2.33 meters and a lower RRE of 2.34 ± 3.29 degrees. This surpasses other image-to-point methods, such as RetrI2P (RTE: 1.61 ± 2.39 meters, RRE: 3.16 ± 2.85 degrees) and VP2P (RTE: 2.05 ± 3.23 meters, RRE: 4.01 ± 6.37 degrees). These results demonstrate that our method provides more accurate alignment with lower error margins compared to previous works. The results on the KITTI dataset confirm the robustness and superiority of M23D over existing methods, highlighting its potential for real-world applications that require high-precision registration. IV-C Ablation Studies We perform ablation studies on the 7Scenes dataset to evaluate the contributions of different components in our model, as shown in Table IV. Compared to the baseline (BL), adding Image-MAE (F1) or Point-MAE (F2) slightly improves the performance. The Dual MAE (F3) leads to further improvement. After introducing Cross-Modal alignment (Full), the model achieves the best results, demonstrating the effectiveness of our full approach. Fig. 8: The visualization of the point cloud projection onto the image demonstrates that our method achieves precise rigid transformation, maintaining alignment without significant discrepancies. We conduct ablation studies to evaluate the impact of different masking strategies on the RGB-D Scenes v2 dataset, as shown in Table V. Compared to the baseline, random masking (M2) improves performance; however, due to its lack of targeted selection, the gains remain limited. Similarity-based masking (M3) further improves the results with only a slight increase in computational cost. Although directly applying reinforcement learning (M4) achieves strong performance, it incurs a significant increase in computation. In contrast, our SRLM strategy (M5) achieves the highest performance while maintaining good efficiency, demonstrating the superiority of our design in balancing accuracy and computational cost. Our strategy is specifically designed to adapt to the MAE framework. Therefore, the effectiveness of the reinforcement learning (RL) component serves as evidence of the strong capability of our MAE-based design. Previous common masking strategies often limit the full potential of the MAE module, whereas our approach enables more effective utilization of its representation learning capacity. The inter-modal MAE serves as the core representation learning mechanism, enforcing cross-modal consistency through reconstruction and establishing a strong feature foundation for 2D–3D correspondence estimation. The RL module is introduced solely to optimize the mask selection process, addressing the non-differentiability of discrete masking and improving training efficiency, rather than replacing or weakening the role of MAE. Importantly, MAE-based supervision alone already brings consistent performance gains compared to the baseline. The RL strategy further refines the masking process by prioritizing informative regions, leading to additional improvements on top of the MAE framework. While similarity evaluation is widely used in registration tasks, it is typically applied at the correspondence or matching stage. In contrast, our similarity-aware masking strategy leverages these cues during the representation learning phase. By guiding the MAE to focus on informative cross-modal regions before correspondence estimation, we shift the usage of similarity. Rather than directly establishing correspondences, similarity is used to prioritize which regions should be preserved or reconstructed during the masked modeling process. This allows the MAE to learn more correspondence-aware representations early in the process, setting the stage for more effective matching and correspondence estimation later. TABLE VI: Efficiency comparison of different methods. Model 2D3D-MATR Flow-I2P Ours FPS ↑ 7.852 6.061 7.684 GFLOPs ↓ 388.2 525.1 535.1 Memory (GB) ↓ 5.637 2.844 5.423 Param (M) ↓ 28.2 28.7 28.6 Fig. 7 reports ablation studies on the key hyperparameters in our framework, where the registration recall (R) is evaluated under different parameter settings. As shown in Fig. 7(a)–(c), the performance consistently improves when increasing the corresponding hyperparameters from small values, and reaches a relatively stable region around the default setting (e.g., γ1=γ2=γ3=1 _1= _2= _3=1). Further increasing these values leads to marginal performance fluctuations or slight degradation, indicating that overly strong weighting may introduce redundancy or reduce robustness. Fig. 7(d) exhibits a similar trend, where moderate values yield the best performance, while extreme settings result in noticeable drops. Overall, these results demonstrate that the proposed method is not overly sensitive to hyperparameter choices and achieves stable performance within a reasonable range, validating the robustness of our design and the practicality of the selected default parameters. Table VI compares the efficiency of different detection-free image-to-point cloud registration methods in terms of inference speed, computational cost, memory usage, and model size. Our method achieves an inference speed of 7.684 FPS, which is comparable to 2D3D-MATR (7.852 FPS) and faster than Flow-I2P (6.061 FPS), indicating that no additional inference latency is introduced. Although our approach requires slightly higher computational cost (535.1 GFLOPs) than 2D3D-MATR, this overhead mainly arises from similarity computation and does not affect inference efficiency, since the proposed intermodal MAE and RL-based masking strategy are applied only during training. Moreover, our method maintains a comparable memory footprint and parameter count (28.6M), demonstrating that the improved registration performance is achieved without significantly increasing model complexity. Compared to our method, Flow-I2P exhibits better performance in terms of GFLOPs and memory usage, but lags slightly in FPS. Overall, the results suggest that our approach strikes a favorable balance between accuracy and efficiency, making it suitable for practical deployment. Fig. 9: Visualization of image and point cloud corresponding regions selected by reinforcement learning. The selected regions are highlighted in yellow in the image and in blue in the point cloud. IV-D Visualization We conduct extensive experiments to validate the effectiveness of our model and visualize M23D’s matching performance for a more intuitive understanding. As shown in Fig. 6, we select seven scene pairs from the 7Scenes and RGB-D Scenes v2 datasets. In each visualization, the top, middle, and bottom rows correspond to M23D, M23D without SRLM, and the baseline model, respectively. We consider matches with a projection distance below 50 pixels as correct (green lines), and others as incorrect (red lines), using this threshold for visualization. The results show that applying ID-MAE improves matching, with more green lines indicating better correspondence. However, random masking still leads to some mismatches (red lines). With SRLM, matching performance improves significantly, demonstrating the effectiveness of our approach in enhancing both accuracy and robustness for image-to-point cloud registration. In Fig. 8, we project the point cloud onto the image using the obtained rigid transformation. It can be seen that the structure aligns well, with no large-scale misalignments. Especially in the Heads scene, which lacks sufficient texture information, the registration results are still maintained effectively. The visualization in Fig. 9 shows the image and point cloud regions selected by the proposed reinforcement learning-based masking strategy during training. The highlighted yellow regions in the image and blue regions in the point cloud indicate the areas considered informative for cross-modal matching. It can be observed that these selected regions tend to concentrate on areas with high texture and prominent structures. This is likely because these regions can provide high-quality features, and applying the Masked Auto-Encoder (MAE) to these regions can significantly improve their representation learning, thereby enhancing the overall registration accuracy. The selective masking strategy focuses the model’s attention on the most informative parts of the input data, allowing it to learn robust cross-modal correspondences from these salient regions. By emphasizing the feature-rich areas, the model can better capture the intricate geometric and visual cues needed for accurate image-to-point cloud registration, leading to the improved performance demonstrated by the method. This approach is a key innovation of the proposed technique, as it enables efficient and effective learning of the cross-modal mapping without requiring the model to process the entire input indiscriminately. The targeted focus on the most discriminative regions contributes to the overall efficiency and accuracy of the registration pipeline. It can be observed that the selected regions in the image and the point cloud do not always exhibit a strict one-to-one correspondence. This behavior is expected, as the masking strategy operates at the feature and region level, aiming to preserve discriminative semantic and geometric regions rather than enforcing precise pixel-to-point alignment. Moreover, differences in viewpoint, sensing modality, and partial visibility naturally lead to asymmetric region selection between the image and point cloud domains. TABLE VII: Generalization ability of SRLM across different datasets. Training Dataset Testing Dataset Mask Strategy IR R 7-Scenes RGB-D Scenes v2 Random Mask 18.1 23.9 7-Scenes RGB-D Scenes v2 SRLM 18.7 24.3 7-Scenes ScanNet Random Mask 15.9 24.1 7-Scenes ScanNet SRLM 16.2 28.7 These visualizations also reveal several limitations of the proposed method. First, similar to most learning-based image-to-point cloud registration approaches, in low-overlap scenarios (e.g., the bottom-right example in Fig. 9), reliable cross-modal cues are inherently scarce, and the selected regions tend to concentrate on limited areas, which weakens the overall discriminative capability of the masking strategy. This phenomenon stems from the intrinsic ambiguity of correspondence supervision under minimal overlap and remains a common challenge in the field. Second, the proposed similarity-aware masking strategy relies on an initial cross-modal similarity map to guide informative region selection. While this design improves training stability and efficiency, its effectiveness may degrade when the initial similarity estimation is severely affected by noise or large viewpoint variations. We view this dependency as a reasonable trade-off, and consider improving the robustness of initial similarity estimation as a promising direction for future work. Finally, regarding scalability to large-scale scenes, the current implementation follows standard detection-free registration pipelines and is primarily evaluated on indoor and moderate-scale outdoor datasets. Although the proposed MAE and RL-based masking strategy are applied only during training and introduce no additional inference overhead, extending the framework to larger-scale environments may require hierarchical processing or more memory-efficient similarity computation strategies. To further evaluate the cross-dataset generalization ability of SRLM, we train the model on 7-Scenes and directly evaluate it on RGB-D Scenes v2 and ScanNet (see in Table VII). Compared with the random masking strategy, SRLM consistently improves both the Inlier Ratio (IR) and Registration Recall (R) across the two target datasets. Specifically, on RGB-D Scenes v2, SRLM increases IR from 18.1% to 18.7% and R from 23.9% to 24.3%. On ScanNet, SRLM improves IR from 15.9% to 16.2%, while achieving a more substantial R improvement from 24.1% to 28.7%, corresponding to a gain of 4.6 percentage points. These consistent improvements indicate that SRLM does not merely overfit the scene distribution of the training dataset. Instead, similarity-based initialization helps suppress irrelevant and non-overlapping regions, while RL-driven refinement adaptively identifies registration-informative regions according to the task-aligned matching reward. Notably, the larger improvement in R than in IR on ScanNet suggests that SRLM enhances not only the proportion of correct correspondences, but also their geometric reliability for pose estimation. Overall, these results demonstrate that SRLM learns a more transferable masking policy and improves registration robustness under considerable cross-dataset domain shifts. Overall, these analyses demonstrate both the effectiveness and the current limitations of the proposed approach in cross-modal region selection and correspondence modeling. Future work will focus on more robust cross-modal cue modeling under low-overlap conditions, enhancing the robustness of initial similarity estimation, and developing hierarchical and efficient registration frameworks for large-scale scenes. V Conclusion We propose M23D, an Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registration. Our method introduces a dual MAE to enhance feature extraction by aligning features and establishing correspondences. Additionally, we propose a similarity-based RL masking strategy that reduces unnecessary masking and focuses on key matching regions, improving registration accuracy and robustness. M23D achieves state-of-the-art performance on the RGB-D Scenes v2 and 7-Scenes datasets. Acknowledgment This work was supported in part by the Fundamental Research Funds for the Central Universities (11040-03742026007). References [1] M. Li, Z. Qin, Z. Gao, R. Yi, C. Zhu, Y. Guo, and K. Xu, “2d3d-matr: 2d-3d matching transformer for detection-free registration between images and point clouds,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 14128–14138, 2023. [2] Z. Cheng, J. Deng, X. Li, B. Yin, and T. Zhang, “Bridge 2D-3D: Uncertainty-aware Hierarchical Registration Network with Domain Alignment,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 3, p. 2491–2499, 2025. [3] B. Wang, C. Chen, Z. Cui, J. Qin, C. X. Lu, Z. Yu, P. Zhao, Z. Dong, F. Zhu, N. Trigoni, et al., “P2-net: Joint description and detection of local features for pixel and point matching,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 16004–16013, 2021. [4] V. Lepetit, F. Moreno-Noguer, and P. Fua, “EPnP: An Accurate O(n) Solution to the PnP Problem,” International Journal of Computer Vision, vol. 81, p. 155–166, 2009. [5] M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Communications of the ACM, vol. 24, no. 6, p. 381–395, 1981. [6] J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou, “LoFTR: Detector-free local feature matching with transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8922–8931, 2021. [7] Y. Wang, X. He, S. Peng, D. Tan, and X. Zhou, “Efficient LoFTR: Semi-dense local feature matching with sparse-like speed,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 21666–21675, 2024. [8] N. Inoue and K. Shinoda, “A fast and accurate video semantic-indexing system using fast MAP adaptation and GMM supervectors,” IEEE Transactions on Multimedia, vol. 14, no. 4, p. 1196–1205, 2012. [9] M. J. Reale, S. Canavan, L. Yin, et al., “A multi-gesture interaction system using a 3-D iris disk model for gaze estimation and an active appearance model for 3-D hand pointing,” IEEE Transactions on Multimedia, vol. 13, no. 3, p. 474–486, 2011. [10] P. An, Y. Yang, J. Yang, M. Peng, Q. Liu, and L. Nan, “Enhance Image-to-Point-Cloud Registration with Beltrami Flow,” International Journal of Computer Vision, vol. 133, no. 12, p. 8589–8616, 2025. [11] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superglue: Learning feature matching with graph neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4938–4947, 2020. [12] B. Drost, M. Ulrich, N. Navab, and S. Ilic, “Model Globally, Match Locally: Efficient and Robust 3D Object Recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 998–1005, 2010. [13] Y. Liu, V. Kılıç, J. Guan, et al., “Audio–visual particle flow smc-phd filtering for multi-speaker tracking,” IEEE Transactions on Multimedia, vol. 22, no. 4, p. 934–948, 2019. [14] R. B. Rusu, N. Blodow, and M. Beetz, “Fast point feature histograms (FPFH) for 3D registration,” in 2009 IEEE International Conference on Robotics and Automation, p. 3212–3217, 2009. [15] X. Ding, Z. Chen, W. Lin, et al., “Towards 3D colored mesh saliency: Database and benchmarks,” IEEE Transactions on Multimedia, vol. 26, p. 3580–3591, 2024. [16] H. Yu, F. Li, M. Saleh, B. Busam, and S. Ilic, “Cofinet: Reliable coarse-to-fine correspondences for robust pointcloud registration,” Advances in Neural Information Processing Systems, vol. 34, p. 23872–23884, 2021. [17] Z. Qin, H. Yu, C. Wang, Y. Guo, Y. Peng, S. Ilic, D. Hu, and K. Xu, “Geotransformer: Fast and robust point cloud registration with geometric transformer,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, p. 9806–9821, 2023. [18] E. D. Sontag, “Comments on integral variants of ISS,” Systems & Control Letters, vol. 34, no. 1–2, p. 93–100, 1998. [19] M. Feng, S. Hu, M. H. Ang, and G. H. Lee, “2d3d-matchnet: Learning to match keypoints across 2D image and 3D point cloud,” in 2019 International Conference on Robotics and Automation (ICRA), p. 4790–4796, 2019. [20] K. Lai, L. Bo, and D. Fox, “Unsupervised feature learning for 3D scene labeling,” in 2014 IEEE International Conference on Robotics and Automation (ICRA), p. 3050–3057, 2014. [21] B. Glocker, S. Izadi, J. Shotton, and A. Criminisi, “Real-time RGB-D camera relocalization,” in 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), p. 173–179, 2013. [22] S. Chen, Y. Chen, X. Yin, X. Liu, H. Lai, and T. Zhang, “PAF: Prototype Adaptive Fusion for Test-Time Adaptation of Vision-Language Models,” Proceedings of the 33rd ACM International Conference on Multimedia, p. 3007–3016, 2025. [23] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention Is All You Need,” Advances in Neural Information Processing Systems, vol. 30, p. 5998–6008, 2017. [24] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 770–778, 2016. [25] J. Deng, J. Lu, Z. Cheng, and W. Yang, “DiffCorr: Conditional Diffusion Model with Reliable Pseudo-Label Guidance for Unsupervised Point Cloud Shape Correspondence,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 3, p. 2690–2698, 2025. [26] X. Li, W. Yang, J. Deng, Z. Cheng, X. Zhou, and T. Zhang, “Implicit Correspondence Learning for Image-to-Point Cloud Registration,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 16922–16931, June 2025. [27] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 2117–2125, 2017. [28] H. Thomas, C. R. Qi, J.-E. Deschaud, B. Marcotegui, F. Goulette, and L. J. Guibas, “Kpconv: Flexible and deformable convolution for point clouds,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 6411–6420, 2019. [29] Z. Cheng, X. Yin, J. Deng, B. Liao, Y. Chen, X. Zhou, B. Yin, and T. Zhang, “Adaptive Agent Selection and Interaction Network for Image-to-Point Cloud Registration,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 5, p. 3335–3343, 2026. [30] J. Li, B. M. Chen, and G. H. Lee, “So-net: Self-organizing network for point cloud analysis,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 9397–9406, 2018. [31] Z. Qin, H. Yu, C. Wang, Y. Guo, Y. Peng, and K. Xu, “Geometric transformer for fast and robust point cloud registration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11143–11152, 2022. [32] H. Durrant-Whyte and T. Bailey, “Simultaneous localization and mapping: part I,” IEEE Robotics & Automation Magazine, vol. 13, no. 2, p. 99–110, 2006. [33] Y. Jia, F. Cao, T. Wang, et al., “CAD-Mesher: A Convenient, Accurate, Dense Mesh-Based Mapping Module in SLAM for Dynamic Environments,” IEEE Transactions on Multimedia, vol. 28, p. 1025–1036, 2026. [34] H. Luo, Y. Gao, Y. Wu, et al., “Real-Time Dense Monocular SLAM With Online Adapted Depth Prediction Network,” IEEE Transactions on Multimedia, vol. 21, no. 2, p. 470–483, 2019. [35] A. Kendall, M. Grimes, and R. Cipolla, “PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization,” in Proceedings of the IEEE International Conference on Computer Vision, p. 2938–2946, 2015. [36] B. Fan, Y. Yang, W. Feng, et al., “Seeing Through Darkness: Visual Localization at Night via Weakly Supervised Learning of Domain Invariant Features,” IEEE Transactions on Multimedia, vol. 25, p. 1713–1726, 2023. [37] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in International Conference on Machine Learning, p. 1180–1189, 2015. [38] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3D classification and segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 652–660, 2017. [39] Y. Li, Q. Hao, J. Hu, et al., “3D3M: 3D Modulated Morphable Model for Monocular Face Reconstruction,” IEEE Transactions on Multimedia, vol. 25, p. 6642–6652, 2023. [40] F. Chu, Y. Cong, Y. Wang, et al., “DetailRecon: Focusing on Detailed Regions for Online Monocular 3D Reconstruction,” IEEE Transactions on Multimedia, vol. 27, p. 3266–3278, 2025. [41] S. Ren, Y. Zeng, J. Hou, and X. Chen, “CorrI2P: Deep Image-to-Point Cloud Registration via Dense Correspondence,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 3, p. 1198–1208, 2023. [42] J. Li and G. H. Lee, “DeepI2P: Image-to-point cloud registration via deep classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15960–15969, 2021. [43] S. Kang, Y. Liao, J. Li, F. Liang, Y. Li, X. Zou, F. Li, X. Chen, Z. Dong, and B. Yang, “CoFiI2P: Coarse-to-Fine Correspondences-Based Image to Point Cloud Registration,” IEEE Robotics and Automation Letters, vol. 9, no. 11, p. 10264–10271, 2024. [44] Z. Qin, H. Yu, C. Wang, Y. Guo, Y. Peng, and K. Xu, “Geometric transformer for fast and robust point cloud registration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11143–11152, 2022. [45] Y. Sun, C. Cheng, Y. Zhang, C. Zhang, L. Zheng, Z. Wang, and Y. Wei, “Circle loss: A unified perspective of pair similarity optimization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6398–6407, 2020. [46] Y. Pang, W. Wang, F. E. H. Tay, W. Liu, Y. Tian, and L. Yuan, “Masked autoencoders for point cloud self-supervised learning,” in European Conference on Computer Vision, p. 604–621, 2022. [47] R. Zhang, Z. Guo, P. Gao, R. Fang, B. Zhao, D. Wang, Y. Qiao, and H. Li, “Point-m2ae: Multi-scale masked autoencoders for hierarchical point cloud pre-training,” Advances in Neural Information Processing Systems, vol. 35, p. 27061–27074, 2022. [48] D. Li, H. Zhao, X. Yan, L. Zhao, and H. Cao, “Long-range attention classification for substation point cloud,” Neurocomputing, vol. 608, p. 128435, 2024. [49] R. Zhang, L. Wang, Y. Qiao, P. Gao, and H. Li, “Learning 3D representations from 2D pre-trained models via image-to-point masked autoencoders,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 21769–21780, 2023. [50] A. Chen, K. Zhang, R. Zhang, Z. Wang, Y. Lu, Y. Guo, and S. Zhang, “Pimae: Point cloud and image interactive masked autoencoders for 3D object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 5291–5301, 2023. [51] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 16000–16009, 2022. [52] G. Li, H. Zheng, D. Liu, C. Wang, B. Su, and C. Zheng, “Semmae: Semantic-guided masking for learning masked autoencoders,” Advances in Neural Information Processing Systems, vol. 35, p. 14290–14302, 2022. [53] L. Bie, S. Pan, S. Li, et al., “GraphI2P: Image-to-Point Cloud Registration with Exploring Pattern of Correspondence via Graph Learning,” in Proceedings of the Computer Vision and Pattern Recognition Conference, p. 22161–22171, 2025. [54] G. Yao, Y. Xuan, Y. Chen, and Y. Pan, “Quantity-Aware Coarse-to-Fine Correspondence for Image-to-Point Cloud Registration,” IEEE Sensors Journal, vol. 24, no. 20, p. 33826–33837, 2024. [55] H. Wang, Y. Liu, B. Wang, Y. Sun, Z. Dong, W. Wang, and B. Yang, “FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators,” in Proceedings of the International Conference on Learning Representations (ICLR), 2024. [56] Y. Yue, H. Yuan, Q. Miao, X. Mao, R. Hamzaoui, and P. Eisert, “EdgeRegNet: Edge Feature-Based Multimodal Registration Network Between Images and LiDAR Point Clouds,” IEEE Transactions on Multimedia, vol. 27, p. 9281–9292, 2025. [57] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, p. 99–106, 2021. [58] R. Rubinstein, “The cross-entropy method for combinatorial and continuous optimization,” Methodology and Computing in Applied Probability, vol. 1, p. 127–190, 1999. [59] S. Ghanavati, D. Amyot, and A. Rifaut, “Legal goal-oriented requirement language (legal GRL) for modeling regulations,” in Proceedings of the 6th International Workshop on Modeling in Software Engineering, p. 1–6, 2014. [60] P. An, Y. Yang, J. Yang, M. Peng, Q. Liu, and L. Nan, “Enhance Image-to-Point-Cloud Registration with Beltrami Flow,” International Journal of Computer Vision, vol. 133, no. 12, p. 8589–8616, 2025. [61] Y. Chen, R. Sun, W. Li, et al., “Beyondmix: Leveraging structural priors and long-range dependencies for domain-invariant LiDAR segmentation,” in Advances in Neural Information Processing Systems, vol. 38, p. 89462–89488, 2026. [62] Z. Cheng, J. Deng, X. Li, X. Yin, B. Liao, B. Yin, W. Yang, and T. Zhang, “CA-I2P: Channel-Adaptive Registration Network with Global Optimal Selection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 27739–27749, 2025. [63] L. van der Maaten and G. Hinton, “Visualizing Data Using t-SNE,” Journal of Machine Learning Research, vol. 9, p. 2579–2605, 2008. [64] X. Zhang, J. Zhou, W. Sun, and S. K. Jha, “A Lightweight CNN Based on Transfer Learning for COVID-19 Diagnosis,” Computers, Materials & Continua, vol. 72, no. 1, p. 1123–1137, 2022. [65] S. Huang, Z. Gojcic, M. Usvyatsov, A. Wieser, and K. Schindler, “Predator: Registration of 3D point clouds with low overlap,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4267–4276, 2021. [66] A. Strehl, J. Ghosh, and R. Mooney, “Impact of similarity measures on web-page clustering,” in Workshop on Artificial Intelligence for Web Search (AAAI 2000), vol. 58, p. 64, 2000. [67] C. Choy, J. Park, and V. Koltun, “Fully convolutional geometric features,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 8958–8966, 2019. [68] S. Huang, Z. Gojcic, M. Usvyatsov, A. Wieser, and K. Schindler, “Predator: Registration of 3D point clouds with low overlap,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4267–4276, 2021. [69] Z. Wang, J. Wang, D. Zuo, Y. Ji, X. Xia, Y. Ma, J. Hao, M. Yuan, Y. Zhang, and F. Wu, “A Hierarchical Adaptive Multi-Task Reinforcement Learning Framework for Multiplier Circuit Design,” in Proceedings of the 41st International Conference on Machine Learning (ICML), vol. 235, p. 51825–51853, 2024. [70] Z. Wang, J. Wang, Q. Zhou, B. Li, and H. Li, “Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Critic,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 8, p. 8612–8620, June 2022. [71] Z. Wang, T. Pan, Q. Zhou, and J. Wang, “Efficient Exploration in Resource-Restricted Reinforcement Learning,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 8, p. 10279–10287, June 2023. [72] Z. Qin, H. Yu, C. Wang, Y. Guo, Y. Peng, and K. Xu, “Geometric transformer for fast and robust point cloud registration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 11143–11152, 2022. [73] J. Deng, C. Wang, J. Lu, et al., “SE-ORNet: Self-ensembling orientation-aware network for unsupervised point cloud shape correspondence,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 5364–5373, 2023. [74] J. Deng, J. Lu, and T. Zhang, “Quantity-Quality Enhanced Self-Training Network for Weakly Supervised Point Cloud Semantic Segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 5, p. 3580–3596, 2025. [75] Y. Wu, J. Liu, M. Gong, P. Gong, X. Fan, A. K. Qin, Q. Miao, and W. Ma, “Self-Supervised Intra-Modal and Cross-Modal Contrastive Learning for Point Cloud Understanding,” IEEE Transactions on Multimedia, vol. 26, p. 1626–1638, 2024. [76] J. Liu, Y. Wu, M. Gong, Z. Liu, Q. Miao, and W. Ma, “Inter-Modal Masked Autoencoder for Self-Supervised Learning on Point Clouds,” IEEE Transactions on Multimedia, vol. 26, p. 3897–3908, 2023. [77] Z. Cheng, J. Deng, X. Li, L. Liu, X. Yin, B. Yin, and T. Zhang, “B2-3D++: Uncertainty-aware Hierarchical Registration Network with Domain Alignment,” Authorea Preprints, 2025. [78] B. Liao, W. Zhai, Z. Wan, Z. Cheng, W. Yang, Y. Cao, T. Zhang, and Z.-J. Zha, “EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting,” Advances in Neural Information Processing Systems, vol. 38, 2025. [79] Z. Guo, R. Zhang, L. Qiu, X. Li, and P.-A. Heng, “Joint-MAE: 2D–3D Joint Masked Autoencoders for 3D Point Cloud Pre-training,” in Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI), p. 791–799, 2023. [80] J. Liu, L. Kong, Y. Wu, M. Gong, H. Li, Q. Miao, W. Ma, and C. Qin, “Triple Point Masking,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 35, no. 9, p. 8466–8477, 2025. Zhixin Cheng received his bachelor’s degree from the School of Electrical and Electronic Engineering, Huazhong University of Science and Technology, Wuhan, China, in 2020, and his Ph.D. degree from the School of Information Science and Technology, University of Science and Technology of China, Hefei, China, through a combined M.S.–Ph.D. program. He is currently a Specially Appointed Associate Professor with Hefei University of Technology, Hefei, China. His research interests include computer vision and machine learning, with a focus on 3D scene registration, multimodal learning, and image enhancement. Jiacheng Deng received the bachelor’s degree in Information Security from the University of Science and Technology of China in 2023. He is now pursuing a master degree in Control Science and Engineering at University of Science and Technology of China. His research interests include computer vision and deep learning, especially image-to-point cloud registration and pose estimation. Xiaotian Yin is currently pursuing a Ph.D. degree at the University of Science and Technology of China, Hefei, China. His research interests include computer vision and machine learning, with a focus on few-shot learning and multi-modal learning. Baoqun Yin received his bachelor’s degree in Mathematics from Sichuan University, Chengdu, China, in 1985, and his master’s degree in Applied Mathematics from the University of Science and Technology of China (USTC), Hefei, China, in 1993. He earned his Ph.D. degree in Pattern Recognition and Intelligent Systems from the Department of Automation, USTC, in 1998. He is currently a Professor in the Department of Automation at the University of Science and Technology of China. His research interests include stochastic systems, system optimization, and information networks, focusing on Markov decision processes, network optimization, and smart energy management. Richang Hong (Senior Member, IEEE) received the Ph.D. degree from the University of Science and Technology of China, Hefei, China, in 2008. From September 2008 to December 2010, he was a Research Fellow with the School of Computing, National University of Singapore, Singapore. He is currently a Professor with the Hefei University of Technology, Hefei, China. He has coauthored more than 300 publications in the areas of his research interests, which include multimedia question answering, video content analysis, and pattern recognition. Dr. Hong is a Member of the Association for Computing Machinery. He was the recipient of the Best Paper Award in the ACM Multimedia 2010. Tianzhu Zhang (M’11) received the bachelor’s degree in Communications and Information Technology from the Beijing Institute of Technology, Beijing, China, in 2006, and the Ph.D. degree in Pattern Recognition and Intelligent Systems from the Institute of Automation, Chinese Academy of Sciences, Beijing, China, in 2011. He is currently a Professor at the Department of Automation, School of Information Science, University of Science and Technology of China. His current research interests include computer vision and multimedia, with a focus on action recognition, object classification, object tracking, and social event analysis.