Paper deep dive
CM-MAE: A Physics-Guided Cross-Modal Self-Supervised Learning Framework for Vision-Wireless Applications
Yubo Zhang, Yiyao Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/22/2026, 3:23:48 AM
Summary
The paper introduces CM-MAE, a self-supervised vision-wireless pretraining framework designed for cross-scenario representation transfer in mmWave communications. It utilizes synchronized RGB images and 64-beam received-power vectors from the DeepSense 6G dataset. The core innovation is a soft contrastive alignment loss that uses measured beam-power profile similarities to define positive pairs, avoiding false negatives common in hard instance matching. This is combined with a masked joint decoder for reconstructing visual patches and wireless angular clusters. The model employs differential-rate fine-tuning, allowing a fusion head to adapt quickly while encoders remain stable, achieving significant improvements in linear-probe transfer and downstream classification accuracy on unseen scenarios.
Entities (10)
Relation Signals (8)
CM-MAE → evaluatedon → DeepSense 6G
confidence 98% · The evaluated real-data model uses only RGB frames and the measured 64-beam received-power vector available in DeepSense 6G
CM-MAE → encodes → RGB frames
confidence 95% · The evaluated real-data model uses only RGB frames... Image patches... are encoded by separate Transformers
CM-MAE → encodes → 64-beam received-power vector
confidence 95% · The evaluated real-data model uses only... the measured 64-beam received-power vector... wireless angular clusters are encoded by separate Transformers
CM-MAE → uses → Soft Contrastive Alignment Loss
confidence 95% · Its central pretraining term is a soft contrastive alignment loss.
CM-MAE → uses → Masked Joint Decoder
confidence 92% · A masked joint decoder provides the complementary local objective by reconstructing hidden visual patches and wireless angular clusters under modality dropout.
CM-MAE → achieves → 77.38% Top-1 accuracy
confidence 90% · Mild fusion fine-tuning reaches 77.38% Top-1 accuracy on unseen Scenarios 6–8
Soft Contrastive Alignment Loss → basedon → 64-beam received-power vector
confidence 90% · this loss builds a target distribution from similarities between measured beam-power profiles
CM-MAE → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Synchronized camera and wireless measurements observe the same scene through different physical channels. The central difficulty is that a representation learned in one deployment can fail when viewpoint, traffic, illumination, and propagation geometry change. This paper presents CM-MAE, a self-supervised vision--wireless pretraining framework for cross-scenario representation transfer. The evaluated real-data model uses only RGB frames and the measured 64-beam received-power vector available in DeepSense 6G; it does not use ray-traced paths, calibrated depth, or beam-index labels during pretraining. Its central pretraining term is a \emph{soft contrastive alignment loss}. Instead of making the synchronized image--wireless pair the only positive pair, this loss builds a target distribution from similarities between measured beam-power profiles, so nonidentical samples with similar directional responses are not forced apart as false negatives. A masked joint decoder provides the complementary local objective by reconstructing hidden visual patches and wireless angular clusters under modality dropout. After pretraining, a differential-rate fine-tuning rule lets a new fusion head adapt quickly while the encoders move slowly. Under a sequence-disjoint DeepSense 6G protocol, adding the soft alignment loss improves a matched linear-probe transfer average from 24.88\% to 29.49\%. Mild fusion fine-tuning reaches 77.38\% Top-1 accuracy on unseen Scenarios 6--8, and optional transductive normalization adaptation reaches 78.69\%. Since the fusion setting uses the contemporaneous 64-beam power vector at inference, these results should be read as representation-transfer diagnostics, not as proactive beam-prediction or reduced-sweeping claims.
Tags
Links
- Source: https://arxiv.org/abs/2608.15972v1
- Canonical: https://arxiv.org/abs/2608.15972v1
Trouble viewing inline? Open PDF directly →
Full Text
46,650 characters extracted from source content.
Expand or collapse full text
CM-MAE: A Physics-Guided Cross-Modal Self-Supervised Learning Framework for Vision-Wireless Applications Yubo Zhang, Yiyao Liu Abstract Synchronized camera and wireless measurements observe the same scene through different physical channels. The central difficulty is that a representation learned in one deployment can fail when viewpoint, traffic, illumination, and propagation geometry change. This paper presents CM-MAE, a self-supervised vision–wireless pretraining framework for cross-scenario representation transfer. The evaluated real-data model uses only RGB frames and the measured 64-beam received-power vector available in DeepSense 6G; it does not use ray-traced paths, calibrated depth, or beam-index labels during pretraining. Its central pretraining term is a soft contrastive alignment loss. Instead of making the synchronized image–wireless pair the only positive pair, this loss builds a target distribution from similarities between measured beam-power profiles, so nonidentical samples with similar directional responses are not forced apart as false negatives. A masked joint decoder provides the complementary local objective by reconstructing hidden visual patches and wireless angular clusters under modality dropout. After pretraining, a differential-rate fine-tuning rule lets a new fusion head adapt quickly while the encoders move slowly. Under a sequence-disjoint DeepSense 6G protocol, adding the soft alignment loss improves a matched linear-probe transfer average from 24.88% to 29.49%. Mild fusion fine-tuning reaches 77.38% Top-1 accuracy on unseen Scenarios 6–8, and optional transductive normalization adaptation reaches 78.69%. Since the fusion setting uses the contemporaneous 64-beam power vector at inference, these results should be read as representation-transfer diagnostics, not as proactive beam-prediction or reduced-sweeping claims. Index Terms: multimodal ISAC, cross-modal self-supervised learning, masked autoencoder, domain generalization, mmWave beam classification, test-time adaptation I Introduction Vision-assisted wireless learning is attractive because the camera observes geometry and occlusion while the wireless receiver observes the directional response induced by the same scene. In a vehicle-to-infrastructure deployment, however, the relation between image pixels and a selected beam is not fixed. It changes with camera placement, street layout, traffic composition, reflectors, blockers, and array response. A model that fits one measured scenario can therefore lose useful structure when moved to another. This paper focuses on the part of the problem that can be studied with real synchronized measurements: learning transferable vision–wireless representations without using discrete beam-index labels during pretraining. Each training sample contains an RGB image and a 64-dimensional received-power vector obtained from a beam sweep. The downstream task is to classify the strongest measured beam index, but that index is withheld during self-supervised pretraining. Two design choices are needed. First, masked reconstruction should not be purely unimodal. If the image branch reconstructs only image patches and the wireless branch reconstructs only wireless features, the model can learn useful local statistics while leaving the two global representations weakly related. Second, cross-modal alignment should not be a hard instance-matching loss. CLIP-style alignment treats every nonpaired sample in the batch as a negative [6], even when two wireless power profiles have nearly the same dominant angular response. For beam-sweep data, this creates false negatives: samples that are different frames but physically similar from the array’s perspective. CM-MAE addresses these two issues directly. Its global objective is a soft contrastive alignment loss: the target for a visual query is not a one-hot sample identity, but a probability distribution over wireless samples computed from measured beam-profile similarity. Its local objective is masked conditional reconstruction through a joint decoder. The evaluated model implements these ideas with camera-ray-indexed image patches and sparse one-dimensional angular clusters extracted from the 64-beam power vector. A geometry-privileged extension would require calibrated wideband CFR and ray-traced correspondence labels; because those signals are not used in the reported experiments, the extension is discussed only as a limitation and future direction. The main contributions are as follows. • We formulate real-data CM-MAE pretraining using only synchronized RGB frames and measured beam-power vectors, with a clear separation between pretraining inputs and downstream beam-index labels. • We introduce a beam-profile-guided soft contrastive alignment loss that matches learned cross-modal retrieval distributions to measured wireless-response neighborhoods. • We combine this global soft alignment with masked conditional reconstruction in a dual-stream Transformer architecture whose masking, decoder, and modality-dropout rules are explicitly gated by the available modalities. • Matched ablations show a 4.61-point average transfer gain from soft alignment. Differential-rate adaptation raises the unseen-scenario average from 51.36% to 77.38%; optional transductive adaptation reaches 78.69%. I Background and Related Work Contrastive and masked self-supervision. Contrastive learning builds representations by making related views close and unrelated views far apart. In cross-modal learning, CLIP-style training uses the paired sample as the positive and all other samples in the batch as negatives [6]. This objective is simple and effective when the correct relation is sample identity, but it can be too rigid for directional wireless measurements, where multiple frames can have similar beam responses. Masked autoencoding instead hides input tokens and trains a decoder to reconstruct them [3]. MultiMAE and CAV-MAE show that masked reconstruction can be extended to multiple modalities and combined with alignment losses [4, 5]. CM-MAE follows this general family but changes the alignment target from hard identity to a wireless-response neighborhood. Wireless and vision–wireless representation learning. DeepSense 6G provides real measured sensory and wireless data for multimodal communication tasks [1]. Vision-position beam prediction demonstrates the usefulness of side information under supervised training [2]. Recent self-supervised or foundation-style wireless models include LWM [7], ContraWiMAE [8], WMFM [9], WiFo-M2 [10], and WiFo-MiSAC [11]. These works motivate masked and multimodal learning for wireless systems. CM-MAE is narrower in input scope but sharper in target construction: for real DeepSense data, it aligns camera and wireless features using the measured 64-beam power-profile geometry available in each synchronized sample. Adaptation across scenarios. Cross-scenario evaluation is essential because source validation accuracy can overstate transfer performance. Test-time adaptation methods such as Tent update normalization parameters on unlabeled target data by minimizing prediction entropy [12]. Domain-adversarial training seeks scenario-invariant features through gradient reversal [13]. CM-MAE differs from these adaptation-only approaches in two ways: it first builds a cross-modal representation through beam-profile-guided pretraining, and then uses a conservative supervised adaptation rule in which new fusion layers learn quickly while pretrained encoders move slowly. In our experiments, Tent-style normalization adaptation is kept as an optional transductive stage, while the adversarial objective was numerically unstable and is not part of the final method. I Problem Setup I-A Synchronized Image and Beam-Power Measurements Consider a base station equipped with a beam codebook =bb=1NbW=\w_b\_b=1^N_b and a co-located RGB camera. A synchronized sample is denoted (Ii,i),Ii∈ℝ3×H×W, (I_i, ρ_i ), I_i ^3× H× W, i=[ρi,1,…,ρi,Nb]⊤∈ℝ+Nb. ρ_i=[ _i,1,…, _i,N_b] _+^N_b. (1) Here IiI_i is the RGB frame and ρi,b _i,b is the measured received power when beam b is used in the sweep. DeepSense provides Nb=64N_b=64. If i,kh_i,k is the effective channel on subcarrier k, the sweep measurement can be written as ρi,b=1K∑k=1K|bHi,k|2+ξi,b,b=1,…,Nb, _i,b= 1K _k=1^K |w_b^Hh_i,k |^2+ _i,b, b=1,…,N_b, (2) where ξi,b _i,b includes noise and measurement perturbations. The downstream class label is the strongest measured beam index bi⋆=argmaxb∈1,…,Nbρi,b.b_i = _b∈\1,…,N_b\ _i,b. (3) Equation (3) is used for downstream supervised training and evaluation only. CM-MAE pretraining does not consume bi⋆b_i as a label. It does consume the full measured power vector i ρ_i, which is task-relevant because it contains the maximizer in (3). The image is divided into P=(H/S)(W/S)P=(H/S)(W/S) nonoverlapping patches of size S×S× S. Let ~p q_p be the homogeneous image coordinate of patch center p and let KcamK_ cam be the camera intrinsic matrix. The normalized camera ray used by the tokenizer is p=Kcam−1~p‖Kcam−1~p‖2.d_p= K_ cam^-1 q_p\|K_ cam^-1 q_p\|_2. (4) In the evaluated DeepSense implementation, KcamK_ cam is an approximate pinhole camera with a 90∘90 field of view. Therefore pd_p is only a stable image-plane direction coordinate. It is not a depth estimate and it is not a propagation-path label. I-B Learning Objective The unlabeled pretraining set is u=(Ii,i)i=1nu,D_u=\(I_i, ρ_i)\_i=1^n_u, (5) and the labeled downstream set is l=(Ii,i,bi⋆)i=1nl.D_l=\(I_i, ρ_i,b_i )\_i=1^n_l. (6) Let EvE_v and EwE_w denote the vision and wireless encoders. Each encoder prepends a learned CLS token to its token sequence; after the final Transformer layer, the output at this CLS position is used as the global modality representation. We denote these CLS embeddings by iv,iw∈ℝdz_i^v,z_i^w ^d. CM-MAE pretraining has two objectives: 1. Beam-profile soft alignment: if two samples have similar normalized beam-power profiles, the model should assign them similar cross-modal retrieval probabilities. 2. Masked conditional reconstruction: hidden visual patches and hidden wireless cluster features should be reconstructed from the visible tokens of the available modality or modalities. For downstream beam-index classification, a task head gϕg_φ is fitted on labeled source data: (θ⋆,ϕ⋆)=argminθ,ϕ(I,,b⋆)∼lℓcls(gϕ(Eθ(I,)),b⋆).(θ ,φ )= _θ,φE_(I, ρ,b ) _l _ cls\! (g_φ(E_θ(I, ρ)),b ). (7) The fusion setting in this paper gives the classifier both the image and the contemporaneous 64-beam power vector at inference. This protocol evaluates cross-scenario multimodal representation transfer. It does not demonstrate proactive prediction before a sweep, nor does it show reduced beam-training overhead. The vision-only ablation is the corresponding camera-only setting. IV Evaluated CM-MAE Architecture RGB image224×224224×224patch + ray tokensP=196P=196vision ViTmask ratio 0.750.7564-beam powervector + scaleangular clustersQ≤10Q≤ 10wireless ViTmask ratio 0.500.50beam-profilesoft alignmentℒctrL_ ctrjoint decodermasked completionpatch/clusterreconstructiondifferential-ratemild fine-tuningoptional norm-onlytarget adaptation Fig. 1: Evaluated CM-MAE pipeline. Image patches and wireless angular clusters are encoded by separate Transformers, decoded jointly for masked reconstruction, and aligned globally by a beam-profile-guided soft contrastive loss. IV-A Tokenization We use superscripts v and w for visual and wireless quantities. Patch indices are p=1,…,Pp=1,…,P; wireless cluster indices are q=1,…,Qiq=1,…,Q_i with Qi≤QmaxQ_i≤ Q_ . Vision tokens. A convolutional patch embedder fvf_v maps image patch Ii,pI_i,p to width d. The token entering the vision encoder is i,pv=fv(Ii,p)+Wrp+p,x^v_i,p=f_v(I_i,p)+W_rd_p+e_p, (8) where pe_p is a learned patch-position embedding and WrpW_rd_p injects the approximate camera-ray coordinate from (4). This gives the model a consistent notion of image direction while avoiding unsupported depth or path claims. Wireless normalization and scale. The raw received-power vector is converted to dB and normalized per sample: ri,b=10log10(ρi,b+ϵ),pi,b=ri,b−miMi−mi+ϵ,r_i,b=10 _10( _i,b+ε), p_i,b= r_i,b-m_iM_i-m_i+ε, (9) where mi=minbri,bm_i= _br_i,b and Mi=maxbri,bM_i= _br_i,b. The normalized vector i=[pi,1,…,pi,Nb]⊤p_i=[p_i,1,…,p_i,N_b] captures angular shape in [0,1]Nb[0,1]^N_b. Since this normalization removes absolute power, the scalar MiM_i is also embedded as a wireless scale token i,0w=WsMi.x^w_i,0=W_sM_i. (10) Sparse angular clusters. Millimeter-wave power is often concentrated around a small number of angular lobes [14]. The evaluated tokenizer therefore converts the dense 64-beam profile into a short cluster sequence. Local maxima of ip_i above η=0.15η=0.15 are detected, the strongest Qi≤Qmax=10Q_i≤ Q_ =10 peaks are retained, and the global maximum is used as a fallback if no local peak passes the threshold. For a retained peak at beam bi,qb_i,q, define energy ei,q=pi,bi,qe_i,q=p_i,b_i,q and a local power-weighted angular centroid θ¯i,q=∑b=bi,q−Δbbi,q+Δbpi,bθb∑b=bi,q−Δbbi,q+Δbpi,b+ϵ,θb∈[−1,1],Δb=2. θ_i,q= _b=b_i,q- _b^b_i,q+ _bp_i,b _b _b=b_i,q- _b^b_i,q+ _bp_i,b+ε, _b∈[-1,1],\; _b=2. (11) The local wireless feature i,q∈ℝ6a_i,q ^6 stores the normalized powers at bi,q−1,bi,q,bi,q+1b_i,q-1,b_i,q,b_i,q+1 together with a zero imaginary channel, matching the real/imaginary feature shape used by the implementation. The cluster token is i,qw=fw(i,q)+We[log(ei,q+ϵ),κi,q]⊤+γ(θ¯i,q),x^w_i,q=f_w(a_i,q)+W_e[ (e_i,q+ε), _i,q] +γ( θ_i,q), (12) where κi,q _i,q is a peak-quality indicator and γ(⋅)γ(·) is a Fourier angular encoding [15]. Padded cluster slots are excluded by the attention mask. This tokenizer records angular location, local lobe shape, relative energy, and absolute scale, but it does not infer physical propagation paths. IV-B Dual Transformer Encoders and Joint Decoder The visual sequence is iv=[v,i,1v,…,i,Pv],X_i^v=[c^v,x^v_i,1,…,x^v_i,P], (13) where vc^v is the learned visual CLS token. The wireless sequence is iw=[w,i,0w,i,1w,…,i,Qiw],X_i^w=[c^w,x^w_i,0,x^w_i,1,…,x^w_i,Q_i], (14) where wc^w is the wireless CLS token and i,0wx^w_i,0 is the scale token from (10). Invalid padded cluster slots are masked out of attention. The two streams use separate Vision Transformer encoders rather than shared weights. Each encoder is a pre-LayerNorm Transformer stack. For either modality, layer ℓ applies ℓ+12 + 12 =ℓ+MHSA(LN(ℓ)), =H +MHSA(LN(H )), ℓ+1 +1 =ℓ+12+FFN(LN(ℓ+12)). =H + 12+FFN(LN(H + 12)). (15) The final CLS outputs are iv=i,0v,L,iw=i,0w,L,z_i^v=H^v,L_i,0, _i^w=H^w,L_i,0, (16) where position zero denotes the CLS token. The evaluated Base configuration uses width d=768d=768, 12 encoder layers, 12 attention heads, and MLP ratio 4 in each encoder. This is a large model, roughly 232M parameters including the decoder and task head. During pretraining, the joint decoder receives the concatenated encoded sequence [CLSv,1:Pv,CLSw,0w,1:Qw][z^v_ CLS,z^v_1:P,z^w_ CLS,z^w_0,z^w_1:Q] (17) with modality-type embeddings. The decoder has eight Transformer layers, width 768, and 12 attention heads. Separate prediction heads map decoder outputs back to denormalized RGB patch vectors and six-dimensional local wireless features. The decoder is used only for self-supervised pretraining; downstream classification uses the unmasked encoders and a separate fusion head. IV-C Soft Contrastive Alignment Loss Why hard instance matching is insufficient. This is the soft contrastive alignment function referred to in the abstract. It is contrastive because it compares visual and wireless CLS embeddings through a batch retrieval distribution, and it is soft because its target is a probability distribution rather than a one-hot identity label. For a visual query i, hard instance matching would minimize −logπiiv→w- _i^v→ w and treat every j≠ij≠ i as a negative. This is inappropriate when two different samples have similar measured beam-power shapes. CM-MAE instead uses the wireless response itself to define how strongly sample i should be associated with other samples in the batch. Wireless-derived soft targets. For batch size B, normalize each power profile as ~i=i/‖i‖2 p_i=p_i/\|p_i\|_2 and define dij=1−~i⊤~j,qij=[j≠i]exp(−dij2/(2σh2))∑k≠iexp(−dik2/(2σh2)),d_ij=1- p_i p_j, q_ij= 1[j≠ i] (-d_ij^2/(2 _h^2)) _k≠ i (-d_ik^2/(2 _h^2)), (18) where σh=0.5 _h=0.5. Since ip_i is nonnegative and normalized, dijd_ij measures angular-pattern discrepancy. The diagonal is set to zero in the evaluated implementation: ℒctrL_ ctr is used for non-self neighborhood learning, while the synchronized pair is still coupled through joint reconstruction. Smaller σh _h concentrates probability on the nearest non-self profiles; larger σh _h approaches a uniform distribution over j≠ij≠ i. Learned cross-modal distributions. Let ivu_i^v and iwu_i^w be ℓ2 _2-normalized projections of the two CLS tokens and let sijv→w=(iv)⊤jw/τs_ij^v→ w=(u_i^v) u_j^w/τ. The visual-to-wireless retrieval distribution is πijv→w=exp(sijv→w)∑k=1Bexp(sikv→w), _ij^v→ w= (s_ij^v→ w) _k=1^B (s_ik^v→ w), (19) with τ=0.07τ=0.07; iw→v π_i^w→ v is defined analogously. The symmetric soft alignment loss is ℒctr=−12B∑i=1B∑j=1Bqij(logπijv→w+logπijw→v).L_ ctr=- 12B _i=1^B _j=1^Bq_ij ( _ij^v→ w+ _ij^w→ v ). (20) What the function optimizes. For one direction, differentiation with respect to the similarity logit gives ∂H(i,iv→w)∂sijv→w=πijv→w−qij. ∂ H(q_i, π_i^v→ w)∂ s_ij^v→ w= _ij^v→ w-q_ij. (21) Thus a nonpaired sample with large qijq_ij is pulled closer until the learned probability matches the measured beam-profile affinity, while a distant sample with qij≈0q_ij≈ 0 remains a negative. The loss is therefore not a many-positive binary contrastive loss; it is distribution matching. The distribution-matching interpretation follows from ℒctr=Cq+12B∑i=1B[ _ ctr=C_q+ 12B _i=1^B [ KL(i∥iv→w) (q_i\| π_i^v→ w) +KL(i∥iw→v)], +KL(q_i\| π_i^w→ v) ], (22) where Cq=B−1∑iH(i)C_q=B^-1 _iH(q_i) is independent of network parameters. Hence minimizing (20) matches learned cross-modal retrieval neighborhoods to the wireless-response neighborhood. If all embeddings collapse, i π_i becomes uniform; whenever iq_i is nonuniform, the excess KL term is strictly positive. In this paper, “physics-grounded” means only that the target distribution is computed from measured propagation responses. It does not claim path recovery or universal electromagnetic invariance. IV-D Masking, Modality Dropout, and Reconstruction Masking follows the information density of the two modalities. Images are dense and spatially redundant, so vision uses a high mask ratio rv=0.75r_v=0.75 with contiguous row-major blocks of four patches. Wireless clusters are already sparse, so wireless uses a lower mask ratio rw=0.5r_w=0.5 over valid angular-cluster tokens. Let mi,pv,mi,qw∈0,1m^v_i,p,m^w_i,q∈\0,1\ indicate whether a token’s content is visible. Masked positions are not removed. Their content embedding is replaced by a learned mask token while their coordinate embedding remains: ~i,pv=mi,pvi,pv+(1−mi,pv)(v+p+Wrp), x^v_i,p=m^v_i,px^v_i,p+(1-m^v_i,p)(m_v+e_p+W_rd_p), (23) ~i,qw=mi,qwi,qw+(1−mi,qw)(w+γ(θ¯i,q)). x^w_i,q=m^w_i,qx^w_i,q+(1-m^w_i,q)(m_w+γ( θ_i,q)). (24) Thus the model knows where a hidden image patch or angular cluster lies, but not its content. Keeping position information is important for the wireless stream: a position-free mask token would turn two missing clusters at different beam angles into the same decoder input. Let Ωiv _i^v and Ωiw _i^w denote masked visual patches and masked valid wireless clusters. The two reconstruction losses are ℒrec,v _ rec,v =1∑i|Ωiv|∑i∑p∈Ωiv‖^i,pv−i,pv‖22, = 1 _i| _i^v| _i _p∈ _i^v\| t_i,p^v-t_i,p^v\|_2^2, ℒrec,w _ rec,w =1∑i|Ωiw|∑i∑q∈Ωiw‖^i,q−i,q‖22. = 1 _i| _i^w| _i _q∈ _i^w\| a_i,q-a_i,q\|_2^2. (25) The joint decoder is used to make reconstruction conditional on both streams when both are present, so hidden visual patches or wireless clusters must be predicted from the visible context rather than from an isolated unimodal branch. This is a pretraining task design, not a claim that reconstruction loss alone guarantees transfer. Modality dropout prevents the decoder from assuming that both modalities are always present. With probability pmd=0.1p_ md=0.1 the vision branch is dropped, and with another 0.10.1 the wireless branch is dropped. Let av,aw∈0,1a_v,a_w∈\0,1\ denote branch presence. The implemented pretraining loss is ℒpre= _ pre= avℒrec,v+awβwℒrec,w+avawλctrℒctr, a_vL_ rec,v+a_w _wL_ rec,w+a_va_w _ ctrL_ ctr, βw=1,λctr=0.2. _w=1, _ ctr=0.2. (26) The product gate removes the alignment term whenever either CLS embedding is absent; reconstruction remains active for whichever branch is present. V Learning and Adaptation Procedure V-A Stage 1: Self-Supervised Pretraining Pretraining uses paired but unlabeled samples (Ii,i)(I_i, ρ_i) from the source scenarios. For each mini-batch, the tokenizer builds visual patch tokens, the scale token, and wireless angular-cluster tokens; independent masks and modality-dropout indicators are sampled; the two encoders and joint decoder are optimized with ℒpreL_ pre in (26). The beam-index label bi⋆b_i is not used in this stage. The soft contrastive loss is active only when both modality branches are present, and reconstruction is active for every available branch. Checkpoint selection is performed on the source validation scenario after downstream adaptation, not by target-scenario labels. V-B Stage 2: Source-Supervised Adaptation After pretraining, no reconstruction mask is applied. The unmasked encoders produce ivz^v_i and iwz^w_i. In fusion mode, a projection of the full normalized power vector gp(i)g_p(p_i) is added to the wireless CLS feature. A two-layer bidirectional residual attention module exchanges information between the visual and wireless global features, and an MLP predicts one of the Nb=64N_b=64 beam indices. Uniform learning rates caused large representation drift in our cross-scenario runs. The final adaptation rule therefore uses two optimizer groups. If θe _e denotes pretrained encoder parameters and θh _h denotes newly initialized fusion and classifier parameters, the updates are θet+1=θet−ηe∇θeℒcls,θht+1=θht−ηh∇θhℒcls, _e^t+1= _e^t- _e _ _eL_ cls, _h^t+1= _h^t- _h _ _hL_ cls, (27) with ηe=3×10−6 _e=3× 10^-6 and ηh=10−3 _h=10^-3. AdamW uses cosine decay, five-epoch warmup, weight decay 0.1, label smoothing 0.05, head dropout 0.15, gradient clipping at 1, and 80 epochs with effective batch size 512. The peak head rate is about 333333 times the encoder rate. The intended two-timescale behavior is simple: adapt the task-specific head quickly while limiting movement away from the pretrained encoder. V-C Stage 3: Optional Test-Time Adaptation For optional transductive adaptation, only affine parameters of normalization layers are updated on unlabeled target-scenario batches by minimizing ℒTTA=−1B∑i=1B∑b=1Nbp^i,blogp^i,b.L_ TTA=- 1B _i=1^B _b=1^N_b p_i,b p_i,b. (28) Adam uses learning rate 5×10−45× 10^-4 for 30 steps. Adaptation is performed separately for each target scenario and the source weights are restored before moving to another scenario. This stage is transductive: it assumes access to unlabeled target-scenario batches, and labels are used only after adaptation to report accuracy. We therefore report the source-only mild-FT result and the mild-FT+TTA result separately. V-D Out-of-Scope Privileged Geometry We reserve the name CM-MAE-W for a possible geometry-privileged extension with complex wideband CFR, 3-D angle–delay clusters, geometry-guided masks, and a supervised correspondence head. That version is not part of the evaluated method because the reported DeepSense experiments do not use ray-traced path labels, calibrated depth, or patch–path association. A future dataset with such labels could add entropic optimal-transport correspondence with dustbins [16], but that would be a different supervision regime. The results below therefore evaluate only the real-data RGB-plus-measured-power setting defined in Sections I–V. VI Experiments VI-A Setup Dataset and protocol. DeepSense 6G Scenarios 1–8 contain real-world vehicle-to-infrastructure (V2I) measurements at 60 GHz with synchronized RGB images (224×224224×224) and 64-beam power vectors. Each scenario corresponds to a different geographic location. We split each scenario 70/15/15 by seq_index to avoid leakage from consecutive frames of the same vehicle. Pretraining uses unlabeled pairs from Sc1–5; fine-tuning uses labeled data from Sc1–4; Sc5 serves as validation; Sc6–8 are held out as the unseen test set. All non-TTA accuracies use the test split of each scenario without target labels during training or model selection. The TTA row additionally uses unlabeled target-scenario batches for entropy minimization and is marked separately as transductive. The test sets contain 488 samples (Sc5), 254 (Sc6), 159 (Sc7), and 652 (Sc8). Implementation. We use ViT-Base encoders (two streams, ∼ 232 M parameters total). The vision stream is initialized from ImageNet-pretrained ViT-B/16; the CSI stream is trained from scratch. Pretraining uses AdamW (peak LR 1.2×10−41.2×10^-4, weight decay 0.050.05, batch size 192, cosine schedule with 10-epoch warmup) on a single A100 GPU with automatic mixed precision. All models in this paper—the main model and every ablation variant—are pretrained for 160 epochs under the identical protocol, so that differences reflect the design choices rather than training duration. The best checkpoint is selected by validation linear probe. Fine-tuning runs on one A100. Images use ImageNet mean/std normalization; beam angles are mapped to [−1,1][-1,1]. Baselines. We compare with the official ASU ResNet-18 vision-only baseline [2] evaluated under the same cross-scenario protocol. To position CM-MAE among mainstream SSL paradigms, we further pretrain three representative baselines under the identical protocol: CLIP-style hard InfoNCE alignment without masking (CLS-InfoNCE), soft-contrastive alignment without masking (CLS-SoftCtr), and masked reconstruction without alignment (MAE). We also report several CM-MAE variants that isolate the contribution of each component: linear probe (frozen encoder), full fine-tuning (uniform LR 10−510^-5), single-modality mild fine-tuning (vision-only and CSI-only), and mild fine-tuning with and without TTA. VI-B Main Results Table I reports cross-scenario Top-1 accuracy. CM-MAE mild fine-tuning reaches 77.4% on Sc6–8, outperforming the ASU baseline by +24.9+24.9 p. TTA adds another +1.3+1.3 p to reach 78.7%. The gain is largest on the hardest scenarios: Sc6 (low-light residential, +22.5+22.5 p) and Sc7 (dense urban blockage, +31.6+31.6 p). TABLE I: Cross-scenario Top-1 accuracy (%) on DeepSense 6G. Labels on Sc1–4; Sc5 validation; Sc6–8 unseen test. CM-MAE variants use fusion unless noted. Method Sc5 Sc6 Sc7 Sc8 Avg6–8 ASU ResNet-18 [2] 58.27 41.67 43.90 71.92 52.50 Linear probe 65.57 21.65 30.82 62.58 38.35 Full FT 85.66 28.74 38.99 86.35 51.36 Vision-only (mild) 43.03 26.77 21.38 40.64 29.60 CSI-only (mild) 81.35 18.90 22.64 63.65 35.06 Mild-FT (ours) 94.26 64.17 75.47 92.48 77.38 Mild-FT + TTA 93.44 66.54 76.73 92.79 78.69 Key observations: (i) Fusion is essential under the diagnostic protocol. Neither single modality generalizes across scenarios: vision-only mild-FT reaches only 29.6% on Sc6–8 (below the ASU vision baseline), and CSI-only reaches 35.1%. Their fusion reaches 77.4%, showing that the learned visual representation and contemporaneous power-vector representation are complementary in this representation-transfer setting. Because the fusion classifier receives the full 64-beam power vector at inference, this row should not be read as a camera-only proactive beam-prediction result. (i) Full fine-tuning destroys generalization. Despite a high Sc5 accuracy of 85.7%, full FT collapses to 51.4% on Sc6–8—at the level of the ASU baseline—because a uniform learning rate overwrites the cross-modal features with scenario-specific cues from Sc1–4. The high Sc5 accuracy is misleading because Sc5 shares the visual domain of the training scenarios. (i) TTA helps where it matters, when target batches are available. Adaptation improves the hardest scenarios Sc6/Sc7 by +2.4+2.4/+1.3+1.3 p while leaving the already-easy Sc8 nearly unchanged, consistent with the entropy-minimization objective targeting uncertain predictions. This gain is not source-only; it uses unlabeled target data and is therefore reported as an optional transductive result. VI-C Ablation Studies We isolate the three most decisive design choices: the pretraining objective, the fine-tuning strategy, and the masking design. (A) Pretraining objectives. We compare six pretraining variants under an identical protocol: ImageNet initialization, Sc1–5 unlabeled data, 160 epochs, batch size 192, evaluated by the same linear-probe procedure (a linear classifier trained on Sc1–4 features and tested on the held-out test split of each scenario; final epoch-159 checkpoint). The variants factorize along two axes—masking (none vs. block masking with reconstruction) and alignment (none, hard InfoNCE [6], or our soft beam-correlation targets)—and include the mainstream SSL paradigms as baselines. Table I yields several findings. First, reconstruction alone (MAE rows) underperforms any alignment-based objective, and block masking without alignment collapses entirely—the contrastive term is the primary representation-learning signal. Second, soft targets beat hard targets at every masking setting: CLS-SoftCtr >> CLS-InfoNCE (+2.6+2.6 p) and CM-MAE-U >> MAE+InfoNCE (+5.9+5.9 p), confirming that physics-grounded partial credit matters because many samples share similar propagation patterns. Third, and most instructive, the ranking reverses between frozen evaluation and fine-tuning. Under a linear probe, CLS-SoftCtr (38.1%) leads our full model (31.7%) because global CLS alignment produces readily separable features. But once the encoder is unfrozen during mild fine-tuning, masked pretraining pulls decisively ahead (Table I): CM-MAE-U reaches 45.0% on Sc6–8 versus 37.3% for CLS-SoftCtr, a +7.7+7.7 p gain concentrated in the hardest unseen scenario (Sc8: 80.1% vs 61.3%). Masked completion forces the encoder to model within-image spatial structure—which patches belong to the same vehicle, which are occluded—and these local features are exactly what mild fine-tuning exploits when adapting to a new scenario. Pure CLS alignment, lacking this local grounding, overfits the global statistics of the training scenarios and transfers poorly. Masked completion is therefore best understood as a regularizer whose benefit materializes at adaptation time, not at the frozen-feature level. TABLE I: Frozen linear probe vs. mild fine-tuning (Avg6–8 Top-1, fusion) for the two strongest pretraining variants. Masked pretraining trails when frozen but leads after adaptation. Pretraining Linear probe Mild FT CLS-SoftCtr (no mask) 38.14 37.34 CM-MAE-U (mask + soft) 31.71 45.04 TABLE I: Ablation on pretraining objectives. All variants pretrained 160 epochs (ImageNet init, Sc1–5, batch 192), linear probe on the test split (fusion features). Variant Mask. Align. Sc6 Sc7 Sc8 Avg CLS-InfoNCE none hard 21.26 32.08 53.37 35.57 CLS-SoftCtr none soft 27.56 27.67 59.20 38.14 MAE block – 11.02 11.32 52.30 24.88 MAE rand. – 12.20 16.98 60.43 29.87 MAE + InfoNCE block hard 11.02 14.47 51.84 25.78 CM-MAE-U block soft 14.17 17.61 63.34 31.71 (B) Fine-tuning strategy. Starting from the same pretrained encoder, we compare three adaptation strategies, all evaluated test-only on Sc6–8 (Table IV): linear probing (frozen encoder, head only), full fine-tuning (uniform LR 10−510^-5 across all parameters), and mild fine-tuning (differential LR). The trend is striking: linear probing is stable but weak (38.4%); full fine-tuning fits the source scenarios well but forgets pretrained knowledge and fails to transfer (51.4%); mild fine-tuning preserves the cross-modal features and reaches 77.4%, a +26.0+26.0 p jump over full FT. This validates our central claim that how the pretrained model is adapted matters as much as the pretraining itself. TABLE IV: Ablation on the fine-tuning strategy (same pretrained encoder, fusion). Strategy Sc5 Sc6 Sc7 Sc8 Avg6–8 Linear probe (frozen) 65.57 21.65 30.82 62.58 38.35 Full FT (uniform LR) 85.66 28.74 38.99 86.35 51.36 Mild FT (ours) 94.26 64.17 75.47 92.48 77.38 (C) Masking design. All masking variants share the full objective (λctr=0.2 _ctr=0.2) and the protocol of (A), changing one design factor at a time from the base configuration (rv=0.75r_v=0.75 block, rc=0.5r_c=0.5, modality dropout p=0.1p=0.1). Table V varies the vision mask ratio rvr_v, the CSI cluster mask ratio rcr_c, the vision masking type (contiguous 4×14×1 blocks vs. uniform random), and modality dropout. Vision masking follows an inverted-U pattern: the base ratio rv=0.75r_v=0.75 (31.7%) outperforms both lighter (rv=0.5r_v=0.5: 28.8%) and heavier (rv=0.9r_v=0.9: 27.4%) masking, because too-few visible tokens prevent the model from inferring scene geometry while too-many make the reconstruction task trivially easy. CSI masking shows a similar pattern, with rc=0.75r_c=0.75 hurting most (−6.3-6.3 p) since aggressive masking destroys the few salient beam clusters. Block masking beats uniform random masking by +2.5+2.5 p: contiguous blocks force the model to fill in larger semantic regions (e.g., an occluded vehicle) from cross-modal context, whereas random tokens can be solved by local interpolation. The no-dropout row is higher in this frozen linear-probe table (34.02% vs. 31.71%), so Table V does not by itself prove a transfer-accuracy gain from modality dropout. We keep dropout in the reported model as a conservative robustness-oriented choice for missing or unreliable modalities, and leave explicit corruption tests as future validation. TABLE V: Ablation on the masking design (full objective, 160 epochs, batch 192), linear probe on the test split (fusion features). Variant Sc6 Sc7 Sc8 Avg6–8 Base (rv=.75r_v=.75, rc=.5r_c=.5, pmd=.1p_md=.1) 14.17 17.61 63.34 31.71 rv=0.50r_v=0.50 10.24 15.72 60.43 28.80 rv=0.90r_v=0.90 15.35 19.50 47.24 27.36 rc=0.25r_c=0.25 12.60 15.72 67.64 31.99 rc=0.75r_c=0.75 8.27 6.29 61.81 25.46 Random masking (no blocks) 12.20 14.47 60.89 29.19 No modality dropout (pmd=0p_md=0) 17.32 16.35 68.40 34.02 What did not work. A domain-adversarial scenario-classification head with a gradient-reversal layer [13] was added to the fused CLS features to encourage scenario-invariant representations. However, the adversarial loss spiked repeatedly during pretraining, causing gradient explosion and NaN; disabling it (λda=0 _da=0) consistently gave the best result. We also found that continuing SSL pretraining on all eight scenarios with a freshly reset optimizer degraded Sc6–8 accuracy (to ∼ 49%), presumably because resetting Adam momentum and the learning-rate schedule disrupted the already-converged features. VI-D Comparison with Published Methods Table VI places CM-MAE among recent published methods. The table is a positioning summary, not a protocol-matched leaderboard: most prior numbers are in-distribution (same-scenario train/test), whereas our main protocol requires generalization to unseen geographic locations. The only directly matched comparison in this table is the ASU ResNet cross-scenario reference. TABLE VI: Positioning against published methods. Cross-scenario: labels on Sc1–4, test on Sc6–8; in-distribution: same-scenario train/test. Method Setting Metric Result ASU ResNet [2] Cross-scenario Top1 Avg6–8 52.5% BeamLLM [18] In-distribution Top1 61.0% Semantic BF [17] Scenario-specific Top1 – Occlusion-Aware [19] In-distribution Top1 50.9% CM-MAE Mild-FT Cross-scenario Top1 Avg6–8 77.4% CM-MAE + TTA Cross-scenario Top1 Avg6–8 78.7% VII Analysis and Discussion VII-A Per-Scenario Analysis Sc8 (highway/urban canyon; 652 test samples) is the easiest scenario at 92.5%, reflecting its larger dataset and more structured propagation environment. Sc6 (254 test samples, low-light residential) and Sc7 (159 samples, dense urban with blockage) are harder but still improve by +22.5+22.5 and +31.6+31.6 p respectively over the ASU baseline. The Sc5 validation accuracy of 94.3% is high because Sc5 shares the visual domain of Sc1–4 (similar lighting and architecture); we caution that Sc5 accuracy is not a reliable proxy for cross-scenario generalization, and the true measure is Sc6–8. The single-modality results reveal an interesting asymmetry: CSI-only mild-FT achieves high Sc5 accuracy (81.4%) but poor Sc6/7 transfer (18.9% / 22.6%), while vision-only is weak everywhere (Sc5 only 43.0%). This suggests that the raw power vector carries strong scenario-specific information that does not transfer, while the pretrained vision stream needs the CSI alignment target to learn transferable features—their combination is what enables generalization. VII-B Effect of the Soft Contrastive Targets The choice of beam-power correlation as the source of soft targets is important. Unlike self-supervised contrastive learning that uses image augmentations to define positives (irrelevant for CSI), our targets are grounded in the physical signal: two samples with similar beam-power profiles genuinely share propagation characteristics. The Gaussian kernel with σh=0.5 _h=0.5 provides a smooth similarity landscape that tolerates the noise in real power measurements. Our ablation (Table I) confirms that this signal is necessary: reconstruction alone leads to vision-stream collapse, while adding the contrastive term yields a representation that continues improving throughout pretraining. VII-C Limitations and Future Work • Path-resolved correspondence. Our current alignment operates at the sample level (CLS tokens). A finer objective—optimal-transport matching between image patches and beam clusters with asymmetric geometry-guided masking (the CM-MAE-W variant in our formulation)—requires ray-traced path labels (AoA/AoD, path powers) that are only available from simulators such as Wireless InSite. Evaluating it on synthetic data and transferring the representation to real measurements is ongoing work. • Temporal modeling. Predictions are single-frame; incorporating short image–CSI sequences (as in the DeepSense challenge task) could add robustness against transient occlusions. • Pretraining scale and evaluation. Our negative result on all-scenario continued pretraining suggests that optimizer-state handling and learning-rate scheduling for continued SSL deserve further study. Evaluation on the hidden DeepSense challenge test set would also provide an unbiased external comparison. • Computational cost. The dual-stream ViT-Base architecture is large (∼ 232 M parameters); distillation into a smaller model would aid deployment. VIII Conclusion We presented CM-MAE, a physics-guided cross-modal masked autoencoder for vision–wireless representation transfer in 6G beam classification. Self-supervised alignment via beam-power soft contrastive learning improves matched transfer ablations, and a mild differential-LR fine-tuning strategy preserves pretrained cross-modal features better than uniform full fine-tuning. Under our DeepSense 6G cross-scenario protocol, CM-MAE reaches 77.4% Top-1 on Sc6–8, a +24.9+24.9 p gain over the protocol-matched ASU ResNet reference; optional transductive normalization adaptation reaches 78.7%. Because the fusion setting uses the contemporaneous 64-beam power vector, these results support cross-scenario representation transfer rather than proactive reduced-sweep beam prediction. References [1] A. Alkhateeb et al., “DeepSense 6G: A large-scale real-world multi-modal sensing and communication dataset,” IEEE Commun. Mag., vol. 61, no. 9, p. 122–128, 2023. [2] G. Charan, T. Osman, A. Hredzak, N. Thawdar, and A. Alkhateeb, “Vision-position multi-modal beam prediction using real millimeter wave datasets,” in Proc. IEEE WCNC, 2022. [3] K. He et al., “Masked autoencoders are scalable vision learners,” in Proc. IEEE/CVF CVPR, 2022. [4] R. Bachmann, D. Mizrahi, A. Atanov, and A. Zamir, “MultiMAE: Multi-modal multi-task masked autoencoders,” in Proc. ECCV, 2022. [5] Y. Gong et al., “Contrastive audio-visual masked autoencoder,” in Proc. ICLR, 2023. [6] A. Radford et al., “Learning transferable visual models from natural language supervision,” in Proc. ICML, 2021. [7] S. Alikhani, G. Charan, and A. Alkhateeb, “Large Wireless Model (LWM): A foundation model for wireless channels,” arXiv:2411.08872, 2024. [8] B. Guler, G. Geraci, and H. Jafarkhani, “A multi-task foundation model for wireless channel representation using contrastive and masked autoencoder learning,” arXiv:2505.09160, 2025. [9] M. Farzanullah et al., “Wireless Multimodal Foundation Model (WMFM): Integrating vision and communication modalities for 6G ISAC systems,” arXiv:2512.23897, 2025. [10] H. Zhang, S. Gao, and X. Cheng, “WiFo-M2: Plug-and-play multi-modal sensing via foundation model to empower wireless communications,” arXiv:2601.09179, 2026. [11] X. Liu et al., “WiFo-MiSAC: A wireless foundation model for multimodal sensing and communication integration via synesthesia of machines,” arXiv:2604.18255, 2026. [12] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell, “Tent: Fully test-time adaptation by entropy minimization,” in Proc. ICLR, 2021. [13] Y. Ganin et al., “Domain-adversarial training of neural networks,” J. Mach. Learn. Res., vol. 17, no. 59, p. 1–35, 2016. [14] R. W. Heath Jr. et al., “An overview of signal processing techniques for millimeter wave MIMO systems,” IEEE J. Sel. Topics Signal Process., vol. 10, no. 3, p. 436–453, 2016. [15] M. Tancik et al., “Fourier features let networks learn high frequency functions in low dimensional domains,” in Proc. NeurIPS, 2020. [16] M. Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transport,” in Proc. NeurIPS, 2013. [17] A. D. Raha, K. Kim, A. Adhikary, M. Gain, and C. S. Hong, “Advancing ultra-reliable 6G: Transformer and semantic localization empowered robust beamforming in mmWave communications,” arXiv preprint arXiv:2406.02000, 2024. [18] C. Zheng, J. He, G. Cai, Z. Yu, and C. G. Kang, “BeamLLM: Vision-empowered mmWave beam prediction with large language models,” arXiv preprint arXiv:2503.10432, 2025. [19] A. Orimogunje, H. Park, K.-J. Cha, I. Orikumhi, S. Kim, and D. Vukobratovic, “Occlusion-aware multimodal beam prediction and pose estimation for mmWave V2I,” arXiv preprint arXiv:2603.25799, 2026.