Paper deep dive

WiFi2Cap: Semantic Action Captioning from Wi-Fi CSI via Limb-Level Semantic Alignment

Tzu-Ti Wei, Chu-Yu Huang, Yu-Chee Tseng, Jen-Jee Chen

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 33

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/26/2026, 1:37:52 AM

Summary

WiFi2Cap is a three-stage framework designed to generate fine-grained natural-language action captions directly from Wi-Fi Channel State Information (CSI). It addresses the semantic gap between wireless signals and language, as well as direction-sensitive ambiguities like left/right limb confusion, by utilizing a vision-language teacher for knowledge transfer, a CSI student for cross-modal alignment, and a prefix-tuned language model for caption generation. The authors also introduce the WiFi2Cap Dataset, a synchronized benchmark for CSI-based semantic captioning.

Entities (5)

WiFi2Cap · framework · 100%WiFi2Cap Dataset · dataset · 100%Wi-Fi CSI · technology · 100%CLIP · model · 95%Mirror-Consistency Loss · algorithmloss-function · 95%

Relation Signals (3)

WiFi2Cap → generates → Action Captions

confidence 98% · WiFi2Cap, a three-stage framework for generating action captions directly from Wi-Fi CSI.

WiFi2Cap Dataset → supports → WiFi2Cap

confidence 95% · We also introduce the WiFi2Cap Dataset, a synchronized benchmark for CSI-based semantic captioning.

WiFi2Cap → uses → Mirror-Consistency Loss

confidence 95% · To improve direction-sensitive captioning, we introduce a Mirror-Consistency Loss

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Privacy-preserving semantic understanding of human activities is important for indoor sensing, yet existing Wi-Fi CSI-based systems mainly focus on pose estimation or predefined action classification rather than fine-grained language generation. Mapping CSI to natural-language descriptions remains challenging because of the semantic gap between wireless signals and language and direction-sensitive ambiguities such as left/right limb confusion. We propose WiFi2Cap, a three-stage framework for generating action captions directly from Wi-Fi CSI. A vision-language teacher learns transferable supervision from synchronized video-text pairs, and a CSI student is aligned to the teacher's visual space and text embeddings. To improve direction-sensitive captioning, we introduce a Mirror-Consistency Loss that reduces mirrored-action and left-right ambiguities during cross-modal alignment. A prefix-tuned language model then generates action descriptions from CSI embeddings. We also introduce the WiFi2Cap Dataset, a synchronized CSI-RGB-sentence benchmark for semantic captioning from Wi-Fi signals. Experimental results show that WiFi2Cap consistently outperforms baseline methods on BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE, demonstrating effective privacy-friendly semantic sensing.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

33,009 characters extracted from source content.

Expand or collapse full text

WiFi2Cap: Semantic Action Captioning from Wi-Fi CSI via Limb-Level Semantic Alignment Tzu-Ti Wei, Chu-Yu Huang, Yu-Chee Tseng, and Jen-Jee Chen Abstract Privacy-preserving semantic understanding of human activities is important for indoor sensing, yet existing Wi-Fi CSI-based systems mainly focus on pose estimation or predefined action classification rather than fine-grained language generation. Mapping CSI to natural-language descriptions remains challenging because of the semantic gap between wireless signals and language and direction-sensitive ambiguities such as left/right limb confusion. We propose WiFi2Cap, a three-stage framework for generating action captions directly from Wi-Fi CSI. A vision–language teacher learns transferable supervision from synchronized video–text pairs, and a CSI student is aligned to the teacher’s visual space and text embeddings. To improve direction-sensitive captioning, we introduce a Mirror-Consistency Loss that reduces mirrored-action and left–right ambiguities during cross-modal alignment. A prefix-tuned language model then generates action descriptions from CSI embeddings. We also introduce the WiFi2Cap Dataset, a synchronized CSI–RGB–sentence benchmark for semantic captioning from Wi-Fi signals. Experimental results show that WiFi2Cap consistently outperforms baseline methods on BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE, demonstrating effective privacy-friendly semantic sensing. I Introduction Generating natural-language descriptions of human activities is a key capability for human-centered sensing and interaction. Recent vision–language models and contrastive pretraining (e.g., CLIP [18]) have substantially improved captioning from visual inputs, but deploying cameras in private indoor spaces (e.g., bedrooms and hospital wards) raises serious privacy concerns. This motivates privacy-preserving alternatives that avoid recording identifiable appearance information. Prior work has explored non-visual modalities such as mmWave radar, infrared sensing, and Wi-Fi signals [13, 21, 20, 6, 5, 19, 26, 25]. Among them, Wi-Fi Channel State Information (CSI) is robust to occlusion and lighting, and has enabled fine-grained perception such as pose estimation [23, 25, 28]. However, existing CSI-based systems largely target low- or mid-level outputs, such as joint coordinates, segmentation masks, or predefined action labels, rather than free-form and fine-grained language generation. As a result, mapping numerical radio signals to natural-language descriptions remains challenging for two reasons. First, there is a substantial semantic gap between CSI and language, together with limited paired CSI–text supervision. Second, action captioning requires preserving direction-sensitive semantics, such as left/right limbs and mirrored movements, which are easily confused during cross-modal alignment. Figure 1: WiFi2Cap framework. (a) Stage 1: Vision–language teacher trained with contrastive learning and Mirror-Consistency. (b) Stage 2: CSI–text student trained via teacher-guided visual alignment and CSI–text contrastive learning with Mirror-Consistency. (c) Stage 3: Prefix-guided language generation. To address these challenges, we study the largely unexplored problem of semantic action captioning directly from Wi-Fi CSI. We propose WiFi2Cap, a three-stage framework in which a data-rich vision–language teacher transfers semantic knowledge to a CSI student through contrastive alignment, followed by caption generation with a frozen autoregressive language model via prefix tuning (Fig. 1). To reduce frequent confusions between mirrored actions, such as left–right limb ambiguity, we further introduce a Mirror-Consistency Loss during alignment. To support both training and evaluation, we also introduce the WiFi2Cap Dataset, a synchronized benchmark for CSI-based semantic captioning that combines Wi-Fi CSI, RGB videos, and sentence-level action descriptions. The dataset covers 100 action classes, with each instance recorded as a 5-second clip and captured with one transmitter and three receivers. Contributions. This paper makes three contributions: • Wi-Fi CSI action captioning: We propose WiFi2Cap, a framework that generates fine-grained action captions directly from CSI by combining visual knowledge transfer with contrastive learning. • Mirror-Consistency for direction-sensitive semantics: We identify left/right limb confusion and mirrored-action ambiguity as a critical failure mode in cross-modal action captioning, and introduce a mirror-consistency objective to explicitly enforce limb-aware, direction-sensitive alignment. • Dataset and evaluation: We introduce the WiFi2Cap Dataset, a synchronized CSI–RGB–sentence benchmark tailored to semantic caption generation, and demonstrate consistent gains over CSI-only baselines on standard captioning metrics. I Related Work I-A RF-Based Human Sensing RF sensing has gained increasing attention for device-free human understanding due to its robustness to lighting and occlusion and its privacy advantage over cameras [16, 3]. Beyond Wi-Fi, other privacy-preserving modalities such as mmWave radar and infrared sensing have also been explored for human-centric semantic understanding [13, 20, 5], and recent radar–language models further suggest a growing interest in language-grounded semantic interpretation of non-visual signals [17, 27]. CSI-based systems can recover rich motion cues for tasks such as segmentation and 2D/3D pose estimation [23, 25, 9], and recent Transformer variants further improve temporal–spectral modeling [29]. However, most RF works still target geometric or categorical outputs (poses or action labels), while free-form language generation from Wi-Fi signals remains largely unexplored. I-B Cross-Modal Knowledge Transfer Cross-modal transfer learning addresses limited supervision in the target modality by leveraging a stronger source modality [30]. Knowledge distillation transfers soft predictions or intermediate representations from a teacher to a student [8]. Recent studies extend this idea to cross-modal representation distillation, where a vision–language teacher defines a semantic embedding space and a student from another modality is trained to align paired samples while separating mismatches [11]. WiFi2Cap follows this paradigm by aligning CSI features to a vision–language teacher to bridge the modality gap under scarce CSI–text pairs. I-C Contrastive Learning Contrastive objectives such as InfoNCE [22] and its scalable variants (SimCLR [4], MoCo [7]) have shown strong transferability, and CLIP [18] demonstrates that large-scale contrastive pretraining can learn highly semantic cross-modal embeddings. A practical challenge for action understanding is mirror ambiguity (e.g., left/right limbs), which motivates our mirror-consistency regularization during alignment. I-D Prefix/Prompt Tuning for Language Conditioning Prefix/prompt tuning conditions a frozen language model using a small number of trainable parameters, avoiding full fine-tuning. Prefix-Tuning [14] injects learnable key/value prefixes into Transformer attention, while Prompt Tuning [12] and P-Tuning v2 [15] learn continuous prompts/prefix-like parameters across layers. Following this line, WiFi2Cap maps CSI embeddings into per-layer prefixes via a lightweight MLP for CSI-conditioned caption generation. I WiFi2Cap Dataset Existing Wi-Fi sensing datasets mainly target pose estimation or action classification and do not provide the multimodal supervision needed for CSI-based caption generation. To support this task, we construct the WiFi2Cap dataset, a synchronized CSI–RGB–sentence benchmark that supports all three stages of our framework. The dataset covers 100 action categories, and each sample is a 5-second clip with synchronized CSI, RGB observations, and a sentence-level description. I-A Data Collection For each action category, participants perform a predefined movement. To introduce controlled spatial diversity, before every recording the participant randomly selects one of 24 standing positions arranged in a 4×64× 6 grid. During recording, CSI is collected with one transmitter and three receivers around the participant, while an RGB camera provides the visual reference. All devices remain fixed, and CSI streams are synchronized with RGB videos for aligned training and evaluation (Fig. 2). Figure 2: Data acquisition setup and multimodal samples. Left: the physical collection setup, including one transmitter (Tx), three receivers (Rx1–Rx3), the RGB camera, and the 4×64× 6 grid of participant positions. Right: synchronized CSI heatmaps, RGB frames, and sentence-level action descriptions. I-B Data Annotation Each action category is paired with a textual description to supervise both semantic alignment and caption generation. We start from concise action prompts (e.g., “standing and waving the left hand”) and use a GPT-based large language model (LLM) for wording refinement, while manually verifying all descriptions to avoid semantic drift. I-C CSI Preprocessing We decode CSI using PicoScenes [10]. For each receiver, CSI is represented as complex channel responses over time and decomposed into amplitude and phase sequences. We remove pilot and guard subcarriers and apply standard phase sanitization [24]. The three synchronized receivers are treated as multi-view observations, producing the CSI inputs used by the student encoder and caption generator. IV WiFi2Cap Framework As shown in Fig. 1, WiFi2Cap generates action captions from Wi-Fi CSI in three stages: (i) a vision–language teacher that learns a shared video–text embedding space with contrastive learning and mirror-consistency regularization; (i) a CSI student that encodes CSI amplitude/phase, fuses them with a gating module, and aligns CSI embeddings to the teacher’s visual/text embeddings; and (i) prefix-guided generation that conditions a frozen language model by mapping CSI embeddings into per-layer key/value prefixes. IV-A Vision–Language Teacher Following [2], we use frozen CLIP encoders for frames and captions and learn lightweight temporal/projection modules to align video and text in a shared space, as illustrated in Stage 1 of Fig. 1. IV-A1 Vision Branch For the i-th video, we uniformly sample L frames xi,1,…,xi,L\x_i,1,…,x_i,L\. Each frame is encoded by the frozen CLIP image encoder ϕv(⋅) _v(·) to obtain ℓ2 _2-normalized features i,j=ϕv(xi,j)∈ℝDz_i,j= _v(x_i,j) ^D. We form i∈ℝL×DZ_i ^L× D, add positional embeddings P, and aggregate temporally using a Transformer gθ(⋅)g_θ(·): i=gθ(i+).H_i=g_θ(Z_i+P). (1) Temporal average pooling yields i=1L∑j=1Li,ju_i= 1L _j=1^LH_i,j, which is projected and normalized to the final video embedding i∈ℝdv_i ^d. IV-A2 Text Branch For caption yiy_i, we extract i=ϕt(yi)∈ℝDq_i= _t(y_i) ^D using the frozen CLIP text encoder and project it to i∈ℝdt_i ^d with ℓ2 _2 normalization. IV-A3 Contrastive Objective Given a minibatch of N pairs (i,i)i=1N\(v_i,t_i)\_i=1^N, we compute cosine similarities sij=i⊤j/τs_ij=v_i t_j/τ and minimize a symmetric InfoNCE loss: ℒconv→t _con^v→ t =−1N∑i=1Nlog⁡exp⁡(sii)∑j=1Nexp⁡(sij), =- 1N _i=1^N (s_i) _j=1^N (s_ij), (2) ℒcont→v _con^t→ v =−1N∑i=1Nlog⁡exp⁡(sii)∑j=1Nexp⁡(sji), =- 1N _i=1^N (s_i) _j=1^N (s_ji), ℒcon _con =12(ℒconv→t+ℒcont→v). = 12 (L_con^v→ t+L_con^t→ v ). IV-A4 Mirror-Consistency Loss To disambiguate mirrored semantics (e.g., left/right limbs), we create a mirrored pair (x~i,y~i)( x_i, y_i) by horizontally flipping frames and swapping directional words in the caption. With margin m>0m>0, we enforce that each visual embedding matches its correct caption more than its mirrored caption: ℒmc(i) _mc^(i) =max⁡(0,m+s(i,~i)−s(i,i)) = (0,\,m+s(v_i, t_i)-s(v_i,t_i) ) (3) +max⁡(0,m+s(~i,i)−s(~i,~i)). + (0,\,m+s( v_i,t_i)-s( v_i, t_i) ). The teacher objective is ℒteacher=ℒcon+λmcℒmcL_teacher=L_con+ _mcL_mc. IV-B CSI–Text Student The student is trained in two steps, as illustrated in Stage 2 of Fig. 1: (1) align CSI embeddings to the frozen teacher’s visual embeddings; and (2) align CSI to text with contrastive learning and mirror-consistency. IV-B1 Vision–CSI Alignment For receiver r, we denote CSI amplitude/phase as i(r),i(r)∈ℝT×Na×NscM_i^(r), _i^(r) ^T× N_a× N_sc. We encode them with two ResNet-18 backbones fampf_amp and fphaf_pha (no weight sharing) and apply global average pooling to obtain i(r),i(r)∈ℝdca_i^(r),p_i^(r) ^d_c: i(r) _i^(r) =famp(i(r)), =f_amp\! (M_i^(r) ), (4) i(r) _i^(r) =fpha(i(r)). =f_pha\! ( _i^(r) ). (5) We fuse amplitude and phase via a gating module [1]. Given i(r)=[i(r);i(r)]u_i^(r)=[a_i^(r);p_i^(r)], a gate i(r)∈[0,1]dcg_i^(r)∈[0,1]^d_c produces i(r)=i(r)⊙i(r)+(−i(r))⊙i(r).f_i^(r)=g_i^(r) _i^(r)+(1-g_i^(r)) _i^(r). (6) A projection WpW_p maps i(r)f_i^(r) to i(r)=norm(Wpi(r))∈ℝdc_i^(r)=norm(W_pf_i^(r)) ^d. We average valid receiver views and normalize to obtain ¯i c_i, then align ¯i c_i to the teacher visual embedding iv_i using a symmetric InfoNCE distillation loss ℒalignL_align. IV-B2 CSI–Text Alignment We further align ¯i c_i to the CLIP text embedding it_i with a symmetric InfoNCE loss ℒconL_con. To improve direction sensitivity, we apply a text-only mirror-consistency loss using swapped captions y~i y_i: ℒmc(i)=max⁡(0,m+s(¯i,~i)−s(¯i,i)).L_mc^(i)= \! (0,\,m+s( c_i, t_i)-s( c_i,t_i) ). (7) The student objective is ℒstudent=ℒcon+λmcℒmcL_student=L_con+ _mc\,L_mc. Figure 3: CSI encoder. Dual ResNet-18 backbones encode amplitude and phase inputs, followed by gated fusion and projection to a CSI embedding. IV-C Prefix-Guided Language Generation We condition a frozen GPT-2 decoder on CSI by prefix tuning. Given CSI embedding ¯i∈ℝd c_i ^d, a lightweight MLP gϕg_φ produces layer-wise key/value prefixes injected into each self-attention block. Let L be the number of layers, prefix length LpL_p, and hidden size dhd_h (dh=768d_h=768). We reshape gϕ(¯i)∈ℝL×Lp×2dhg_φ( c_i) ^L× L_p× 2d_h into prefixes Ki(ℓ),Vi(ℓ)ℓ=1L\K_i^( ),V_i^( )\_ =1^L. For tokenized caption i=(yi,1,…,yi,Ti)y_i=(y_i,1,…,y_i,T_i), the conditional likelihood is pθ(i∣¯i)=∏t=1Tipθ(yi,t∣yi,1:t−1,gϕ(¯i)),p_θ(y_i c_i)= _t=1^T_ip_θ (y_i,t y_i,1:t-1,\,g_φ( c_i) ), (8) and we minimize the standard autoregressive loss ℒLM=−1N∑i=1N∑t=1Tilog⁡pθ(yi,t∣yi,1:t−1,gϕ(¯i)).L_LM=- 1N _i=1^N _t=1^T_i p_θ (y_i,t y_i,1:t-1,\,g_φ( c_i) ). (9) In practice, GPT-2 remains frozen and we optimize only gϕg_φ. V Experiment Results V-A Main Results on the WiFi2Cap Dataset We first evaluate caption generation on the proposed WiFi2Cap dataset, which is specifically constructed to support CSI-based semantic captioning. We compare two systems: (i) a baseline (CSI→ ) that directly conditions the language model on the CSI encoder without vision–language alignment or mirror-consistency training; and (i) the full WiFi2Cap pipeline. TABLE I: Captioning results on the WiFi2Cap and Person-in-WiFi 3D datasets. Best scores within each dataset block are in bold. Dataset Method BLEU-4 METEOR ROUGE-L CIDEr SPICE WiFi2Cap Baseline (CSI→ ) 14.85 25.86 32.38 0.12 0.28 Full WiFi2Cap 51.78 57.48 64.32 0.52 0.63 Person-in-WiFi 3D Baseline (CSI→ ) 12.15 20.38 26.12 0.10 0.23 Full WiFi2Cap 47.07 58.80 57.26 0.43 0.51 Table I shows that WiFi2Cap clearly outperforms the baseline on all captioning metrics on the proposed WiFi2Cap dataset. BLEU-4 improves from 14.85 to 51.78, METEOR from 25.86 to 57.48, and ROUGE-L from 32.38 to 64.32, with consistent gains in CIDEr and SPICE. These results indicate that the proposed teacher-guided alignment and prefix-based generation strategy substantially improve CSI-to-text captioning quality on our dataset. V-B Transferability on the Person-in-WiFi 3D Dataset We further evaluate transferability on the public Person-in-WiFi 3D dataset [25]. To avoid confounding factors from multi-person scenes and strong view changes, we select a single indoor scene and filter clips to a single-person subset. Each clip is associated with one of eight action categories. Because the dataset does not provide free-form captions, we convert these category labels into natural sentences using a lightweight GPT prompting recipe (1 sentence, present tense, concise, no background details) and manually spot-check the outputs. We reuse the same architectures and hyperparameters as in Sec. V-A. As shown in Table I, WiFi2Cap also generalizes well to this external dataset. Relative to the baseline, WiFi2Cap improves BLEU-4 from 12.15 to 47.07, METEOR from 20.38 to 58.80, and ROUGE-L from 26.12 to 57.26, with consistent improvements in CIDEr and SPICE. These results suggest that the proposed alignment-then-generation pipeline remains effective under a different capture setup, even when captions are synthesized from categorical labels. V-C Qualitative Validation and Examples Representative qualitative examples are summarized in Table I. We show one successful case and one partial mismatch case. In the successful case, the generated caption preserves the core action semantics and differs only in a near-synonymous verb choice. In the partial mismatch case, the model correctly captures the pose and balance but misses the directional phrase. Overall, these examples suggest that WiFi2Cap usually produces specific and fluent action descriptions, while remaining errors are mainly fine-grained directional confusions. TABLE I: Qualitative examples of generated captions. Ground Truth Prediction Sample 1 (correct) The person raises both arms and moves them back and forth in a synchronized waving motion while staying in a squat. The person raises both arms and swings them back and forth in a synchronized waving motion while staying in a squat. Sample 2 (partially correct) A man stands upright on his left foot, facing forward, and maintains his balance with a steady posture. A man stands upright on his left foot, facing away, while maintaining his balance and keeping his body steady. V-D Ablation Study We analyze the following axes: (i) the relative importance of different training stages; (i) the contribution of the mirror-consistency loss; (i) the choice of language model for prefix-guided caption generation; and (iv) the text-side backbone (CLIP B/32 vs. L/14). All results are reported using BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE. V-D1 Ablation of Training Stages Table I quantifies the contribution of each stage. Using only Stage 2-2 (CSI–Text Alignment) performs poorly (e.g., BLEU-4 = 22.57, METEOR = 35.02, ROUGE-L = 40.39). Adding Stage 1 (Vision–Language Teacher), Stage 2-1 (Vision–CSI Alignment), and Stage 3 (Prefix-Guided Tuning) yields a large jump (BLEU-4 = 47.10, METEOR = 51.08, ROUGE-L = 57.74). Enabling the full pipeline brings the best scores (BLEU-4 = 51.78, METEOR = 57.48, ROUGE-L = 64.32), showing that the stages are complementary: the teacher establishes a stable semantic target, the student aligns CSI to text, and the prefix injects CSI semantics directly into generation. TABLE I: Ablation on training stages. A check mark indicates the stage is enabled. Stages Metrics S1 S2-1 S2-2 S3 BLEU-4 METEOR ROUGE-L CIDEr SPICE – – ✓ ✓ 22.57 35.02 40.39 0.21 0.38 ✓ ✓ – ✓ 47.10 51.08 57.74 0.47 0.61 ✓ ✓ ✓ ✓ 51.78 57.48 64.32 0.52 0.63 V-D2 Mirror-Consistency Analysis Table IV compares training without the mirror-consistency loss, with teacher-only mirroring (Stage 1 only), and with full mirror-consistency (Stages 1 and 2). The mirror term yields consistent improvements in overall caption quality while specifically helping preserve direction-sensitive semantics, such as left/right limb usage and mirrored action directions. By lifting all five captioning metrics from the no-mirror baseline to the full model, these results support mirror-consistency as a core component for semantic disambiguation rather than a minor auxiliary regularizer. TABLE IV: Effect of the mirror-consistency loss. Setting BLEU-4 METEOR ROUGE-L CIDEr SPICE w/o Mirror-Consistency 35.25 44.53 51.04 0.36 0.46 Teacher-only Mirror 44.38 52.29 58.75 0.47 0.59 Full Mirror-Consistency 51.78 57.48 64.32 0.52 0.63 V-D3 Choice of Language Model Table V compares GPT-2, Qwen, and microsoft/phi-2 as the decoder in Stage 3. Qwen attains the strongest overall caption quality (highest BLEU-4 and METEOR, with solid CIDEr/SPICE), GPT-2 achieves the best ROUGE-L but lags on BLEU-4/METEOR, and phi-2 trails on most metrics. These trends indicate that both model capacity and pretraining data materially affect CSI-conditioned generation. TABLE V: Effect of the language model in Stage 3 (CSI-conditioned generation). Language Model BLEU-4 METEOR ROUGE-L CIDEr SPICE GPT-2 51.78 57.48 64.32 0.52 0.63 Microsoft/phi-2 47.97 52.75 59.55 0.46 0.62 Qwen 55.11 60.13 63.83 0.53 0.66 V-D4 Text-Side Backbone Table VI contrasts CLIP B/32 vs. L/14 on the text side. L/14 gives small but consistent gains over B/32 across BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE while preserving the same ranking among language models. This suggests that a stronger text backbone modestly improves the semantic target used to supervise CSI. TABLE VI: Choice of backbone for the text encoder. Text Backbone BLEU-4 METEOR ROUGE-L CIDEr SPICE CLIP B/32 51.78 57.48 64.32 0.52 0.63 CLIP L/14 53.76 58.37 65.21 0.53 0.67 V-E Additional Analysis V-E1 Left–Right Confusion Analysis of VLMs To further verify that direction-sensitive ambiguity is a real failure mode rather than an artifact of our dataset, we conduct a hand-side recognition test on Qwen2-VL-2B-Instruct. We query the model with a binary question: “Is the person using their left hand or right hand? Answer with ‘left’ or ‘right’.” Using the original pretrained weights, Qwen2-VL-2B-Instruct attains 53.3% accuracy, indicating substantial ambiguity in distinguishing mirrored limb semantics. We then fine-tune Qwen2-VL-2B-Instruct on the UTD-MHAD dataset using the proposed Mirror-Consistency loss by pairing each training image with its horizontally flipped version. The accuracy increases to 73.3%, demonstrating that mirror-consistency supervision effectively improves direction-sensitive understanding and supporting our use of this objective in WiFi2Cap. V-E2 Visualization of Modality Alignment Fig. 4 visualizes the cosine similarity matrix between CSI and text embeddings. Before Stage 2-2, the matrix shows weak structure and a low top-1 matching accuracy of 0.067. After the proposed training, the diagonal becomes much clearer and the top-1 accuracy increases to 0.600, indicating substantially improved alignment between CSI and text representations. Figure 4: Visualization of modality alignment between CSI and text embeddings. Each row denotes a CSI embedding and each column denotes a text embedding; brighter values indicate higher cosine similarity. A clearer diagonal pattern after training indicates stronger CSI–text correspondence. VI Conclusions We introduced WiFi2Cap, a three-stage framework for generating natural-language action descriptions directly from Wi-Fi CSI. The framework addresses two central challenges in CSI-based semantic captioning: bridging the modality gap between wireless signals and language, and preserving direction-sensitive semantics such as left/right limb descriptions. To this end, WiFi2Cap transfers semantic supervision from synchronized video–text pairs to CSI through teacher-guided alignment and CSI–text contrastive learning, while the proposed Mirror-Consistency Loss mitigates mirrored-action and left–right ambiguities during cross-modal alignment. We also introduced the WiFi2Cap Dataset, a synchronized CSI–RGB–sentence benchmark for semantic captioning from Wi-Fi signals. Experiments and ablations show consistent gains in caption quality and direction-sensitive disambiguation. Overall, WiFi2Cap establishes a privacy-friendly bridge from wireless sensing to fine-grained semantic understanding. References [1] J. Arevalo, T. Solorio, M. Montes, and F. A. González (2017) Gated multimodal units for information fusion. arXiv preprint arXiv:1702.01992. External Links: Document Cited by: §IV-B1. [2] M. Benavent-Lledo, D. Mulero-Pérez, D. Ortiz-Perez, and J. Garcia-Rodriguez (2025) Text-driven online action detection. Integrated Computer-Aided Engineering 32 (4), p. 415–423. Cited by: §IV-A. [3] L. Biase, P. Pecoraro, G. Pecoraro, M. L. Caminiti, and V. D. Lazzaro (2022) Markerless radio frequency indoor monitoring for telemedicine: gait analysis, indoor positioning, fall detection, tremor analysis, vital signs and sleep monitoring. Sensors 22, p. 8486. External Links: Document Cited by: §I-A. [4] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton (2020) A simple framework for contrastive learning of visual representations. In ICML, Cited by: §I-C. [5] S. Chiu, Y. Huang, C. Lin, Y. Tseng, J. Chen, M. Tu, B. Tung, and Y. Nieh (2023) Privacy-preserving video conferencing via thermal-generative images. In 2023 IEEE International Conference on Robotics and Automation (ICRA), p. 9478–9485. Cited by: §I, §I-A. [6] R. Gade and T. B. Moeslund (2014) Thermal cameras and applications: a survey. Machine vision and applications 25 (1), p. 245–262. Cited by: §I. [7] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick (2020) Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on CVPR, p. 9726–9735. External Links: Document Cited by: §I-C. [8] G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: §I-B. [9] W. Jiang, H. Xue, C. Miao, S. Wang, S. Lin, C. Tian, S. Murali, H. Hu, Z. Sun, and L. Su (2020) Towards 3d human pose construction using wifi. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking (MobiCom), p. 1–14. External Links: Document Cited by: §I-A. [10] Z. Jiang, T. H. Luan, H. Hao, J. Wang, X. Ren, K. Zhao, W. Xi, Y. Xu, and R. Li (2020) Eliminating the barriers: demystify wi-fi baseband design and introduce picoscenes wi-fi sensing platform. arXiv preprint arXiv:2010.10233. External Links: Document Cited by: §I-C. [11] S. Kim, R. Xiao, M. Georgescu, S. Alaniz, and Z. Akata (2025) COSMOS: cross-modality self-distillation for vision language pre-training. In Proceedings of the IEEE/CVF Conference on CVPR, p. 14690–14700. External Links: Document Cited by: §I-B. [12] B. Lester, R. Al-Rfou, and N. Constant (2021) The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 3045–3059. Cited by: §I-D. [13] W. Li, W. Lei, K. Shi, Z. Shi, Y. Wang, and J. Zhou (2024) MmSkeleton: 3d human skeleton estimation using millimeter wave radar sparse point clouds. In 2024 IEEE/CIC International Conference on Communications in China (ICCC), p. 307–312. Cited by: §I, §I-A. [14] X. L. Li and P. Liang (2021) Prefix-tuning: optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), p. 4582–4597. External Links: Document Cited by: §I-D. [15] X. Liu, K. Ji, Y. Fu, W. Tam, Z. Du, Z. Yang, and J. Tang (2022) P-tuning: prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), p. 61–68. Cited by: §I-D. [16] L. Nguyen, P. Susarla, A. Mukherjee, M. Cañellas, C. Á. Casado, X. Wu, O. Silvén, D. Jayagopi, and M. B. Lopez (2024) Non-contact multimodal indoor human monitoring systems: a survey. Information Fusion 110, p. 102457. External Links: Document Cited by: §I-A. [17] M. Pushkareva, Y. Feldman, C. Domokos, K. Rambach, and D. Di Castro (2024) Radar spectra-language model for automotive scene parsing. In 2024 International Radar Conference (RADAR), p. 1–6. Cited by: §I-A. [18] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §I, §I-C. [19] Y. Ren, Z. Wang, Y. Wang, S. Tan, Y. Chen, and J. Yang (2022) GoPose: 3d human pose estimation using wifi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6 (2), p. 1–25. Cited by: §I. [20] M. Ryoo, B. Rothrock, C. Fleming, and H. J. Yang (2017) Privacy-preserving human activity recognition from extreme low resolution. In Proceedings of the AAAI conference on artificial intelligence, Vol. 31. Cited by: §I, §I-A. [21] A. Sengupta, F. Jin, R. Zhang, and S. Cao (2020) Mm-pose: real-time human skeletal posture estimation using mmwave radars and cnns. IEEE sensors journal 20 (17), p. 10032–10044. Cited by: §I. [22] A. van den Oord, Y. Li, and O. Vinyals (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. External Links: Document Cited by: §I-C. [23] F. Wang, S. Zhou, S. Panev, J. Han, and D. Huang (2019) Person-in-wifi: fine-grained person perception using wifi. In Proceedings of the IEEE/CVF ICCV, p. 5451–5460. External Links: Document Cited by: §I, §I-A. [24] X. Wang, L. Gao, and S. Mao (2015) PhaseFi: phase fingerprinting for indoor localization with a deep learning approach. In Proceedings of the IEEE Global Communications Conference (GLOBECOM), p. 1–6. External Links: Document Cited by: §I-C. [25] K. Yan, F. Wang, B. Qian, H. Ding, J. Han, and X. Wei (2024) Person-in-wifi 3d: end-to-end multi-person 3d pose estimation with wi-fi. In Proceedings of the IEEE/CVF Conference on CVPR, p. 969–978. External Links: Document Cited by: §I, §I-A, §V-B. [26] Y. Yang, P. Hu, J. Shen, H. Cheng, Z. An, and X. Liu (2024) Privacy-preserving human activity sensing: a survey. High-Confidence Computing 4 (1), p. 100204. Cited by: §I. [27] J. Yuan, S. Chen, S. Yao, R. Zhang, H. C. Feng, K. Jiang, Y. Shang, Y. Zhao, and B. Tang (2026) Sig2text: a vision-language model for non-cooperative radar signal parsing. IET Radar, Sonar & Navigation 20 (1), p. e70113. Cited by: §I-A. [28] M. Zhao, T. Li, M. A. Alsheikh, Y. Tian, H. Zhao, A. Torralba, and D. Katabi (2018) Through-wall human pose estimation using radio signals. In Proceedings of the IEEE/CVF Conference on CVPR, p. 7356–7365. External Links: Document Cited by: §I. [29] Y. Zhou, C. Xu, L. Zhao, A. Zhu, F. Hu, and Y. Li (2023) CSI-former: pay more attention to pose estimation with wifi. Entropy 25 (1), p. 20. External Links: Document Cited by: §I-A. [30] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He (2020) A comprehensive survey on transfer learning. Proceedings of the IEEE P, p. 1–34. External Links: Document Cited by: §I-B.

Instant research discovery