Paper deep dive
RAVEN: Radar Adaptive Vision Encoders for Efficient Chirp-wise Object Detection and Segmentation
Anuvab Sen, Mir Sayeed Mohammad, Saibal Mukhopadhyay
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 4/10/2026, 2:50:52 AM
Summary
RAVEN is a computationally efficient deep learning architecture for FMCW radar perception that processes raw ADC data in a chirp-wise streaming manner. It utilizes independent receiver state-space encoders and a learnable cross-antenna attention module to preserve MIMO structure, enabling compact virtual-array feature extraction. The model supports early-exit inference based on latent state stabilization, significantly reducing computation and latency compared to traditional frame-based radar pipelines.
Entities (5)
Relation Signals (3)
RAVEN → processes → ADC data
confidence 95% · The method processes raw ADC data in a chirp-wise streaming manner
RAVEN → uses → State Space Model
confidence 95% · independent receiver state-space encoders (SSM)
RAVEN → evaluatedon → RaDICaL
confidence 90% · Across automotive radar benchmarks, the approach reports strong object detection... RaDICaL [16] provides synchronized 77 GHz
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This paper presents RAVEN, a computationally efficient deep learning architecture for FMCW radar perception. The method processes raw ADC data in a chirp-wise streaming manner, preserves MIMO structure through independent receiver state-space encoders, and uses a learnable cross-antenna mixing module to recover compact virtual-array features. It also introduces an early-exit mechanism so the model can make decisions using only a subset of chirps when the latent state has stabilized. Across automotive radar benchmarks, the approach reports strong object detection and BEV free-space segmentation performance while substantially reducing computation and end-to-end latency compared with conventional frame-based radar pipelines.
Tags
Links
- Source: https://arxiv.org/abs/2604.04490v1
- Canonical: https://arxiv.org/abs/2604.04490v1
Trouble viewing inline? Open PDF directly →
Full Text
72,626 characters extracted from source content.
Expand or collapse full text
RAVEN: Radar Adaptive Vision Encoders for Efficient Chirp-wise Object Detection and Segmentation Anuvab Sen, Mir Sayeed Mohammad, Saibal Mukhopadhyay Georgia Institute of Technology, Atlanta, Georgia, USA asen74@gatech.edu, mirsayeedmohammad@gatech.edu, saibal.mukhopadhyay@ece.gatech.edu Abstract We introduce RAVEN, a deep learning architecture for processing frequency-modulated continuous-wave (FMCW) radar data that is designed for high computational efficiency. RAVEN reduces computation by using a learnable antenna mixer module on independent receiver state space encoders (SSM) to compress the virtual MIMO array into a compact set of learned features and by performing per-chirp inference with a calibrated early-exit rule, so the model reaches a decision using only a subset of chirps in a radar frame. These design choices yield up to 170× lower computation and 4× lower end-to-end latency than conventional frame-based radar backbones, while achieving state-of-the-art detection and BEV free-space segmentation performance on automotive radar datasets. 1 Introduction Millimeter–wave radars are increasingly central to perception on autonomous ground and aerial platforms. Compared to cameras and LiDAR, they remain robust under adverse weather/lighting and directly sense relative velocity via Doppler. These qualities, combined with radar’s lower size, weight, and power profile, make it an attractive sensing modality for mobile platforms [8, 4]. Recent 4D imaging radars further extend range and reliability under fog and rain, offering stronger velocity tracking [41, 21, 1]. Yet higher spatial and Doppler resolution comes with a steep cost: data volume and compute scale rapidly with antenna count and bandwidth [25, 35, 2], making them infeasible for embedded platforms and high-speed applications [23, 33, 8]. Most existing deep learning pipelines for radar perception follow a frame-based paradigm. They first collect all ADC samples for an entire radar frame, apply a sequence of fast Fourier transforms (FFTs) along range, angle, and Doppler dimensions to construct high-resolution range–angle–Doppler (RAD) tensors, and then run dense convolutional or transformer backbones on these tensors. This design exposes rich spatial–Doppler structure, but it also fixes latency to at least one frame interval and requires expensive dense processing on large 3D feature maps, which is problematic on resource-constrained platforms. Sequential models that operate directly on streaming analog-to-digital converter (ADC) signals have emerged as a promising alternative: by processing chirps as they arrive, they reduce peak memory usage and can, in principle, make decisions earlier than frame-based models [29] (Figure˜1 (a)). However, existing lightweight sequential approaches often struggle on more complex tasks such as object detection. We identify two key reasons. First, they typically compress or mix receiver channels early in the pipeline, discarding the explicit spatial localization information provided by a multiple-input multiple-output (MIMO) array. Second, in Doppler-division multiplexed (DDM) systems, they do not explicitly separate the contributions of different transmit antennas that are spectrally interleaved but remain latent in each receiver stream (Figure˜2). Ignoring this structure causes the virtual-array elements of different transmitters to be aliased together, which degrades angle estimation and, in turn, detection accuracy. We propose an efficient radar data processing architecture that keeps this MIMO structure explicit while remaining streaming-friendly (Figure˜1 (b)). In a MIMO radar, NtxN_tx transmitter antennas and NrxN_rx receiver antennas form a large “virtual array”: each receiver element views a target with a distinct phase profile determined by the array geometry, and the combination of transmitter–receiver pairs yields many virtual elements with fine angular resolution (Figure˜2). If the model mixes receiver channels too early or ignores which transmitter generated which echo, this virtual-array structure is lost, and recovering angle information becomes difficult, leading to degraded detection performance or the need for heavier decoders. To avoid this, we first process samples from each receiver channel independently so the encoder can learn per-antenna chirp features. We then introduce a lightweight cross-antenna attention module that learns how to combine per-chirp features across receiver channels using a small set of learnable virtual-array queries. This module effectively acts as a learnable beamformer: it reconstructs virtual-array features directly from the streaming signals without constructing range–angle–Doppler (RAD) tensors or using computationally expensive FFT based pipelines. The attention mixer adds negligible overhead to a streaming state space backbone while maintaining strong spatial localization information. We also take advantage of the evolving nature of the scene motion between radar chirps to enable early decisions in object detection. Because adjacent chirps contribute primarily differential motion (Doppler) information, detection performance saturates after a small number of chirps in a radar frame, beyond which additional chirps yield diminishing returns. We therefore train with early-chirp supervision and deploy a calibrated stopping rule that triggers as soon as the latent state of the sequence model stabilizes, reducing both encoder FLOPs and latency. Figure 1: (a) Comparison of traditional radar processing paradigms: frame-wise CNN encoders, chirp-wise recurrent models, and sample-wise streaming SSM pipelines. (b) Our spatial-aware hybrid architecture preserves per-RX structure, performs cross-antenna attention and extracts chirp-wise virtual-array features for lightweight detection. (c) Runtime–performance characterization showing our method achieves higher accuracy at significantly lower latency and compute compared to existing radar perception models [25, 5, 29, 28]. Figure 2: MIMO radar virtual antenna formation and multiplexing. (a) NtxN_tx transmitters and NrxN_rx receivers form Ntx×NrxN_tx\!×\!N_rx virtual antennas. RX channels read simultaneously. (b) TDM: TX elements fire sequentially. (c) DDM: TX elements fire spectrally interleaved FMCW pulses; virtual-array information is mixed in frequency per receiver. The key contributions of this paper are: 1. Physics-inspired spatial mixing for streaming ADC: A cross-antenna attention module following independent RX processors that separates latent TX structure (DDM) to recover virtual-MIMO cues without reconstructing full RAD cubes. 2. Hybrid design for efficiency: Optimal placement of a cross-attention module between chirp-modeling channel-SSMs and frame-encoding chirp-SSM for the best compromise between computation and spatial resolving capacity. 3. Sub-frame low-latency detection: A chirp-wise SSM backbone that updates online and supports early decision via a calibrated stopping rule on chirp states, reducing encoder FLOPs and end-to-end latency (Figure˜1 (c)). 2 Related Work Classical object detection pipelines in radar vision first extract sparse point clouds (PC) via CFAR from range/angle/Doppler tensors and then run point-based or pseudo-image detectors (e.g. PointPillars variants) [27, 14]. While bandwidth-friendly and easy to fuse, these miss weak returns and often require multi-frame accumulation, which increases latency and can create motion ghosts. To preserve structural density, recent works process range–Doppler maps, range–angle heatmaps, or full RAD tensors with CNNs/Transformers [2, 25, 5, 22]. However, these 3D grids (e.g., 256×64×12256\!×\!64\!×\!12 in a 3TX 4 RX configuration) are costly to transfer and infer on [27]. For high-antenna imaging radars, forming and processing full RAD cubes in real time is especially demanding [25]. These limitations motivate sequential ADC processing models that avoid constructing large tensors and reduce peak memory/latency. End-to-end models on time-domain ADC aim to learn task-optimal transforms beyond fixed FFTs [40]. Chirp-wise sequential encoders further reduce peak memory by updating an internal state as samples arrive; ChirpNet, for instance, reports ∼15× \!15× fewer parameters than CNN baselines [29]. However, many lightweight sequential designs compress or mix receiver (RX) channels early, weakening spatial localization and angle cues latent in the MIMO array. Deep state space models (SSMs) offer linear-time streaming with long-range dependence via structured state updates [7, 32, 6], making them natural time-series backbones; yet without explicit cross-antenna correlation, even SSM-based encoders risk discarding geometry that originates from the radar physics. Beyond radar, there is a broad literature on anytime and early-exit inference for deep models. CNNs such as MSDNet add intermediate classifiers with confidence- or entropy-based stopping to trade accuracy for compute on a per-input basis [9]. Transformer variants (DeeBERT, FastBERT) attach lightweight heads and use entropy/consistency criteria to decide when to stop [37, 18]. Radar-specific work has also begun to exploit temporal structure across frames for better perception [15], but typically assumes full-frame access and dense feature maps, whereas our focus is on chirp-wise, sub-frame decisions from streaming ADC. In contrast, RAVEN combines these two: we design a radar physics inspired encoder that sequentially processes raw ADC that preserves MIMO structure via cross-antenna mixing and enables chirp-wise early decisions by applying a calibrated stopping rule on the slow-time SSM state, rather than introducing heavy auxiliary heads. 3 Methodology 3.1 Design Motivation We explicitly leverage the signal and array physics of FMCW MIMO radar when designing RAVEN’s encoder, instead of treating ADC samples as generic time-series data. In an FMCW radar, each target generates a beat frequency tied to its range and Doppler, and an NrxN_rx-element array encodes angle through deterministic phase shifts across antennas. These spatial phase patterns form the steering vector that enables angular resolution [31, 10]. Conventional sequential encoders often ignore this structure. For example, if each RX channel is reduced to a scalar and then averaged or passed through a shared 1×11× 1 mix to form a per-chirp token, the operation is equivalent to applying a fixed uniform beamformer. This collapses the NrxN_rx dimensional receiver array response into a single value, discarding the relative phase differences that encode angle. Downstream layers then receive tokens stripped of spatial diversity, making angle recovery significantly harder. The problem becomes worse in Doppler-division multiplexed (DDM) MIMO radars, where RX channels already contain linear mixtures of multiple TX waveforms. If the encoder does not explicitly preserve or disentangle these TX-specific components (akin to matched filters), early tokenization further mixes virtual-array responses. This additional entanglement degrades the network’s ability to learn angular structure and ultimately reduces detection accuracy. This motivates two design choices in RAVEN : • Per-RX fast-time processing: maintain separate encoders for each RX channel so that per-antenna phase and amplitude structure is preserved. • Explicit cross-antenna mixing: use a lightweight attention-based module that learns steering-like weights across RX channels and latent TX structure, instead of relying on implicit spatial learning in deep backbones. These physics-driven constraints keep the encoder compact while retaining the spatial information needed for accurate localization. Figure 3: RAVEN Architecture: (1) Fast-time per-RX SSMs compress I/Q into compact 2-D tokens; (2) cross-antenna attention fuses RX channels and expands to virtual-MIMO features; (3) a chirp-wise SSM updates the state online across chirps; (4) a learned projection maps features to a T×H×WT× H× W grid; (5) lightweight decoders produce detection heatmaps/boxes and segmentation. 3.2 RAVEN Model Architecture RAVEN turns streaming ADC samples into BEV detections and freespace maps through five stages (Figure˜3): • Fast-time SSMs: each RX channel is processed independently by a small state space model to produce a compact token per RX and chirp. • Cross-antenna attention: per-chirp RX tokens are fused with a lightweight attention module that learns spatial correlations and forms virtual MIMO features. • Chirp-wise SSM: a slow-time SSM reads the chirp sequence, maintaining a hidden state so the model can update online and support anytime inference. • Spatial projection: sequence features are mapped into a T×H×WT× H× W grid suitable for 2D decoding. • Lightweight decoders: shallow CNN decoders produce detection heatmaps/boxes and freespace segmentation maps. Notation. Each radar frame is a sequence of NcN_c chirps (slow-time), and each chirp has NsN_s fast-time samples. The receiver (RX) has NrxN_rx channels and the transmitter (TX) has NtxN_tx channels. We use complex (I/Q) samples, so the input channel dimension is 2Nrx2N_rx. A frame is ∈ℝNc×Ns×2NrxX ^N_c× N_s× 2N_rx , with axes (slow -time, fast-time, channels). Bold capitals denote tensors, bold lower-case vectors, and plain symbols denote dimensions. We write LNLN for LayerNorm, σ(⋅)σ(·) for SiLU, and PoolKPool_K for adaptive average pooling to length K. 3.2.1 Parallel RX-channel SSM encoders (fast-time) For receiver r∈1,…,Nrxr\!∈\!\1,…,N_rx\ and chirp k∈1,…,Nck\!∈\!\1,…,N_c\, let r,k∈ℝNs×2x_r,k ^N_s× 2 be the fast-time I/Q sequence for RX r at chirp k. Each RX uses its own state space encoder SSMr:ℝNs×2→ℝNs×2SSM_r:R^N_s× 2 ^N_s× 2 (implemented with a Mamba block), and we compute ~r,k=SSMr(r,k)∈ℝNs×2,r,k=Pool1(~r,k⊤)∈ℝ2. z_r,k=SSM_r(x_r,k) ^N_s× 2,f_r,k=Pool_1\! ( z_r,k ) ^2. Stacking all receivers gives k=[1,k,…,Nrx,k]∈ℝNrx×2,∈ℝNc×Nrx×2.F_k= [f_1,k,…,f_N_rx,k ] ^N_rx× 2,F ^N_c× N_rx× 2. Intuition. Each RX stream is summarized to a tiny per-chirp token that still carries per-antenna range/phase information, providing a compact but geometry-aware input to the cross-antenna attention stage. 3.2.2 Cross-antenna attention & virtual MIMO expansion For chirp k, let k∈ℝNrx×2F_k\!∈\!R^N_rx× 2 be the per-RX summaries from the fast-time SSMs. We first expand each RX to d-dim tokens and add a learnable RX embedding: krx=ink+rx∈ℝNrx×d,rx=[1rx,…,Nrxrx]⊤.H^rx_k=W_inF_k+E^rx\;∈\;R^N_rx× d,E^rx=[e^rx_1,…,e^rx_N_rx]^\! . We introduce a bank of learnable TX queries ∈ℝNtx×dQ\!∈\!R^N_tx× d that probe the RX tokens via cross-attention (queries == TX, keys/values == RX). With pre-norm, =LN(),=LN(krx),=krx,q=LN(Q), =LN(H^rx_k), =H^rx_k, the TX-updated tokens are Attn(,,)=softmax(⊤d)∈ℝNtx×d.Attn(q,k,v)=softmax\! ( qk d )v\;∈\;R^N_tx× d. We apply TX-side residual and feed-forward: ~=+Attn(,,) T=Q+Attn(q,k,v) =~+FFN(LN(~))∈ℝNtx×d.T= T+FFN\! (LN( T) )\;∈\;R^N_tx× d. Next, for every (r,t)(r,t) pair we concatenate the corresponding RX and TX tokens and project to a compact two-dimensional feature: r,t=pair[rrx;t]∈ℝ2,pair∈ℝ2×(2d).p_r,t=W_pair\,[\,h^rx_r;\,t_t\,]\;∈\;R^2, _pair ^2×(2d). Stacking over r and t yields k∈ℝNrx×Ntx×2P_k ^N_rx× N_tx× 2, which we vectorize and normalize to form the per-chirp output: k=LN(vec(k))∈ℝ2NrxNtx.y_k=LN\! (vec(P_k) )\;∈\;R^2N_rxN_tx. Over all chirps this gives =[1,…,Nc]∈ℝNc×(2NrxNtx).Y= [y_1,…,y_N_c ] ^N_c×(2N_rxN_tx). Intuition. In Figure˜4 (a) the TX queries act like learnable steering vectors that search the field of RX tokens, producing TX-specific summaries T. Pairwise fusion [rrx;t]↦ℝ2[h^rx_r;t_t] ^2 then yields a compact feature for every virtual MIMO pair (r,t)(r,t), enabling the network to emphasize phase-consistent returns across antennas (DDM compatible) without constructing range–angle–Doppler (RAD) tensors. Figure 4: (a) Attention Mixer: Learnable transmitter queries are used to extract Doppler-division multiplexed information from the receiver signal in the time domain. These are fused together to form the virtual antenna array for retrieving the MIMO information. (b) Early Decision Supervision: During training, decoders take outputs from multiple chirp levels, and loss is computed simultaneously [13], forcing the model to converge on earlier chirps. 3.2.3 Chirp-wise (slow-time) SSM backbone We compress the channel dimension and prepare slow-time features: k=σ(preσ(redk))∈ℝD,z_k=σ\! (W_pre\;σ(W_redy_k) ) ^D, ∗=[1,…,Nc]∈ℝNc×D,Z_ =[z_1,…,z_N_c] ^N_c× D, with red∈ℝD×(2NrxNtx)W_red ^D×(2N_rxN_tx), pre∈ℝD×DW_pre ^D× D and an optional shallow MLP. We use Mamba-style structured SSMs that support both streaming updates and parallel training. The final slow-time representation is ∗=SSM()∈ℝNc×D.Z_ =SSM(Z) ^N_c× D. Intuition: the SSM keeps a compact state while reading chirps in order, enabling online/anytime decisions without needing the full frame. 3.2.4 Encoder–decoder projection and heads Detection branch. We project slow-time features to a compact spatio–temporal grid and decode object heatmaps and box offsets. =Conv1DC=D→HW(∗⊤)⏟∈ℝHW×Nc U=Conv1D_C\!=\!D→ HW\! (Z_ )_U ^HW× N_c →PoolTdetdet∈ℝHW×Tdet \ Pool_T_det\ U_det ^HW× T_det →reshapedet∈ℝTdet×H×W. \ reshape\ S_det ^T_det× H× W. A shallow Conv–LN–SiLU stack with bilinear upsampling maps detS_det to a high-resolution feature map A, from which 1×11×1 heads predict classification scores and box offsets, yielding Det=[,]Det=[P,R]: =sigmoid(Convcls()),=Convreg(),P=sigmoid\! (Conv_cls(A) ), =Conv_reg(A), Segmentation branch. We use an analogous projection to produce features for drivable-area (freespace) segmentation. An analogous projection with temporal pooling of length TsegT_seg produces seg∈ℝTseg×H×WS_seg ^T_seg× H× W, followed by a similar Conv–LN–SiLU + upsampling stack to obtain segmentation logits =Conv(Up(σ(LN(Conv(seg)))))M=Conv (Up(σ(LN(Conv(S_seg)))) ) 3.3 Sub-frame Decision Framework For a constant-velocity target, information about v accumulates via coherent computation across the chirps. Doppler FFT needs NcN_c chirps to achieve velocity resolution Δv v, but detection often tolerates coarser Δv v. This observation suggests that a sequential detector need not wait for all NcN_c chirps: it can stop once its internal state has stabilized to a sufficient prediction. 3.3.1 Training approach to enable sub-frame decision We implement this idea via multi-prefix supervision (Figure˜4-b). Let ℒ=L1,…,LM⊆1,…,NcL=\L_1,…,L_M\ \1,…,N_c\ be a set of chirp-prefix lengths (with LM=NcL_M=N_c). For a frame, the encoder produces ∗∈ℝNc×DZ_ ^N_c× D. For each L∈ℒL we take the prefix ∗(L)=∗[:,1:L,:]∈ℝL×DZ_ ^(L)=Z_ [:,1:L,:] ^L× D and pass it through the same projection and decoders to obtain Det^(L)∈ℝ3×H′×W′,Seg^(L)∈ℝ1×H′×W′. Det^(L) ^3× H × W , Seg^(L) ^1× H × W . ℓdet _det combines classification and box regression, and ℓseg _seg is segmentation loss. All prefixes are supervised against the same ground-truth targets (Det⋆,Seg⋆)(Det ,Seg ) for the frame, yielding a deep-supervision objective ℒtask=∑L∈ℒ[ℓdet(Det^(L),Det⋆) _task= _L [\, _det ( Det^(L),\,Det ) +ℓseg(Seg^(L),Seg⋆)] + _seg ( Seg^(L),\,Seg )\, ] 3.3.2 Early inference rule Let ∗(L)=z1,…,zL∈ℝL×DZ_ ^(L)=\z_1,…,z_L\ ^L× D denote the chirp-wise latent states. For each new chirp zLz_L, we measure its novelty relative to the earlier bag of chirps via the minimum cosine distance dL=min1≤j<L(1−zL⊤zj‖zL‖‖zj‖).d_L\;=\; _1≤ j<L\! (1- z_L z_j\|z_L\|\,\|z_j\| ). When dLd_L falls below a calibrated threshold τ (Figure˜6), the latent dynamics have saturated and additional chirps offer negligible benefit. Because the decoder operates on blocks of K pooled chirps, we compute a block-averaged score d¯m=1K∑L=(m−1)K+1mKdL, d_m\;=\; 1K _L=(m-1)K+1^mKd_L, and select the earliest block satisfying d¯m≤τ d_m≤τ. The final early-exit index is therefore Lexit=Kminm:d¯m≤τ.L_exit\;=\;K\, \\,m:\, d_m≤τ\,\. 4 Experimental Results 4.1 Datasets RaDICaL (Radar, Depth, IMU, Camera). RaDICaL [16] provides synchronized 77 GHz frequency-modulated continuous-wave (FMCW) radar, stereo RGB-D, and inertial measurement unit (IMU) measurements. The radar uses a 4 RX × 2 TX TDM-MIMO configuration. For each frame we use complex ADC cube with (Nc,Ns,Nrx)=(64,192,8)(N_c,N_s,N_rx)=(64,192,8), where TX-RX pairs are interleaved along NrxN_rx The frame labels are generated from synchronized camera detections: tiled images are processed by RetinaNet [17, 34], and detections are merged into bird’s-eye view (BEV) occupancy masks. RADIal (High-Definition Radar Multi-Task) RADIal [25] contains about two hours of synchronized driving with RGB video, a 16-beam LiDAR, and a 77 GHz imaging radar over 91 sequences (urban, highway, rural). The radar uses a 12 TX × 16 RX DDM configuration (192 virtual antennas). Around 8,252 of ∼ 25,000 frames are annotated with vehicle centroids in polar and Cartesian coordinates and drivable-area (freespace) masks obtained from LiDAR. Since the TX are Doppler division multiplexed, the complex ADC cubes have size (Nc,Ns,Nrx)=(256,512,16)(N_c,N_s,N_rx)=(256,512,16). We follow an 80/20 train/validation split during training process. 4.2 Baselines and Implementation Details We compare RAVEN against radar-only CNN/FFT-based and UNet-style models [25, 26, 38, 40], attention/Transformer-based architectures including FFT–Transformer hybrids [5, 2], chirp-wise sequential models such as ChirpNet [29], and ultra-light SSM-based encoders such as SSMRadNet [28]. All baselines are trained on the same ADC representation as RAVEN. For RADIal, we train jointly for drivable-area segmentation and vehicle detection using Adam (learning rate 1×10−41× 10^-4, weight decay 5×10−65× 10^-6), batch size 8, and 200 epochs; segmentation uses a Jaccard (IoU) loss and detection uses Focal loss plus Smooth L1 regression. For RaDICaL, we train for BEV occupancy segmentation with Adam (learning rate 1×10−41× 10^-4, weight decay 5×10−65× 10^-6), batch size 8, and 300 epochs, using binary cross-entropy (BCE) as the primary loss. Figure 5: Qualitative ablation of the adaptive decision module across four scenarios. Each example shows the RGB view, segmentation evolution over chirps (white - true positive, green - false positive, red - false negative), detection evolution (point-level RA predictions), and the chirp-state contribution signal. Sample (a) complex multi-vehicle scene where early-chirp assumptions about distant objects are refined into accurate detections. Sample (b) early-chirp false positives (“hallucinated” obstacles) are suppressed as more chirps arrive. Sample (c) early hallucinations fade but segmentation remains unreliable throughout. Sample (d) an object briefly emerges in clutter before vanishing, and a noisy chirp-similarity score depicts the irregularity of the data, resulting in poor segmentation and detection. Figure 6: Design motivation for adaptive chirp selection. (Left) Minimum cosine-distance aggregate across all frames in train-set reveals a clear knee point beyond which new chirps add little novel information. (Middle) Validation-set scores show consistent gains from 32→64 chirps, with negligible improvements thereafter. (Right) Memory and latency scale strongly with chirp count, showing that reducing chirps provides substantial efficiency gains with minimal performance loss. 4.3 Evaluation Metrics Segmentation: On RADIal, we follow [25] and report mean intersection-over-union (mIoU) for drivable-area masks. For RaDICaL we additionally report Dice coefficient measuring overlap between predicted and ground-truth masks and Chamfer distance (CD) - measuring the average bidirectional nearest-neighbor Euclidean distance between occupied pixels (or points) in the predicted and ground-truth BEV masks [3, 39]. While mIoU and Dice capture area overlap, Chamfer distance explicitly evaluates contour and boundary alignment. Using both Dice and Chamfer thus evaluates how much of the area is correct and how well object boundaries are localized. Detection: For RADIal detection we report mean average precision (mAP), mean average recall (mAR), and F1-score following the official protocol [25]. Efficiency: Computational efficiency is summarized by multiply–accumulate operations (MACs), parameter count in millions (M), and end-to-end latency per frame measured on an NVIDIA RTX 4060 mobile GPU. 4.4 Qualitative Results Figure˜5 illustrates how the adaptive decision module behaves across diverse scenarios. The chirp cosine distance score has a general downward trend in the first few chirps, after which it drops below a threshold value. In structured scenes with multiple vehicles, early chirps form coarse hypotheses that later chirps refine into stable detections, while inconsistent early hallucinations are naturally suppressed as more chirps accumulate. Conversely, cluttered or noisy scenes expose the limits of early-stage inference: segmentation may remain unreliable, objects may briefly appear and disappear, and the chirp-state contribution signal becomes erratic when the underlying data quality is poor. These examples highlight how the module integrates temporal evidence to stabilize predictions while avoiding unnecessary processing. Figure˜6 motivates this design by showing that chirp-wise information exhibits diminishing returns. The feature cosine distance analysis across train-set (average cosine distance score) reveals a downward saturation trend with a distinct knee point where new chirps become highly redundant, enabling a natural threshold τ=0.2τ=0.2 for early stopping. Consistent with this, performance trends on the validation set show clear improvements only up to roughly 64 chirps, after which mIoU and F1 gains flatten. At the same time, both memory usage and inference latency scale closely with chirp count—dropping the chirp budget from 256 to the 32–64 range yields more than a 2×2× speedup with minimal accuracy loss. Together, these trends justify using an adaptive chirp-termination strategy to maintain accuracy while reducing computation. 4.5 Quantitative Results Table 1: Comparison with prior works on RaDICaL [16]. Best values per row are highlighted in bold. Metric / Model ChirpNet ChirpNetLite ChirpNet-SSM ChirpNet-Attn T-FFTRadNet FFT-RadNet UNet SSMRadNet RAVEN [29] [29] [30] [30] [5] [25] [26] [28] (Ours) Computational Complexity GMACs ↓ 1.480 0.320 0.340 0.350 15.990 41.740 15.140 0.108 0.053 Params (M) ↓ 3.780 3.761 3.761 3.761 9.000 4.250 17.270 0.566 0.347 Accuracy Metrics Dice Coefficient ↑ 0.986 0.989 0.990 0.991 0.995 0.996 0.996 0.996 0.997 Chamfer ↓ 0.097 0.095 0.088 0.091 0.108 0.076 0.078 0.086 0.082 • Note: Dice coefficient: higher is better (↑ ); Chamfer distance: lower is better (↓ ). Table 2: Overall segmentation and detection performance on RADIal[25]. Best values (global) in bold. Class Model mIoU ↑ F1 ↑ mAP ↑ mAR ↑ RE (m) ↓ AE (∘) ↓ GMACs ↓ Params (M) ↓ Lat. (ms) ↓ Convolution Pixor (PC) [38] — — 0.96 0.32 0.17 0.25 — — — Pixor (RA) [38] — — 0.96 0.82 0.12 0.20 — — — PolarNet [20] 0.61 — — — — — — — — Conv3D + FFT-RadNet [36] 0.75 0.47 0.58 0.39 0.19 0.33 — — — FFT-RadNet [25] 0.74 0.88 0.97 0.82 0.14 0.17 146.82 3.80 53.59 RLSM [24] 0.71 0.86 0.91 0.82 — — — — — FFT-RadUNeta 0.75 0.80 0.83 0.77 0.16 0.10 134.40 18.48 44.92 ADCNet [40] 0.79 0.89 0.93 0.86 0.14 0.11 — 2.50 18.13 ADC UNet [40] 0.77 0.85 0.88 0.82 0.18 0.11 — 17.50 8.18 ADC UNet (NPT) [40] 0.73 0.80 0.83 0.77 0.19 0.10 — — — FourierNet-FFT-RadUNetb 0.78 0.86 0.84 0.87 0.16 0.11 134.41 19.13 48.73 FourierNet-FFT-RadNetc 0.79 0.88 0.87 0.89 0.14 0.12 146.59 4.45 57.44 C–M DNN [11] 0.80 0.89 0.97 0.83 0.45 — 179.00 7.70 68.00 Attention ChirpNet (Self-Attn) [30] 0.65 — — — — — 33.00 50.95 20.37 T-FFTRadNet [5] 0.79 0.87 0.88 0.87 0.16 0.13 97.00 9.60 52.90 TransRadar [2] 0.82 0.93 0.95 0.91 0.15 0.10 171.50 3.70 — EchoFusion [19] — 0.93 0.96 0.92 0.12 0.18 — — — Recurrent ChirpNet (GRU) [29] 0.64 — — — — — 12.35 5.77 27.33 SSM ChirpNet (SSM) [30] 0.66 — — — — — 15.50 45.85 9.32 SSMRadNet [28] 0.79 0.77 0.83 0.71 0.14 0.15 1.67 0.31 14.20 GNN + Convolution SparseRadNet [36] 0.78 0.93 0.96 0.91 0.13 0.10 129.50 6.90 — SSM + Attention RAVEN (Sub-frame, Ours) 0.85 0.89 0.88 0.89 0.17 0.25 0.27 1.51 9.15 RAVEN (Full Frame, Ours) 0.90 0.93 0.95 0.92 0.12 0.10 1.02 1.51 20.08 • RE = range error; AE = azimuth error; RA = range–azimuth. ↑ higher is better; ↓ lower is better. Blue rows denote our models. a, b, c [a] FFT-RadNet[25]+UNet[26]; FourierNet[42] FFT fed to [b] FFT-RadUNet, [c] FFT-RadNet. RAVEN delivers state-of-the-art (SOTA) performance on both RADIal and RaDICaL datasets. On RaDICaL [16], RAVEN achieves a Dice coefficient of 0.997 with Chamfer distance 0.082 at just 0.053 GMACs (see Table˜1). Compared to FFT-RadNet [25], which reaches 0.996 Dice at 41.74 GMACs, this corresponds to nearly 790×790× lower compute and about 12×12× fewer parameters (0.35 M vs. 4.25 M), while maintaining near state-of-the-art mask quality and boundary alignment with a tiny fraction of the computational and parameter budget. On RADIal [25], our model attains an mIoU of 0.90, F1 of 0.93, and the lowest range/angle errors (RE 0.12 m, AE 0.10∘0.10 ), while using only 1.02 GMACs (see Table˜2), about 170×170× less compute than TransRadar [2] (171.5 GMACs) and 95×95× less than T-FFTRadNet [5] (97 GMACs), yet matching or surpassing their segmentation and detection accuracy, while enabling chirp-wise decisions at a fraction of the compute. 5 Conclusion & Future Work We introduced RAVEN, an end-to-end machine learning architecture for radar-based perception that models FMCW radar chirp sequences using lightweight channel-wise and chirp-wise SSMs paired with cross-antenna attention. By aggregating information along the chirp dimension while preserving MIMO structure, RAVEN produces BEV freespace and object detections with state-of-the-art accuracy on RADIal and RaDICaL datasets, yet requires orders of magnitude fewer GMACs and parameters than prior radar-only models. Our analysis shows that the chirp-wise backbone yields stable prefix representations, causing the latent state to saturate early; this enables effective chirp subsampling and points toward compute-adaptive radar pipelines. Physics-aware sequential encoders can thus match heavy frame-based models under edge constraints. Looking forward, incorporating RAVEN into multimodal stacks and evaluating it across broader driving conditions offers a path toward robust, efficient multi-modal perception systems for real-world deployment. RAVEN therefore offers a practical, edge-friendly radar perception backbone for high-resolution object detection and freespace segmentation. 6 Acknowledgement This material is based upon work supported in part by SRC JUMP 2.0 (CogniSense, #2023-JU-3133) and in part by DARPA through the OPTIMA program. Any opinions, or recommendations expressed in this material are those of the author(s) and do not reflect the views of SRC or DARPA. This research was also supported through research cyber infrastructure resources and services provided by the Partnership for an Advanced Computing Environment (PACE) at the Georgia Institute of Technology, Atlanta, Georgia, USA. References [1] K. Burnett, Y. Wu, D. J. Yoon, A. P. Schoellig, and T. D. Barfoot (2022) Are we ready for radar to replace lidar in all-weather mapping and localization?. IEEE Robotics and Automation Letters 7 (4), p. 10328–10335. External Links: Document Cited by: §1. [2] Y. Dalbah, J. Lahoud, and H. Cholakkal (2024-01) TransRadar: adaptive-directional transformer for real-time multi-view radar semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 353–362. Cited by: §1, §2, §4.2, §4.5, Table 2. [3] H. Fan, H. Su, and L. Guibas (2017) A point set generation network for 3d object reconstruction from a single image. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 2463–2471. External Links: Document Cited by: §4.3. [4] L. Fan, J. Wang, Y. Chang, Y. Li, Y. Wang, and D. Cao (2024) 4D mmwave radar for autonomous driving perception: a comprehensive survey. IEEE Transactions on Intelligent Vehicles 9 (4), p. 4606–4620. External Links: Document Cited by: §1. [5] J. Giroux, M. Bouchard, and R. Laganiere (2023-10) T-fftradnet: object detection with swin vision transformers from raw adc radar signals. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, p. 4030–4039. Cited by: Figure 1, Figure 1, §2, §4.2, §4.5, Table 1, Table 2. [6] A. Gu and T. Dao (2024) Mamba: linear-time sequence modeling with selective state spaces. In First conference on language modeling, Cited by: §2. [7] A. Gu, K. Goel, and C. Ré (2022) Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, External Links: Link Cited by: §2. [8] Z. Han, J. Wang, Z. Xu, S. Yang, L. He, S. Xu, J. Wang, and K. Li (2023) 4d millimeter-wave radar in autonomous driving: a survey. arXiv preprint arXiv:2306.04242. Cited by: §1. [9] G. Huang, D. Chen, T. Li, F. Wu, L. van der Maaten, and K. Weinberger (2018) Multi-scale dense networks for resource efficient image classification. In International Conference on Learning Representations, External Links: Link Cited by: §2. [10] Y. Huang, P. V. Brennan, D. Patrick, I. Weller, P. Roberts, and K. Hughes (2011) FMCW based MIMO imaging radar for maritime navigation. Progress In Electromagnetics Research 115, p. 327–342. External Links: Document, Link Cited by: §3.1. [11] Y. Jin, A. Deligiannis, J. Fuentes-Michel, and M. Vossiek (2023) Cross-modal supervision-based multitask learning with automotive radar raw data. IEEE Transactions on Intelligent Vehicles 8 (4), p. 3012–3025. External Links: Document Cited by: Table 2. [12] H. Kumawat and S. Mukhopadhyay (2022) Radar guided dynamic visual attention for resource-efficient rgb object detection. In 2022 International Joint Conference on Neural Networks (IJCNN), Vol. , p. 1–8. External Links: Document Cited by: §7.1.1. [13] A. Kusupati, G. Bhatt, A. Rege, M. Wallingford, A. Sinha, V. Ramanujan, W. Howard-Snyder, K. Chen, S. Kakade, P. Jain, et al. (2022) Matryoshka representation learning. Advances in Neural Information Processing Systems 35, p. 30233–30249. Cited by: Figure 4, Figure 4. [14] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom (2019) Pointpillars: fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 12697–12705. Cited by: §2. [15] P. Li, P. Wang, K. Berntorp, and H. Liu (2022) Exploiting temporal relations on radar perception for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 17071–17080. Cited by: §2. [16] T. Y. Lim, S. A. Markowitz, and M. N. Do (2021) RaDICaL: a synchronized fmcw radar, depth, imu and rgb camera dataset with low-level fmcw radar signals. Note: https://doi.org/10.13012/B2IDB-3289560_V1 Cited by: §4.1, §4.5, Table 1, Table 1, Figure 7, Figure 7, §7.1.1. [17] T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár (2020) Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 42 (2), p. 318–327. External Links: Document Cited by: §4.1, §7.1.1. [18] W. Liu, P. Zhou, Z. Wang, Z. Zhao, H. Deng, and Q. Ju (2020) Fastbert: a self-distilling bert with adaptive inference time. In Proceedings of the 58th annual meeting of the association for computational linguistics, p. 6035–6044. Cited by: §2. [19] Y. Liu, F. Wang, N. Wang, and Z. Zhang (2023) Echoes beyond points: unleashing the power of raw radar data in multi-modality fusion. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: Table 2. [20] F. E. Nowruzi, D. Kolhatkar, P. Kapoor, F. Al Hassanat, E. J. Heravi, R. Laganiere, J. Rebut, and W. Malik (2020) Deep open space segmentation using automotive radar. In 2020 IEEE MTT-S International Conference on Microwaves for Intelligent Mobility (ICMIM), p. 1–4. Cited by: Table 2. [21] D. Paek, S. Kong, and K. T. Wijaya (2022) K-radar: 4d radar object detection for autonomous driving in various weather conditions. Advances in Neural Information Processing Systems 35, p. 3819–3829. Cited by: §1. [22] A. Palffy, J. Dong, J. F. Kooij, and D. M. Gavrila (2020) CNN based road user detection using the 3d radar cube. IEEE Robotics and Automation Letters 5 (2), p. 1263–1270. Cited by: §2. [23] S. M. Patole, M. Torlak, D. Wang, and M. Ali (2017) Automotive radars: a review of signal processing techniques. IEEE Signal Processing Magazine 34 (2), p. 22–35. Cited by: §1. [24] M. Pushkareva, Y. Feldman, C. Domokos, K. Rambach, and D. D. Castro (2024) Radar spectra-language model for automotive scene parsing. In 2024 International Radar Conference (RADAR), Vol. , p. 1–6. External Links: Document Cited by: Table 2. [25] J. Rebut, A. Ouaknine, W. Malik, and P. Pérez (2022) Raw high-definition radar for multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 17000–17009. Note: Paper: https://doi.org/10.1109/CVPR52688.2022.01651. Dataset: https://github.com/valeoai/RADIal External Links: Document Cited by: Figure 1, Figure 1, §1, §2, item a, b, c, §4.1, §4.2, §4.3, §4.3, §4.5, §4.5, Table 1, Table 2, Table 2, Table 2, Figure 7, Figure 7, §7.1.2. [26] O. Ronneberger, P. Fischer, and T. Brox (2015) U-net: convolutional networks for biomedical image segmentation. Medical Image Computing and Computer Assisted Intervention, p. 234–241. Cited by: item a, b, c, §4.2, Table 1. [27] N. Scheiner, F. Kraus, N. Appenrodt, J. Dickmann, and B. Sick (2021) Object detection for automotive radar point clouds—a comparison. AI Perspectives 3, p. 6. External Links: Document, Link Cited by: §2. [28] A. Sen, M. S. Mohammad, and S. Mukhopadhyay (2026-03) SSMRadNet : a sample-wise state-space framework for efficient and ultra-light radar segmentation and object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 4365–4374. Cited by: Figure 1, Figure 1, §11, §4.2, Table 1, Table 2. [29] S. Sharma, H. Kumawat, and S. Mukhopadhyay (2024) ChirpNet: noise-resilient sequential chirp-based radar processing for object detection. In IEEE International Microwave Symposium, Cited by: Figure 1, Figure 1, §1, §2, §4.2, Table 1, Table 1, Table 2, Figure 7, Figure 7. [30] S. Sharma, H. Kumawat, A. Sen, J. Park, and S. Mukhopadhyay (2025) Toward efficient and robust sequential chirp-based data-driven radar processing for object detection. IEEE Transactions on Radar Systems 3 (), p. 1435–1448. External Links: Document Cited by: Table 1, Table 1, Table 2, Table 2. [31] H. Singh and A. Chattopadhyay (2023) Multi-target range and angle detection for mimo-fmcw radar with limited antennas. In 2023 31st European Signal Processing Conference (EUSIPCO), Vol. , p. 725–729. External Links: Document Cited by: §3.1. [32] J. T.H. Smith, A. Warrington, and S. Linderman (2023) Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §2. [33] S. Sun, A. P. Petropulu, and H. V. Poor (2020) MIMO radar for advanced driver-assistance systems and autonomous driving: advantages and challenges. IEEE Signal Processing Magazine 37 (4), p. 98–117. Cited by: §1. [34] Z. Tian, C. Shen, H. Chen, and T. He (2019) FCOS: fully convolutional one-stage object detection. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Vol. , p. 9626–9635. External Links: Document Cited by: §4.1. [35] Y. Wang, Z. Jiang, Y. Li, J. Hwang, G. Xing, and H. Liu (2021) RODNet: a real-time radar object detection network cross-supervised by camera-radar fused object 3d localization. IEEE Journal of Selected Topics in Signal Processing 15 (4), p. 954–967. External Links: Document Cited by: §1, §7.1.1. [36] J. Wu, M. Meuter, M. Schöler, and M. Rottmann (2024) SparseRadNet: sparse perception neural network on subsampled radar data. arXiv preprint arXiv:2406.10600. Cited by: Table 2, Table 2. [37] J. Xin, R. Tang, J. Lee, Y. Yu, and J. Lin (2020-07) DeeBERT: dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, p. 2246–2251. External Links: Link, Document Cited by: §2. [38] B. Yang, W. Luo, and R. Urtasun (2018) Pixor: real-time 3d object detection from point clouds. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, p. 7652–7660. Cited by: §4.2, Table 2, Table 2. [39] S. Yao, R. Guan, X. Huang, Z. Li, X. Sha, Y. Yue, E. G. Lim, H. Seo, K. L. Man, X. Zhu, and Y. Yue (2024) Radar-camera fusion for object detection and semantic segmentation in autonomous driving: a comprehensive review. IEEE Transactions on Intelligent Vehicles 9 (1), p. 2094–2128. External Links: Document Cited by: §4.3, §7.1.1. [40] B. Zhang, I. Khatri, M. Happold, and C. Chen (2023) ADCNet: learning from raw radar data via distillation. arXiv preprint arXiv:2303.11420. Cited by: §2, §4.2, Table 2, Table 2, Table 2. [41] Y. Zhang, A. Carballo, H. Yang, and K. Takeda (2023) Perception and sensing for autonomous vehicles under adverse weather conditions: a survey. ISPRS Journal of Photogrammetry and Remote Sensing 196, p. 146–177. Cited by: §1. [42] P. Zhao, C. X. Lu, B. Wang, N. Trigoni, and A. Markham (2023) CubeLearn: end-to-end learning for human motion recognition from raw mmwave radar signals. IEEE Internet of Things Journal 10 (12), p. 10236–10249. External Links: Document Cited by: item a, b, c. Supplementary Material Figure 7: (a) RaDICaL [16]: label generation from RGB frames using a tiled RetinaNet detector (adapted from [29]). (b) RADIal [25]: FFT of raw ADC data produces range–azimuth maps; CFAR yields radar point clouds; segmentation maps mark drivable (white) vs. non-drivable (black) areas; nearest and second-nearest vehicles are highlighted in red and green, respectively. 7 Experimental Details 7.1 Datasets 7.1.1 RaDICaL dataset and annotation We use the RaDICaL dataset [16], which provides synchronized measurements from a 44-Rx, 33-Tx 7777 GHz FMCW radar, an RGB camera, a depth camera, and an inertial measurement unit (IMU). The depth camera produces reliable depth estimates only up to approximately 1010 m, making it less effective for distant objects, whereas the radar remains sensitive to far-range targets. Scenes are recorded from a vehicle-mounted sensor rig across urban streets, country roads, and highways. Unlike many prior radar datasets [39], RaDICaL releases raw ADC samples in addition to preprocessed range–Doppler or range–angle maps. This preserves the full semantic content of the radar data and enables efficient raw chirp-wise processing. While the radar hardware supports a 33-Tx×4-Rx MIMO configuration, the dataset was collected using a 22-Tx×4-Rx TDM MIMO setup, yielding 88 virtual channels per chirp. Our pipeline uses all available virtual channels. RaDICaL Annotation pipeline. Supervised radar learning is limited by the difficulty of generating high-quality labels directly in the radar domain (e.g., range–azimuth–Doppler tensors or sparse point clouds). Instead of annotating radar data manually or relying on CFAR-based heuristics, we derive supervision from synchronized RGB images. We use a RetinaNet [17] detector with a ResNet-50 backbone pre-trained on COCO on the camera images. To improve detection of small and distant objects, we adopt a tiling strategy [12]: each image is split into overlapping tiles, inference is run independently on each tile, and detections are stitched back into the original resolution. This improves recall for far objects compared to a single-pass detector. We restrict COCO classes to person, bicycle, car, motorcycle, bus, and truck. From the stitched detections, we generate a binary mask in image space. This mask serves as the ground-truth signal during training. Importantly, we do not use radar-to-camera calibration matrices to project annotations across modalities; instead, the model learns cross-modal alignment implicitly through its architecture. This avoids dependence on calibration, eliminates the need to store radar-domain labels, and minimizes the alignment noise and sparsity issues seen in RODNet-style labels [35]. The tiling strategy can produce duplicate detections when a single object spans multiple tiles; we mitigate this with non-maximum suppression (NMS) after stitching. An overview of the RaDICaL annotation pipeline is shown in Fig. 7(a). 7.1.2 RADIal dataset overview For comparison and context, we briefly summarize the RADIal dataset [25]. RADIal uses a high-definition imaging radar with NRx=16N_Rx=16 receiver antennas and NTx=12N_Tx=12 transmitter antennas, giving NRxNTx=192N_RxN_Tx=192 virtual channels. This dense virtual array provides fine azimuth resolution and supports elevation estimation. The radar is accompanied by a 16-layer automotive-grade LiDAR, a 5 Mpix RGB camera mounted behind the windshield, and synchronized GPS and CAN traces for vehicle pose and kinematics. The three sensors have parallel horizontal lines of sight in the driving direction, and their extrinsic calibration is provided. RADIal contains 9191 sequences of 11–44 minutes each (city, highway, and countryside driving), for a total of roughly 2525k synchronized frames, of which 8,2528,252 frames are labeled with about 9,5509,550 vehicles. The RADIal signal-processing and labeling pipeline is summarized in Fig. 7(b). Figure 8: Per-block latency (ms) on a single GPU. The channel SSM is the main sequential bottleneck because it processes long fast-time sequences; the mixer and decoders are highly parallelizable. Nevertheless, RADIal is the only large-scale dataset that provides raw analog-to-digital converter (ADC) radar signals rather than only preprocessed FFT cubes. This makes it possible to train foundation models directly on raw radar data streams, yielding competitive performance and architectures like RAVEN that can exploit raw signals efficiently for on-edge deployment. 8 RAVEN Block-Wise Analysis RAVEN’s encoder–decoder pipeline consists of four logical components: (i) per-RX channel SSMs that operate along fast time, (i) an antenna attention mixer that reconstructs virtual-MIMO features, (i) a chirp-wise SSM backbone along slow time, and (iv) lightweight decoders for detection and segmentation. We profile them individually. Figure 8 summarizes the per-block parameter count, GMACs, and latency contributions, normalized to the full model. These plots show a consistent picture: most parameters reside in the 2D decoders, most MACs in the combination of chirp-wise SSM and decoders, and most latency in the channel SSM. The antenna mixer, despite encoding detailed virtual-MIMO structure, contributes only a small fraction of total compute and parameter count. This means we can afford spatial reasoning without compromising efficiency, provided that the fast-time block remains narrow and the slow-time backbone operates on sufficiently compressed tokens. 9 Physics-guided Encoder Design The design of RAVEN’s encoder is guided directly by the signal and array physics of FMCW MIMO radar. In this section, we move from the basic chirp model to the virtual-array view and then to architectural choices: (i) how fast-time structure suggests 1D state space models, (i) how MIMO geometry encodes angle, (i) why naive channel mixing destroys that information, and (iv) how our channel SSMs and antenna mixer modules implement a physics-aligned, end-to-end encoder for object detection. Section 11 then validates these choices empirically. 9.1 FMCW chirp and beat signal A single FMCW chirp of duration TcT_c and bandwidth B has instantaneous transmit frequency ftx(t)=f0+St,S=BTc,0≤t≤Tc,f_tx(t)=f_0+St, 28.80008ptS= BT_c, 0≤ t≤ T_c, (1) and complex baseband signal stx(t)=exp(j2π(f0t+12St2)).s_tx(t)= \! (j2π (f_0t+ 12St^2 ) ). (2) A target at range R and radial velocity v yields an echo delayed by τ=2Rcτ= 2Rc and Doppler-shifted by fD=2vλf_D= 2vλ, where c is the speed of light and λ is the carrier wavelength. After mixing with the transmit signal and low-pass filtering, the resulting beat signal can be approximated as sb(t)≈exp(j2π(fbt+const)),fb≈fr+fD,s_b(t)≈ \! (j2π (f_bt+const ) ), 28.80008ptf_b≈ f_r+f_D, (3) with range-dependent frequency fr=2SRcf_r= 2SRc and Doppler frequency fD=2vλf_D= 2vλ. Thus, range manifests as a linear frequency along fast time, while velocity appears as phase evolution across chirps. Let TsT_s denote the ADC sampling period and n∈0,…,Ns−1n∈\0,…,N_s-1\ the fast-time index. For chirp index k with repetition interval TRT_R, a single-target beat sample at one receiver is approximately xk[n]∝exp(j2π(frnTs+fDkTR)),x_k[n] \! (j2π(f_rnT_s+f_DkT_R) ), (4) which is the starting point for our fast-time state space encoders: the fast-time dimension is a 1D sequence whose frequency encodes range, motivating SSMs along fast time (ADC samples). 9.2 MIMO virtual array and angle encoding For an NRxN_Rx-element receive array with inter-element spacing d, a plane wave from azimuth θ induces a spatial steering vector (θ)=[ 1,ejϕ,…,ej(NRx−1)ϕ]⊤,ϕ=2πdλsinθ.a(θ)= [\,1,e^jφ,…,e^j(N_Rx-1)φ ] , φ=2π dλ θ. (5) Stacking the beat samples across antennas for chirp k and fast-time index n yields k[n]=∑ℓAℓej2π(fr,ℓnTs+fD,ℓkTR)(θℓ)+k[n],x_k[n]= _ A_ e^j2π(f_r, nT_s+f_D, kT_R)a( _ )+w_k[n], (6) where AℓA_ and θℓ _ denote the complex amplitude and angle of the ℓ -th target and k[n]w_k[n] represents noise and clutter. In TDM/DDM MIMO, each receiver additionally sees echoes from multiple transmitters, so the virtual array combines TX and RX patterns. The virtual steering vector becomes a Kronecker product of TX and RX steering vectors, and different transmitters are separated in either time (TDM) or Doppler (DDM). Crucially, angle information is encoded in relative phase differences across antennas and transmitters; any operation that averages these channels too early risks collapsing the array response to a single beam. Architecturally this means we should preserve per-antenna channels until we have a mechanism that can explicitly reason over them. 9.3 Why naive channel mixing loses angle To see how early mixing harms angular resolution, consider a simplistic encoder that first maps each receiver’s fast-time sequence into a scalar summary and then averages across receivers. Let xr,k[⋅]x_r,k[·] denote the fast-time samples for receiver r and chirp k, and let g(⋅)g(·) be a (near-linear) temporal encoder. We define ur,k=g(xr,k[⋅]),k=[u1,k,…,uNRx,k]⊤,u_r,k=g\! (x_r,k[·] ), 28.80008ptu_k= [u_1,k,…,u_N_Rx,k ] , (7) and obtain a per-chirp token via uniform averaging zk=1NRx∑r=1NRxur,k=Hk,=1NRx.z_k= 1N_Rx _r=1^N_Rxu_r,k=w^Hu_k, 28.80008ptw= 1N_Rx1. (8) If the scene is dominated by a single far-field target, then ku_k is approximately proportional to the steering vector (θ)a(θ), so the token becomes zk∝H(θ)=1NRxH(θ).z_k ^Ha(θ)= 1N_Rx1^Ha(θ). (9) This is precisely the output of a fixed beamformer with weights w: all spatial information is compressed into one scalar, and only that one beam pattern is available to the downstream network. Relative phase shifts ejrϕe^jrφ across antennas, which distinguish different angles θ, no longer appear explicitly in the representation. In DDM/TDM MIMO, where TX waveforms are interleaved in Doppler or time, this problem becomes more severe: the virtual array structure is already entangled across chirps and frequencies, and early channel mixing further entangles it, making it difficult for later layers to recover angle-of-arrival (AoA) cues without reconstructing RAD tensors. This motivates an encoder that first models each channel’s fast-time dynamics and then performs explicit, learned spatial mixing across antennas. 9.4 Per-RX Channel Fast Time SSMs and Antenna mixer as a radar physics-friendly alternative RAVEN avoids this pitfall by inserting two carefully structured stages before the slow-time backbone. Per-RX channel fast time SSMs: Instead of aggregating channels immediately, we maintain a separate fast-time encoder for each receiver. For receiver r and chirp k, we collect the I/Q sequence r,k∈ℝNs×2,x_r,k ^N_s× 2, (10) and feed it to a Mamba-style state space model SSMrSSM_r: ~r,k z_r,k =SSMr(r,k)∈ℝNs×2, =SSM_r(x_r,k) ^N_s× 2, (11) r,k _r,k =Pool1(~r,k⊤)∈ℝ2, =Pool_1\! ( z_r,k ) ^2, (12) where Pool1Pool_1 adaptively averages the fast-time dimension to length 1. Stacking across receivers yields k=[1,k,…,NRx,k]∈ℝNRx×2,F_k=[f_1,k,…,f_N_Rx,k] ^N_Rx× 2, (13) so each antenna contributes a compact per-chirp descriptor that retains its relative phase and amplitude structure. This implements the “first compress fast time per channel” step suggested by the physics above. Attention-based antenna mixer: The antenna mixer then interprets kF_k as a set of tokens and learns how to combine them in a way analogous to a set of learnable beams. After projecting from ℝ2R^2 to ℝdR^d and adding RX embeddings, we obtain krx=Wink+rx∈ℝNRx×d,H_k^rx=W_inF_k+E^rx ^N_Rx× d, (14) and introduce NTxN_Tx TX queries ∈ℝNTx×dQ ^N_Tx× d. Multi-head attention produces a set of TX-aligned features k=Attn(,krx,krx)∈ℝNTx×d,T_k=Attn(Q,H_k^rx,H_k^rx) ^N_Tx× d, (15) which can be interpreted as learnable steering patterns over the RX tokens. To expose joint TX–RX information to the downstream SSM, we form small pairwise features for every (r,t)(r,t) combination (e.g., by concatenation and a linear layer) and compress each pair to a two-dimensional vector: r,t,k _r,t,k =Wpair[r,krx;t,k]∈ℝ2, =W_pair [h^rx_r,k;t_t,k ] ^2, (16) k _k =LN(vec(r,t,kr,t))∈ℝ2NRxNTx. =LN\! (vec (\p_r,t,k\_r,t ) ) ^2N_RxN_Tx. (17) Thus, each chirp is represented by a 2NRxNTx2N_RxN_Tx-dimensional learned virtual-antenna feature vector. Crucially, this representation is obtained directly from time-domain ADC signals through learned projections and attention; we never perform explicit 2D/3D FFTs across range or angle, and we never construct dense range–azimuth–Doppler (RAD) tensors. Stacking over chirps gives =[1,…,Nc]∈ℝNc×(2NRxNTx),Y=[y_1,…,y_N_c] ^N_c×(2N_RxN_Tx), (18) which preserves the structure of the MIMO array in a compact latent space and feeds directly into the chirp-wise SSM. This gives a physics-inspired end-to-end encoder: fast-time SSMs for range, attention for angle, and chirp-wise SSMs for temporal evolution. Model Variant Channel SSM Antenna Mixer mIoU F1 mAP GMACs (A) Shared Fast-time SSM ✗ ✗ 0.79 0.77 0.81 1.67 (B) Cross-Antenna Attention + Shared Fast-time SSM ✗ ✗ 0.80 0.79 0.83 38.89 (C) Shared Fast-time SSM + Cross-Antenna Attention ✗ ✓ 0.79 0.80 0.83 1.62 (D) Cross-Antenna Attention + Channel SSM ✓ ✓ 0.84 0.88 0.88 34.56 !8 (E) Channel SSM + Cross-Antenna Attention (RAVEN, full-frame) ✓ ✓ 0.90 0.93 0.95 1.02 !8 (F) Channel SSM + Cross-Antenna Attention (RAVEN, sub-frame) ✓ ✓ 0.85 0.89 0.88 0.27 Table 3: Ablation of channel SSM and antenna mixer on RADIal. All variants share the same chirp-wise SSM backbone and decoders. Our physics-guided RAVEN encoders (blue rows), which apply per-RX channel SSMs before the antenna mixer, achieve the best trade-off between accuracy and compute; the sub-frame variant further improves efficiency for early-exit decisions. 10 Ablation: Role and Ordering of Per RX Channel Fast Time SSM and Antenna Mixer The radar physics discussion suggests that both the per-RX channel SSMs and the cross-antenna attention mixer are important, and that their ordering should follow the natural flow of information. Our hypothesis is to first compress ADC samples across each receiver channel along fast time, then isolate angle information from the channels. To validate this, we compare the model variants in Table 3, which all share the same chirp-wise SSM and decoders but differ in how they model fast time and cross-antenna interactions. Figure 9: Segmentation and detection maps across driving scenes with and without multi-chirp supervision. Without supervision across chirp levels, segmentation gradually approaches the ground truth, but detection remains unstable throughout the sequence, consistent with Figure˜10. In (a), the model forgets real objects mid-frame that only reappear at the end. In (b), it initially identifies one set of objects but later predicts an entirely different set. In (c), it begins to hallucinate obstacles near the final chirps. With multi-chirp supervision, these issues disappear: detection becomes consistent across the sequence, and both segmentation and detection remain accurate through the final frame. Model Variants: We briefly restate what each row does and why some variants are much heavier: • (A) Shared fast-time SSM. A single fast-time SSM operates on all 2NRx2N_Rx input channels jointly. There is no per-RX channel SSM and no dedicated antenna mixer; the model treats the ADC samples as a generic multichannel sequence. This gives a reasonable baseline in both accuracy and compute (1.67 GMACs). • (B) Cross-antenna attention + shared fast-time SSM. Here we augment the shared fast-time SSM with global cross-antenna interactions inside the same block, but still without a separate channel SSM module or a structured antenna mixer head. In our implementation, this attention is applied at full fast-time resolution: it sees roughly 512×NRx512× N_Rx tokens per chirp instead of a compressed set of per-antenna summaries. Because attention scales at least quadratically with the sequence length, this makes the block extremely expensive (38.89 GMACs) even though the accuracy gain over (A) is small. • (C) Fast-time SSM + cross-antenna attention. A shared fast-time SSM is followed by a dedicated cross-antenna attention mixer. This introduces an explicit mixer module, but because the fast-time SSM is still shared across all channels, it does not produce clean per-antenna summaries. The mixer therefore operates on features that partially blur channel structure, and accuracy improves only marginally over (A)/(B), while compute (1.62 GMACs) remains comparable to (A). • (D) Cross-antenna attention + channel SSM. Both channel SSMs and the mixer are present, but in the reverse order: cross-antenna attention is applied first on lightly projected I/Q samples, and the resulting mixed features are then processed by per-RX SSMs. As in (B), the attention still runs on long fast-time sequences (≈512≈ 512 samples per antenna), so it sees a large number of tokens and dominates the compute. This explains why (D) achieves good accuracy (mIoU 0.84, F1 0.88) but remains very heavy at 34.56 GMACs. • (E) Channel SSM + cross-antenna attention (RAVEN, full-frame). Our proposed hybrid encoder: per-RX channel SSMs first compress each fast-time sequence into a low-dimensional token, so each chirp is represented by only NRxN_Rx channel tokens instead of 512×NRx512× N_Rx time samples. The cross-antenna attention mixer then operates on this compressed set of tokens, reconstructing virtual MIMO structure at a much smaller sequence length. This ordering is motivated by the physics analysis in Section 9. • (F) Channel SSM + cross-antenna attention (RAVEN, sub-frame). This variant uses the hybrid encoder as (E) but trains with our early-chirp criterion and decodes from a sub-frame subset of chirps. It maintains strong accuracy while reducing compute to 0.27 GMACs, providing an efficient early-exit option for on-edge deployment. Optimal placement of the hybrid design for efficiency. The trends in Table 3 support the physics-guided design. Variants (B) and (D) show that simply adding cross-antenna attention on top of raw fast-time sequences is not a good trade-off: attention over ∼512×NRx 512× N_Rx tokens per chirp is powerful but incurs tens of GMACs. Variants (A) and (C), which avoid that extreme cost, either lack structured cross-antenna reasoning or do not preserve clean per-channel summaries and therefore underperform in accuracy. Our RAVEN encoders in (E) and (F) implement a hybrid placement: channel SSMs first compress each fast-time stream into a single token per antenna, and the cross-antenna attention mixer then operates on this compact set of tokens. This reduces the attention sequence length by roughly a factor of 512 while preserving the virtual-array structure, and is exactly why (E) and (F) achieve the best balance between accuracy and compute, turning the SSM-first mixer placement into a core physics-inspired architectural contribution rather than just another variant. 11 Early Chirp State Saturation Experiment (a) (b) Figure 10: Design motivation for adaptive chirp selection. (a) Validation curves illustrate how detection and segmentation performance evolve with increasing chirp count. Without explicit early-convergence constraints, detection remains suboptimal until the full chirp frame is processed. (b) Introducing multi-chirp supervision during training encourages the model to saturate earlier and learn temporal continuity across chirps, yielding smoother convergence and higher overall detection–segmentation performance. We evaluate the impact of enforcing early state convergence by decoding from partial chirp sets, compared to training without this constraint. Consistent with the observations in [28], we find that mIoU improves rapidly in the early chirp regime. However, this behavior does not naturally extend to detection, where performance depends on information accumulated across the full chirp sequence (Figure˜10(a)). To encourage the latent states to saturate earlier—enabling reliable early-exit decisions—we decode intermediate outputs from multiple chirp subsets and supervise each with detection targets (Section 3.3): ℒtask=∑L∈ℒ[ℓdet(Det^(L),Det⋆) _task= _L [\, _det ( Det^(L),\,Det ) +ℓseg(Seg^(L),Seg⋆)] + _seg ( Seg^(L),\,Seg )\, ] Supervising detection at multiple chirp depths forces the model to extract complete spatial cues from early temporal observations. Learning this temporal–spatial continuity within radar frames improves overall performance, as shown in Figure˜10(b). We further visualize the effect of this training strategy on sub-frame decisions (Figure˜9). Without early-decision supervision, the temporal progression of the radar frame is poorly preserved: detection heatmaps fluctuate significantly across chirps, with the model sometimes forgetting strong reflectors or hallucinating obstacles mid-frame. With the early-chirp constraint, these inconsistencies largely disappear. Both detection and segmentation exhibit smoother evolution across chirps, and the latent states stabilize much earlier. This leads not only to improved detection performance but also to earlier state saturation, which directly contributes to computational savings under our early-exit framework. 12 Additional Results 12.1 Architecture Hyperparameters Table 4 lists the key architectural hyperparameters of RAVEN. The antenna mixer is deliberately narrow (64 dims, 8 heads) so that it adds negligible GMACs on top of the channel SSMs; the Mamba state dimension of 16 keeps per-RX encoders lightweight; and the 1×11×1 Conv1D projection maps chirp features to a 32×5632×56 BEV grid before the detection and segmentation decoders. Component Configuration Antenna Mixer Dim 64, 8 heads, expansion 4×4×, init ∼(0,1) \!N(0,1) SSM (Mamba) State dim 16, conv kernel 4, expansion 2 Spatial Proj. 1×11×1 Conv1D → 1792 ch. (grid 32×5632×56) Table 4: RAVEN architectural hyperparameters. To quantify the impact of compressing per-RX ADC samples to a single token, we test RAVEN variants where the fast-time SSM condenses each RX channel into K∈1,4,8,16K∈\1,4,8,16\ tokens before the cross-antenna mixer. Table 5 shows that the marginal gain from K=1K=1 to K=16K=16 is only 0.3%0.3\% F1, confirming that fast-time samples are highly compressible and that a single token captures the essential range/phase information needed downstream, validating our design choice. Tokens per RX (K) 1 (RAVEN) 4 8 16 F1 (Detection) 0.934 0.936 0.936 0.937 Table 5: Fast-time token compression ablation on RADIal. Marginal gains from K=1K=1 to K=16K=16 confirm that a single token per RX channel sufficiently captures range and phase information. 12.2 Early-Exit Decision Rule: Cosine Similarity vs. Entropy We compare two chirp-stopping criteria: (i) minimum cosine similarity between the new chirp latent state and all prior states (our default), and (i) entropy of the chirp-state distribution. Although entropy produces a smoother signal, cosine similarity yields better validation performance (+0.85% mAP, +0.67% mIoU) at similar compute, as shown in Table 6 and Figure 11. The cosine rule directly measures the novelty of each new chirp in the latent space, providing a more reliable and interpretable stopping condition. Figure 11: Cosine distance vs. entropy as chirp-stopping signals. Cosine similarity (blue) produces a cleaner knee-point, enabling more consistent early-exit decisions than entropy (orange). Stopping Rule mAP mAR F1 mIoU Cosine (Ours) 94.5 95.1 94.8 89.5 Entropy 93.6 94.0 93.8 88.8 Table 6: Early-exit decision rule comparison on RADIal (all metrics in %). Cosine similarity outperforms entropy on every metric. 12.3 Adaptive Chirp Selection vs. Scene Velocity Although static scenes nominally require less Doppler resolution, multiple chirps are still needed to form the virtual MIMO aperture for angular cues; fewer chirps shrink the virtual array and degrade spatial localization. Figure 12 shows no correlation between selected chirp count and object velocity, confirming that our adaptive stopping rule is driven by prediction stability in the latent space rather than scene motion. Figure 12: Velocity distribution and adaptive chirp count. (Left) Velocity histogram of annotated objects in RADIal. (Right) Scatter plot of per-frame selected chirp count vs. object velocity. The absence of correlation confirms that adaptive stopping is stability-driven, not velocity-driven. 12.4 Multi-Task vs. Task-Specific Performance Joint training does not introduce gradient interference. RAVEN trained jointly outperforms task-specific single-head baselines on both objectives: detection (0.95 vs. 0.93 mAP) and segmentation (90.2% vs. 90.1% mIoU). We attribute this to the shared chirp-SSM backbone learning complementary spatial features that benefit both heads simultaneously.